Interactive Analytics with Kyligence Augmented OLAP for Azure HDInsight

With over 2.5 quintillion bytes of data being created each day(1), there has never been a better time to work with Big Data. Opportunities abound for enterprises able to capture this data and transform it into actionable insights. But for many organizations, even ones putting data first, their Big Data efforts don’t meaningfully develop beyond data capture. It’s why, of the 85% of organizations that have invested in building a data-driven culture, only 37% have had any kind success(2).

In order for enterprises to claim their data as a strategic advantage over the competition, business decision makers and their analytics teams must make insight extraction a priority. And while every business leader will assure you that insights are a priority, they’re often undermined by a business intelligence and analytics infrastructure that’s too slow. To truly become data-driven and stay above the increasingly competitive landscape, organizations must extract insights in real time.

Take the Next Step Towards Faster Big Data Insights

For many, moving to cloud platforms (like Microsoft Azure) for their Big Data operations has been a critical step toward obtaining greater speed and agility in their insight generation, but it’s not always enough. For organizations working with extremely massive datasets, query times, concurrency, and BI tool management can continue to be a drag on productivity. Kyligence, an AI Augmented data management and analytics platform, was created to solve these persistent barriers to actionable data-driven insights.

Powered by Apache Kylin, a leading open source OLAP engine for Big Data, Kyligence works to automatically accelerate insights from datasets of any size, significantly improving the productivity of data engineers and the analysts they work with.

To achieve this, Kyligence implements classic multidimensional analytics technology on the Hadoop platform, but with significant enhancements in intelligent modeling, query routing, and performance optimizations.

Sub-Second Query Latency for 100,000s’ of Concurrent Users at Petabyte Scale

For enterprise customers with Hadoop data lakes, getting value out of their data is often struggle. One technical issue is query-latency. Most query engines on Hadoop have response times in the tens-of-seconds, minutes, and even hours in some cases. Interactive analytics, or analytics at the speed of thought, requires that query-latency be at the sub-second level.

Many years ago, cube-based OLAP technology was created to address query latency issues, but traditional OLAP data cubes just can’t handle today’s data volumes. Their size is limited by the configuration of the servers they run on. Also, traditional cubes usually have a lengthy build-time and are difficult to maintain. This makes them a poor fit in the modern business environment.

Kyligence, however, implements a modernized OLAP analytics technology on the Hadoop platform. Hive tables on Hadoop or Kafka streams can be used as source data, and the cube building jobs are Spark or MapReduce jobs running on Hadoop. The actual cubes are stored in the Hadoop File System.

Hadoop gives Kyligence a platform to process and store petabytes of data at high speeds. For example, clients serving 3.8 million queries per day on 8.9 trillion records, experience 50% of their queries finishing within 0.5 seconds, and 90% of their queries finishing within 1.2 seconds.

The Leading BI Accelerator Built to Run Natively on HDInsight

Backed by a 99.9% SLA, Azure HDInsight is  optimized cloud-based managed platform for running the full spectrum of open source analytics frameworks including Apache Spark, Hive, MapReduce, HBase, Storm, Kafka and Microsoft ML Services.

The Azure HDInsight architecture separates compute and storage, and since the data is persisted in Azure Blob Storage or Azure Data Lake Storage, customers can easily start, stop, scale, and shrink the cluster on demand, without worrying about data loss. Azure Blob Storage and Azure Data Lake Storage are highly available, scalable, HDFS compatible, and low-cost storage services.

Kyligence worked with Microsoft to build integrations with Azure HDInsight, SQL Data Warehouse, Blob Storage, and Azure Data Lake Storage. Kyligence also developed a one-stop analysis platform, Kyligence Cloud, based on Azure’s elastic architecture.

Customers now have the flexibility to choose deployment of Kyligence Cloud from the Azure Marketplace or installing Kyligence Enterprise on HDInsight. This blog can walk you through the process of deploying Kyligence on Azure Marketplace.

Optimized Productivity for Modern Business Applications

HDInsight users can use Kyligence Enterprise’s Big Data platform to take advantage of its intelligent modeling capabilities and significantly improve their productivity. Artificial intelligence assists enterprises with building data models by using both historical business needs and ongoing user behavior. Engineers can use this intelligent modelling to quickly create and modify data models for various topics and business lines, and then publish them for business analysts.

Technical tables and fields are transformed into the user-friendly metrics and dimensions of data models, easily understood by business users. AI-based data modeling algorithms not only provide enhanced data modeling, but also processing, accelerated data mining, reduced reliance of business users on IT teams, and increased productivity for those IT teams. Users can also run ad-hoc queries with Smart Pushdown, analyzing specific records from the original data source.

Support for Every BI Tool, Including Power BI and Excel

With Kyligence Enterprise and Azure HDInsight, business analysts can easily access relevant datasets through popular front-end business intelligence tools like Power BI and Excel, executing a natural flow of analysis using intuitive drag-and-drop features. With high-performance queries on massive datasets, users can quickly generate results and further refine analysis ideas and business decisions.

Expect (And Get) More From Your Big Data Analytics

The secret is out when it comes to Big Data. The more of it you have, the better your insights will be. Businesses have become experts at developing the infrastructure necessary to collect the data they need from anywhere they may need to get it from, but it is becoming clear that this is only half the battle.

For every business, future success hinges on Big Data analytics tools and the insights that come from them. But the only way for those insights to matter is if you can arrive at them faster than your competition. Ensuring your BI strategy and Big Data analytics platform is built for speed (and not just collection) is critical for making sure that happens.

It’s why Kyligence is the leading high-performance extreme OLAP solution trusted by global enterprises for data collection, processing, modeling, and analysis. Now, as a Microsoft Gold Partner, Kyligence can further enable business users to easily conduct data-analysis and make decisions without coding, so they can instead focus on using the other Microsoft tools they know and love. With streamlined deployment, and native integration with HDInsight, it has never been easier to get started with Kyligence.

With streamlined deployment, and native integration with HDInsight, it has never been easier to get started with Kyligence. Try it today!

Have questions? Feedback? Would you just like to keep up with what’s new in Augmented Analytics? Join us on LinkedIn and Twitter. And if you're looking for more information about Apache Kylin and how it stacks up against Kyligence, take a look at our Apache Kylin Comparison page.

References:

(1) https://www.domo.com/learn/data-never-sleeps-5?aid=ogsm072517_1&sf100871281=1

(2) http://newvantage.com/wp-content/uploads/2017/01/Big-Data-Executive-Survey-2017-Executive-Summary.pdf