Kyligence – Augmented OLAP for the Big Data Era

In a previous article, we talked about what Apache Kylin and Kyligence are and how they differ. To date, Kyligence has made significant improvements to the traditional extreme OLAP technology Kylin provides. Today, we will look at a few of Kyligence’s more popular improvements as they relate to common doubts surrounding OLAP for Big Data engines and OLAP cube technology.

Doubt #1: OLAP Cubes Are Not a Good Fit for Big Data Analysis

Traditional OLAP cube technologies usually have a size limits, whether it’s several GB or several hundred GBs. This limit is defined either by the design of, or physical memory size on, the server. In Big Data processing, the data stored on the Hadoop platform is usually at terabyte or even petabyte level. This is way more than a traditional cube can handle.    

With Kyligence, OLAP for Big Data cube building jobs are Spark or MapReduce jobs running on Hadoop, with the cubes themselves stored in columnar files on Hadoop. A Kyligence cube can grow to 100s of terabytes, and most of the time, query latency is still under 1 second. Further optimizations can improve cube building speed, saved storage space, and increase query speeds.

For example, ByteDance, the Chinese startup that recently surpassed Uber and became the world most valuable unicorn, uses Apache Kylin on its TouTiao news site to help them better serve their over 120 million daily active users. According to the Apache Kylin website, their OLAP cube (at toutiao.com) has more than 2.4 trillion source records. Taking 4+ TB of storage (and source files of more than 100 TB), most queries can be finished in less than 1 second. For reference, a similar query would need a couple hours with Hive.

Doubt #2: It’s Hard to Build and Manage OLAP Cubes

The reality is that most data engineers who are familiar with Hadoop technologies have never worked with OLAP cube technologies. On the other side of that, most cube designers in the business intelligence (BI) world are not familiar with Hadoop. At Kyligence, we provide ‘Auto Modeling’ capabilities to shorten the learning curve related to building data models and cubes.

Auto Modeling takes your queries as inputs and automatically builds data models. You can import a SQL file, paste SQL statements, or select from your query histories. These query inputs are analyzed by machine learning algorithms to identify measures, dimensions, and the relationships between tables. You can modify the models and cube definitions to reflect your business’ requirements, but the default model is a good starting point.

Not only will Kyligence help you build the cubes, it also helps you manage those cubes. Over time, Kyligence will analyze cube utilizations and make recommendations to further optimize your cube. If a new cube should be created, Kyligence will also let the user know.

Doubt #3: I Can Only Find Aggregated Results in the Cube

Sometimes, after drilling down into aggregated queries, you’ll want to see the original transactions to better understand the the issue. You may even think that you will need another query tool to look at the source databases. That is true with traditional OLAP cubes, but it’s not the case with Kyligence Enterprise. Kyligence offers Smart Query Routing, so users can query both aggregated results and detailed records through Kyligence Enterprise.

Kyligence does this in two ways:

1. Table Index: You can think of a table index as the local cache of original transaction data, stored with cubes in the Hadoop file system.
2. Pushdown: Pushdown is the process of handing over the query to a Big Data query engine like Hive for answers.

Here is a high-level example of query routing:

Doubt #4: OLAP Cubes Are Tied to a Specific Big Data Analytics Tool Only

Unlike traditional OLAP cubes, Kyligence was built with openness in mind from the very beginning. Kyligence supports ODBC, JDBC, Rest API, and MDX queries. We partner with all major business intelligence and analytics vendors to provide the best user experience. In October of 2018, Microsoft published an official Power BI connector for Kyligence.

More recently, in November of 2018, Kyligence partnered with Qlik to release Kyligence Data Connector for Qlik. And for Tableau users, Kyligence automatically syncs models with Tableau to avoid duplicating effort and human error. If your favorite business intelligence tool does not come with a pre-built Kyligence connector, you can simply point it to Kyligence using our ODBC driver and start seeing improved query performance immediately.

Get Started With Augmented OLAP Analytics for Big Data Today

In this article we focused on the four most common questions people ask about OLAP on Big Data cube technologies, OLAP tools, and Kyligence. But there is far more to Kyligence than its OLAP on Hadoop BI technology. In future articles, we’ll continue to take a deeper look at the product, its features, and what else it has to offer OLAP analytics. We've also recently published a presentation - Augmented OLAP for Big Data explaining the capabilities that augmented OLAP analytics makes possible.

Until then, I encourage you to explore the Kyligence website for more information about Kyligence’s big data analytics platform and its Cloud Big Data platform: Kyligence Cloud.

TEST DRIVE with $500 worth of free usage

Also, be sure to follow us on LinkedIn and Twitter for the latest Kyligence product updates and announcements.