By Use Cases
By BI Tool
Subscribe to our newsletter>
Get the latest products updates, community events and other news.
In a previous article, we talked about what Apache Kylin and Kyligence are and how they differ. To date, Kyligence has made significant improvements to the traditional extreme OLAP technology Kylin provides. Today, we will look at a few of Kyligence’s more popular improvements as they relate to common doubts surrounding OLAP for Big Data engines and OLAP cube technology.
Traditional OLAP cube technologies usually have a size limits, whether it’s several GB or several hundred GBs. This limit is defined either by the design of, or physical memory size on, the server. In Big Data processing, the data stored on the Hadoop platform is usually at terabyte or even petabyte level. This is way more than a traditional cube can handle.
With Kyligence, OLAP for Big Data cube building jobs are Spark or MapReduce jobs running on Hadoop, with the cubes themselves stored in columnar files on Hadoop. A Kyligence cube can grow to 100s of terabytes, and most of the time, query latency is still under 1 second. Further optimizations can improve cube building speed, saved storage space, and increase query speeds.
For example, ByteDance, the Chinese startup that recently surpassed Uber and became the world most valuable unicorn, uses Apache Kylin on its TouTiao news site to help them better serve their over 120 million daily active users. According to the Apache Kylin website, their OLAP cube (at toutiao.com) has more than 2.4 trillion source records. Taking 4+ TB of storage (and source files of more than 100 TB), most queries can be finished in less than 1 second. For reference, a similar query would need a couple hours with Hive.
The reality is that most data engineers who are familiar with Hadoop technologies have never worked with OLAP cube technologies. On the other side of that, most cube designers in the business intelligence (BI) world are not familiar with Hadoop. At Kyligence, we provide ‘Auto Modeling’ capabilities to shorten the learning curve related to building data models and cubes.
Auto Modeling takes your queries as inputs and automatically builds data models. You can import a SQL file, paste SQL statements, or select from your query histories. These query inputs are analyzed by machine learning algorithms to identify measures, dimensions, and the relationships between tables. You can modify the models and cube definitions to reflect your business’ requirements, but the default model is a good starting point.
Not only will Kyligence help you build the cubes, it also helps you manage those cubes. Over time, Kyligence will analyze cube utilizations and make recommendations to further optimize your cube. If a new cube should be created, Kyligence will also let the user know.
Sometimes, after drilling down into aggregated queries, you’ll want to see the original transactions to better understand the the issue. You may even think that you will need another query tool to look at the source databases. That is true with traditional OLAP cubes, but it’s not the case with Kyligence Enterprise. Kyligence offers Smart Query Routing, so users can query both aggregated results and detailed records through Kyligence Enterprise.
Kyligence does this in two ways:
Here is a high-level example of query routing:
Unlike traditional OLAP cubes, Kyligence was built with openness in mind from the very beginning. Kyligence supports ODBC, JDBC, Rest API, and MDX queries. We partner with all major business intelligence and analytics vendors to provide the best user experience. In October of 2018, Microsoft published an official Power BI connector for Kyligence.
More recently, in November of 2018, Kyligence partnered with Qlik to release Kyligence Data Connector for Qlik. And for Tableau users, Kyligence automatically syncs models with Tableau to avoid duplicating effort and human error. If your favorite business intelligence tool does not come with a pre-built Kyligence connector, you can simply point it to Kyligence using our ODBC driver and start seeing improved query performance immediately.
In this article we focused on the four most common questions people ask about OLAP on Big Data cube technologies, OLAP tools, and Kyligence. But there is far more to Kyligence than its OLAP on Hadoop BI technology. In future articles, we’ll continue to take a deeper look at the product, its features, and what else it has to offer OLAP analytics. We've also recently published a presentation - Augmented OLAP for Big Data explaining the capabilities that augmented OLAP analytics makes possible.
Until then, I encourage you to explore the Kyligence website for more information about Kyligence’s big data analytics platform and its Cloud Big Data platform: Kyligence Cloud.
Also, be sure to follow us on LinkedIn and Twitter for the latest Kyligence product updates and announcements.
Learn about the fundamentals of a data product and how we help build better data products with real customer success stories.
Learn about the components of Kyligence Cloud platform that enable it to be a production-ready self-managed distributed computing system.
See how data engineers can leverage the newly launched Data Assets Dashboard to identify the most valuable data, dimensions, and metrics.
Learn about SQL Server Analysis Services (SSAS), an online analytical processing (OLAP) and data mining tool in Microsoft SQL Server.
Learn about the Distributed MDX Engine, an enhanced semantic layer architecture on the cloud, and how it can boost your data analytics.
A detailed step-by-step guide on how to connect Excel to ClickHouse within Kyligence Tiered Storage and achieve sub-second query latencies.
The fact that OLAP software can be utilized to gather these big data insights is proof of its relevancy today. And it will continue to persist because, again, it is rooted in sound theory.
99 Almaden Boulevard Suite #663
San Jose, CA 95113
+1 (669) 256-3378
Ⓒ 2022 Kyligence, Inc. All rights reserved.
Already have an account? Click here to login