Excel Your KPIs with AI Copilot Start for free today
Your AI Copilot for Data
Definitive Guide to Decision Intelligence
Subscribe to our newsletter>
Get the latest products updates, community events and other news.
In a previous article, we talked about what Apache Kylin and Kyligence are and how they differ. To date, Kyligence has made significant improvements to the traditional extreme OLAP technology Kylin provides. Today, we will look at a few of Kyligence’s more popular improvements as they relate to common doubts surrounding OLAP for Big Data engines and OLAP cube technology.
Traditional OLAP cube technologies usually have a size limits, whether it’s several GB or several hundred GBs. This limit is defined either by the design of, or physical memory size on, the server. In Big Data processing, the data stored on the Hadoop platform is usually at terabyte or even petabyte level. This is way more than a traditional cube can handle.
With Kyligence, OLAP for Big Data cube building jobs are Spark or MapReduce jobs running on Hadoop, with the cubes themselves stored in columnar files on Hadoop. A Kyligence cube can grow to 100s of terabytes, and most of the time, query latency is still under 1 second. Further optimizations can improve cube building speed, saved storage space, and increase query speeds.
For example, ByteDance, the Chinese startup that recently surpassed Uber and became the world most valuable unicorn, uses Apache Kylin on its TouTiao news site to help them better serve their over 120 million daily active users. According to the Apache Kylin website, their OLAP cube (at toutiao.com) has more than 2.4 trillion source records. Taking 4+ TB of storage (and source files of more than 100 TB), most queries can be finished in less than 1 second. For reference, a similar query would need a couple hours with Hive.
The reality is that most data engineers who are familiar with Hadoop technologies have never worked with OLAP cube technologies. On the other side of that, most cube designers in the business intelligence (BI) world are not familiar with Hadoop. At Kyligence, we provide ‘Auto Modeling’ capabilities to shorten the learning curve related to building data models and cubes.
Auto Modeling takes your queries as inputs and automatically builds data models. You can import a SQL file, paste SQL statements, or select from your query histories. These query inputs are analyzed by machine learning algorithms to identify measures, dimensions, and the relationships between tables. You can modify the models and cube definitions to reflect your business’ requirements, but the default model is a good starting point.
Not only will Kyligence help you build the cubes, it also helps you manage those cubes. Over time, Kyligence will analyze cube utilizations and make recommendations to further optimize your cube. If a new cube should be created, Kyligence will also let the user know.
Sometimes, after drilling down into aggregated queries, you’ll want to see the original transactions to better understand the the issue. You may even think that you will need another query tool to look at the source databases. That is true with traditional OLAP cubes, but it’s not the case with Kyligence Enterprise. Kyligence offers Smart Query Routing, so users can query both aggregated results and detailed records through Kyligence Enterprise.
Kyligence does this in two ways:
Here is a high-level example of query routing:
Unlike traditional OLAP cubes, Kyligence was built with openness in mind from the very beginning. Kyligence supports ODBC, JDBC, Rest API, and MDX queries. We partner with all major business intelligence and analytics vendors to provide the best user experience. In October of 2018, Microsoft published an official Power BI connector for Kyligence.
More recently, in November of 2018, Kyligence partnered with Qlik to release Kyligence Data Connector for Qlik. And for Tableau users, Kyligence automatically syncs models with Tableau to avoid duplicating effort and human error. If your favorite business intelligence tool does not come with a pre-built Kyligence connector, you can simply point it to Kyligence using our ODBC driver and start seeing improved query performance immediately.
In this article we focused on the four most common questions people ask about OLAP on Big Data cube technologies, OLAP tools, and Kyligence. But there is far more to Kyligence than its OLAP on Hadoop BI technology. In future articles, we’ll continue to take a deeper look at the product, its features, and what else it has to offer OLAP analytics. We've also recently published a presentation - Augmented OLAP for Big Data explaining the capabilities that augmented OLAP analytics makes possible.
Until then, I encourage you to explore the Kyligence website for more information about Kyligence’s big data analytics platform and its Cloud Big Data platform: Kyligence Cloud.
Also, be sure to follow us on LinkedIn and Twitter for the latest Kyligence product updates and announcements.
Learn about the fundamentals of a data product and how we help build better data products with real customer success stories.
Unlock potentials of analytics query accelerators for swift data processing and insights from cloud data lakes. Explore advanced features of Kyligence Zen.
Optimize data analytics with AWS S3. Leverage large language models and accelerate decision-making.
Optimize data analytics with Snowflake's Data Copilot. Leverage large language models and accelerate decision-making.
Discover the 7 top AI analytics tools! Learn about their pros, cons, and pricing, and choose the best one to transform your business.
Discover operational and executive SaaS metrics that matter for customers success, importance, and why you should track them with Kyligence Zen.
Unlock the future of augmented analytics with this must-read blog. Discover the top 5 tools that are reshaping the analytics landscape.
What website metrics matter in business? Learn about categories, vital website metrics, how to measure them, and how Kyligence simplifies it.
99 Almaden Boulevard Suite #663
San Jose, CA 95113
+1 (669) 256-3378
Ⓒ 2023 Kyligence, Inc. All rights reserved.
Already have an account? Click here to login
A complete product experience
A guided demo of the whole process, from data import, modeling to analysis, by our data experts.
Q&A session with industry experts
Our data experts will answer your questions about customized solutions.
Please fill in your contact information.We'll get back to you in 1-2 business days.