Excel Your KPIs with AI Copilot Start for free today
Your AI Copilot for Data
Definitive Guide to Decision Intelligence
Subscribe to our newsletter>
Get the latest products updates, community events and other news.
Special thanks to Nikhil Jain of Kyligence who co-wrote this article with me and provided many helpful technical insights.
When it comes to business intelligence and analytics, Qlik is a clear leader in helping businesses uncover critical new insights. Few Big Data tools for analytics offer the granular control and feature breadth Qlik provides. And now, in this era of Big Data insights, the ability of organizations to dig deeper into their datasets and extract insights quickly is becoming a leading concern for CIOs and their teams worldwide.
As your datasets grow to accommodate more complex business scenarios, you may begin to struggle with query speeds and the time it takes to load reports. This is not an uncommon occurrence when working with data across massive data lakes and data warehouses. The good news is that there’s an easy way to solve this, especially for Qlik users. So, if you’d like to ensure that Qlik continues to play a key role in your enterprise business intelligence strategy long into the future, read on.
Qlik Sense is a powerful business intelligence (BI) product that helps analysts uncover insights that they never even knew existed. The core technology that enables Qlik is its Associative Engine. The Qlik Associative Engine takes data from multiple sources, compresses and indexes them in memory, and builds relationships among the datasets.
Thanks to the Qlik Associative Engine, analysts can explore their data freely and get answers at the speed of thought. The following diagram, from Qlik’s white paper, The Associative Differences, illustrates how the Qlik Associative Engine differs from Query based tools.
Since Qlik reads data into memory for
indexing, the size of data it can process is limited by the server memory
configuration. Even with today’s advanced servers, the available memory is
usually at the 100s’ of gigabyte level. Modern data lakes usually have terabytes
to petabytes of data. Qlik Associative Engine’s in-memory processing model
cannot process all of that data.
Qlik also has a Direct Discovery mode. With
Direct Discovery, Qlik queries the underlying data sources directly using SQL.
But, because the data is not in memory, Direct Discovery has to calculate
aggregations on the fly while executing these queries. This can take a long
time with large datasets or many concurrent users, which is a common limitation
that other BI tools share.
So now we have the Qlik Associative Engine, which is great for uncovering hidden information yet struggles with data volumes that exceed available memory. We also have Direct Discovery that can query Big Data but may have performance issues when dealing lots of data. What has to happen so that users can enjoy the best of both worlds?
The optimal approach for solving this challenge is to add Kyligence and Qlik’s On Demand Application Generation (ODAG) to the mix. These are the two missing pieces that give Qlik users the ability to see the bigger picture and ask the right questions with data at scale.
Kyligence serves as the leading query acceleration engine for many popular BI products (Qlik, Power BI, Tableau, etc.). Unlike other analytics products, Kyligence intelligently identifies the queries that can be accelerated and pre-calculates the aggregations used by these queries. It leverages the storage and processing power of the data lake to process petabyte-scale datasets. Built on top of the Apache Kylin open source project, Kyligence’s OLAP engine is used worldwide by some of the largest enterprises with the most demanding data processing needs.
Qlik’s ODAG allows users to first select the
data they are interested in and then interactively generate an on-demand app
with which they can analyze the data with Qlik’s full in-memory capabilities.
Now, Qlik users can get the entire picture by querying the aggregations generated by Kyligence interactively, and then zoom in and explore the data more granularly with ODAG. The following diagram illustrates this architecture.
At the Qlik Qonnections 2019 conference in Dallas, Texas, we shared a powerful demo of how Qlik and Kyligence work together. Our demo scenario was of a retailer analyzing its customer profiles and related behaviors. The retailer in this example had over one billion customers in 6 different regions across a time window of 15 months (from Jan 2018 until March 2019). The retailer wanted to analyze their purchase history and demographic information.
We used Qlik’s Direct Discovery mode to source the pre-computed data from Kyligence OLAP cubes which represented the “Retail Dashboard”. The dashboard was comprised of customer counts spread across the entire timeframe and several charts depicting a variety of information which can be seen in the image below.
It is evident that the dashboard contains a
highly aggregated view on a huge dataset. With such a huge data volume, digging
into the data and telling a story has always been a nightmare for BI users and
analysts. However, thanks to the power of Kyligence, with its pre-computed data
cubes, analysts get a response to their queries in the blink of an eye.
In general, the dashboards are designed to
support users from different departments and provide different points of view.
Users can zoom into different sections of the dashboard to analyze aspects
pertaining to their areas of interest. This is where Qlik’s ODAG wizard plays an
important role as it allows users to define criteria & constraints to
generate on-demand app(s), thus providing a precise view for further analysis.
We demonstrated this feature by drilling down from an aggregated view with several filters like region, customer type, last product bought, buying date etc. Once all conditions were met, an on-demand app was ready for generation as shown below.
As a result, an on-demand app was generated which opens up a predefined dashboard on a focused dataset. This enables users to have a view tailored to meet their demands in order to discover hidden insights. As shown below, the on-demand application generated during our demo contains fine-grained customer information which offers details about customer age, their financial and educational status, and more. Such information plays an important role in effective campaigning and sales predictions.
In this demo, we showed how users dealing with
BI on Big data can greatly benefit from the combined power of Kyligence and
Qlik. We started with a look at the big picture using the entire dataset and
then zoomed in to a subset of the data with Qlik to show us what we couldn’t
spot before. ODAG served as the bridge connecting these two scenarios.
Ready to get started on the path to faster analytics and rapid insights? Get a complete overview of how your business intelligence tools can be enhanced with Kyligence here. You can also learn more about our OLAP for Big Data technology from this recently published presentation on Augmented Analytics we shared at this year’s Strata London.
For more examples of how Kyligence enhances Qlik, check out this video and visit our Qlik resources page:
If you're wondering how Kyligence compares to Apache Kylin, our Apache Kylin Comparison page is full of useful information. Also, be sure to follow us on LinkedIn and Twitter for the latest Kyligence product updates and announcements.
wanted to thank David Freriks, Emerging Technology Evangelist at Qlik, and Hugo
Sheng, Senior Director of Partner Engineering at Qlik, for their guidance and
help. Qlik has been a great partner to work with and we look forward to more
joint success stories!
Learn about the fundamentals of a data product and how we help build better data products with real customer success stories.
Discover the 7 top AI analytics tools! Learn about their pros, cons, and pricing, and choose the best one to transform your business.
Discover operational and executive SaaS metrics that matter for customers success, importance, and why you should track them with Kyligence Zen.
Unlock the future of augmented analytics with this must-read blog. Discover the top 5 tools that are reshaping the analytics landscape.
What website metrics matter in business? Learn about categories, vital website metrics, how to measure them, and how Kyligence simplifies it.
Unlock potentials of analytics query accelerators for swift data processing and insights from cloud data lakes. Explore advanced features of Kyligence Zen.
Unlock power of data storytelling in business. Learn how to convey insights using narrative and visual representations, examples, and benefits.
Explore these exceptional cloud analytics tools. Assess their pros, cons, and pricing to pinpoint the optimal one for your business.
Learn what natural language query is and how it transforms your data analytics. Explore examples of natural language queries in Kyligence Zen.
Discover how AI shapes banking, healthcare, and data analytics sectors. Get insights into the future of industry disruption to guide your decisions.
99 Almaden Boulevard Suite #663
San Jose, CA 95113
+1 (669) 256-3378
Ⓒ 2023 Kyligence, Inc. All rights reserved.
Already have an account? Click here to login
A complete product experience
A guided demo of the whole process, from data import, modeling to analysis, by our data experts.
Q&A session with industry experts
Our data experts will answer your questions about customized solutions.
Please fill in your contact information.We'll get back to you in 1-2 business days.