Build a Faster Big Data Platform with Qlik and Kyligence

Special thanks to Nikhil Jain of Kyligence who co-wrote this article with me and provided many helpful technical insights.

When it comes to business intelligence and analytics, Qlik is a clear leader in helping businesses uncover critical new insights. Few Big Data tools for analytics offer the granular control and feature breadth Qlik provides. And now, in this era of Big Data insights, the ability of organizations to dig deeper into their datasets and extract insights quickly is becoming a leading concern for CIOs and their teams worldwide.

As your datasets grow to accommodate more complex business scenarios, you may begin to struggle with query speeds and the time it takes to load reports. This is not an uncommon occurrence when working with data across massive data lakes and data warehouses. The good news is that there’s an easy way to solve this, especially for Qlik users. So, if you’d like to ensure that Qlik continues to play a key role in your enterprise business intelligence strategy long into the future, read on.

Getting More from Your Big Data Analytics Tools: Qlik Associative Engine

Qlik Sense is a powerful business intelligence (BI) product that helps analysts uncover insights that they never even knew existed. The core technology that enables Qlik is its Associative Engine. The Qlik Associative Engine takes data from multiple sources, compresses and indexes them in memory, and builds relationships among the datasets.

Thanks to the Qlik Associative Engine, analysts can explore their data freely and get answers at the speed of thought. The following diagram, from Qlik’s white paper, The Associative Differences, illustrates how the Qlik Associative Engine differs from Query based tools.

Since Qlik reads data into memory for indexing, the size of data it can process is limited by the server memory configuration. Even with today’s advanced servers, the available memory is usually at the 100s’ of gigabyte level. Modern data lakes usually have terabytes to petabytes of data. Qlik Associative Engine’s in-memory processing model cannot process all of that data.

Qlik also has a Direct Discovery mode. With Direct Discovery, Qlik queries the underlying data sources directly using SQL. But, because the data is not in memory, Direct Discovery has to calculate aggregations on the fly while executing these queries. This can take a long time with large datasets or many concurrent users, which is a common limitation that other BI tools share.

So now we have the Qlik Associative Engine, which is great for uncovering hidden information yet struggles with data volumes that exceed available memory. We also have Direct Discovery that can query Big Data but may have performance issues when dealing lots of data. What has to happen so that users can enjoy the best of both worlds?

Optimizing Your Big Data Analytics Platform with Kyligence and ODAG

The optimal approach for solving this challenge is to add Kyligence and Qlik’s On Demand Application Generation (ODAG) to the mix. These are the two missing pieces that give Qlik users the ability to see the bigger picture and ask the right questions with data at scale.

Kyligence serves as the leading query acceleration engine for many popular BI products (Qlik, Power BI, Tableau, etc.). Unlike other analytics products, Kyligence intelligently identifies the queries that can be accelerated and pre-calculates the aggregations used by these queries. It leverages the storage and processing power of the data lake to process petabyte-scale datasets. Built on top of the Apache Kylin open source project, Kyligence’s OLAP engine is used worldwide by some of the largest enterprises with the most demanding data processing needs.

Qlik’s ODAG allows users to first select the data they are interested in and then interactively generate an on-demand app with which they can analyze the data with Qlik’s full in-memory capabilities.

Now, Qlik users can get the entire picture by querying the aggregations generated by Kyligence interactively, and then zoom in and explore the data more granularly with ODAG. The following diagram illustrates this architecture.

Extreme OLAP Analytics in Action with Qlik and Kyligence

At the Qlik Qonnections 2019 conference in Dallas, Texas, we shared a powerful demo of how Qlik and Kyligence work together. Our demo scenario was of a retailer analyzing its customer profiles and related behaviors. The retailer in this example had over one billion customers in 6 different regions across a time window of 15 months (from Jan 2018 until March 2019). The retailer wanted to analyze their purchase history and demographic information.

We used Qlik’s Direct Discovery mode to source the pre-computed data from Kyligence OLAP cubes which represented the “Retail Dashboard”. The dashboard was comprised of customer counts spread across the entire timeframe and several charts depicting a variety of information which can be seen in the image below.

It is evident that the dashboard contains a highly aggregated view on a huge dataset. With such a huge data volume, digging into the data and telling a story has always been a nightmare for BI users and analysts. However, thanks to the power of Kyligence, with its pre-computed data cubes, analysts get a response to their queries in the blink of an eye.

In general, the dashboards are designed to support users from different departments and provide different points of view. Users can zoom into different sections of the dashboard to analyze aspects pertaining to their areas of interest. This is where Qlik’s ODAG wizard plays an important role as it allows users to define criteria & constraints to generate on-demand app(s), thus providing a precise view for further analysis.

We demonstrated this feature by drilling down from an aggregated view with several filters like region, customer type, last product bought, buying date etc. Once all conditions were met, an on-demand app was ready for generation as shown below.

As a result, an on-demand app was generated which opens up a predefined dashboard on a focused dataset. This enables users to have a view tailored to meet their demands in order to discover hidden insights. As shown below, the on-demand application generated during our demo contains fine-grained customer information which offers details about customer age, their financial and educational status, and more. Such information plays an important role in effective campaigning and sales predictions.

Enhance Your Qlik Experience with the World’s Leading Big Data OLAP Technology

In this demo, we showed how users dealing with BI on Big data can greatly benefit from the combined power of Kyligence and Qlik. We started with a look at the big picture using the entire dataset and then zoomed in to a subset of the data with Qlik to show us what we couldn’t spot before. ODAG served as the bridge connecting these two scenarios.

Ready to get started on the path to faster analytics and rapid insights? Get a complete overview of how your business intelligence tools can be enhanced with Kyligence here. You can also learn more about our OLAP for Big Data technology from this recently published presentation on Augmented Analytics we shared at this year’s Strata London.

For more examples of how Kyligence enhances Qlik, check out this video and visit our Qlik resources page:

If you're wondering how Kyligence compares to Apache Kylin, our Apache Kylin Comparison page is full of useful information. Also, be sure to follow us on LinkedIn and Twitter for the latest Kyligence product updates and announcements.

We wanted to thank David Freriks, Emerging Technology Evangelist at Qlik, and Hugo Sheng, Senior Director of Partner Engineering at Qlik, for their guidance and help. Qlik has been a great partner to work with and we look forward to more joint success stories!