Scale Your Big Data OLAP on Hadoop to Petabytes on Google Cloud Platform

Kyligence – created by the early contributors to Apache Kylin – announces that its enterprise OLAP (online analytical processing) engine on Hadoop – Kyligence Enterprise – can now run on Google Cloud Platform. Highly integrated with Google Cloud Storage and DataProc, Kyligence scales your extreme OLAP analytics to petabytes of data, and makes your business intelligence and analytics applications work on your data lakes.

Kyligence Enterprise can achieve sub-second query latency on petabyte-scale datasets on Google Cloud Hadoop by building pre-aggregated OLAP cubes. And with an easy drag and drop user interface and intelligent data modeling assistant that Kyligence Enterprise offers, it greatly simplifies the OLAP cube building process.

By deploying on Google Cloud Platform, Kyligence Enterprise helps you matching the dynamic computing and analytics requirements, reducing the operation cost and accelerating business development on cloud.

In this article, we will introduce you a step-by-step guide of how to run Kyligence on Google Cloud Platform. We will have 3 sections:

1. Intro of Kyligence
2. Prepare Google Account
3. Deploy and run Kyligence on Google Cloud Platform

About Google Cloud Platform

Google Cloud Platform, offered by Google, is a suite of cloud computing services that runs on the same infrastructure that Google uses internally for its end-user products.  Alongside a set of management tools, it provides a series of modular cloud services including computing, data storage, data analytics and machine learning.

About Kyligence Extreme OLAP Analytics Platform

Kyligence is a data-tech company focusing on Big Data analytics and founded by the team who were early contributors to Apache Kylin – the first top-level project in Apache Software Foundation (ASF) from China. Powered by Apache Kylin, Kyligence provides Kyligence Enterprise – an intelligent enterprise big data analytics platform for the on-premise market, and also Kyligence Cloud for the cloud solution.

Kyligence has won lots of well-known customers across different industries including Huawei, China Unicom, OPPO, SAIC, Pacific Insurance Group, China UnionPay, Guotai Junan Securities.

Prerequisite

1. Apply a Google account
2. Apply free trial of Kyligence via https://cloud.kyligence.io/#/cloudapply and select Google Cloud Platform as platform to run your analytic workloads

Prepare Your Google Account

Create related resources on Google Cloud Platform

To run Kyligence on Google Cloud Platform, it relies on the following Google cloud resources and services. You need to prepare your Google account as well as related resources beforehand.

• DataProc
• Compute Engine
• Storage
• VPC
• Cloud SQL

1. Create a Project

In Google Cloud Platform console, click the current project name in the upper left corner.

Select NEW PROJECT in the pop-up window, enter a new project name, and click CREATE.

1. Enable API to access the Project

Please ensure the following APIs are enabled:

• Stackdriver Logging API
• Compute Engine API
• Cloud Dataproc Control API
• Cloud Dataproc API
• Cloud SQL Admin API

To see API usage in the current Project, select API and Services in the menu, then select Dashboard.

To find the API and enable it, select the library in the API and services, search the API in the API library, select the required API and click Enable.

1. Create a Service Account and authorize

Select IAM & admin and in the menu, then select Services accounts.

At the top of the page, select CREATE SERVICE ACCOUNT and fill in the service account name on the right. Select the owner in the project role.

Check the box below to Furnish a new private key, and select the key type as JSON, and click Save.

1. Create Storage

Select Storage - Browser - CREATE BUCKET

Name the new bucket, select the storage category Multi-Regional, and finally click create.

The differences between the different storage types are shown below:

1. Create VPC

Select the VPC network in the VPC network on the left side of the console and click CREATE VPC NETWORK.

Fill in the VPC network name, select the subnet creation mode as automatic, then click create.

Deploy Kyligence on Google Cloud Platform

Kyligence offers an online service - Kyligence Cloud to ease the deployment of Kyligence Enterprise on Google Cloud Platform. You can finish the Hadoop as well as Kyligence service deployment via simple clicks within 30 minutes.

1. Login https://cloud.kyligence.io using your own account and click Create Cluster.

1. Fill in the cluster name.

1. At cluster topology section, specify your cluster size with input the number of worker nodes.

Note: Edge node is where Kyligence service runs

1. Click +Account and fill in your Google account in the first blank, and private key (please select the private key, launch the saved private key file in note text format, copy and paste all the contents of the private key into the second blank) in the second blank, click submit.

1. After selecting the entered Google account, select the region and select VPC, subnet, storage space.

Note: Kyligence Cloud will read the list of VPCs, subnets, and storage accounts in your Google account. If it is not possible, please check if the Google account and private key you entered are correct.

1. Choose the version of Kyligence Enterprise you want to deploy. You can also choose to install KyAnalyzer and enable email notification. Then click Submit.

1. Start the cluster. In the cluster page of Kyligence Cloud Portal, click the start button 

   

   and wait until the cluster status is changed to RUNNING.

Note: The new cluster startup will take about 20 minutes

1. After the cluster is successfully started, you can launch Kyligence Enterprise for OLAP modeling and analysis. For more details about how to use Kyligence Enterprise, please visit HERE.

Summary

Via seamless integration with Google Cloud Platform, Kyligence Enterprise scales extreme augmented OLAP analytics to petabytes of data, making your business intelligence tools and applications work on your data lakes. It helps you matching the dynamic computing and analytics requirements, reducing the operation cost and accelerating business development on cloud.

Also, if you happen to be evaluating Apache Kylin against Kyligence for your OLAP analytics on Big Data, then we recommend you check out our Apache Kylin Comparison page.

If you have questions or feedback, we'd love to hear from you on LinkedIn and Twitter.