Kyligence Cloud Big Data Platform on AWS: Preparing and Creating Clusters

1. Introduction to Kyligence Cloud: Extreme OLAP Analytics Platform

1.1 What is Kyligence Cloud?

Kyligence Cloud is a platform-as-a-service (PaaS) product for big data OLAP analytics in the cloud that combines Kyligence’s original and interactive extreme OLAP on Big Data technology with best practices in cloud based big data analytics for easy-to-use, high performance, high concurrency, auto scaling, high security, and high availability features. This provides users with a simple, fast, scalable, and low-cost solution.

1.2 Kylignece Cloud architecture

Kyligence Cloud acts on your behalf to communicates with AWS / Azure. When you create a cluster through the Kyligence Cloud platform, it uses the AWS / Azure account information you provide to create clusters and other resources under the account.

For data modeling and analysis, all of which occur within your virtual private cloud (VPC). Kyligence Cloud will not touch your data.

With a compute-and-storage-separated architecture, when no computation is needed, you can easily scale and shrink the cluster, or even safely stop the cluster without worrying about losing data. Metadata such as table structures and cube models is stored in an RDS or SQL Server instance under your account and Cube data is stored in an S3 or Azure blob store.

The figure below shows the architecture of Kyligence Cloud.

Kyligence Cloud Architecture

In this diagram, Hadoop / Spark cluster, KAP, KyAnalyzer, and Zeppelin are on-demand services that have the same life cycle as your cluster. Cloud storage and metadata are long persistent and will not be deleted when the cluster stops.

2. Preparation

Kyligence Cloud supports AWS global, AWS China, Azure global and Azure China. This article takes AWS global as an example to demonstrate how to use Kyligence Cloud for rapid deployment on the cloud. Deployment processes for AWS global and AWS China are basically the same, therefore this tutorial also applies to AWS China.

Before creating a Stack, there’s some preparation work to do. You need to verify that in one region of your AWS account you have:

• 1 VPC (need to enable DNS resolution, DNS hostnames)
• 2 subnets of different available zones on the same VPC
• 1 S3 bucket
• 1 key pair

2.1 Create VPC and subnet

Kyligence Cloud will create EMR, EC2, and RDS in two subnets on the VPC.

If you do not currently have any VPC available, or would like to use a separate VPC for Kyligence Cloud, you can create a VPC and its subnets in the following ways:

I. Install the AWS CLI Tools (see: https://docs.aws.amazon.com/cli/latest/userguide/installing.html)

II. execute the “aws configure” command to configure AWS authentication information

“AWS Configure” Command

III. Get the CloudFormation template that created your VPC (https://docs.kyligence.io/cloud/en-us/appendix/vpc_template.html), and copy its contents into file create_vpc.template. Then execute the following AWS command：

aws cloudformation create-stack \

--stack-name <Cloudformation stack name> \

--template-body <CloudFormation template file> \

--region <region> \

--parameters \

ParameterKey=VPCNAME,ParameterValue=<VPC name> \

ParameterKey=AZ1,ParameterValue=<Availability Zone 1> \

ParameterKey=AZ2,ParameterValue=<Availability Zone 2>
An example of the above command:

Create a VPC Command

The parameter values circled in the picture should be replaced by your own values. After the command is executed, if the StackId information is output, it means that the creation is successful.

2.2 Creating S3 Bucket

Kyligence Cloud uses S3 buckets to persist KAP cube data and EMR logs.

First log in to your AWS account and enter the AWS S3 console.

I. Click “Create bucket”

Create a bucket

II. Select the bucket name and region, and the region should be consistent with the VPC region.

Create a bucket

2.3 Creating a Key Pair

The key pair is used to SSH into the EMR nodes and EC2 node where KAP is located. You can use an existing key pair or create a new one.

First log in to the AWS EC2 console.

I. Click “Key pairs” or “X Key Pairs”, as shown:

Key Pair Entrance

II. Click “Create Key Pair”

Create a Key Pair

III. Enter the key pair name and click “Create”. After successful creation, the browser will automatically download the “.pem” file, which will be used by SSH to log in to the cluster node. Be sure to keep this file properly. , so you do not have to worry about security.

Create a key pair

3. Login / Register Kyligence Account

3.1 Login Kyligence account

Kyligence Account is a Kyligence unified identity account that you can use to log in to all Kyligence products and services.

After entering https://cloud.kyligence.io/ in the browser address bar, the Kyligence account login screen will be displayed. If you have registered Kyligence account, you can log in directly to the site; Otherwise, you can click on the left side of the page “Sign Up” button to enter the registration page, fill in the relevant information to register.

3.2 Registration

Fill in all the information on the registration page and click “Submit”.

Register Account

Afterward, you will receive a verification email (If you do not receive the email, you can check the spam box or click “Send Again”), click the link in the email to complete the verification. After verification, you’ll be redirected back to login page.

Verify the Mailbox

3.3 Apply for a trial version

Once your Kyligence account application is complete, enter https://cloud.kyligence.io/ in the address bar to access the Kyligence Cloud console. If it’s the first time you log in Kyligence Cloud, you will be taken to the trial page automatically.

Apply for Trial

Kyligence Cloud offers a 60-day free trial of one fully functional stack to get you familiar with and understand Kyligence Cloud, experience and discover its value.

Fill the questionnaire form and click the “Apply” button to submit. We will review your application within 3 business days. After the application is approved, you will be notified via email.

4. Start to Deploy the Stack

In the Kyligence Cloud, stacks are the main objects for operations. A stack consists of one or more Hadoop clusters. After completing previous preparations, we can now start the deployment of the stack. The operation flow is very simple. First go to the Kyligence Cloud Console (https://cloud.kyligence.io)

If you are logging into Kyligence Cloud for the first time, you will need to select a basic cloud platform. Here we choose AWS Global, then click “Continue”.

Select Cloud Platform

Select AWS provider

4.1 Create a stack

I. In the stack list page, click the “+ Stack” button to enter the Create Stack page.

New Stack

II. Fill in the stack name, select the EMR version and the number of nodes / hardware configuration for each node type.

Fill in the basic information of the cluster

III. Fill in the Access Key and Secret Key for your AWS account, select the region to deploy, and pick the AWS resources that you have already prepared in the previous section.

Fill in AWS resource information

IV. Select the version of KAP to install, along with two other optional supporting tools KyAnalyzer and Zeppelin. And you can toggle e-mail notification of the stack operation result on/off.

Choose KAP

V. Finally, click on the submit button at the bottom of the page, this will create a stack in the NEW state, and displayed in the stack list.

4.2 start the stack

I. In the stack list, click the Start button to the right of the stack entry.

Start the Stack

II. After startup, the stack enters the CREATING state where Kyligence Cloud is deploying the cluster and installing KAP and its dependencies. The process lasts about 25 to 50 minutes, and Kyligence Cloud will notify you of the deployment result as soon as it is done.

Stack Created

III. After the stack is deployed, it will be in the RUNNING state. Congratulations, the cluster and KAP is ready for you now.

Stack Is Created

5. Using Kyligence Extreme OLAP Analytics Platform

After the stack is deployed, you can officially start using Kyligence Analytics Platform (KAP).

The Kyligence Analytics Platform (KAP) Big Data Intelligence Analytics Platform, based on Apache Kylin, is an enterprise data warehouse offering sub-second analytics on tremendous large data sets, providing business users, analysts, and engineers with easy and fast Big Data Analysis Services.

5.1 Visit KAP

Click the collapse button to the left of the stack name to see KAP’s entrance. If you have chosen to install the KyAnalyzer and Zeppelin when you deployed the stack, the entrances for both tools are also displayed here.

KAP Entrance

Note: KAP deployed through Kylignce Cloud defaults to the latest KAP Plus version.

Click the KAP button, the browser will redirect you to the KAP login page; for first time visit, you can use the default user name ADMIN and the default password KYLIN to log in; after initial login you need to change the password to ensure access security. KAP comes with a sample cube that you can build and execute queries on.

Once the cube is built, you can log in to KyAnalyzer (using the same username and password as KAP), synchronize the cube, and conduct multidimensional visualization analytics.

5.2 KAP management

Kyligence Cloud provides basic management capabilities for KAP, allowing users to start, stop, configure, and upgrade KAP directly on Kyligence Cloud’s interface.

I. You can click the “View” button in the menu on the right side of the stack entry to enter the stack detail page.

Details Page Entry

II. On stack detail page, you can see the KAP related information, click on the right side of the “stop” / “start” to manage KAP services. You can also adjust the KAP configuration properties directly through the interface here.

KAP Management

6. Conclusion

So far, this article describes how to use Kyligence Cloud for easy and quick deployment on AWS. Features like Kyligence Cloud’s auto scaling are yet to be explored, and we’ll cover more Kyligence Cloud features in future articles. If you have questions with your use, you can get help through the “Support” button at the bottom left of the Kyligence Cloud console page and the “Contact Us” button at the top right.