Build the Common Data Language with the Metrics Platform Start Now
By Use Cases
By BI Tools
Subscribe to our newsletter>
Get the latest products updates, community events and other news.
Author: Ruzong Zhang, Senior Engineer of Beike Data Platform, Beike; Responsible for the development, operation, and maintenance of OLAP engines and the metrics platform. Editor: Coco Li
What Beike is doing is to promote the deep Internetization of the traditional housing services industry. In this process, data is a vital pillar.——rusong zhang
What Beike is doing is to promote the deep Internetization of the traditional housing services industry. In this process, data is a vital pillar.
In this article, we’ll dive into the unified Metrics Platform at Beike, introduce Beike’s practice of building the Metrics Platform infrastructure using Apache Kylin and some real use cases at Beike.
We aspire to provide comprehensive and trusted housing services to 300 million families.— — Beike’s Vision
We aspire to provide comprehensive and trusted housing services to 300 million families.
— — Beike’s Vision
KE Holdings Inc. (“Beike”)(NYSE: BEKE) is the leading integrated online and offline platform for housing transactions and services in China.
The company is a pioneer in building the industry infrastructure and standards in China to reinvent how service providers and housing customers efficiently navigate and consummate housing transactions, ranging from existing and new home sales, home rentals, to home renovation, real estate financial solutions, and other services.
In the following years, Beike aims to cover over 300 cities in China and franchise more than 100 partners, linking 100 thousand branches and 1 million property agents and serving over 200 million households.
Beike has been recognized as China’s Combo of Zillow and MLS.
As a technology-driven housing service platform, Beike aggregates and empowers top service providers across the sectors to create an open and high-quality housing service ecosystem. Beike is committed to providing 300 million families with a full range of housing and property services, including second-hand property, new property, rental property, renovation, and community services.
In the second half of 2016, Beike started to plan the building of its metrics system to make clear the definition of metrics and statistical coverage and improve data sharing and security. The company also planned the metrics platform and started using Apache Kylin as the data engine for data services. To address the multidimensional analysis demands of the business lines, Beike has been using Apache Kylin for over four years.
Starting in late 2016, the company used a Hive + MySQL platform (internal codename: Geodynamics.)
The diagram shows the data flow on the platform. The colleague responsible for the data warehouse performs the initial pre-aggregation of data, which then answers queries through the relational database. However, the rapidly growing volume of data prolongs query response times, which in turn places extra pressure on the operation and maintenance of the underlying database. To solve these issues and build a metrics system for Beike, it is necessary to have an engine that supports large-scale data computation and a short query response time. Market research shows that Apache Kylin can be used to build our metrics system with its support to massive data computation, sub-second query response, standard SQL, maintainability, technology stack, and community activeness.
In March 2017, Apache Kylin 1.6 was released. With the launch of the metrics platform, Apache Kylin began offering its services in Beike. By the end of 2017, Beike had created over 300 Cubes and answered over 200,000 queries per day.
In early 2018, the metrics platform was rolled out across business lines, and more data products started to use Apache Kylin.
For example, data products, such as Merlin and Turing, cover a wide range of data needs from PC to mobile phones and involve all levels of a company’s organizational structure. To guarantee the output and query of key data, we deployed another cluster to support key business queries.
At the end of 2018, Beike had two clusters in total and built over 600 cubes, with daily queries reaching 2 million.
At the beginning of 2019, our Apache Kylin Team set 2 KPIs, which are to ensure key data could be generated before 9 am every day, and the response time is less than 3s for 99.7% of queries. We upgraded Apache Kylin to version 3.1 to implement the real-time multidimensional analysis.
To achieve both goals, we upgraded the cluster from 1.6.0 to 2.5.2 on the computation side and introduced Spark to shift the focus of Cube building from MapReduce to Spark.
The chart above shows performance before and after the optimization. The average build time of a key Cube has dropped from 70 minutes to 43 minutes, an improvement of about 40%. And less than 3s query response time for 99.7% of the queries was achieved in December.
The chart below shows the daily statistics at that time. By the end of 2019, Beike had two clusters, version 2.5.2, with 700+ Cubes and answered more than 10 million queries per day.
In early 2020, Apache Kylin was upgraded to version 3.1.0 and integrated Flink.
The following chart shows the build time comparison before and after using Flink for the level-1 metrics. The improvement is quite obvious. By the end of 2020, Beike had two Apache Kylin 3.1 clusters, 800+ Cubes, and answered over 23 million daily queries.
The diagram below shows the Apache Kylin cluster in our architecture.
We divided the cluster nodes into three roles. The Master node schedules the cube building and metadata query services. It does not handle the Build or Query services directly. There are multiple builders (Job nodes) and query machines (Query nodes) providing building or query services, which are divided on the cluster nodes side. The cluster itself is not available to the end-users and provides services through the upper-level platform.
On the platform side, we focused on API, tasks, queries, and metadata. We repackaged Apache Kylin’s API, simplified the process of Cube building, and integrated the company’s permissions system to control Model’s access.
Regarding job management, the platform controls job submission, including priority, operations, status monitoring, and alarms for abnormal data. In terms of queries, the platform performs routing to the cluster where the Cubes are located as well as real-time monitoring and analysis of queries. As for metadata management, we apply life cycle management to Cubes. As long as rules are met, the process of taking Cube offline will be initiated. Metadata management also includes migration of Cube between clusters, cluster version control, configuration management, etc.
The figure below shows a very interesting feature of the platform called Cube Query Analysis. With this feature, Apache Kylin’s query log is analyzed once an hour to count the number of times these Cubes are queried and which products use the data of the cube. In the chart below, you can see the percentage of data products using cubes. In this case, 7 this one cube.
The figure below shows the percentage of cube query response time. This cube was queried 690,000 times, and response time was less than 3s for 99.99% of queries.
We looked into each SQL query parsed by cube to see the dimension groups used by SQL and the corresponding response times. The following figure covers three types of data:
These data help Kylin users to better understand data usage and to make targeted optimizations in building and querying.
The efficient application of Apache Kylin at Beike is not possible without the contribution of our colleagues. The following chart shows some of the records contributed to the community by Beike colleagues over the past years. Four colleagues have contributed code to Apache Kylin, covering various aspects such as job scheduling, Web UIs, optimization of build, and query from version 1.6 to 3.1.
The following diagram shows the architecture of the Apache Kylin-based metrics platform. After modeling, the data is then used to serve the metrics platform. The metrics platform provides services to the users in the form of APIs, which are based on the metrics defined by the business lines.
The following is the process of calculating and using Apache Kylin-based metrics.
Next, I will show you two metric examples with two different calculations, one is SUM and the other is COUNT DISTINCT.
Accurate count distinct, an advantage of Apache Kylin, is also a key requirement of Beike’s metrics system, especially for some performance-related metrics, such as the number of site visits taken by agents.
The two pictures on the left and right sides show reports displayed on a mobile phone, and the one in the middle is the report displayed on a computer. These products all obtain corresponding metrics data through fixed dimension groups. The report is quickly generated by doing quick filtering.
The other is a self-service analysis scenario, which allows for flexible dimension selection. The figure below shows the Odin platform developed by our company. The two red boxes on the left are the dimensions and metrics, which the user can select. The figure on the right is a chart generated by Apache Kylin based on the user’s choice. After determining the dimensions and metrics, the user can save the configuration as a fixed report.
Whether it is a fixed report or self-service analysis, the underlying query process remains the same.
First, the business line calls metrics through API. They need to specify the dimensions, the time frame, and filtering conditions to trigger API calls.
Then, the metrics platform accepts API calls, converts API parameters into standard SQL, and submits the query to the Apache Kylin cluster for execution. After the query is completed, the results will be returned to the metrics platform, which packs the data into a fixed format and returns it to the business line. This is the underlying query process for Beike’s various data products using Apache Kylin.
The figure below shows the usage of Apache Kylin at Beike. It is connected to the company’s metrics system to cover all business lines. Apache Kylin provides query services for more than 30 data products and supports the calculation demand of 10,000+ metrics. The maximum daily query volume is more than 23 million. We promise to achieve less than 3s query response time for 99.7% of the queries. Currently, we are able to deliver as promised.
Currently, changes involving Cube are cumbersome. We did a brief test of Apache Kylin 4.0 at Beike. We hope that with Apache Kylin 4.0, the modeling process can be more streamlined and flexible, such as supporting dynamic schema updates.
In the meantime, we hope to support multi-tenancy at the query level to avoid interaction among different business lines. We have a large number of business providers and this problem occurs from time to time.
We are also planning on deploying Apache Kylin to Kubernetes to bring down the costs. Currently, the number of machines and instances in Kubernetes is relatively high, so are the O&M costs.
Read on to learn the key competencies and critical features to look for when evaluating a semantic layer offering for your BI tool.
Kyligence Zen intelligently manages data in the retail industry. Read to learn how to develop the "North Star Metric" system to track goals and progress.
Kyligence introduces the deployment of OLAP on top of Azure, including data sources, features, benefits, and prerequisites. Learn more about Kyligence for Azure.
What's OLAP on big data? What're its benefits? Here's everything you need to know about OLAP.
Learn how one big fast-food brand leveraged Kyligence capabilities and implemented precision marketing to maximize profit opportunities.
99 Almaden Boulevard Suite #663
San Jose, CA 95113
+1 (669) 256-3378
Ⓒ 2023 Kyligence, Inc. All rights reserved.
Already have an account? Click here to login
您还可以在云平台中 部署 Kyligence
直接获得 30 天免费试用
请填写真实信息，我们会在 1-2 个工作日内电话与您联系。