Excel Your KPIs with AI Copilot Start for free today
Your AI Copilot for Data
Definitive Guide to Decision Intelligence
Subscribe to our newsletter>
Get the latest products updates, community events and other news.
There has been a surge of interest in the metrics store, which provides a standard set of definitions on top of the data warehouse. In this article, I will explain the good, the bad, and the ugly part of the metrics store. And how you can apply DataOps practices to your business metrics to avoid the bad and ugly parts and enjoy the good part of the metrics store.
The metrics store provides a virtualized middle layer between upstream data warehouses/data sources and downstream business applications, allowing users to create and define business metrics as code or using a graphical user interface (GUI).
It abstracts away the complexity of underlying data warehouses and provides end-users with easy-to-understand metrics to reuse and analyze in downstream analytics, data science, and business applications. This can bring businesses increased efficiency, agility, and collaboration in metric consumption.
However, the bad part of metrics store is businesses' potential loss of freedom. IT-driven governance, control, and consistency may conflict with business-driven agility, self-service, and autonomy.
If metrics are solely created and curated by IT, the business will need to constantly request service from IT to build, test, and deploy the metrics. This process will slow down business in delivering value quickly and agilely for analytics.
Furthermore, it is worth noting that the potential for overwhelming and disordered metrics as a business grows is just one issue plaguing metrics stores. This can result in difficulty in tracking key performance indicators (KPIs), leading to suboptimal decision-making.
The key to maintaining the good and avoiding the bad and ugly parts of metrics store is to balance IT-driven governance, control, and consistency of data with business-driven agility, self-service, and autonomy. To make metrics changes more agile, organizations need to extend DataOps practices to metrics management. This can be achieved by defining metrics in human and machine-readable configuration files, integrating them into version control systems, and automating the integration and testing stages with CI/CD platforms and testing tools. By doing so, organizations can achieve automated, collaborative, and agile metrics management.
According to Gartner, DataOps is a collaborative data management practice that aims to improve communication, integration, and automation of data flows between data managers and data consumers within an organization.
DataOps takes inspiration from DevOps, a set of tools, practices, and philosophies used to support the continuous delivery of software applications in the face of constant changes.
With the obvious values of DataOps, and its potential to harness the ugly part of metrics store, it makes sense for your organization to build DataOps practices. What’s more, we propose to take one step further to implementing such practices in your metric management by simply following two steps:
Metrics are the common data language for business. They help businesses build a consensus on the measurable results of business objectives. That is why you must enable unified metrics definitions across the business that enables trust, observability, and productivity.
As experts in data and analytics, we know that the traditional BI approach has limitations, including the need for a common language and the coupling of visualization with business metrics.
With a centralized metrics store, the metrics logic will be defined repetitively in different tools, causing metrics consistency and discrepancy.
A standalone metrics layer acts as a semantic layer that converts technical data into business metrics, bridging the gap and enabling non-technical users to understand, explore, and gain insights from traditional data warehouses.
To apply DataOps practice to your business metrics, you need to transform your metrics into manageable pieces of code by defining them in human and machine-readable configuration files. These configuration files should be integrated into version control systems to track, review, and monitor changes. With CI/CD platforms and testing tools, you can automate the integration and testing stages and deploy metrics to end-users faster, with higher quality and lower error rates.We call this idea Metrics-as-Code, which involves managing metrics as if they were code. It extends the DataOps practice to your business metrics.The benefits of Metrics-as-Code are clear: you can achieve automated, collaborative, and agile data management, and all the software development best practices that were previously applied to building software are now possible with managing metrics.
By implementing dataOps practices, the data team gains governance and control. However, how can business users perform self-service analytics without relying too heavily on the data team? Here is how one of our customers approaches achieving a perfect balance of both.
The data team could define the base metrics and trigger the data warehouse to compute the metrics automatically. Then, the business users can derive new metrics from the existing ones by using drag-and-drop GUI. This workflow allows the business to create new metrics without relying on data team, which saves time and increases efficiency. Additionally, it allows IT to focus on creating new metrics rather than implementing existing ones.
For example, a top commercial bank successfully rolled out a self-service metrics platform — Pandora and adapted metric-driven DataOps practices. Since Pandora went live, the end-to-end delivery time of business metrics has been reduced to 5 workdays because 30 out of 50 metrics are already available in the repository and ready to ship. The other 15 metrics could be derived from the existing ones by applying simple filtering or mathematical transformations. So BI engineers only need to focus on creating the 5 new metrics instead of implementing all 50 metrics.
At Kyligence, we help businesses improve their metric delivery efficiency by extending DataOps practices to business metrics. We've created ZenML, a YAML-based language that Kyligence Zen uses for defining metrics, dimensions, and relational datasets. With ZenML, data engineers can define, manage, and govern metrics as code. Kyligence Zen uses it for storing metric definitions and providing data services like SQL-based queries, BI integrations, metrics templating, goal management, progress tracking, alerting, anomaly detection, and metrics data lineage.
ZenML is a common data language that enables data engineers to apply engineering practices on metrics, leading to more efficient and effective management of data assets.You can use ZenML to store all metric definitions and their relations and mappings to relational data models. With this metadata, Kyligence Zen can automatically translate all business requests and tasks into SQL executions. A variety of data services are then provided around ZenML, including.
You can also maintain ZenML/YAML files in git for version control and publish (or rollback to) a certain version of metric definitions to Kyligence Zen service for business operations.
In this article, I discuss the benefits of using DataOps practices to manage business metrics. By building a metric layer on top of your data infrastructure and applying DataOps practices to your business metrics, your organization can achieve automated, collaborative, and agile data management.I also introduce ZenML, a YAML-based language for defining metrics, dimensions, and relational datasets, which enables data engineers to define, manage, and govern metrics as code.If you're interested in extending DataOps to your business metrics, here's what you can do next:
2. Read the following posts to learn more about Kyligence Zen and the metrics store:
Understanding the Metrics Store
Metrics Store vs Semantics Layer
Video: What is Metrics Store
Learn about the fundamentals of a data product and how we help build better data products with real customer success stories.
Discover the 7 top AI analytics tools! Learn about their pros, cons, and pricing, and choose the best one to transform your business.
Discover operational and executive SaaS metrics that matter for customers success, importance, and why you should track them with Kyligence Zen.
Unlock the future of augmented analytics with this must-read blog. Discover the top 5 tools that are reshaping the analytics landscape.
What website metrics matter in business? Learn about categories, vital website metrics, how to measure them, and how Kyligence simplifies it.
Unlock potentials of analytics query accelerators for swift data processing and insights from cloud data lakes. Explore advanced features of Kyligence Zen.
Unlock power of data storytelling in business. Learn how to convey insights using narrative and visual representations, examples, and benefits.
Explore these exceptional cloud analytics tools. Assess their pros, cons, and pricing to pinpoint the optimal one for your business.
Learn what natural language query is and how it transforms your data analytics. Explore examples of natural language queries in Kyligence Zen.
Discover how AI shapes banking, healthcare, and data analytics sectors. Get insights into the future of industry disruption to guide your decisions.
99 Almaden Boulevard Suite #663
San Jose, CA 95113
+1 (669) 256-3378
Ⓒ 2023 Kyligence, Inc. All rights reserved.
Already have an account? Click here to login
A complete product experience
A guided demo of the whole process, from data import, modeling to analysis, by our data experts.
Q&A session with industry experts
Our data experts will answer your questions about customized solutions.
Please fill in your contact information.We'll get back to you in 1-2 business days.