Metrics Store: The Good, The Bad, and Ugly

There has been a surge of interest in the metrics store, which provides a standard set of definitions on top of the data warehouse. In this article, I will explain the good, the bad, and the ugly part of the metrics store. And how you can apply DataOps practices to your business metrics to avoid the bad and ugly parts and enjoy the good part of the metrics store.

The Good, The Bad, and The Ugly of Metrics Store

The Good of Metrics Store

The metrics store provides a virtualized middle layer between upstream data warehouses/data sources and downstream business applications, allowing users to create and define business metrics as code or using a graphical user interface (GUI).

It abstracts away the complexity of underlying data warehouses and provides end-users with easy-to-understand metrics to reuse and analyze in downstream analytics, data science, and business applications. This can bring businesses increased efficiency, agility, and collaboration in metric consumption.

The Bad of Metrics Store

However, the bad part of metrics store is businesses' potential loss of freedom. IT-driven governance, control, and consistency may conflict with business-driven agility, self-service, and autonomy.

If metrics are solely created and curated by IT, the business will need to constantly request service from IT to build, test, and deploy the metrics. This process will slow down business in delivering value quickly and agilely for analytics.

The Ugly of Metrics Store

Furthermore, it is worth noting that the potential for overwhelming and disordered metrics as a business grows is just one issue plaguing metrics stores. This can result in difficulty in tracking key performance indicators (KPIs), leading to suboptimal decision-making.

The key to maintaining the good and avoiding the bad and ugly parts of metrics store is to balance IT-driven governance, control, and consistency of data with business-driven agility, self-service, and autonomy. To make metrics changes more agile, organizations need to extend DataOps practices to metrics management. This can be achieved by defining metrics in human and machine-readable configuration files, integrating them into version control systems, and automating the integration and testing stages with CI/CD platforms and testing tools. By doing so, organizations can achieve automated, collaborative, and agile metrics management.

What is DataOps, then?

According to Gartner, DataOps is a collaborative data management practice that aims to improve communication, integration, and automation of data flows between data managers and data consumers within an organization.

DataOps takes inspiration from DevOps, a set of tools, practices, and philosophies used to support the continuous delivery of software applications in the face of constant changes.

Extending DataOps Practices to Managing Your Business Metrics

With the obvious values of DataOps, and its potential to harness the ugly part of metrics store, it makes sense for your organization to build DataOps practices. What’s more, we propose to take one step further to implementing such practices in your metric management by simply following two steps:

Step 1: Build a metric layer on top of your data infrastructure

Why a standalone metric layer?

Metrics are the common data language for business. They help businesses build a consensus on the measurable results of business objectives. That is why you must enable unified metrics definitions across the business that enables trust, observability, and productivity.

Why is BI failing to provide the metric layer?

As experts in data and analytics, we know that the traditional BI approach has limitations, including the need for a common language and the coupling of visualization with business metrics.

With a centralized metrics store, the metrics logic will be defined repetitively in different tools, causing metrics consistency and discrepancy.

A standalone metrics layer acts as a semantic layer that converts technical data into business metrics, bridging the gap and enabling non-technical users to understand, explore, and gain insights from traditional data warehouses.

Step 2: Applying DataOps Practice to Your Business Metrics

To apply DataOps practice to your business metrics, you need to transform your metrics into manageable pieces of code by defining them in human and machine-readable configuration files. These configuration files should be integrated into version control systems to track, review, and monitor changes. With CI/CD platforms and testing tools, you can automate the integration and testing stages and deploy metrics to end-users faster, with higher quality and lower error rates.We call this idea Metrics-as-Code, which involves managing metrics as if they were code. It extends the DataOps practice to your business metrics.The benefits of Metrics-as-Code are clear: you can achieve automated, collaborative, and agile data management, and all the software development best practices that were previously applied to building software are now possible with managing metrics.

What you can do with Metrics-as-Code as a data developer

• Version control for metrics: With metrics-as-code, you can version the entire metrics solution and each metric in it, enabling you to track, review, and monitor changes made to the metrics over time.
• Test Automation: Metrics-as-code allows you to automate the testing of your metrics, ensuring that they are accurate and reliable before deploying them to end users.
• Environment Management: Metrics can be easily migrated from one environment to another by simply transferring the code. This makes it easy to maintain consistency across different environments and ensures that the metrics are the same in each environment.

For businesses, there are benefits too

• Metrics templating makes it easy to build a group of metrics as manageable code, and package them into reusable and shareable templates. This helps businesses quickly jump-start their metrics, saving up to 80% of the effort required to build metrics from scratch.
• Metrics lineage allows businesses to easily track data lineage, from data warehouse tables to metrics. Information such as the data source used to create the metric, the date it was created, and any transformations or calculations performed is available in code as metadata.

How about business-driven agility, self-service, and autonomy?

By implementing dataOps practices, the data team gains governance and control. However, how can business users perform self-service analytics without relying too heavily on the data team? Here is how one of our customers approaches achieving a perfect balance of both.

The data team could define the base metrics and trigger the data warehouse to compute the metrics automatically. Then, the business users can derive new metrics from the existing ones by using drag-and-drop GUI. This workflow allows the business to create new metrics without relying on data team, which saves time and increases efficiency. Additionally, it allows IT to focus on creating new metrics rather than implementing existing ones.

For example, a top commercial bank successfully rolled out a self-service metrics platform — Pandora and adapted metric-driven DataOps practices. Since Pandora went live, the end-to-end delivery time of business metrics has been reduced to 5 workdays because 30 out of 50 metrics are already available in the repository and ready to ship. The other 15 metrics could be derived from the existing ones by applying simple filtering or mathematical transformations. So BI engineers only need to focus on creating the 5 new metrics instead of implementing all 50 metrics.

Kyligence’s Approach to Enabling DataOps for your Business Metrics

At Kyligence, we help businesses improve their metric delivery efficiency by extending DataOps practices to business metrics. We've created ZenML, a YAML-based language that Kyligence Zen uses for defining metrics, dimensions, and relational datasets. With ZenML, data engineers can define, manage, and govern metrics as code. Kyligence Zen uses it for storing metric definitions and providing data services like SQL-based queries, BI integrations, metrics templating, goal management, progress tracking, alerting, anomaly detection, and metrics data lineage.

ZenML is a common data language that enables data engineers to apply engineering practices on metrics, leading to more efficient and effective management of data assets.You can use ZenML to store all metric definitions and their relations and mappings to relational data models. With this metadata, Kyligence Zen can automatically translate all business requests and tasks into SQL executions. A variety of data services are then provided around ZenML, including.

• SQL-based metric queries and BI integrations via open APIs like JDBC / ODBC / RestAPI
• Metric templates of business domain know-how
• Causal analysis, anomaly detection
• Goal management, progress tracking, and alerting
• Metrics data linage
• And more...

You can also maintain ZenML/YAML files in git for version control and publish (or rollback to) a certain version of metric definitions to Kyligence Zen service for business operations.

Closing

In this article, I discuss the benefits of using DataOps practices to manage business metrics. By building a metric layer on top of your data infrastructure and applying DataOps practices to your business metrics, your organization can achieve automated, collaborative, and agile data management.I also introduce ZenML, a YAML-based language for defining metrics, dimensions, and relational datasets, which enables data engineers to define, manage, and govern metrics as code.If you're interested in extending DataOps to your business metrics, here's what you can do next:

1. Try out Kyligence Zen and ZenML for free at our website. Follow this quick start guide to build your first Metric Store using ZenML.

2. Read the following posts to learn more about Kyligence Zen and the metrics store:

Understanding the Metrics Store

Metrics Store vs Semantics Layer

Video: What is Metrics Store

Reference

• Gartner. (n.d.). DataOps. Retrieved from https://www.gartner.com/en/information-technology/glossary/dataops#:~:text=DataOps
• Aslett, M. (2021, March 10). DataOps: Understanding the definition and differentiation. Ventana Research. Retrieved from https://mattaslett.ventanaresearch.com/dataops-understanding-the-definition-and-differentiation
• Kyligence. (n.d.). ZenML Intro. Retrieved from https://zen-docs.kyligence.io/en/zenml-intro
• Kyligence. (2021, January 15). Understanding the Metrics Store. Retrieved from https://kyligence.io/blog/understanding-the-metrics-store/
• Kyligence. (2021, July 2). Metrics Store vs Semantics Layer. Retrieved from https://kyligence.io/blog/metrics-store-vs-semantics-layer/
• Kyligence. (2019, May 16). What is Metrics Store [Video]. YouTube. https://www.youtube.com/watch?v=QqNHprbMzpg&feature=youtu.be
• China E-business Network. (2021, April 22). Ping An Bank: Pandora Metrics Platform. https://www.cebnet.com.cn/20210422/102743969.html
• Monte Carlo. (2023, March 22). What is DataOps? Retrieved from https://www.montecarlodata.com/blog-what-is-dataops/