BI Dashboards are Creating a Technical Debt Black Hole

The Journey Toward Data Democratization

How to avoid technical debt during cloud transformation

Is Migrating Legacy BI Dashboards to Cloud Making Any Sense?

As businesses are rushing to the public cloud, recreating BI dashboards in the cloud sounds like common sense, but is it REALLY the right thing to do? Now, let’s take a step back and think really hard before moving forward.

BI Dashboards are the Technical Debt of Yesterday and Today

Experienced BI engineers have long been aware that business dashboards behave like a financial debt that piles up over time. They are usually counted as an asset, business-wise. But behind the glitz and acclaim are the messy, dirty but interconnected data silos.

Imagine, now, you are on a mission to create a single source of truth by strategically moving legacy dashboards to the cloud. With hundreds of or even thousands of dashboards scattered amongst different locations being used for different purposes in different business units, there’s no wonder that it’s hard to wrap your head around that… Let alone create a single source of truth. This is literally another twisted form of “spaghetti code” in the Business Intelligence space.

How Does the Dashboard Debt Mount Over Time?

From a high-level perspective, this chaos could be traced back to a traditional mindset on BI architecture — data flows through pipelines to data warehouses and gets visualized in dashboards. Every time a new dashboard is requested, a new data pipeline and dashboard will be added to the existing pool; a pattern has been formed and followed again and again.

As business continues to evolve, more business users get access to data, more dashboards get implemented, more data silos, overlapping and inconsistency get created.

In this traditional BI architecture, enterprise data is tightly coupled to dashboards and being managed at a relatively low granular level — dashboard by dashboard; it affects the viability of component reuse across dashboards.

Moving to cloud, data practitioners need to pay the debt from the past and hope to reduce them all through a one-time endeavor. They probably will be drawn to another Technical Debt Black Hole if they choose to stay in the comfort zone by crafting TONS of dashboards for business stakeholders?

Anatomy of Business Intelligence Dashboards

Before sharing a solid battle-tested solution with you, let’s first deconstruct BI dashboards and understand their “Anatomy”.

"We’re not designing pages, we’re designing systems of components."

by Stephen Hay

In searching for inspiration and parallels, I kept coming back to Brad Frost’s Atomic Design model. My thought is that data products (whether dashboards or data applications, etc.) are comprised of metrics.

We’re not designing dashboards; we’re designing a collection of metrics.

We’re not building dashboards; we’re building a centralized repository of shared metrics.

We do not own dashboards; we empower non-technical data consumers to assemble dashboards in a self-service headless manner.

“Atomic Design” applied in Business Intelligence gives us the flexibility to traverse from dashboard to dashboard and improve metric consistency and agility while simultaneously achieving low-cost data democracy at scale.

How to Avoid a Technical Debt Black Hole for the Future

Previously, we were doing version control at the dashboard level, so upstream and downstream components are tight-coupled, and most metrics could hardly be shared; Applying the Metric Design Mindset, we are introducing the middle layer to decouple upstream data pipelines from downstream data consumers and enable the metrics to be reused across dashboards; instead of worrying about the ongoing dashboard creation workload, the dashboard implementation process gets simplified — just assemble predefined certified metrics into a self-serve dashboard if no new metrics required.

Some Kyligence customer has successfully implemented and operated a metrics store powered by Kyligence offerings to “bypass” this dashboard black hole. There are some thoughtful designs and key learnings to share with you in the upcoming blogs. I’ll take you through their ENTIRE journey, business-wise, technology-wise and people-wise!

The Facts About the Metric Platform — Dec 2021

• Total Active Metrics: 9700+
• Total Self-Service Dashboards Created: 7k+
• Daily Unique Visitor: 5k+
• Daily Query Count: 40k+
• Daily Query Performance: 90% less than 5s