Semantic Layer Explained: The BI Trend for 2023

October 16, 2023: When I first penned down this article on semantic layers in 2020, the concept was fairly fresh in the market landscape. Over the span of three years, the data industry has evolved, and I've updated this article to reflect the latest developments.

One important development is the metrics store concept, which is quickly becoming popular in the modern data stack. It changes how we see and use data. This new version aims to help you understand semantic layers and metrics stores, which are important in today's data-focused world.

 

In the digital age, businesses are inundated with data, yet starved for insights. As enterprises embrace technologies like data lakes and cloud computing, they often find themselves lost in a sea of data silos. The key metadata within data warehouses remains an enigma to business users, making data analysis a herculean task. What's the missing link between business users and the treasure trove of insights locked away in data warehouses?

 

 The 'Semantic Layer' emerges as a solution to navigate these challenges. Acting as a bridge between technical data infrastructure and business users, it provides a unified and comprehensible view of data assets. 

This article delves into the concept, importance, and implementation of a semantic layer in today's business intelligence landscape.

 

What exactly is a semantic layer?

What is a Semantic Layer?

A semantic layer is a translation layer that sits between data and business users. It converts complex data into familiar business terms.

 

The semantic layer simplifies and organizes business logic, hierarchies, calculations, and more. It is a data model layer separate from the technical implementation layer. This frees business users from concerns about the technical complexity and implementation of the underlying data source.

   

Anyone who uses data should be able to find, understand, and use it easily, regardless of their knowledge of data. The semantic layer provides business users with an easy way to understand the data.

The semantic layer is built on the source data, such as a data warehouse, data lake, or data mart. It defines metadata to enrich the data model and make it simple enough for business users to understand. 

 

Gartner points out the importance of a semantic layer in their report “How to Use Semantics to Drive the Business Value of Your Data”:

 

“Unprecedented levels of data scale and distribution are making it almost impossible for organizations to effectively exploit their data assets. Data and analytics leaders must adopt a semantic approach to their enterprise data assets or face losing the battle for competitive advantage.” [1]

 

Benefits of Semantic Layer

 

• Self-service analytics: It is a valuable enabler of self-service analytics that balances agility and control.
• Integrability: It allows for integration with different data stores and file formats, making data more accessible and standardized.
• Development Efficiency: It makes development efficient for technical professionals and easy enough for citizen developers by allowing integration with various data sources and enabling the building of abstractions, definitions, metrics, and measures.
• Centralized Maintenance: It can be maintained by a central data management or analytical development team, ensuring consistency, governance, and the ability to deliver trusted data.
• Leveraging Existing Technology and Skills: Building a semantic layer at the edge of the data layer leverages existing technology, architecture, and in-house skills, making it a cost-effective option for implementation.

 

Who is Semantic Layer for?

 

Business User

• Can view analytic content periodically and use it to make data-driven decisions.
• The ability to query using natural language and receive autogenerated insights streamlines their analysis process.

Data Analyst

• Has the ability to select from available fields in the semantic layer, aiming for diagnostic analytics.
• Seeks answers to "why" questions through deeper analysis.
• Utilizes natural language processing, SQL generation, automatic visualization generation, and machine learning services for insights.

Data Scientist

• Can mash up multiple certified data sources.
• Queries against large datasets, create visualizations, and generate insights on impactful data.
• Uses automatic data profiling, data classification recommendations, visual lineage, impact analysis, and advanced analytics.

Data Engineer

• Can introduce data from entirely new data sources, even those not traditionally linked with analytics.
• Centralized maintenance of a semantic layer ensures consistency and governance, while development efficiency allows data engineer to easily build abstractions and metrics on top of different data sources.

Examples of Semantic Layers

 

Retail Business: Amidst the COVID-19 pandemic, a major Food and Beverage (F&B) retail company leveraged a semantic layer to fast-track its digital transformation on Microsoft Azure. The setup facilitated seamless data integration from various sources, processed through Apache Spark. The data, once processed, was funneled into Kyligence's Semantic Layer for analysis. This architecture simplified data management by auto-collecting and categorizing data into cubes based on predefined metrics, enabling non-technical users to conduct analysis using familiar tools like Excel.

 

Manufacturing: In the realm of automobile manufacturing, a prominent manufacturer leveraged a semantic layer to amplify the potential of its data lake on AWS, especially focusing on data analytics concerning the Internet of Vehicles. Car owner data was systematically collected and analyzed through the semantic layer, enabling precise service marketing by grouping car owners based on their behaviors. Additionally, it facilitated a detailed analysis of services-related churn rates to optimize user data access processes, thereby reducing churn. Furthermore, the collection and analysis of vehicle trajectory data through the semantic layer ushered in derivative services like designated driving, thus enriching the data dimensions and enhancing service offerings. The semantic layer played a quintessential role in translating technical data into actionable insights, propelling service optimization, and fostering innovation in service delivery.

 

Financial Services: Consider the transformation at a top commercial bank with the roll-out of Pandora, a self-service metrics platform, acting as a semantic layer. Initially, a data dashboard with 50 metrics took 12 workdays to deliver. With Pandora, the delivery time shrunk to 5 workdays, as 30 metrics were readily available and 15 were easily derived, leaving only 5 new metrics to be created. This platform, akin to a semantic layer, democratized data by enabling non-technical users to assemble dashboards effortlessly, while IT experts focused on creating new metrics. This scenario vividly illustrates the efficiency and simplification a semantic layer brings to data management and business self-service.

Critical Capabilities of a Semantic Layer

 

How should data and analytics leaders implement the semantic layer to take full advantage of its capabilities?

 

Below are the key competencies you should consider:

 

• Shared Business Logic in Metrics

 

A semantic layer contains the core logic required for business analysis, transforming the underlying data model into familiar business definitions (dimensions, metrics, hierarchies) and easy-to-understand terms. It can contain commonly used derived metrics, such as year-over-year, month-over-month, month-to-date, etc. Users can directly consume the metrics and reuse the semantics in downstream applications.

 

Through a variety of query interfaces, the universal semantic layer serves as the endpoint for data analysis. This endpoint may be a BI tool or a customized application.

 

• Unified Security Policy

 

A semantic layer ensures users and data access controls are uniformly applied in all downstream analysis or business applications, so IT doesn’t need to configure data access control for individual downstream systems.

 

• High-Performance Backend Engines 

 

The semantic layer must have a powerful built-in engine or be able to connect to a big data engine such as Apache Spark. It can give businesses a more comprehensive view of their data to conduct analysis on massive, detailed datasets. This cannot be done without a powerful backend engine.

 

Several vendors have introduced their own semantic layers. Let’s take a look at a few BI leaders and their offerings.

     

Architecture of a Semantic Layer 

 

Feature	Semantic Layer in Data Warehouse	Standalone Semantic Layer	Semantic Layer in BI Tools
Integration with Existing Data Services	Yes	Depends on Vendors	Seamless
Flexibility and Customization	Limited	High	Limited
Vendor Lock-in	No	No	Yes
Development Effort	High	Moderate	Low
Performance	Potential Issues	Good	Depends on BI Tool
Governance and Centralization	Good	High	Poor
End-User Friendliness	Poor	Good	High
Support for Massive Data Processing	Yes	Yes	Depends on BI Tool
Cloud Deployment	Yes	Yes	Yes

Semantic Layer Architecture in Data Warehouse

This option involves integrating the semantic layer within the existing data warehouse or data layer. It allows for the creation of abstractions, definitions, metrics, and measures on top of different data stores and file formats. The semantic layer connects to various layers, making data more accessible and standardized.

Example tools and vendors: Oracle, SAP, Amazon Web Services, Microsoft Google, IBM, Snowflake, Teradata, Incorta

Pros:

• Building views on top of data warehouses or virtual mart environments provides trusted data for analytics models.
• Virtual marts can be useful for candidate and contender approaches.
• Managing a semantic layer through a central location can be an efficient way to handle multiple marts and concurrent analytics users.

Cons:

• Scaling and development challenges due to the need for building multiple views.
• Limited end-user friendliness of data warehouses.
• Potential performance issues with querying and optimization.
• Additional development effort is required for implementing calculations and business logic.

Standalone Semantic Layer Architecture

In this option, the semantic layer exists as its own architectural element within the data and analytics stack. It offers a dedicated semantic layer independent of specific data services or technologies. Users can build abstractions, definitions, metrics, and measures by connecting to various data layers.

Example Tools and vendors: Kyligence, Cloudera, Microsoft, AtScale, Cambridge Semantics, Kyvos Insights

Pros:

• Offers capabilities to connect disparate data across an organization's data landscape
• Provides governance and centralization, allowing for consistent development and maintenance
• Simplifies data access for analytics and improves reuse of metrics
• Creates a consumption-tool-agnostic, reusable semantic layer

Cons:

• Requires additional effort for integration with different data stores and file formats.
• May require additional maintenance and governance efforts.
• Lack of integration with existing data services within the data layer.

What's happening in the market

• Kyligence introduced Kyligence Zen, a Metrics Platform, to centralize metrics, automate data pipelines, and support multiple BI tools, advancing its standalone semantic layer offerings for improved business analytics and data trust across organizations.
• AtScale announced new capabilities within its semantic layer platform to support code-first data modelers, including developers, analytics engineers, and data scientists.
• Kyvos Insights introduced an Analytics Acceleration Semantic Layer on Azure Marketplace, a feature mirroring offerings by competitors, positioning Kyvos as a follower in this aspect of cloud analytics accessibility.

Semantic Layer Architecture in BI Tools

This option involves utilizing the semantic layer provided by analytics and business intelligence (A&BI) tools. These tools often provide built-in capabilities for creating and managing the semantic layer, allowing users to define and organize data models, measures, and hierarchies.

Example tools and vendors: Info Birst, IBM, Microsoft Power BI, MicroStrategy, Oracle, Salesforce, SAP, Sisense, Tableau, Google Cloud Looker

Pros:

• Seamless integration with A&BI tools, providing a unified user experience.
• Simplified development and management of the semantic layer within the tool.
• Leverages the capabilities and features of the A&BI tool for data modeling and analysis.

Cons:

• Vendor Lock-in: Dependency on the specific A&BI tool for the semantic layer.
• Limited flexibility and customization options compared to stand-alone semantic layers.
• It may require additional licensing costs for the A&BI tool.

What's happening in the market

• In version 2020.2, Tableau introduced a logical model layer for complex modeling. Additionally, the new VizQL Data Service allows embedding Tableau into automated workflows, simplifying query construction and data modeling, acting as a semantic layer. However, it's limited to Tableau's ecosystem.

1. Power BI: Introduced read-write XMLA endpoints in Power BI Premium, facilitating broader tool compatibility and third-party integration.
2. Looker: In March 2023, Google Cloud launched Looker Modeler as a standalone metrics layer that acts as a semantic layer allowing metrics to be consumed across various BI tools.
3. MicroStrategy: Emphasized federated analytics since 2019, enabling unified analysis and IT governance through reusable objects and definitions.

Best Tool for Building a Universal Semantic Layer

 

At Kyligence, we recognize the immense value provided by a semantic layer. Based on years of experience working with our valued customers, we have concluded that a platform that converts technical data into business metrics is the best way to enable business users to explore, understand, and gain insights.

   

A unified metrics layer or metric store, built through a graphical user interface (GUI), empowers business users to define business terms and calculation logic precisely. This creates a universal data language for all businesses to align and collaborate in one convenient location, ensuring that everyone in the organization is on the same page.

 

Let's look at some key capabilities that Kyligence Zen, our metrics store, can provide:

 

• Low-code Metrics Catalog

 

The Metrics Catalog is a central location where users can define, compute, organize, and share metrics using GUI. It allows users to define both basic metrics like overall sales and derived/composite metrics like year-to-date (YTD) and year-over-year (YOY)sales.

 

• Automated Metric Computation

 

Our product can automatically compute metrics, freeing users to focus on the business impact and outcomes. With Kyligence Zen’s automated metrics calculation by the built-in augmented OLAP engine, business users can spend less time calculating and more time taking action on the data that matters most.

 

• Open APIs

 

Kyligence Zen offers Open APIs to connect your organization's data and metrics with any BI or SaaS tool. This agnostic metrics store enables businesses to align data consumers and decision-makers on one unified platform, offering various use cases such as connecting BI for in-house data analytics and decision-making, IM for pushing alerts and collaborating among teams, and SaaS tools for realizing different business use cases or streamlining business workflow.

 

What Types of Semantic Layers Are BI Vendors Offering? 

 

Tableau's Semantic Layer Capability: Enhanced Complex Modeling Capability

 

In the release version of Tableau 2020.2, Tableau introduced a logical (semantic layer) model layer to help users associate more data models. The introduction of this function enables each Tableau data source to support the analysis of multiple fact tables and complex analysis scenarios such as many-to-many relationships (previously, it could only support a single fact table). 

   

As you can see, Tableau provides a new semantic layer that enhances its ability to perform complex modeling and analysis. This newly launched data source can be used in Tableau's software ecosystem. By publishing this logical model layer to Tableau Server, more business users can use the logical model in the shared data source through their browsers, and IT can monitor the published data source and control/authorize user rights to data.

   

The semantic layer provided by Tableau assumes IT-centered model management with self-service capabilities in mind. The modeling process is simple and easy to use with a low learning curve. This transparent and seamless modeling method makes Tableau's semantic layer very appealing. However, Tableau's semantic layer does not work with other BI tools. It is common for large enterprises and their different business units to use different BI tools, and so, for these organizations, the Tableau semantic layer can be very limiting.

 

Power BI's Semantic Layer Capability: Unified Semantics With Support for Multiple Applications

 

In March 2020, Power BI released a public preview of read-write XMLA endpoints in Power BI Premium. Power BI Premium provides open-platform connectivity for Power BI datasets, enabling customers to leverage a semantic layer compatible with a wide range of data-visualization tools from different vendors. This means any third party can consume (read) or synchronize (write) the Power BI semantic layer by reading and writing XMLA endpoints.

 

The Power BI semantic layer can be used with other BI tools. From the below diagram by Power BI, we can see that Power BI supports third-party tools to define, manage, and diagnose the semantic layer of Power BI on the write side, and other visualization tools (including Tableau and Excel, as shown in the diagram) on the read side.

   

We can see that the latest read-write XMLA endpoints capability of Power BI enhances the capability of Power BI Premium with reusable models. Combined with its ability to perform ultra-complex modeling, its semantic layer can be a very good choice for enterprise-wide BI deployment.

 

MicroStrategy: Federated Analytics Enables a Unified Analysis Platform With IT Governance

 

MicroStrategy has emphasized its Federated Analytics capability since 2019. The semantic layer can be reused with different BI tools through reusable objects and definitions, realizing the unification of underlying discrete data sources and providing a single source of truth.

   

BI tools are continuously evolving, so it’s important to take a long-term investment view; enterprises should avoid vendor lock-in of the semantic layer. Separating the semantic layer from a particular BI provider allows for higher flexibility and scalability. Ideally, the semantic layer should be BI tool agnostic.

 

Other Important Considerations:

 

• Open Ecosystem

 

One of the main purposes of the semantic layer is to give users easy access to a set of unified business definitions. If the semantic layer can only be used with a particular vendor’s tools, it defeats a primary purpose of a semantic layer. The semantic layer should work with various tools for maximum value and investment payback.

 

• Cloud Deployment

 

Data is now stored in various locations, including the cloud, which is increasingly becoming the preferred choice for enterprise IT. A semantic layer that can be deployed in the cloud ensures it will be able to meet both current and future changes in IT architecture.

   

Factors to Consider When Adopting the Semantic Layer

 

Enterprises should consider the following aspects when selecting a semantic layer:

• Does it integrate well with popular BI tools and other interfaces?
• Does it support massive data processing or have the ability to integrate with big data compute engines?
• Does it support cloud deployment, which is becoming increasingly popular?

 

Sources for evaluating potential vendors:

• Gartner's peer insight review for analytics query accelerators
• Read Gartner's report on Demystifying Semantic Layers for Self-Service Analytics(Gartner's client access)
• Recommended reading: Demystifying the Metrics Store and Semantic Layer

Summary

Adopting a Semantic Layer is pivotal for enterprises aiming to bridge the gap between complex data and insightful analytics. It serves as a catalyst in democratizing data, simplifying data access, and fostering a unified understanding across various business units. By translating technical data into user-friendly terms, it accelerates decision-making processes, maximizes data asset utilization, and propels enterprises toward a data-driven culture. The flexibility it offers in integration with popular BI tools and big data engines, coupled with its cloud deployment capability, makes it a future-proof investment for evolving IT architectures.
Explore the transformative potential of a Semantic Layer by booking a demo with Kyligence to delve deeper into our robust solution tailored for your enterprise’s analytical needs.

Frequently Asked Questions on the Semantic Layer

 

We cover some of the hitching questions on the Semantic data layer

 

What is the semantic layer?

 

A semantic layer constitutes a data interpretation that aligns with business perspectives and delivers an integrated, comprehensive data view within a business entity. Diverse data descriptions from various data repositories can be efficiently interconnected through a semantic layer, providing a cohesive, reliable, and unique data perspective for analytical tasks and other commercial objectives.

 

Comparing Semantic Layer with Metrics Store

 

Both Semantic Layer and Metrics Store are pivotal in bridging the gap between raw data and actionable insights. A Semantic Layer acts as a translator, simplifying complex data into business-friendly terms, aiding in easier comprehension and utilization. Conversely, a Metrics Store centralizes and standardizes metrics definitions ensuring consistency across various tools and dashboards. While the Semantic Layer focuses on data translation and simplification, Metrics Store emphasizes on consistency and standardization of metrics. Both play crucial roles in enhancing data accessibility and understanding, albeit through different approaches, contributing to a more robust and effective data analytics infrastructure.

 

Sources and Additional Resources:

 

[1] How to Use Semantic to Drive the Business Value of Your Data

[2] 10 Enterprise Analytics Trends to Watch in 2020

[3] Tableau for the Enterprise: An Overview for IT 

[4] The Tableau Data Model

[5] MicroStrategy 2019 Whitepaper

[6] Announcing Read/Write XMLA Endpoints in Power BI Premium Public Preview

 

---

START A FREE TRIAL TODAY

---