Using semantic layers with data marts

 

Data management is fundamental to organizational success in the rapidly evolving digital landscape. Mastering the intricate dance between technical complexity and user-friendly accessibility can be challenging. However, crucial structures like semantic layers and data marts provide the necessary bridge. 

Semantic layers serve as a business-friendly lens, transforming technical data nuances into understandable business terms, while data marts provide targeted data subsets tailored to specific business domains. Their synergy can create a powerful data management solution, significantly enhancing data accessibility, analysis, and decision-making processes. 

This article delves into the in-depth exploration of semantic layers and data marts, their significance, potential integration, and the benefits of their combined use. 

We will also illustrate real-world applications across various industries, demonstrating how these powerful tools can transform data operations and contribute to business growth. 

Before we dive deeper, let’s quickly explore semantic layer and data marts definitions.

 

What is a semantic layer?

 

A semantic layer is a business abstraction derived from the technical implementation layer – a model layer that uniformly maintains business logic, hierarchies, calculations, and others. This frees business users from concerns about the technical complexity and implementation of the underlying data source. A data consumer (regardless of their data literacy level) must easily discover, understand, and utilize the data. The semantic layer provides business users with an easy way to understand the data.

 

A semantic layer contains the core logic required for business analysis. It can also transform the underlying data model into familiar business definitions (dimensions, measures, hierarchies) and easy-to-understand terms. Also, it can contain commonly used derived measures, such as year-over-year, month-over-month, month-to-date, and others. Users can directly consume the calculated measures and reuse the semantics in different downstream applications.

 

The unified semantic layer serves as the endpoint for data analysis through various query interfaces. This endpoint may be a BI tool or a customized application. So the end-users of the semantic layer should be the ones to analyze the data. They may be data analysts, business analysts, decision-makers, and report designers. Not engineers and developers.

 

In the same digital world, there are diverse users, for example, business analysts, data analysts, data engineers, and data scientists. Each tells their own data story and needs a unified data definition to translate their insights into data. The semantic layer is one such platform, storing not data but metadata. It acts as a single version of the truth, serving diverse users.

 

What is a data mart?

 

Traditionally, a data mart is a structure/access pattern used to retrieve client-facing data in data warehouse setups. A data mart is a subset of a data warehouse that is typically focused on a single business line or team.

 

Nowadays, it is also possible to build a data mart structure on top of a data lake. Notably, some vendors use the term "lakehouse" to refer to building a data mart on top of the data lake.

 

Whereas data warehouses or data lakes have enterprise-wide depth, the information in data marts pertains to a single department. Each department or business unit may be considered the owner of its data mart, which includes all hardware, software, and data in some installations.

 

Data mart vs. Data warehouse

 

In the realm of data management and analytics, both data warehouses and data marts play significant roles. However, their functions, scope, and uses set them apart.

• Scope and Purpose

 

A data warehouse acts as a central hub of preprocessed, structured data for the entire organization. It serves as a unified source for analytics and business intelligence on an enterprise-wide scale. On the other hand, a data mart caters to a specific business unit or department, such as finance, marketing, or sales. It is, in essence, a focused subset of a data warehouse.

• Data Structuring

 

Data warehouses are designed to handle structured data, i.e., data that has been cleaned, processed, and organized into a format ready for analysis. This contrasts with data marts, which may often deal with a narrower set of data tailored for specific use cases or teams.

• Ownership and Accessibility

 

While data warehouses are typically enterprise-wide resources managed at an organizational level, data marts can often be maintained by individual departments. This decentralized approach allows faster access to relevant data and facilitates quicker decision-making processes within specific teams.

• Cost and Implementation

 

Compared to building and managing a full-fledged data warehouse, setting up a data mart can often be more cost-effective and quicker to implement, given its smaller scale and focused scope. However, the overall cost-effectiveness would depend on the number of isolated data marts an organization may need to manage.

• Historical Data Storage

 

Both data warehouses and data marts store historical data, enabling analysts to identify trends and patterns over time.

• Data Lake Considerations

 

It's worth mentioning the concept of a data lake—a repository for raw and unstructured data—in this context. Unlike data warehouses that store preprocessed, structured data, data lakes allow for the storage of raw data, which can be processed as and when needed. This flexibility makes data lakes a valuable asset alongside data warehouses and data marts in an organization's data strategy.

Choosing between a data mart and a data warehouse—or deciding how to incorporate a data lake into your data strategy—depends on your organization's specific needs, resources, and the nature of the data you deal with. While data marts offer quick, targeted insights for specific teams, data warehouses provide a comprehensive view of the organization's data, beneficial for wide-scale, cross-functional analytics.

 

Often data mart is the last layer in the traditional data warehouse structure. Its position locates before connecting with business intelligence tools.

 

Advantages of the data mart

• A fraction of an organization's data is stored in data marts. This information is useful to a specific group of people within a company.
• It is a cost-effective alternative to a data warehouse, which can take high costs to build.
• Data mart allows faster access to data.
• Data mart is simple to use since it is tailored to the needs of its customers. As a result, a data mart can speed up corporate processes.
• Data marts need less implementation time compared to data warehouse systems. It is faster to implement a data mart as you only need to concentrate the only subset of the data.
• It contains historical data that allows the analyst to spot patterns in the data.

 

How a semantic layer integrates with data marts

 

The semantic layer can be located in many layers, for example, in BI, on top of the logical data warehouse (LDW) or data mart. Defining the semantic layer with the data mart is an option for the semantic layer on top of the logical data warehouse (LDW).

 

The traditional semantic layer, linked to traditional A&BI tools, functions as a data mart, providing a logic layer and a store of analytics-ready data with the context to support self-service by unskilled users. Data collection, on the other hand, can only go so far. As a result, new approaches centered on connecting to data have gained popularity.

 

For example, Tableau 2020.2 introduced a logical (semantic layer) model layer to assist users in associating more data models. With the addition of this function, each Tableau data source can now support the analysis of multiple fact tables  and complex analysis scenarios like many-to-many relationships (previously, it could only support a single fact table).

   

Tableau now has a new semantic layer that improves its capacity to execute complicated modeling and analysis, as you can see. Tableau's software ecosystem may take advantage of this freshly released data source. Moreover, more business users can utilize the logical model in the shared data source through their browsers as a result of publishing this logical model layer to the Tableau Server. Additionally, IT can monitor the published data source and control/authorize user rights to data.

   

Tableau's semantic layer is designed for IT-centered model management with self-service features. The modeling technique is simple and straightforward, with a short learning curve. Tableau's semantic layer appeals to me because of its transparent and seamless modeling style. However, it does not operate with other BI products. Large corporations and their many business segments frequently use different BI products. Therefore the Tableau semantic layer can be highly restricted for these organizations.

 

The traditional IT-built semantic layer failed for two fundamental reasons:

• They were difficult to set up and manage;
• They were difficult for end-users to use, alter, and update, indicating that now the semantic layer was often a platform exclusively utilized by IT. This reliance on IT slowed down the semantic layer platform's deployment and usability.

 

As a result, we advocate putting semantics on the edge of a logical data warehouse. Because the logical data warehouse (LDW) is built to meet 95% of analytics requirements. LDWs offer a wide range of analytic engines, allowing them to accommodate a diverse collection of users and applications. As a result, including the semantic layer in the LDW is frequently recommended.

 

In this approach, the semantic layer functions as a data mart sitting on top of an LDW. It can source data from other data stores, but the data warehouse is specifically modeled as a star schema or snowflake schema to support the semantic layer. Modelers can enhance the data in the semantic layer by adding hierarchies, calculated measures, etc.

   

Also, users can build a semantic layer with a data mart to ensure data definition and consistency inside a department, unit, or set of users in an organization. Examples include marketing, sales, HR, and finance. Let's look at the user cases of data mart with semantic layers within different industries and what business value can be generated.

 

Benefits of using a semantic layer with a data mart

 

A data mart is defined as a subset of a data warehouse focused on a single functional area of an organization. The semantic layer is the layer that BI tools usually connect to. Building a semantic layer with a data mart can implement many key benefits below.

 

Shared Business Logic

A semantic layer contains the core logic required for business analysis, transforming the underlying data model into familiar business definitions (dimensions, measures, hierarchies) and easy-to-understand terms. It can contain commonly used derived measures, such as year-over-year, month-over-month, month-to-date, etc. A semantic layer with a specific data mart can unify the data definitions in a specific area. Users in their specific areas can directly consume the calculated measures and reuse semantics in different downstream applications.

 

Unified Security Policy

A semantic layer ensures users and data access controls are uniformly applied in all downstream analysis or business applications. Data mart makes sure different areas of data get isolated, so IT doesn't need to configure data access control for individual downstream systems.

 

High-Performance Backend Engines

The semantic layer must have a powerful built-in engine or be able to connect to a big data engine such as Spark. The unified semantic layer can bring businesses a more comprehensive view of their data so that businesses can conduct analysis on massive, detailed datasets. This cannot be done without a powerful backend engine.

 

Enable interactive analytics for the end-user

A data mart can unify data from multiple sources. It provides an interface that BI tools use to enable ad-hoc analysis with drag and drop. Through a variety of query interfaces, the unified semantic layer with a data mart serves as the endpoint for data analysis. This endpoint may be a BI tool or a customized application.

 

Use cases for semantic layers with data marts

 

Here are the use cases on how to integrate data marts into semantic layers:

 

Finance

The financial industry has many data analysts who find the best portfolio of investments and calculate the risk factors of different markets. Financial analysts can use semantic layers to calculate the aggregated returns of multiple investment products, as well as risk factor assessments for large portfolios.

 

Retail

The semantic layer enables retailers across channels to integrate all data from POS systems, e-commerce, customer service, and marketing programs into one source. This enables analysts to assist marketers in creating better campaigns and experiences that meet customer expectations.

 

Manufacturing

One of the biggest pain points in the manufacturing sector is finding the production processes that need the most optimization. With easy access to data, manufacturing companies can build process forecasting tools to calculate the "best process" and execute against it.

 

Medical

With access to all relevant data, analysts can use semantic layers to analyze deteriorating patients.  Subsequently, they can allocate medical resources to the right patients, improving the management of medical resources.

   

Learn more about Kyligence Zen here or try out Kyligence Zen for free.

---