5 megatrends that will shift the data and analytics market
Over the past year, as we interacted with enterprise customers of all sizes, we have seen 5 unmistakable megatrends that are shifting the data and analytics market. We have confirmed these trends by examining the perspectives of investors, research firms, and industry practitioners.
Trend 1: From Data Center/Cloud to Distrbuted Cloud
The shift from data center deployments to the cloud continues apace.
"New data from Synergy Research Group shows that enterprise spending on cloud infrastructure services continued to ramp up aggressively in 2020, growing by 35% to reach almost $130 billion.
In 2020 COVID-19 helped to further fuel this major shift in worldwide IT operations. Over the decade, the average annual spending growth for a data center was just 2% and for cloud services (IaaS, PaaS and hosted private cloud) was 52%."Source
On the other hand, the regulatory needs of some countries with data still needing to reside in a physical data center have pushed the public cloud services to be available in different physical locations, resulting in different deployment models, such as private cloud, public cloud, and multi-cloud.
Trend 2: Great Convergence of Analytics and AI/ML
Large enterprises and tech companies with sophisticated, complex data needs usually require their data infrastructure to serve two purposes: 1. help internal users make better decisions through the use of data, 2. build operational AI/ML systems to support data intelligence into customer-facing applications.
Historically, companies had data lakes on one side serving AI use cases and then data warehouses on the other side supporting transactional analytics and business intelligence. In reality, many organizations suffer from the duplicated data pipelines, complex infrastructures, and high costs coming along with this kind of data stack. Companies are now trying to merge both sides, creating a unified experience for all types of data analytics, including both BI and ML.
According to Andreessen Horowitz, a venture capital firm in Silicon Valley, California, that backs entrepreneurs building the future through technology, in an article called "The Emerging Architecture for Modern Data Infrastructure," some experts believe that data warehouses and data lakes are on a path toward convergence to simplify the technology and vendor landscape for both use cases. With the debut of the lakehouse concept, Databricks, an organization and big data processing platform founded by the creators of Apache Spark, also believes that the data infrastructure of the future belongs to the one that can support both BI and ML use cases.
Trend 3: From IT-Centric Data Workflows to Self-Service Analytics
Diverging from the on-demand manual delivery method that relied on IT in the past, business users in enterprises will now meet their own analysis requirements through data assets provided by a self-served analytics platform, shortening the time period from data to business insight, and improving business decision-making efficiency.
Using the self-service analytics platform to serve business data analysis, 80% of the needs of business users can be independently met based on the existing datasets, while the remaining 20% of new requirements can be quickly defined based on the existing data assets. Behind the scenes, the datasets are still well governed by IT to prevent wrong usage and high cost. This new workflow of delivering data is well optimized with the right balance between efficiency and cost.
Trend 4: Manual Analytics & Insight to Augmented Analytics
The traditional approach uses visual-based data discovery to enable business analysts and business users to uncover important insights. However, when the data is complex, large, and highly dimensional, users have to either focus on exploring their own hypothesis or they have to manually explore all possible combinations of their data to get an accurate analysis conclusion. This approach is very time-consuming and prone to bias.
Gartner believes that Augmented Analytics, a new paradigm of data and analytics, has emerged. Augmented Analytics uses machine-learning-based automation to assist human users to uncover insights and provides direct actions from data. The beauty of Augmented Analytics is that it applies a range of algorithms to the data in parallel and reduces the risk of missing important insights in data.
Trend 5: Convergence of Delayed Insights and Real-time Analytics
In the past, the refresh of the data always comes in batches. More timeliness of business decision-making has called for a lower latency of data. The ability to take immediate action at the first indication of an incident is a key factor in the success of business operations especially in use cases such as fraud detection and campaign monitoring.
Consumers are also demanding a more engaging and seamless user experience that requires applications and businesses to provide response and feedback at the speed of thought. When it comes to business decision-making, there is no way to identify problems and anomalies within the current situation without knowledge of what has happened in the past. This need has driven the trend of convergence of batch data and real-time data.
In summary, the end goal of this overall trend is the democratization of data analytics to empower all levels of a business to get faster, more timely, and better-informed insights.
Future-ready capability that you can leverage from Kyligence now
As more and more companies move to a cloud-first or cloud-only strategy, they are discovering a new practical reality for getting the answers, knowing the questions, and predicting the future. For data analytics in the cloud era, Kyligence offers a compelling set of functions that help ask the right questions to get immediate answers.
Cloud Native Design
To truly benefit from the cloud, customers need to adopt a cloud native design architecture. Cloud native systems take full advantage of the cloud service model. Designed to thrive in a dynamic, virtualized cloud environment, these systems make extensive use of Platform as a Service (PaaS) compute infrastructure and managed services. They treat the underlying infrastructure as disposable - provisioned in minutes and resized, scaled, moved, or destroyed on-demand – via automation. With Kyligence you can get the benefit of cloud native design by enjoying these features with agility and automation.
- Separate compute from storage to achieve elastic scalability
- Scale-out architecture with read and write separation
- On-demand or usage-based automated resource allocation to control cloud costs
- Ingest, transform, compute, and store using cloud services
AI-Augumented Data Modeling
Traditionally, data modeling is a manual and tedious process that requires data modeling experts. From model creation to model optimization, Kyligence's engine does all the work in the background. The user reaps the benefits without ever having to know the details of how it all works. For example, the engine is smart enough to generate recommendations, whether it needs to build a new model or simply modify an existing model, to answer the query with pre-built indexes.
Unified Semantic Layer
A data consumer (no matter his/her data literacy) needs to be able to easily discover, understand, and utilize the data. The semantic layer provides business users with an easy way to understand the data without knowing the underlying complexity of the data structure. At the same time, the Unified Semantic Layer provides IT with the benefits of centralized governance and security.
What you can expect next from Kyligence
Unified Analytics Platform for Data, BI, and AI
Kyligence believes with the unified, end-to-end, business intelligence and machine learning orchestration platform, organizations can enable innovation with speed and flexibility. The unified analytics platform should be able to provide the following:
- Extract, clean, and reformat data to make it ready for analytics
- Allow data analysts to perfrom traditional BI analytics, generate dashboards and reports
- Help the data analysts go much deeper into the AI territory that was traditionally handled by machine learning engineers with a PhD
- Use SQL-like programming syntax to make AI-enabled predictive analytics
Kyligence's unified analytics platform will promote much broader adoption of BI and AI across the enterprise, leading to easier integration, closer collaboration, and faster innovation.
As the demand of existing customers and the overall market trend to gain more timeliness of business decision-making continues to grow, Kyligence provides one platform to serve real-time, batch, and hybrid analysis. The unified batch and streaming data with a single architecture eliminate the need for separate infrastructures and tools, resulting in significant savings in cost and time. For the end-user, the introduction of streaming data is agnostic, as both batch and streaming data are orchestrated under the same unified semantic layer.
The key benefit of introducing augmented analytics is to enable office workers and business users who do not have a sufficient skill set, or simply are not efficient enough to analyze on their own, to uncover business problems intelligently with the assistance of ML-backed application/systems.
Many of Kyligence's use cases are empowering business users in large enterprises by giving them opportunities to get more value from their data by transforming them into citizen data scientists and assisting every business decision-maker in the organization.
Instead of manually performing drag-and-drop analysis and visualizations, with Kyligence's future augmented analytics capability, the system will automatically help business users identify key drivers of business trends, anomalies, and the possible root cause of the data patterns so that business users can focus on adjusting business decisions and escalate problems instead of spending time exploring the answers.
In short, as we examined the perspectives from data field investors, researchers, practitioners, and data and analytic customers, we identified the following megatrends that will shift the data and analytics market in the near future.
- From on-prem/hadoop to cloud insfrastructure
- From siloed insfrastrucutre to analytics and AL/ML in one platform
- From ETL to ELT
- From manual analytics & insights to augmented analytics that boost analytics effiency and enable citizen data scientists
- Delayed Insights in batches to Real-time analytics
As Gartner points out, in the "Gartner Top 10 Trends in Data and Analytics for 2021":
"Business leaders are beginning to understand the importance of using data and analytics to accelerate digital business initiatives. Instead of being a secondary focus — completed by a separate team — data and analytics is shifting to a core function."
Thus, in order to gain a competitive edge, making the right choices in data and analytics is more important now than ever. We highly recommend business leaders at companies of all sizes should consider implementing an architecture that best fits the needs of their business and considers following these trends to prepare their business to be future-ready. In response to these trends and our customer requests, Kyligence is on the way to shifting its capabilities to also provide:
- Unified analytic platform for data, BI, and AI
- Realtime, batch, and hybrid analytics in one platform
- Augmented analytics to serve broader data consumers
- What is a Lakehouse? - The Databricks Blog
- The Emerging Architectures for Modern Data Infrastructure
- How Tech Stacks Up in B2B - Andreessen Horowitz
- Gartner: Data Fabric Architecture is Key to Modernizing Data Management and Integration
- Synergy Research Group: 2020 – The Year that Cloud Service Revenues Finally Dwarfed Enterprise Spending on Data Centers
- Gartner Top 10 Trends Impacting Infrastructure & Operations for 2020
- Rockset: Popular Use Cases for Real-Time Analytics
- Microsoft: Defining Cloud Native