By Use Cases
By BI Tool
Subscribe to our newsletter>
Get the latest products updates, community events and other news.
Amazon Web Services (AWS) Chief Evangelist Jeff Barr published an article on his official blog entitled "Migration Complete – Amazon's Consumer Business Just Turned off its Final Oracle," officially announcing the completion of the migration of the core trading system database. This isn’t the first migration the enterprise has undertaken. Previously, Amazon completed the migration of traditional data warehouses to Redshift in 2018.
This blog will discuss the significance of this
major milestone and why businesses are migrating traditional data warehouses to
the cloud. I’ve worked in the field of data warehouse and BI for more than a
decade and participated in many traditional data warehouse migration projects for
As early as the AWS re: Invent conference in 2018, AWS announced that all Oracle-based data warehouses would be turned off and migrated to Redshift-based cloud data warehouses. In the keynote speech “Under the Hood: How Amazon Uses AWS Services for Analytics at a Massive Scale,” AWS introduced the migration process for the Oracle data warehouse. Luckily, I listened to this speech firsthand and have shared some thoughts below on data warehouse migration based on my own personal experience.
Chief among reasons to migrate are the limitations of performance, scalability, and concurrency. Once the DW is hit by more than a handful of users, the headaches begin. DWs also have high price tags and high maintenance and support costs.
Amazon reduced database costs by over 60% and AWS customers regularly report cost savings of 90% by switching from Oracle to the cloud. Latency of our consumer-facing applications was reduced by 40%. The use of managed cloud services reduced database admin overhead by 70%.
technical, business, and financial challenges as the categories of hurdles they
needed to overcome:
Amazon shared three major reasons why Oracle-based traditional data warehouses should be migrated:
Oracle is based on an all-in-one architecture with tight coupling of computing and storage. It can’t quickly respond to its given requirements when the amount of data increases explosively.
It takes more than 100 hours of downtime per month to upgrade and patch the system.
Many companies will spend billions of dollars on data warehouses every year (the exact cost of AWS is unknown).
In recent years, the migration of the traditional data warehouse has spread like a prairie fire across many industries, which is most readily apparent in the internet industry. In addition to the above points, the following ones may also be the main reasons why enterprises need to migrate traditional data warehouses:
traditional data warehouse technology system is relatively closed. Enterprises need
technology that is more flexible, independent, and controllable, so they can
make enhancements or optimizations according to their needs, instead of being
restricted to the product development roadmap of suppliers.
data warehouse is mainly used to store and process structured data and often
serves enterprise report applications. With the continuous push for digital transformation, enterprises have increasingly
higher requirements for their data warehouse or data platform and require them to
support more innovative applications such as self-service analysis, real-time
calculation, graph calculation, machine learning, etc.
in the domestic financial industry, traditional data warehouse systems and software are essentially monopolized
by foreign products. With the continuous escalation of this trade war, the
domestication of core financial technologies has been raised to a strategic
level, and the domestication of data warehouses in the financial industry has
become imperative for the future.
warehouse migration is a huge project, especially for a large-scale data
warehouse, which requires a lot of manpower and material resources, and the
process will face various challenges. Some enterprises have lost an entire team
due to the hard and boring work in the logarithmic phase of migration.
data warehouse migration needs to overcome the following three major
we reasonably design the technical setup strategy to ensure the smooth
migration of models, data, scripts, and upper-level applications? In
particular, it’s necessary to address the technical differences between the old
and new systems, such as the differences in data types, functions, SQL syntax,
stored procedures, result consistency, etc. This is especially important when
the tech stacks of the two systems are inconsistent, such as the current
mainstream migration from traditional databases to big data platforms.
process of migration, how can we reduce the impact on the business and achieve as
smooth a migration as possible? For this, you’ll need not only technical
support, but also communication with management. Especially when the new
platform has just launched, system instability may occur from time to time. It
is essential to first establish a rapid response mechanism and strategy to
reduce the impact on the business.
migration process requires you to invest a lot of manpower and resources. How
can we improve the automation of migration, boost the migration efficiency, and
effectively control the migration cost? This requires the development of
various automated tools, such as data lineage analysis tools and SQL script
migration tools, to greatly reduce the dependence on manpower.
following figure shows the five best practices that are essential for an
optimal AWS migration experience. Ensure communication systems are in place
throughout the enterprise and you have a solid strategy, ready to implement, that
accounts for the expected adjustment period and also minimizes the impact of
the transition on the business.
successful migration experiences of internet enterprises represented by AWS vary
in strategy, methods, and processes, so we can’t settle on one standard
migration practice. However, one thing they all have in common is a team with solid
technical strength that has gone through careful design, careful planning,
continuous verification, iteration, and even trial and error.
Since joining Kyligence, I have participated in a number of projects for
migrating traditional data warehouses to big data platforms. We have
accumulated a lot of valuable first-hand experience on the migration of
traditional data warehouses and have formed an overall plan for products,
tools, methodologies, and services.
Here are some additional related resources:
The Good, Bad, and the Beautiful
The Evolution of Precomputation Technology
Complete – Amazon’s Consumer Business Just Turned off its Final Oracle Database
Under the Hood: How Amazon Uses AWS Services for Analytics at a Massive Scale
Learn about the fundamentals of a data product and how we help build better data products with real customer success stories.
Kyligence introduces the deployment of OLAP on top of Azure, including data sources, features, benefits, and prerequisites. Learn more about Kyligence for Azure.
What's OLAP on big data? What're its benefits? Here's everything you need to know about OLAP.
Learn how one big fast-food brand leveraged Kyligence capabilities and implemented precision marketing to maximize profit opportunities.
Come to see the Next Generation of SQL Query Engine
Learn how to achieve alternatives to SSAS.
99 Almaden Boulevard Suite #663
San Jose, CA 95113
+1 (669) 256-3378
Ⓒ 2022 Kyligence, Inc. All rights reserved.
沪 ICP 备 16026036 号 -1
沪公网安备 31011502006713 号
Already have an account? Click here to login