Data Migration Strategy Guide

Making the decision to move to a new system is never easy. But when your old system no longer meets your needs, you have no choice but to do it. The question is: How do you go about migrating all of your data? In this guide, we'll walk you through the different steps involved in data migration and show you how to create a strategy that works for your business. Let's get started.

Data migration is the process of moving data from one storage location to another, such as from an on-premises data centre to the cloud, or from a database to a data warehouse. The data migration process includes planning, mapping, extraction, and formatting to ensure the data is accessible in the new storage environment. 

Being able to quickly and easily migrate data is becoming increasingly important as the amount and types of data we have to handle increases exponentially.

Most data migrations generally involve these steps:

• Analysing the data you want to migrate to determine any compatibility issues between the current storage environment and the one you’re migrating to
• Backing up the data to minimize the chances of data loss during the data migration
• Testing your migration code or application using a copy of your production environment to validate the data on the new source system
• Extracting the data from the source system using a data loader or an ETL application
• Transforming the data, if necessary, to the target system format
• Loading the data into the new system
• Verifying and testing to ensure your data transfer has been successful

Of course, there can be variations on the above depending on what you’re migrating and from where/to where.

These are the most common risks and challenges associated with data migrations. 

1. Going Over Budget

According to The Bloor Group, more than 60% of data migration projects go over time and over budget, with cost overruns averaging 30% and time overruns averaging 41%. Time is, of course, interconnected with budget and it’s usually delays related to one of the other risks explained below that result in budget overruns.

2. Data Loss

Data loss is a common issue in data migrations. Data loss is, of course, a major business killer, which is why backing up data (mentioned above) is a key part of data migrations. 

3. Downtime

Downtime is another common data migration risk. Without the proper systems in place, such as geo-replication, it can be hard to migrate data without shutting servers down in a way that affects application performance. 

4. Data Corruption

With data corruption, unnecessary data can be transferred into the new system, leading to potential crashes and “bad data” that ultimately affects the way your applications perform. 

5. Data Gravity

Perhaps the biggest risk for all data migrations is data gravity, which results from bringing disparate or once unconnected data sets together in the same environment. Data gravity is the idea that all data and applications have a natural attraction to each other, which creates a “heaviness” of data that makes it harder to move around and much more difficult to disentangle for the sake of simplifying, applying, and using it. 

It’s possible to avoid all of the above, including data gravity, with proper data migration planning, and that’s where your data migration strategy comes into play.

The general goal of every data migration is to improve performance and competitiveness, but you’ll accomplish neither of these if your migration results in inaccurate data that contains unknowns and redundancies. Having a comprehensive data migration strategy, or plan, will prevent a data migration that ends up creating more problems than it solves.

There are various types of data migrations, which we’ll get to shortly, and your overall plan should match the type of migration you’re doing.

General speaking, though, there are two main types of data migration strategies:

1. All at Once 

Otherwise known as a “Big Bang” migration, an all-at-once transfer, where you migrate all your data at once, typically involves taking systems down while data goes through ETL processing and transitions to the new database.

The advantage of this method is that it all happens quickly in one go. The disadvantage is that your business’s systems will have to be down for a bit, introducing security and data loss risks and also the risk of a compromised or failed migration.

2. “Trickle”

In a trickle data migration, the data migration process is completed in phases by running both systems—the old one and the new one—in parallel during execution of the migration. This eliminates downtime or operational interruptions.

Compared to all-at-once-migrations, trickle implementations can be complicated, but if done right, this added complexity more often than not reduces risks instead of adding them.

In addition to there being various types of data migration strategies, there are also different types of data migrations, depending on where you’re migrating your data from and to, and what type of data you’re migrating.

Database Migrations

A database migration is the transferring of data or applications between two database systems, either for the sake of moving from one vendor to another or for upgrading the software currently being used for the database.

Cloud Migrations 

Cloud migrations involve moving data or applications from either an on-premises data centre to the cloud or from one cloud to another. In many cases, cloud migrations involve storage migrations.

Storage Migrations

Storage migrations involve moving data or applications from existing storage arrays to new ones.

Application Migrations

Application migrations involve moving applications from one environment to another. This could mean moving the application from an on-premises data centre to the cloud, moving the application from one cloud to another, or moving the application's underlying data to a new form of the application hosted by a software provider.

All data migrations involve some form of ETL, but the exact form of your data centre migration plan will depend on the unique needs of your business. That said, there are some general steps you can follow to ensure your plan has what you need to succeed.

A good data centre migration project plan will keep your migration on time and within budget. Here’s a quick, step-by-step guide, courtesy of Data Migration Pro.

1. Pre-migration Planning

You need to perform a pre-migration impact assessment to verify the cost of the migration. This assessment should examine if your cost estimates are based on guesswork or concrete details and facts. You should also make executives and IT aware of any involvement they will or might have with the migration. You’ll also need to obtain in advance a formal agreement from the relevant security governance teams around the project’s security restrictions, determine the optimal project delivery structure (i.e., agile vs. waterfall), make sure everyone involved understands their role in the migration, design a training plan, and make sure you have a configuration management policy in place.

2. Project Initiation

The project initiation phase is all about getting your “back office” in order. In this part of the data migration planning, you should create a stakeholder communication plan to publish and circulate your project’s policies, set up your project collaboration platform, create your standard project documents, formalize third-party supplier agreements, and define hardware and software requirements for the later phases of the project.

3. Landscape Analysis

Landscape analysis is potentially the most important phase of your data migration planning because it’s where you dig into the structure, meaning, content, and context of your data. In this phase, you should create a detailed data dictionary, a high-level source-to-target mapping specification, and a high-level scoping report. You should also determine high-level volumetrics, share the risk management process with the team, create a data quality management process and impact report, develop and share a first-cut system retirement strategy, design your conceptual and common models, and refine your project estimates.

4. Solution Design

The solution design process involves mapping out your source-to-target transformations and creating the final design for build. In this phase, you should create detailed mapping design specifications, an interface design specification, and a data quality management specification. You should also define your production hardware requirements and agree to the service level agreements for the migration.

5. Building and Testing

In the building and testing phase, you’ll implement your data migration architecture and use rigorous testing to ensure it’s fit for purpose. In this phase, you should make sure your team has documented the migration logic. You’ll also need to test the migration with a mirror of the live environment, develop an independent migration validation engine, define your reporting strategy and associated technology, make sure you have an ongoing data quality monitoring solution, create a migration fallback policy, establish your legacy decommissioning strategy, complete any relevant execution training, ensure you understand the data quality issues that may arise and get sign off from stakeholders on anticipated issues, define and lay out your data migration execution strategy, and create a gap analysis for measuring actual vs. current progress. 

6. Migration and Validation

This is where you actually perform the migration in one of the aforementioned approaches: all at once or trickle down. In this part, expect to have to demonstrate your migration’s compliance to auditors and business sponsors. You should also independently and objectively validate the migration to ensure its success and that the resulting data quality is at a high enough level to support the target services.

7. Decommissioning and Monitoring

The final stage of the data migration is where you sunset your legacy environment and transition your data quality assets. In this phase, you should complete a system retirement validation and hand over ownership of data quality monitoring.

As already mentioned, data storage is a key part of every data migration. Without proper storage, you run the risk of sinking your migration’s success before it even starts because your data won’t be ready or it won’t be moveable. 

Built on Evergreen architecture, Pure Storage® subscription offerings make data migrations easier and more affordable:

• Evergreen//One™ reduces the complexity and cost associated with storage administration and support and helps you achieve financial flexibility and operational simplicity while mitigating IT risk.
• Evergreen//Flex™ gives you the flexibility to respond to changes in demand and use, increase your storage agility, and maximise ROI on capacity usage via lower upfront costs.
• Evergreen//Forever™ provides real IT agility, enabling you to buy your storage once and scale seamlessly and non-disruptively, without penalty, virtually forever.

Explore Pure’s Evergreen portfolio and make your next data migration easier and more cost-effective.