Data Migration: Plan, Run and Check a System Move

Data migration is the work of moving records from one system to another without losing meaning, history or trust. It is also where most enterprise projects quietly fail, long after the demo looked good. The plan matters more than the tool.

Buyers often treat data migration as a line item under the platform build. Vendors often treat it as a short phase at the end. Both are wrong. Data migration is a project in its own right, with its own scope, budget and acceptance tests.

This guide covers how to scope the move, how to check the data before you start, how to run a cutover, and what to put in the contract. It applies whether you are moving to a new ERP, a cloud warehouse or a replacement line-of-business system.

Scope the data migration before you pick a platform

The platform decision gets all the attention, but the scope decides the cost. Settle what moves, what stays and what gets retired before you compare products. How to scope a data platform project sets out the questions worth answering first.

Start with an inventory. List every source system, its owner, its record count and its age. Include the spreadsheets and Access databases that nobody admits to running. Those are usually the hardest part of the job.

Then classify each data set. Some records must move in full, with history. Some need only the current state. Some should be archived and never loaded. Write the decision down for each one, with a name against it.

A useful test: for every table, ask who would notice if it arrived empty. If nobody would notice, question whether it should move at all.

Set a clear end state. The goal of data migration is not a copy of the old system in a new one. It is a smaller, cleaner data set that the business agrees to own.

Profile and clean the data first

Most data migration delays come from data quality, not from tooling. Profile the source before you write a single mapping. Count nulls, duplicates, orphaned child records and values outside the allowed list.

Expect surprises. Fields that were free text for twenty years will contain addresses, phone numbers and notes in the same column. Decide the rule for each and record it.

Cleaning is a business decision, not a technical one. Someone on the business side must own the rule for duplicates, the rule for incomplete records and the rule for records that fail validation.

• Agree a rule for duplicate customers and suppliers, and who merges them.
• Agree what happens to records that fail validation: fix, park or drop.
• Agree the retention rule for history the new system cannot hold.
• Agree who signs off the cleaned extract before it is loaded.

Keep a data quality log. Every rule, every exception and every decision goes in it. When the audit comes, that log is your defence. A short written method beats a long verbal one.

Map, transform and test in cycles

Mapping is where the real design of a data migration happens. For each target field, record the source field, the transformation rule and the owner. Where a field has no source, say so and agree what fills it.

Build in cycles, not one big bang. A typical pattern is three or more full loads: a technical load to prove the pipeline, a business load for user testing, and a final load for cutover. Each cycle should get faster and cleaner.

Test with reconciliation, not with eyeballing. For every table, compare record counts and key financial totals between source and target. A data migration that balances to the penny is one you can defend.

Run the load against a full-size copy of production data at least once. Small test sets hide performance problems that only appear at volume.

Plan the cutover and the downtime

Cutover is a schedule, not a hope. Work backwards from the go-live date and list every task, its owner and its duration. Include the steps that depend on other teams, such as network changes or interface switches.

Decide the downtime window early and tell the business what it means. If the old system must stay live during the data migration, plan how you will capture changes made after the extract. A delta load or a freeze period both work; picking neither does not.

Have a rollback plan that has been tested. If the new system fails its checks on the night, the team must be able to return to the old one without debate.

• Freeze the source system at an agreed time and publish the freeze.
• Run the final extract and load against the frozen data.
• Reconcile counts and totals before anyone logs in.
• Release users in waves, starting with the least risky group.
• Keep the old system read-only for an agreed period.

Staff the night shift properly. The people who built the pipeline should be on the call, not just the people who operate it.

Keep the old system until the new one is proven

Retirement is a separate decision from go-live. Keep the legacy system available, at least read-only, until the business has run a full cycle on the new one. For finance, that usually means a month-end close.

Set exit criteria in advance. Examples: one clean month-end close, no open severity-one defects, and a signed reconciliation for every migrated table. When the criteria are met, retire. Until then, do not.

Watch the cost of running both. Dual running is expensive, and the pressure to switch off early is real. A written exit date keeps that pressure honest.

Write the data migration into the contract

If you are buying the work, the contract should name the data migration deliverables, not just the implementation. Ask for the mapping document, the reconciliation reports and the data quality log as contractual outputs.

Payment should follow proof. Tie a portion of the fee to a successful reconciliation and a clean first close, not to the go-live date alone. That aligns the vendor with the outcome you care about.

When you compare providers, look for evidence of past data migration work at your scale, in your sector and with your source systems. Browse IT services vendors by service, technology and country to shortlist firms that have done this before.

Describe the move precisely when you go to market: source systems, record volumes, data quality known issues and the cutover window you can accept. Post a project with that detail and you will get proposals you can compare.

Frequently asked questions

How long does a data migration take?

It depends on volume and data quality, not on the tool. A single system with clean data can take a few months; a multi-source move with poor quality often takes six to twelve. Plan the profiling phase first and let its findings set the schedule.

What is the biggest cause of data migration failure?

Poor source data and unclear ownership. Teams that profile early, agree rules with the business and reconcile every load tend to finish. Teams that skip those steps discover the problems on cutover night.

Should we migrate history?

Only what the business will use or is required to keep. Moving twenty years of history adds cost and risk. Archive what is not needed and record the decision.

Can we run the old and new systems at the same time?

Yes, and for finance you usually should, at least for one close. Dual running costs money, so set exit criteria and a retirement date in advance.

What this means for buyers and vendors

For buyers, the message is simple: scope the data before you scope the platform, and make reconciliation the acceptance test. Put the data migration deliverables in the contract and pay against proof.

For vendors, data migration is where you win or lose the reference. Bring a profiling method, a reconciliation template and a tested rollback plan to the first meeting. Firms that can show those artefacts get shortlisted faster.

Either way, treat the move as its own project. The platform will be replaced again; a clean, documented data set will serve you across every system that follows.