Migration

Migration

Data migration.jpg
Unless you're a startup, you likely have historical records stored somewhere that you may need moved to your new system. If they're on paper, you may or may not decide to digitize them for transfer, which is fine. But if your records consist of results spreadsheets, training documentation, and other important digital files, you'd probably like them in the new database so you can easily retrieve them. And of course, if you're transitioning from an old database system, you'll likely need to import to and retrieve from your new LabLynx system those records, meaning that you'll want to price in data migration services from LabLynx.

We can do all or part of it, depending on your resources. The price is strictly based on the number of hours it will take, so the condition of your existing data is the largest determiner. If there's a lot of cleanup to be done, you may want to weigh the migration cost against the value of bringing them over. If you do decide to go ahead, we'll supply the experienced professionals to make sure your new system has all of the legacy data you require, cleanly and as efficiently as possible.

Data Migration Considerations

Data that have been around awhile in a legacy system tend to have been tailored, over time if not in the beginning, to work in ways that suit the existing informatics infrastructure. That is, an individual record, for instance, may only make sense when combined with a particular software application or other pieces of data. The potentially convoluted ways that your data have been managed within various applications may be a nightmare, but it's a scenario that is relatively common, one that many are familiar with. This concept of data attracting more applications and processing power, creating more challenges for data management and future migration, is known as "data gravity."

Getting around the challenges of data gravity requires careful planning and execution.

Planning

Migration Team.jpg
With the opportunity to update to a new and better data management solution, it's important to be able to step back from the existing data and how they are used, spending some time and thought on re-conceptualizing what data must be managed, and how. It is important to include all stakeholders so that their user requirements are fully understood and incorporated into the planning phase. In some cases it turns out that much of the digital data that have been so carefully gathered over the years is no longer necessary, requiring some data cleanup. In other cases, the lab is required by law or corporate policy to retain certain data for a designated amount of time. However, that data may come in the form of paper records, spreadsheets or proprietary files, which doesn't need to be digitized because they are rarely referred back to (and if an auditing body does need them, the data can be sufficiently accessed in a timely manner, if filed well). As such, we're reminded that data migration can be a tedious, time-consuming business (and therefore costly), since typically a significant amount of "cleaning" (eliminating duplicate records, re-formatting, identifying errors, etc.) must be done to the old data so that junk isn't carried over into the new database. Take time to decide if (a) it's necessary and (b) sufficient time and money are allocated to cleaning and transferring the data properly.

Execution

Data migration is typically done using a three-step process called "extract, transform, and load" (ETL).

Extract

The first step of the ETL process is to import structured and unstructured data into a single repository. There are ETL tools that make this step fairly quick and easy. Once done, the boundary has been drawn around which data are being migrated. However, they must be scrutinized and cleaned up before they are suitable to transfer into your new cloud SQL Server or PostgreSQL database.

Transform

If your old data are already in a compatible format (i.e., SQL Server database records), than they may only require basic cleaning without transformation, and then they can be migrated as a "pass-through." Otherwise, several processes may need to take place during the transformation step. The first step in data transformation is data discovery, which identifies the meaning and projected use of the data in their source format. A data profiling tool helps accomplish this. Assessing the data in this way helps define the re-formatting that needs to occur. Then data mapping, the actual mapping of what records go where, takes place. And then programming code is generated to run the specifics of the transformation job, typically using a data transformation tool or platform.

In addition to these basic steps—or instead of re-formatting—the data may need to be prepared in one or more of the following ways:

  • Filter: This involves selecting the fields or columns to be migrated.
  • Enrich: Format data according to planned usage. For example mm/dd/yy may change to mm/dd/yyyy.
  • Split: Similar to rearranging how records are formatted, a column may need to be split into multiple columns, or two combined, etc.
  • Join: Data from multiple sources may need to be combined into a single table or field.
  • De-duplicate. Duplicate records need to be removed, obviously.

After all necessary transformation operations have been performed, the data are ready to be loaded into the new database so they can be accessed in the new system.

Load

The load process gets underway as the old records are loaded into the data warehouse of the new database. The data transformation process that has been planned and coded is now put into action, and the data are converted to the correct output. They now exist in the data warehouse of your system's cloud-hosted database (or on your own servers if that has been your chosen option). Afterwards, transformed data are checked to make sure the process has produced the desired results.

Once the data is reviewed, the migration has been accomplished. The legacy data are accessible and can be searched in the system's data warehouse at any time using multiple search criteria.

Pricing for Services

See the Unit Price List page for more about our competitive, transparent pricing for services.