NDMP: Not just for backups anymore!

Nate Oyer

by Nate Oyer

NDMP (Network Data Management Protocol) has traditionally been a great way to backup NAS data.  This purpose built backup protocol was created to move data from source to target without the data needing to pass through a backup server.  NDMP passes the data in a block like fashion instead of the traditional file level movement.  These two features make NDMP much faster than a traditional CIFS file backup.   But what if we could utilize NDMP for something else?

It seems such a waste to have this technology and not utilize it, especially when NAS repositories are getting larger and larger and backing them up becomes unrealistic.  For more on that, check out my blog on Protecting Your Big Data with Isilon.

EMC Isilon has found a way to utilize NDMP for data migrations.  This is great news, because now we can migrate from older NAS appliances (NetApp/EMC Celerra or VNX) to Isilon in a much faster and more efficient manner.

Here is a high level process:

  • Create an NDMP user on the source box
  • Setup target directories on the Isilon
  • Run the copy command from the Isilon cluster
  • The system creates a snapshot on the source box to track changes
  • Isilon copies the data via NDMP directly from the source (no “middle man” transporter)
  • Incremental copies to be run as needed

The system utilizes snapshots to track changes on the source and transfers the data directly from the source to the Isilon target.  This is great because we are no longer performing the “tree-walk-of-death” that traditional CIFS migrations require.  We also gain the benefit of taking the “middle-man” transporter out of the equation as a bottleneck.  We no longer are dependent on a server running EMCopy, Rsync or Robocopy to clog up the migration process.

In my testing in the lab, I copied 100,000 files, 1MB in size, in 17mins and 34 seconds.  That is 0.72 gpbs (737 mbps/92MBps).  That is very close to the realistic max throughput of a 1gbps network link.  This is, however,  much faster than the projected published copy rates (81GB/hr for 1MB files). Other factors can affect copy speeds such as file size, load on the source/target NAS and available network throughput.  Testing in your environment should be done to determine a realistic copy rate for your data set.

This copy utility also copies security permissions along with the data.  It does not, however, copy shares and CIFS server information, but that is to be expected.  Because the NDMP copy session utilizes snapshot technology on the source box, incremental data copies can be run as well.  This can really help during a data migration to keep the cutover time to a minimum.  A full copy can be run prior to the migration window, and then incremental copies can be run after to keep the source and target in sync.  Finally, during the cutover window, one final copy can be run to catch up any out of sync data.

NDMP isn’t dead.  Sometimes we can recycle older technology for a new purpose.