I’ve had a Synology NAS for a very very long time, and not too long ago I began transferring its contents to a brand new Ubiquiti UniFi UNAS Professional 8. This appeared prefer it must be a reasonably boring operation. Each gadgets communicate SMB, I’ve a quick community (not too long ago upgraded to 10 gigabit internally), and Home windows has had instruments for copying recordsdata reliably between machines for many years. Naturally, it become an entire night of studying issues I assumed I already knew, which is why I began a weblog lol.
There was a pleasant little bit of historical past right here for me as a result of again in 2007 (good lord!) I wrote a submit referred to as “XCopy thought of dangerous – Robocopy or XXCopy or SyncBack.” My argument on the time was principally that when you might be transferring sufficient recordsdata, Explorer stops being the transfer and Robocopy begins trying fairly good. I even used /Z, Robocopy’s restartable mode, as a result of with the ability to resume {a partially} transferred file was helpful on unreliable connections.
Nearly twenty years later, /Z turned out to be one of the necessary issues I wanted to take away as a result of it made every part hella sluggish.
The migration
The fundamental job was simple. I had shares on the Synology equivalent to:
servermusic
and matching shares on the UNAS:
UNAS-Professional-8music
I initially used Explorer, principally as a result of it was there and since typically the simple factor actually is the simple factor. That lasted till Explorer began producing errors on particular person recordsdata:
The requested operation couldn't be accomplished resulting from a file system limitation
My first thought was filenames. NAS migrations are filled with alternatives to find that one filesystem is extra permissive than one other, and there have been filenames with parentheses and different punctuation in them.
Then this failed:
servermusicAthleteTourist 5 Wires.m4p
There’s nothing particularly unique about 05 Wires.m4p, so I moved over to Robocopy to get a bit extra info. It persistently obtained to 92% and returned Home windows error 665:
92% New File 4.3 m 05 Wires.m4p
ERROR 665 (0x00000299) Copying File
The requested operation couldn't be accomplished resulting from a file system limitation
At this level the helpful query was now not “what’s improper with that filename?” however “which a part of the trail is refusing this file?”
I copied the file from the Synology to my native Home windows desktop. That labored. I then copied the native file from Home windows to the UNAS, and that failed with the identical filesystem limitation.
That remoted the issue so the Synology may learn the file, Home windows may retailer it, and one thing about scripting this specific file to the UNAS was inflicting bother.
Alternate Knowledge Streams, once more
NTFS recordsdata can comprise named Alternate Knowledge Streams along with the peculiar unnamed stream that we normally consider because the contents of a file. That is an outdated Home windows filesystem function, and it occurs to be one I wrote about in 2007 when discussing Zone.Identifier, which Home windows can use to document the place a downloaded file got here from. I even blogged about Alternate Knowledge Streams in 2003!!! Home windows can expose these streams with DIR /R.
So I ran:
dir /r "%USERPROFILEpercentDesktop 5 Wires.m4p"
and obtained:
11/30/2011 02:17 PM 4,576,368 05 Wires.m4p
360,456 05 Wires.m4p:01APIC_03.jpg:$DATA
There it’s. Alongside the traditional 4.5 MB music file was a roughly 360 KB named information stream referred to as 01APIC_03.jpg.
That additionally defined the unusual 92% failure. Robocopy was efficiently getting by the principle contents of the file after which encountering the extra stream. What had regarded like a failure someplace in the course of an peculiar .m4p file was really occurring when Home windows tried to take care of the extra filesystem information.
Robocopy has assist for precisely this case. Microsoft paperwork X as one of many /COPY flags, that means “skip alternate information streams.” So:
/COPY:DATX
means copy the file’s information, attributes, and timestamps, however don’t copy the alternate streams. /DCOPY:DATX applies the corresponding habits to directories. I retried the identical file:
robocopy "servermusicAthleteTourist" "UNAS-Professional-8musicAthleteTourist" "05 Wires.m4p" /R:0 /W:0 /COPY:DATX /DCOPY:DATX /V
and it accomplished efficiently. The necessary distinction right here is that DATX doesn’t take away metadata saved inside an MP3, M4A, M4P, JPEG, or different file format. It tells Robocopy to not reproduce separate filesystem streams related to the file. In my case these additional streams weren’t one thing I wanted to protect on the brand new NAS.
The copy labored, however it was sluggish
As soon as the ADS concern was understood, I began the bigger migration with a reasonably conventional-looking Robocopy command:
robocopy "servermusic" "UNAS-Professional-8music" /E /Z /MT:16 /R:2 /W:2 /COPY:DATX /DCOPY:DATX /XJ /TEE /LOG:"%USERPROFILEpercentDesktopsynology-to-unas.log"
It ran, however efficiency was everywhere. Generally I’d see a couple of hundred megabits per second, then it might drop dramatically. A small file may seem to sit down there for a very long time. I began questioning whether or not I used to be taking a look at buffering, sluggish disks, parity calculations, SMB habits on the UNAS, or perhaps my Synology had lastly reached its limits.
So now it is “simply attempt random stuff (bisect)” time. I decreased the variety of threads. I attempted single threaded. None of that helped. Then I eliminated /Z.
Microsoft’s Robocopy documentation describes /Z as restartable mode, which lets an interrupted file resume fairly than beginning once more from byte zero. What I had forgotten is that Microsoft’s present migration steerage particularly warns that /Z ought to be used cautiously as a result of the additional logging required for restartability can considerably scale back copy efficiency.
My profitable music run ended up utilizing:
robocopy "servermusic" "UNAS-Professional-8music" /E /MT:4 /R:2 /W:2 /COPY:DATX /DCOPY:DATX /XJ /TEE /LOG:"%USERPROFILEpercentDesktopsynology-to-unas-DATX.log"
The abstract from that run was:
Whole Copied Skipped Mismatch FAILED
Recordsdata : 15940 6991 8949 0 0
Bytes : 70.907 g 47.917 g 22.989 g 0 0
Velocity : 187,185,171 Bytes/sec.
So the run that copied virtually 48 GB of remaining information averaged about 187 MB/sec, with no failed recordsdata.
This was not a managed benchmark the place I modified precisely one variable whereas every part else remained an identical, so I’m not going to faux the quantity proves that /Z accounted for each little bit of the sooner slowdown. The sensible distinction was massive sufficient, nevertheless, that /Z is now not one thing I’ll robotically put in a LAN migration command simply because restartability sounds fascinating. On a steady native community I’d begin with out it and add it solely once I really need its semantics.
/MT is beneficial, however it helps a selected type of drawback
Robocopy’s /MT:n choice runs copies utilizing a number of threads. It helps values from 1 by 128, with eight threads because the default if /MT is equipped with out a quantity. Microsoft’s personal migration steerage additionally factors out that extra threads don’t robotically translate right into a quicker migration and recommends measuring thread counts in opposition to the precise workload.
This made extra sense as soon as I finished pondering of /MT:4 as “make one file 4 occasions quicker.”
Think about a music assortment with 1000’s of recordsdata of questionable provenance (I ripped them, simply kidding). There’s work related to opening a file, creating the vacation spot file, studying and writing its contents, coping with metadata, and shutting it once more. A single-threaded copy has durations the place the community or storage may be ready whereas a kind of operations completes. Having a number of recordsdata in progress without delay offers Robocopy alternatives to overlap that work.
For this specific assortment, 4 threads turned out to be a great match. Sixteen wasn’t clearly serving to extra, and one thread wasn’t an enchancment. I’d resist turning /MT right into a magic worth that belongs in each command line, as a result of a listing containing 50,000 images presents a special workload from 4 900 GB disk photographs.
There’s additionally a logging price price remembering. Microsoft recommends redirecting Robocopy output to a log when utilizing multithreaded copies, and its migration steerage makes use of switches equivalent to /NP, /NFL, and /NDL when the target is throughput fairly than watching each filename scroll by.
For a migration I’m not actively watching, I’d most likely use one thing like:
robocopy "servershare" "UNAS-Professional-8share" /E /MT:4 /R:2 /W:2 /COPY:DATX /DCOPY:DATX /XJ /NP /NFL /NDL /LOG:"%USERPROFILEpercentDesktopnas-migration.log"
Then got here the massive recordsdata
Later I began copying some recordsdata that have been a whole bunch of gigabytes every. Microsoft describes /J as unbuffered I/O and recommends it for big recordsdata, so it appeared like the apparent choice to attempt.
With /J enabled, nevertheless, NAS-to-NAS switch slowed dramatically. The helpful factor about having a Home windows machine within the center is that I may take a look at every half of the journey individually. I took one of many very same massive recordsdata and copied it straight from the Synology to my native machine. That ran at roughly 250 MB/sec, so the Synology was completely able to studying the file at excessive pace.
I then eliminated /J from the direct Synology-to-UNAS Robocopy command:
robocopy "servershare" "UNAS-Professional-8share" "huge-file.ext" /R:0 /W:0 /COPY:DATX /NP
and the pace got here again.
I don’t suppose the helpful conclusion is that /J is dangerous. Microsoft recommends it for large-file copies for a motive, and it’s fully attainable that it’s precisely what you need when copying from native disk to native disk or in one other community configuration. What mattered right here was that Home windows was concurrently studying from one SMB server and writing to a different SMB server, and on this specific path buffered I/O carried out significantly better.
That could be a good reminder that command-line switches describe habits, not assured efficiency enhancements. /J modifications the I/O mannequin. /MT modifications concurrency. /Z provides restartability. Whether or not these modifications enhance a migration is determined by the remainder of the system.
The Synology was quicker than I gave it credit score for
At a number of factors I blamed the growing old Synology. It’s an outdated machine with spinning disks, so it was straightforward to imagine that a couple of hundred megabits per second was merely all it had left. Then I remembered that the Synology has 4 1 GbE interfaces and that SMB 3 helps Multichannel. I’m nonetheless shocked this labored so effectively.
SMB Multichannel permits an SMB session to make use of a number of community paths concurrently. Microsoft paperwork this particularly as a option to mixture accessible community bandwidth, and Synology helps SMB3 Multichannel for a similar motive.
Home windows makes the energetic channels straightforward to examine:
Get-SmbMultichannelConnection -ServerName server |
Format-Desk ServerName,Chosen,ClientIpAddress,ServerIpAddress,ClientLinkSpeed,ServerLinkSpeed,CurrentChannels
My machine reported:
ServerName Chosen ClientIpAddress ServerIpAddress ClientLinkSpeed ServerLinkSpeed
---------- -------- --------------- --------------- --------------- ---------------
server True 192.168.1.45 192.168.1.210 1000000000 1000000000
server True 192.168.1.45 192.168.1.198 1000000000 1000000000
server True 192.168.1.45 192.168.1.197 1000000000 1000000000
server True 192.168.1.45 192.168.1.26 1000000000 1000000000
All 4 1 GbE interfaces on the Synology have been taking part within the SMB connection.
My Home windows machine presently has a 2.5 GbE adapter (10 gig coming quickly), and throughout the quick copy I used to be seeing roughly 250 MB/sec arriving from the Synology. That all of a sudden made the habits of the system a lot much less mysterious. The Synology was not restricted to the throughput of 1 gigabit Ethernet connection as a result of SMB Multichannel was permitting Home windows to make use of the 4 accessible server-side paths, whereas the two.5 GbE hyperlink on the PC was turning into the smaller community pipe.
Synology’s documentation makes an necessary distinction right here between SMB Multichannel and peculiar hyperlink aggregation. Multichannel can improve SMB efficiency for one consumer through the use of a number of community connections, whereas standard hyperlink aggregation is mostly about mixture throughput throughout a number of purchasers and providers.
Like I mentioned, I’ve a ten GbE adapter on the way in which for the Home windows machine, so there’s one other experiment accessible after the migration. The Synology nonetheless solely has 4 1 GbE interfaces, which supplies it 4 Gb/sec of community hyperlinks in mixture, however eradicating the present 2.5 GbE consumer bottleneck ought to present how a lot farther the disks and the Synology itself can go. For an older NAS that I had already mentally demoted to “the sluggish backup NAS,” it carried out surprisingly effectively.
I additionally tried rsync
Sure, I do know, what about rsync? You’re saying Home windows as a intermediary is pointless. The Synology can present rsync, and UniFi Drive can pull from an rsync server utilizing daemon mode. Ubiquiti paperwork the rsync path underneath Drive’s Backup Duties and requires daemon mode for one of these supply.
I attempted a separate film share with rsync whereas the opposite experiments have been occurring. It labored, and I noticed about 67 MB/sec. It additionally gave me a vacation spot format with some further listing nesting that I would want to scrub up afterward.
That isn’t an argument that rsync is sluggish basically, nor that its listing habits can’t be configured accurately. It’s simply what occurred on this specific Synology-to-UNAS take a look at. As soon as the Robocopy path was reaching roughly 187 MB/sec and preserving precisely the UNC share format I wished, there wasn’t a lot incentive for me to make rsync the first migration mechanism.
The marginally amusing consequence was that the apparently oblique path:
Synology -> SMB -> Home windows -> SMB -> UNAS
was significantly quicker in my setting than asking the 2 NAS gadgets to switch the take a look at share straight with the rsync implementation uncovered by UniFi Drive. My guess is as a result of rsync wasn’t utilizing the 4 1gig connections linked. Let me know what you suppose within the feedback.
The Robocopy command I ended up with
For the traditional shares containing a number of recordsdata, that is the model I’d begin with now:
robocopy "servershare" "UNAS-Professional-8share" /E /MT:4 /R:2 /W:2 /COPY:DATX /DCOPY:DATX /XJ /NP /NFL /NDL /LOG:"%USERPROFILEpercentDesktopnas-migration.log"
The switches have pretty particular jobs:
/E Copy subdirectories, together with empty ones
/MT:4 Enable 4 file-copy threads
/R:2 Retry a failed copy twice
/W:2 Wait two seconds between retries
/COPY:DATX Copy information, attributes and timestamps, however skip ADS
/DCOPY:DATX Apply the corresponding listing copy flags
/XJ Exclude junction factors
/NP Do not print share progress
/NFL Do not log each filename
/NDL Do not log each listing
/LOG Write the helpful output to a file
I intentionally do not need /Z in there. I additionally wouldn’t robotically add /J; for a workload dominated by very massive recordsdata I’d take a look at the identical switch each with and with out it earlier than committing to a multi-terabyte run.
The /MT worth is equally empirical. 4 labored extraordinarily effectively for this Synology and this mixture of recordsdata, however I’d attempt 1, 4, 8, or one other smart quantity in opposition to a consultant slice of the actual information fairly than assuming that the most important accessible thread rely is finest.
Checking the consequence
For the recordsdata the place I care sufficient to show that the vacation spot incorporates precisely the identical bytes because the supply, PowerShell’s Get-FileHash is a handy remaining verify:
Get-FileHash "serversharehuge-file.ext" -Algorithm SHA256
Get-FileHash "UNAS-Professional-8sharehuge-file.ext" -Algorithm SHA256
Get-FileHash makes use of SHA-256 by default, and if the SHA-256 values match then the 2 inputs produced the identical digest. For huge recordsdata this requires studying the whole file once more at each ends, so I’m unlikely to hash each tune in a music assortment, however it’s a straightforward option to confirm the notably beneficial multi-hundred-gigabyte recordsdata after a migration.
Just a few issues I’d verify earlier than blaming the NAS
What made this migration fascinating was that the signs may have supported a number of believable explanations. The UNAS has a brand new RAID array, the Synology is outdated, Home windows is performing as an SMB consumer in each instructions, the recordsdata got here from years of various functions, and Robocopy has sufficient switches to make virtually any command line look authoritative.
When a file failed, I copied it Synology-to-local after which local-to-UNAS, which uncovered the destination-side ADS drawback. When massive recordsdata have been sluggish, I copied the identical file Synology-to-local, which proved the outdated NAS may nonetheless ship about 250 MB/sec. When the community all of a sudden grew to become a lot quicker, Get-SmbMultichannelConnection confirmed that each one 4 Synology Ethernet interfaces have been taking part. When /J regarded sluggish, eradicating simply that habits restored the throughput I used to be anticipating.
The ultimate efficiency was not the results of discovering a secret “quick Robocopy” command from a discussion board. It got here from fascinated by which options I really wished for this migration and eradicating a couple of that have been helpful in different circumstances however costly in mine.
That’s most likely the half I’ll need to keep in mind the following time I do that. /Z, /J, and /MT aren’t ranges on a efficiency slider. They alter restartability, buffering, and concurrency. Alternate Knowledge Streams are actual information even when Explorer usually hides them. SMB Multichannel could make an outdated NAS with a number of gigabit interfaces rather more succesful than one would possibly assume from taking a look at any single Ethernet port.
Robocopy ended up being the quickest factor I attempted. As with all recommendation, this labored for me. Ideally you may gind extra worth within the feedback as Hacker Information of us and Home windows consultants will drop in with higher instruments and methods. Simply keep in mind, there’s multiple option to saturate a community and I fully saturated this one, so I am fairly proud of the results of my migration.
About Scott
Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, advisor, father, diabetic, and Microsoft worker. He’s a failed stand-up comedian, a cornrower, and a e-book writer.
