I’ve had a Synology NAS for a very very long time, and not too long ago I began transferring its contents to a brand new Ubiquiti UniFi UNAS Professional 8. This appeared prefer it should be a reasonably boring operation. Each units communicate SMB, I’ve a quick community (not too long ago upgraded to 10 gigabit internally), and Home windows has had instruments for copying recordsdata reliably between machines for many years. Naturally, it was an entire night of studying issues I assumed I already knew, which is why I began a weblog lol.
There was a pleasant little bit of historical past right here for me as a result of again in 2007 (good lord!) I wrote a put up referred to as “XCopy thought-about dangerous – Robocopy or XXCopy or SyncBack.” My argument on the time was principally that after you might be transferring sufficient recordsdata, Explorer stops being the transfer and Robocopy begins wanting fairly good. I even used /Z, Robocopy’s restartable mode, as a result of with the ability to resume {a partially} transferred file was helpful on unreliable connections.
Nearly twenty years later, /Z turned out to be one of the vital necessary issues I wanted to take away as a result of it made all the pieces hella gradual.
The migration
The fundamental job was easy. I had shares on the Synology reminiscent of:
servermusic
and matching shares on the UNAS:
UNAS-Professional-8music
I initially used Explorer, largely as a result of it was there and since generally the simple factor actually is the simple factor. That lasted till Explorer began producing errors on particular person recordsdata:
The requested operation couldn't be accomplished resulting from a file system limitation
My first thought was filenames. NAS migrations are filled with alternatives to find that one filesystem is extra permissive than one other, and there have been filenames with parentheses and different punctuation in them.
Then this failed:
servermusicAthleteTourist 5 Wires.m4p
There may be nothing particularly unique about 05 Wires.m4p, so I moved over to Robocopy to get just a little extra info. It constantly bought to 92% and returned Home windows error 665:
92% New File 4.3 m 05 Wires.m4p
ERROR 665 (0x00000299) Copying File
The requested operation couldn't be accomplished resulting from a file system limitation
At this level the helpful query was not “what’s unsuitable with that filename?” however “which a part of the trail is refusing this file?”
I copied the file from the Synology to my native Home windows desktop. That labored. I then copied the native file from Home windows to the UNAS, and that failed with the identical filesystem limitation.
That remoted the issue so the Synology may learn the file, Home windows may retailer it, and one thing about penning this explicit file to the UNAS was inflicting hassle.
Alternate Knowledge Streams, once more
NTFS recordsdata can comprise named Alternate Knowledge Streams along with the abnormal unnamed stream that we often consider because the contents of a file. That is an previous Home windows filesystem characteristic, and it occurs to be one I wrote about in 2007 when discussing Zone.Identifier, which Home windows can use to file the place a downloaded file got here from. I even blogged about Alternate Knowledge Streams in 2003!!! Home windows can expose these streams with DIR /R.
So I ran:
dir /r "%USERPROFILEpercentDesktop 5 Wires.m4p"
and bought:
11/30/2011 02:17 PM 4,576,368 05 Wires.m4p
360,456 05 Wires.m4p:01APIC_03.jpg:$DATA
There it’s. Alongside the conventional 4.5 MB music file was a roughly 360 KB named information stream referred to as 01APIC_03.jpg.
That additionally defined the unusual 92% failure. Robocopy was efficiently getting by the principle contents of the file after which encountering the extra stream. What had seemed like a failure someplace in the midst of an abnormal .m4p file was truly occurring when Home windows tried to take care of the extra filesystem information.
Robocopy has assist for precisely this case. Microsoft paperwork X as one of many /COPY flags, that means “skip alternate information streams.” So:
/COPY:DATX
means copy the file’s information, attributes, and timestamps, however don’t copy the alternate streams. /DCOPY:DATX applies the corresponding conduct to directories. I retried the identical file:
robocopy "servermusicAthleteTourist" "UNAS-Professional-8musicAthleteTourist" "05 Wires.m4p" /R:0 /W:0 /COPY:DATX /DCOPY:DATX /V
and it accomplished efficiently. The necessary distinction right here is that DATX doesn’t take away metadata saved inside an MP3, M4A, M4P, JPEG, or different file format. It tells Robocopy to not reproduce separate filesystem streams related to the file. In my case these further streams weren’t one thing I wanted to protect on the brand new NAS.
The copy labored, but it surely was gradual
As soon as the ADS situation was understood, I began the bigger migration with a reasonably conventional-looking Robocopy command:
robocopy "servermusic" "UNAS-Professional-8music" /E /Z /MT:16 /R:2 /W:2 /COPY:DATX /DCOPY:DATX /XJ /TEE /LOG:"%USERPROFILEpercentDesktopsynology-to-unas.log"
It ran, however efficiency was far and wide. Typically I might see a number of hundred megabits per second, then it might drop dramatically. A small file may seem to sit down there for a very long time. I began questioning whether or not I used to be buffering, gradual disks, parity calculations, SMB conduct on the UNAS, or perhaps my Synology had lastly reached its limits.
So now it is “simply strive random stuff (bisect)” time. I decreased the variety of threads. I attempted single threaded. None of that helped. Then I eliminated /Z.
Microsoft’s Robocopy documentation describes /Z as restartable mode, which lets an interrupted file resume moderately than beginning once more from byte zero. What I had forgotten is that Microsoft’s present migration steering particularly warns that /Z needs to be used cautiously as a result of the additional logging required for restartability can considerably cut back copy efficiency.
My profitable music run ended up utilizing:
robocopy "servermusic" "UNAS-Professional-8music" /E /MT:4 /R:2 /W:2 /COPY:DATX /DCOPY:DATX /XJ /TEE /LOG:"%USERPROFILEpercentDesktopsynology-to-unas-DATX.log"
The abstract from that run was:
Complete Copied Skipped Mismatch FAILED
Information : 15940 6991 8949 0 0
Bytes : 70.907 g 47.917 g 22.989 g 0 0
Velocity : 187,185,171 Bytes/sec.
So the run that copied virtually 48 GB of remaining information averaged about 187 MB/sec, with no failed recordsdata.
This was not a managed benchmark the place I modified precisely one variable whereas all the pieces else remained similar, so I’m not going to faux the quantity proves that /Z accounted for each little bit of the sooner slowdown. The sensible distinction was massive sufficient, nevertheless, that /Z is not one thing I’ll mechanically put in a LAN migration command simply because restartability sounds fascinating. On a secure native community I might begin with out it and add it solely once I really need its semantics.
/MT is beneficial, but it surely helps a selected type of drawback
Robocopy’s /MT:n possibility runs copies utilizing a number of threads. It helps values from 1 by 128, with eight threads because the default if /MT is provided with no quantity. Microsoft’s personal migration steering additionally factors out that extra threads don’t mechanically translate right into a quicker migration and recommends measuring thread counts in opposition to the precise workload.
This made extra sense as soon as I finished considering of /MT:4 as “make one file 4 instances quicker.”
Think about a music assortment with 1000’s of recordsdata of questionable provenance (I ripped them, simply kidding). There may be work related to opening a file, creating the vacation spot file, studying and writing its contents, coping with metadata, and shutting it once more. A single-threaded copy has durations the place the community or storage will be ready whereas a type of operations completes. Having a number of recordsdata in progress without delay offers Robocopy alternatives to overlap that work.
For this explicit assortment, 4 threads turned out to be a superb match. Sixteen wasn’t clearly serving to extra, and one thread wasn’t an enchancment. I might resist turning /MT right into a magic worth that belongs in each command line, as a result of a listing containing 50,000 images presents a special workload from 4 900 GB disk photographs.
There may be additionally a logging value value remembering. Microsoft recommends redirecting Robocopy output to a log when utilizing multithreaded copies, and its migration steering makes use of switches reminiscent of /NP, /NFL, and /NDL when the target is throughput moderately than watching each filename scroll by.
For a migration I’m not actively watching, I might in all probability use one thing like:
robocopy "servershare" "UNAS-Professional-8share" /E /MT:4 /R:2 /W:2 /COPY:DATX /DCOPY:DATX /XJ /NP /NFL /NDL /LOG:"%USERPROFILEpercentDesktopnas-migration.log"
Then got here the big recordsdata
Later I began copying some recordsdata that have been tons of of gigabytes every. Microsoft describes /J as unbuffered I/O and recommends it for giant recordsdata, so it appeared like the plain choice to strive.
With /J enabled, nevertheless, NAS-to-NAS switch slowed dramatically. The helpful factor about having a Home windows machine within the center is that I may take a look at every half of the journey individually. I took one of many very same massive recordsdata and copied it instantly from the Synology to my native machine. That ran at roughly 250 MB/sec, so the Synology was completely able to studying the file at excessive velocity.
I then eliminated /J from the direct Synology-to-UNAS Robocopy command:
robocopy "servershare" "UNAS-Professional-8share" "huge-file.ext" /R:0 /W:0 /COPY:DATX /NP
and the velocity got here again.
I don’t suppose the helpful conclusion is that /J is dangerous. Microsoft recommends it for large-file copies for a cause, and it’s solely potential that it’s precisely what you need when copying from native disk to native disk or in one other community configuration. What mattered right here was that Home windows was concurrently studying from one SMB server and writing to a different SMB server, and on this explicit path buffered I/O carried out a lot better.
That may be a good reminder that command-line switches describe conduct, not assured efficiency enhancements. /J adjustments the I/O mannequin. /MT adjustments concurrency. /Z provides restartability. Whether or not these adjustments enhance a migration will depend on the remainder of the system.
The Synology was quicker than I gave it credit score for
At a number of factors I blamed the growing older Synology. It’s an previous machine with spinning disks, so it was straightforward to imagine that a number of hundred megabits per second was merely all it had left. Then I remembered that the Synology has 4 1 GbE interfaces and that SMB 3 helps Multichannel. I’m nonetheless stunned this labored so effectively.
SMB Multichannel permits an SMB session to make use of a number of community paths concurrently. Microsoft paperwork this particularly as a approach to combination accessible community bandwidth, and Synology helps SMB3 Multichannel for a similar cause.
Home windows makes the energetic channels straightforward to examine:
Get-SmbMultichannelConnection -ServerName server |
Format-Desk ServerName,Chosen,ClientIpAddress,ServerIpAddress,ClientLinkSpeed,ServerLinkSpeed,CurrentChannels
My machine reported:
ServerName Chosen ClientIpAddress ServerIpAddress ClientLinkSpeed ServerLinkSpeed
---------- -------- --------------- --------------- --------------- ---------------
server True 192.168.1.45 192.168.1.210 1000000000 1000000000
server True 192.168.1.45 192.168.1.198 1000000000 1000000000
server True 192.168.1.45 192.168.1.197 1000000000 1000000000
server True 192.168.1.45 192.168.1.26 1000000000 1000000000
All 4 1 GbE interfaces on the Synology have been collaborating within the SMB connection.
My Home windows machine at present has a 2.5 GbE adapter (10 gig coming quickly), and in the course of the quick copy I used to be seeing roughly 250 MB/sec arriving from the Synology. That all of a sudden made the conduct of the system a lot much less mysterious. The Synology was not restricted to the throughput of 1 gigabit Ethernet connection as a result of SMB Multichannel was permitting Home windows to make use of the 4 accessible server-side paths, whereas the two.5 GbE hyperlink on the PC was changing into the smaller community pipe.
Synology’s documentation makes an necessary distinction right here between SMB Multichannel and abnormal hyperlink aggregation. Multichannel can enhance SMB efficiency for one consumer by utilizing a number of community connections, whereas typical hyperlink aggregation is usually about combination throughput throughout a number of purchasers and companies.
Like I stated, I’ve a ten GbE adapter on the way in which for the Home windows machine, so there’s one other experiment accessible after the migration. The Synology nonetheless solely has 4 1 GbE interfaces, which provides it 4 Gb/sec of community hyperlinks in combination, however eradicating the present 2.5 GbE consumer bottleneck ought to present how a lot farther the disks and the Synology itself can go. For an older NAS that I had already mentally demoted to “the gradual backup NAS,” it carried out surprisingly effectively.
I additionally tried rsync
Sure, I do know, what about rsync? You might be saying Home windows as a intermediary is pointless. The Synology can present rsync, and UniFi Drive can pull from an rsync server utilizing daemon mode. Ubiquiti paperwork the rsync path beneath Drive’s Backup Duties and requires daemon mode for one of these supply.
I attempted a separate film share with rsync whereas the opposite experiments have been happening. It labored, and I noticed about 67 MB/sec. It additionally gave me a vacation spot structure with some extra listing nesting that I would wish to wash up afterward.
That’s not an argument that rsync is gradual usually, nor that its listing conduct can’t be configured appropriately. It’s simply what occurred on this explicit Synology-to-UNAS take a look at. As soon as the Robocopy path was reaching roughly 187 MB/sec and preserving precisely the UNC share structure I needed, there wasn’t a lot incentive for me to make rsync the first migration mechanism.
The marginally amusing consequence was that the apparently oblique path:
Synology -> SMB -> Home windows -> SMB -> UNAS
was significantly quicker in my surroundings than asking the 2 NAS units to switch the take a look at share instantly with the rsync implementation uncovered by UniFi Drive. My guess is as a result of rsync wasn’t utilizing the 4 1gig connections linked. Let me know what you suppose within the feedback.
The Robocopy command I ended up with
For the conventional shares containing plenty of recordsdata, that is the model I might begin with now:
robocopy "servershare" "UNAS-Professional-8share" /E /MT:4 /R:2 /W:2 /COPY:DATX /DCOPY:DATX /XJ /NP /NFL /NDL /LOG:"%USERPROFILEpercentDesktopnas-migration.log"
The switches have pretty particular jobs:
/E Copy subdirectories, together with empty ones
/MT:4 Enable 4 file-copy threads
/R:2 Retry a failed copy twice
/W:2 Wait two seconds between retries
/COPY:DATX Copy information, attributes and timestamps, however skip ADS
/DCOPY:DATX Apply the corresponding listing copy flags
/XJ Exclude junction factors
/NP Do not print share progress
/NFL Do not log each filename
/NDL Do not log each listing
/LOG Write the helpful output to a file
I intentionally don’t have /Z in there. I additionally wouldn’t mechanically add /J; for a workload dominated by very massive recordsdata I might take a look at the identical switch each with and with out it earlier than committing to a multi-terabyte run.
The /MT worth is equally empirical. 4 labored extraordinarily effectively for this Synology and this mixture of recordsdata, however I might strive 1, 4, 8, or one other wise quantity in opposition to a consultant slice of the actual information moderately than assuming that the most important accessible thread rely is finest.
Checking the consequence
For the recordsdata the place I care sufficient to show that the vacation spot comprises precisely the identical bytes because the supply, PowerShell’s Get-FileHash is a handy remaining examine:
Get-FileHash "serversharehuge-file.ext" -Algorithm SHA256
Get-FileHash "UNAS-Professional-8sharehuge-file.ext" -Algorithm SHA256
Get-FileHash makes use of SHA-256 by default, and if the SHA-256 values match then the 2 inputs produced the identical digest. For huge recordsdata this requires studying your entire file once more at each ends, so I’m unlikely to hash each track in a music assortment, however it’s a simple approach to confirm the notably priceless multi-hundred-gigabyte recordsdata after a migration.
Just a few issues I’d examine earlier than blaming the NAS
What made this migration attention-grabbing was that the signs may have supported a number of believable explanations. The UNAS has a brand new RAID array, the Synology is previous, Home windows is performing as an SMB consumer in each instructions, the recordsdata got here from years of various purposes, and Robocopy has sufficient switches to make virtually any command line look authoritative.
When a file failed, I copied it Synology-to-local after which local-to-UNAS, which uncovered the destination-side ADS drawback. When massive recordsdata have been gradual, I copied the identical file Synology-to-local, which proved the previous NAS may nonetheless ship about 250 MB/sec. When the community all of a sudden turned a lot quicker, Get-SmbMultichannelConnection confirmed that every one 4 Synology Ethernet interfaces have been collaborating. When /J seemed gradual, eradicating simply that conduct restored the throughput I used to be anticipating.
The ultimate efficiency was not the results of discovering a secret “quick Robocopy” command from a discussion board. It got here from fascinated by which options I truly needed for this migration and eradicating a number of that have been helpful in different circumstances however costly in mine.
That’s in all probability the half I’ll need to bear in mind the subsequent time I do that. /Z, /J, and /MT aren’t ranges on a efficiency slider. They modify restartability, buffering, and concurrency. Alternate Knowledge Streams are actual information even when Explorer usually hides them. SMB Multichannel could make an previous NAS with a number of gigabit interfaces far more succesful than one may assume from any single Ethernet port.
Robocopy ended up being the quickest factor I attempted. As with all recommendation, this labored for me. Ideally you may gind extra worth within the feedback as Hacker Information of us and Home windows consultants will drop in with higher instruments and techniques. Simply bear in mind, there’s a couple of approach to saturate a community and I fully saturated this one, so I am fairly pleased with the results of my migration.
About Scott
Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, guide, father, diabetic, and Microsoft worker. He’s a failed stand-up comedian, a cornrower, and a ebook creator.
