Copying Single Source to Multiple Destinations in Parallel
Posted: 05 Jun 2019 22:33
Most of us that have worked with computers have run into this problem in some form or another--you have one source and it needs to be copied to multiple destinations. And the normal solution is to run these copy operations serially, ie one at a time, so the execution time is N*NumberOfDestinations.
This is fine and dandy with small batch copies or high-bandwidth copies. But what about when the source is massive and/or the bandwidth is limited? Having to repeat N for each destination can drag out a 12hr copy to 5 sources into a 60hr week long job.
But in my research there is actually a solution that no one or no program seems to takes advantage of--drive caching.
Whenever a file is initially read and then read again, there's a strong chance that the file will be re-read from the cache at lightning speed. It simple to try, just copy a file that's 20MB or so and then compare it--the compare finishes almost instantly--especially if you repeat the compare command again.
When copies are done serially, each file being copied is read multiple times because each copy job is treated separately. But what if you copied each file in parallel. Each subsequent read on the same file would be a cache hit, thereby allowing the file to be read at much faster speeds for all the other copies. In fact, the only additional time it would take is for the write times on the destinations, which would also be in parallel, so your time would only increase by the time to write the file on the slowest destination.
For example, I have source A that I want to copy to drives 1, 2, and 3. To do so the logic would be something like:
copy sourceA drive1
copy sourceA drive2
copy sourceA drive3
Each of these would run after the other is executed.
Now these can be paralleled by using the start (or even call?) command
start copy sourceA drive1
start copy sourceA drive2
start copy sourceA drive3
Which would start each of these in parallel. However, in my experience with this, when you have a large number of source items, the file being copied simultaneously can get out of sync causing cache misses. And as this progresses, instead of improving cache hits, you've drastically increased source drive seek times which actually makes the whole operation slower. (I don't have any SSDs I've tried this with as those should not exhibit any issues with drive seek, just the same issue of a cache miss.)
But what if the parallelization was done on a file-by-file basis. Then you can almost surely have a cache hit and the multiple copies won't get out of sync. This is pretty each to implement when you have a simple directory of files as you can just loop through them using for, executing a parallel copy on each file. I think the logic would be something like this:
for %f in (*.*) do (
start copy %f drive1
start copy %f drive2
start copy %f drive3
)
But once you introduce multiple directories, and sub-directories, etc., I don't know how this would be implemented. I found the 'Additional Task for each File to be copied' here to be a useful guide in understanding where to start by changing the logic to creating f file list first, but I can't figure out what that logic would look like for traversing and entire source drive or tree branch. Perhaps using 'for /r' would be a solution here, but I'm just not sure.
And then there's also the question of which copy command to use on each file? copy? xcopy? robocopy? What would be the best for the overall goal, which is to reduce the execution time from N*NumberOfDestinations to N*SlowestDestination?
Any thoughts or ideas appreciated.
This is fine and dandy with small batch copies or high-bandwidth copies. But what about when the source is massive and/or the bandwidth is limited? Having to repeat N for each destination can drag out a 12hr copy to 5 sources into a 60hr week long job.
But in my research there is actually a solution that no one or no program seems to takes advantage of--drive caching.
Whenever a file is initially read and then read again, there's a strong chance that the file will be re-read from the cache at lightning speed. It simple to try, just copy a file that's 20MB or so and then compare it--the compare finishes almost instantly--especially if you repeat the compare command again.
When copies are done serially, each file being copied is read multiple times because each copy job is treated separately. But what if you copied each file in parallel. Each subsequent read on the same file would be a cache hit, thereby allowing the file to be read at much faster speeds for all the other copies. In fact, the only additional time it would take is for the write times on the destinations, which would also be in parallel, so your time would only increase by the time to write the file on the slowest destination.
For example, I have source A that I want to copy to drives 1, 2, and 3. To do so the logic would be something like:
copy sourceA drive1
copy sourceA drive2
copy sourceA drive3
Each of these would run after the other is executed.
Now these can be paralleled by using the start (or even call?) command
start copy sourceA drive1
start copy sourceA drive2
start copy sourceA drive3
Which would start each of these in parallel. However, in my experience with this, when you have a large number of source items, the file being copied simultaneously can get out of sync causing cache misses. And as this progresses, instead of improving cache hits, you've drastically increased source drive seek times which actually makes the whole operation slower. (I don't have any SSDs I've tried this with as those should not exhibit any issues with drive seek, just the same issue of a cache miss.)
But what if the parallelization was done on a file-by-file basis. Then you can almost surely have a cache hit and the multiple copies won't get out of sync. This is pretty each to implement when you have a simple directory of files as you can just loop through them using for, executing a parallel copy on each file. I think the logic would be something like this:
for %f in (*.*) do (
start copy %f drive1
start copy %f drive2
start copy %f drive3
)
But once you introduce multiple directories, and sub-directories, etc., I don't know how this would be implemented. I found the 'Additional Task for each File to be copied' here to be a useful guide in understanding where to start by changing the logic to creating f file list first, but I can't figure out what that logic would look like for traversing and entire source drive or tree branch. Perhaps using 'for /r' would be a solution here, but I'm just not sure.
And then there's also the question of which copy command to use on each file? copy? xcopy? robocopy? What would be the best for the overall goal, which is to reduce the execution time from N*NumberOfDestinations to N*SlowestDestination?
Any thoughts or ideas appreciated.