Need help synchronizing files between servers.

[foxidrive]

These two disk arrays at work have 171 thousand files on drives T: and U: and the results of the total time taken to process the two forms of using for don't seem to be conclusive.

In one case "for /r" is a few seconds faster and in the other case "for /f using a file" is a few seconds faster.

Code: Select all

T: drive

reboot 1

T:\>gett for /r %a in (*) do @rem
0 days 00:00:36.268

reboot 2

T:\>gett dir /b /s /a-d ^>"%temp%\a.txt"
0 days 00:00:39.942

T:\>gett for /f "usebackq delims=" %a in ("%temp%\a.txt") do @rem
0 days 00:00:00.205

================================================

U: drive

reboot 1

U:\>gett for /r %a in (*) do @rem
0 days 00:00:48.827

reboot 2

U:\>gett dir /b /s /a-d ^>"%temp%\a.txt"
0 days 00:00:45.575

U:\>gett for /f "usebackq delims=" %a in ("%temp%\a.txt") do @rem
0 days 00:00:00.222

[mcnd]

More tests, some process profiling and some debug later.

Instead of trying to remove the efect of the cache, I've made repetitive queries to ensure "all"? of the file/folder structure is cached by the OS, suspended almost anything else and then executed the tests to see if there is a significative difference in how both commands handle the data.

Code: Select all

[W:\]:# gett dir d:\ /s /b ^> list
0 days 00:00:08.315

[W:\]:# gett dir d:\ /s /b ^> list
0 days 00:00:08.269

[W:\]:# dir list | find "list"
22/06/2015  11:01        28.016.937 list

[W:\]:# find /c /v "" < list
351858

[W:\]:# gett for /f "usebackq delims=" %a in ("list") do @rem
0 days 00:00:00.656

[W:\]:# gett for /f "usebackq delims=" %a in ("list") do @rem
0 days 00:00:00.663

[W:\]:# gett for /r "d:\" %a in (*) do @rem
0 days 00:00:06.880

[W:\]:# gett for /r "d:\" %a in (*) do @rem
0 days 00:00:06.671

"procmon" shows that both "dir" and "for /r" commands use similar queries to retrieve the recursive file/folder information, but the debugger shows more code in the "dir" command to deal with all the available output options. This and the file write operation gives a little avantage to the "for /r" approach over the combined "dir" + "for /f" with a temporary file. But unless a few seconds are critical, the difference is not really significative. The bigger cost is the iteration over the file system both commands need.

But

Then, just to measure the cost of the file generation, as it is inexistent in the "for /r" test, I have found something I didn't expect (I don't know if it is a known thing, I didn't knew it).

Code: Select all

[W:\]:# gett dir d:\ /s /b ^> list
0 days 00:00:08.371

[W:\]:# gett dir d:\ /s /b ^> nul
0 days 00:00:52.664

[W:\]:# gett dir d:\ /s /b ^> nul
0 days 00:00:50.996

[W:\]:# gett dir d:\ /s /b ^> list
0 days 00:00:08.341

[W:\]:# gett dir d:\ /s /b ^> list
0 days 00:00:08.366

[W:\]:# gett type list ^> nul
0 days 00:00:04.197

[W:\]:# gett type list ^> nul
0 days 00:00:05.917

[W:\]:# gett type list ^> nul
0 days 00:00:05.873

[W:\]:# gett type list ^> list2
0 days 00:00:00.789

[W:\]:# gett type list ^> list2
0 days 00:00:00.768

[W:\]:# gett type list ^> list2
0 days 00:00:00.762

At least in my systems (XP, 32b W7, 64b W7) , the "dir" and "type" commands are very inefficient writing to the nul device. Something that does not happen with piped versions or external commands

Code: Select all

[W:\]:# gett dir d:\ /s /b ^| find /v "" ^> nul
0 days 00:00:07.849

[W:\]:# gett dir d:\ /s /b ^| find /v "" ^> nul
0 days 00:00:07.611

[W:\]:# gett dir d:\ /s /b ^| find /v "" ^> list2
0 days 00:00:07.764

[W:\]:# gett dir d:\ /s /b ^| find /v "" ^> list2
0 days 00:00:07.795

[W:\]:# gett findstr "^" ^< list ^>nul
0 days 00:00:00.237

[W:\]:# gett findstr "^" ^< list ^>nul
0 days 00:00:00.245

[W:\]:# gett findstr "^" ^< list ^>list2
0 days 00:00:00.208

[W:\]:# gett findstr "^" ^< list ^>list2
0 days 00:00:00.195

[foxidrive]

This kettle of fish is more weird than it appears. My tests are on Win 8.1 32 bit

Here are my results from your tests but performed in a script, batch file below:


dir \ /s /b > list
0 days 00:00:05.206
dir \ /s /b > nul
0 days 00:00:04.345
dir \ /s /b > nul
0 days 00:00:04.390
dir \ /s /b > list
0 days 00:00:05.358
dir \ /s /b > list
0 days 00:00:05.457
type list > nul
0 days 00:00:00.559
type list > nul
0 days 00:00:00.562
type list > nul
0 days 00:00:00.563
type list > list2
0 days 00:00:00.971
type list > list2
0 days 00:00:00.947
type list > list2
0 days 00:00:00.992

Code: Select all

@echo off
for %%a in (
"dir \ /s /b > list"
"dir \ /s /b > nul"
"dir \ /s /b > nul"
"dir \ /s /b > list"
"dir \ /s /b > list"
"type list > nul"
"type list > nul"
"type list > nul"
"type list > list2"
"type list > list2"
"type list > list2"
) do (
echo %%~a
call :part1
cmd /c %%~a
call :part2
)
pause
goto :EOF

:part1
call getTimestamp -f {ums} -r t1
goto :EOF
:part2
call getTimestamp -f {ums} -r t2
call getTimestamp -d %t2%-%t1% -f "{ud} days {hh}:{nn}:{ss}.{fff}" -u
goto :EOF

And in part two, the results are:

dir \ /s /b | find /v "" > nul
0 days 00:00:04.654
dir \ /s /b | find /v "" > nul
0 days 00:00:04.600
dir \ /s /b | find /v "" > list2
0 days 00:00:05.400
dir \ /s /b | find /v "" > list2
0 days 00:00:05.410
findstr "^" < list >nul
0 days 00:00:00.174
findstr "^" < list >nul
0 days 00:00:00.179
findstr "^" < list >list2
0 days 00:00:00.155
findstr "^" < list >list2
0 days 00:00:00.149
Press any key to continue . . .

Code: Select all

@echo off
call :part1
echo dir \ /s /b ^| find /v "" ^> nul
dir \ /s /b | find /v "" > nul
call :part2
call :part1
echo dir \ /s /b ^| find /v "" ^> nul
dir \ /s /b | find /v "" > nul
call :part2
call :part1
echo dir \ /s /b ^| find /v "" ^> list2
dir \ /s /b | find /v "" > list2
call :part2
call :part1
echo dir \ /s /b ^| find /v "" ^> list2
dir \ /s /b | find /v "" > list2
call :part2
call :part1
echo findstr "^" ^< list ^>nul
findstr "^" < list >nul
call :part2
call :part1
echo findstr "^" ^< list ^>nul
findstr "^" < list >nul
call :part2
call :part1
echo findstr "^" ^< list ^>list2
findstr "^" < list >list2
call :part2
call :part1
echo findstr "^" ^< list ^>list2
findstr "^" < list >list2
call :part2
pause
goto :EOF

:part1
call getTimestamp -f {ums} -r t1
goto :EOF
:part2
call getTimestamp -f {ums} -r t2
call getTimestamp -d %t2%-%t1% -f "{ud} days {hh}:{nn}:{ss}.{fff}" -u
goto :EOF