Page 1 of 2
Fastest Recursive Searching
Posted: 23 Jul 2021 12:46
by atfon
Hello Folks,
I have been working on a system agnostic (Win 7 and above, 32 or 64-bit) script which will capture the file name and path of a set of frequently used applications. The location of these applications can vary from %localappdata% (for items like OneDrive) to %programfiles% and %programfiles(x86)% for items like chrome.exe or winword.exe. I have been doing some tests with for /r, dir, robocopy, where and wmic. I have also tested with writing the data to a temporary file and then just using findstr to type the results to the console and/or a file. I'm sure there are benefits and drawbacks to different methodologies as well as the powershell scripts to Get-ItemProperty of the two Uninstall locations.
I know it can be difficult to provide suggestions without any code or the set of applications in question, but is there one methodology you might recommend over the others?
Thank you in advance and I apologize if this question is too vague.
Edit: I forgot to include the where command and forfiles.
Re: Fastest Recursive Searching
Posted: 23 Jul 2021 14:20
by aGerman
viewtopic.php?f=3&t=10140
Turns out that robocopy works nicely because it supports the exclusion of heavy folders like C:\Windows, at least if you have to begin searching in the root folder of a drive.
It can also be used if you only have a few locations where you expect the apps to be found. You can put them in a nested loop (just like the drive letters in the linked thread).
If you already tried out all the possibilities you listed, what have been your experiences?
Steffen
Re: Fastest Recursive Searching
Posted: 23 Jul 2021 16:28
by atfon
Thanks for the recommendation, Steffen (@aGerman). I agree that robocopy does seem best. I've been running it a few times against a few locations recursively and using the /MT multithreaded option (along with a couple of the other performance tweak options you listed). l then send the output to a text file. I parse that text file for the items I need with type piped with findstr. I can use the results with wmic to capture the version too.
In all, it takes about 3.8 seconds, so not bad. I haven't tried a nested loop yet, but that will be next. When I get a chance to test, I will post whether that improves performance.
Re: Fastest Recursive Searching
Posted: 24 Jul 2021 06:31
by aGerman
The number of threads defaults to 8 if you don't specify the /MT option. Keep in mind that multithreading is not for free. There is a break-even. Rule of thumb is to use as many threads as the number of logical cores on your machine.
There's a predefined variable you might want to take instead.
/MT:%NUMBER_OF_PROCESSORS%
Just compare the performance with the default, or with the value that you previously specified.
Steffen
Re: Fastest Recursive Searching
Posted: 26 Jul 2021 06:16
by atfon
Hello Steffen (@aGerman),
Thank you for the feedback. I agree that you do use more resources with the /MT option, but it does seem to provide a small performance boost as long as your are judicious with it. I have kept it to 4X%NUMBER_OF_PROCESSORS%. In my case that equates to /MT:32. I tested it a few times compared to the default:
Test 1:
/MT:8 - 4.16
/MT:32 - 3.92
Test 2:
/MT:8 - 3.91
/MT:32 - 3.80
Test 3:
/MT:8 - 3.77
/MT:32 - 3.74
Re: Fastest Recursive Searching
Posted: 26 Jul 2021 10:48
by atfon
I had a chance to do some testing with a nested loop to generate a text file with paths of the executable files as @aGerman suggested. This resulted in some improved performance. To generate the list took 0.70 seconds. To run the complete script took 2.90 seconds.
Re: Fastest Recursive Searching
Posted: 26 Jul 2021 10:50
by aGerman
Your CPU must have been quite bored before
If you run the /MT:32 before /MT:8, is it still faster? (Beware of caching
)
Steffen
Re: Fastest Recursive Searching
Posted: 26 Jul 2021 11:40
by atfon
Could be! Good point about caching @aGerman. I ran a few tests with /MT:32 before /MT:8 for just the robocopy portion of the script and the results weren't all that conclusive:
Test 1
/MT:32 - 0.39 seconds
/MT:8 - 0.69 seconds
Test 2
/MT:32 - 0.81 seconds
/MT:8 - 0.68 seconds
Test 3
/MT:32 - 0.77 seconds
/MT:8 - 0.78 seconds
Test 4
/MT:32 - 0.70 seconds
/MT:8 - 0.77 seconds
In 3 of the 4 cases, /MT:32 was the faster of the two.
Edit to add some specification information:
Windows 10 Pro
Processor: Intel(R) Core(TM) i5-10210U CPU @ 1.60GHz 2.11 GHz
Installed RAM: 32.0 GB (31.8 GB usable)
Re: Fastest Recursive Searching
Posted: 26 Jul 2021 12:56
by aGerman
Code: Select all
@echo off &setlocal EnableDelayedExpansion
for %%n in (64 %NUMBER_OF_PROCESSORS% 2) do (
echo /mt:%%n
echo !time!
for /f "tokens=*" %%i in (
'robocopy "C:\." " nul" "wordpad.exe" /xd "C:\Windows" /l /s /mt:%%n /xx /r:1 /w:1 /ns /nc /ndl /np /njh /njs'
) do echo "%%i"
echo !time!
echo ~~~~~~~~~~~~~~~~~~~~~
)
pause
Typical result
Code: Select all
/mt:64
20:51:01,52
"C:\Program Files (x86)\Windows NT\Accessories\wordpad.exe"
"C:\Program Files\Windows NT\Accessories\wordpad.exe"
20:51:03,68
~~~~~~~~~~~~~~~~~~~~~
/mt:4
20:51:03,69
"C:\Program Files (x86)\Windows NT\Accessories\wordpad.exe"
"C:\Program Files\Windows NT\Accessories\wordpad.exe"
20:51:05,83
~~~~~~~~~~~~~~~~~~~~~
/mt:2
20:51:05,83
"C:\Program Files (x86)\Windows NT\Accessories\wordpad.exe"
"C:\Program Files\Windows NT\Accessories\wordpad.exe"
20:51:08,62
~~~~~~~~~~~~~~~~~~~~~
Drücken Sie eine beliebige Taste . . .
Slow little 14 inch notebook which is comfortable to have on my knees in the evening
Windows 10 Home 64 Bit
AMD Ryzen 3 3200U with Radeon Vega Mobile Gfx 2.60 GHz
4,00 GB RAM (3,44 GB usable)
For me it doesn't make much of a difference how many threads are defined, as long as it is at least the number of processors. I don't know if robocopy only schedules a reasonable number of threads depending on how it divides the work into packages (which is opaque for us since it isn't open source).
Steffen
Re: Fastest Recursive Searching
Posted: 26 Jul 2021 13:24
by atfon
Oddly, on my system this script stalls after displaying the following:
It never gets around to displaying the location of wordpad.exe, although it is found with this command:
Re: Fastest Recursive Searching
Posted: 26 Jul 2021 13:30
by aGerman
That's weird
Too many threads already?
What if you remove the 64 from the list?
Re: Fastest Recursive Searching
Posted: 26 Jul 2021 13:38
by atfon
Maybe!
I tried this bare command:
Code: Select all
robocopy "C:\." " nul" "wordpad.exe" /xd "C:\Windows" /l /s /xx /r:1 /w:1 /njh /njs /fp /ndl /ns /nc /np /mt:8
It seems to just sit there after locating the same two files your system found:
Code: Select all
C:\Program Files\Windows NT\Accessories\wordpad.exe
C:\Program Files (x86)\Windows NT\Accessories\wordpad.exe
Re: Fastest Recursive Searching
Posted: 26 Jul 2021 13:47
by aGerman
Then it's probably still searching in the remaining folder structures. Maybe C:\Users (which is only 7.6 GB here).
Re: Fastest Recursive Searching
Posted: 26 Jul 2021 13:54
by atfon
I agree, but it was taking several minutes. For comparison, I modified your script slightly to the following:
Code: Select all
robocopy "%programfiles%\ " "%tmp%\ " "wordpad.exe" /xd "C:\Windows" /l /s /mt:%%n /xx /r:1 /w:1 /njh /njs /fp /ndl /ns /nc /np
I received these results. Not too surprising since it is just recursing Program Files.
Code: Select all
/mt:64
15:52:34.33
"C:\Program Files\Windows NT\Accessories\wordpad.exe"
15:52:34.69
~~~~~~~~~~~~~~~~~~~~~
/mt:8
15:52:34.69
"C:\Program Files\Windows NT\Accessories\wordpad.exe"
15:52:35.03
~~~~~~~~~~~~~~~~~~~~~
/mt:2
15:52:35.03
"C:\Program Files\Windows NT\Accessories\wordpad.exe"
15:52:35.58
~~~~~~~~~~~~~~~~~~~~~
Re: Fastest Recursive Searching
Posted: 26 Jul 2021 14:15
by aGerman
I refrain from asking you how you got it bloated that much
FWIW I'm a little amazed that defining the source with a trailing space still works
I mean, it's a known issue that
"%programfiles%\" failes because robocopy uses the default C tokenizing for the command line. The backslash escapes the following quotation mark and the resulting token is something like
C:\Program Files" which is invalid. You can work around this by doubling the trailing backslash
"%programfiles%\\".
Just in case you're wondering why I used
" nul" as target: You can't create a folder with a leading space, at least not on Windows. However, robocopy still accepts it. If you use an existing folder and a file which is to be copied (even with option /L which doesn't actually copy anything) already exists, it wouldn't be listed.