Why does SET performance degrade as environment size grows?

[dbenham]

This large memory issue cropped up again. I ran into a massive performance degradation problem while working on a StackOverflow question - http://stackoverflow.com/a/39672776/1012053. Within a loop of 1000, I was creating hundreds of variables between SETLOCAL and ENDLOCAL. The performance was totally non-linear, to the point of quickly becoming unusable.

I believe I have made a significant new discovery related to environment memory management. We had been thinking that the environment never shrinks. But I discovered that I could get the environment to shrink if I put the SETLOCAL, massive allocation, and ENDLOCAL within a CALLed subroutine, or within a new cmd.exe process. The new process wasn't much of a surprise, but the CALL solution was.

I haven't had time to fully investigate, but this looks promising.


Dave Benham

[Ed Dyreen]

This is great info

@Aacini, I used your analysis results to optimize my code.

I have a few hundreds of variables stored on disk that all follow the same analogy; set variable=data. where variable is fileName and data can be anything from 0 to 8k of data.

I used sort.EXE to get an alphabetical filename list and then call the variables into memory alphabetically and reverse alphabetically. I expected the reverse alphabet list to load variables much more slowly in memory but in fact speed remained exactly the same. So there seem to be more involved that add up to the speed than just the Set command. ( The names of the variables are sorted but now call will need to jump from one directory to the other whereas before sorting was by output of for /R )

Then I increase the environment by 1 MB using 256x4096kb variables and unload them in reverse order. This takes only a few seconds however it takes exponentially more time as environment grows to preserve more memory and mem.EXE /P crash above 1MB on windows XP.

Code: Select all

	setlocal enableDelayedExpansion
	:: (
		set "$4096=12345678" &for /L %%i in ( 1, 1, 9 ) do set "$4096=!$4096!!$4096!"
	:: )
	endlocal &set "a=%$4096:~7%"%=						minus 7 for variable name		=%

	(echo. ) &set "oldSize=" &for /F "skip=1 tokens=1-3" %%a in (

		'mem.EXE /P ^|findstr.EXE /I "COMMAND"'

	) do 	if not defined oldSize echo.  occupied range: '%%a-%%c' &set /A oldSize = 0x%%c
	::
	echo.  oldSize: '%oldSize%' bytes.

	for /L %%u in ( 1, 1, %$reserveMemorySize% ) do for /L %%x in ( 1, 1, 4 ) do (%=1 megaByte / 4 	= 256 kiloBytes	=%

		for /L %%y in ( 1, 1, 8 ) do (%=				256 kiloBytes / 8 	= 32 kiloBytes	=%

			for /L %%z in ( 1, 1, 8 ) do (%=			32 kiloBytes / 8 	= 4096 	bytes	=%

				set "$[%%u%%x%%y%%z]=%a%"
			)
		)
	)

	set /A edp = 128 * $reserveMemorySize
	echo.  accomodating up to %edp% fast definitions in %$reserveMemorySize% megabytes.

	if +%$reserveMemorySize% LSS 2 ( %=					mem.EXE crash above 1mb on winXPSP3	=%

		(echo. ) &set "newSize=" &for /F "skip=1 tokens=1-3" %%a in (

			'mem.EXE /P ^|findstr.EXE /I "COMMAND"'

		) do 	if not defined newSize echo.  occupied range: '%%a-%%c' &set /A newSize = 0x%%c
		::
		set /A size = newSize - oldSize
	)
	if +%$reserveMemorySize% LSS 2 ( %=					mem.EXE crash above 1mb on winXPSP3	=%

		echo.  newSize: '%newSize%' bytes.
		echo.  growth : '%size%' bytes.
	)

	(echo. ) &echo.  clearing %$reserveMemorySize% megabytes in reverse order...
	::
	for /L %%u in ( %$reserveMemorySize%, -1, 1 ) do for /L %%x in ( 4, -1, 1 ) do (%=1 megaByte / 4= 256 kiloBytes	=%

		for /L %%y in ( 8, -1, 1 ) do (%=				256 kiloBytes / 8 	= 32 kiloBytes	=%

			for /L %%z in ( 8, -1, 1 ) do (%=			32 kiloBytes / 8 	= 4096 	bytes	=%

				set "$[%%u%%x%%y%%z]="
			)
		)
	)

But this one did improve performance of the Set command by a whopping 275%.

[einstein1969]

after many years I need to understand why in dbenham's expansion tests there are no problems, while those of aacini show a degradation.

[einstein1969]

I'm doing some tests to delve into the issue of the environment.

I discovered a new bug.

This script populates the env first with variables whose name is of the form I_* and then after emptying it fills it with other variables of the type I*.
Well, the tests lead to an increase in performance of 3 times as much. This behavior occurs on windows 10 but is fixed on windows 11.

Other forms of the variable that reduce performance are i[*] or i.*. More tests will follow.

Code: Select all

@echo off
setlocal EnableDelayedExpansion

for /F "Tokens=1 delims==" %%v in ('set') do set "%%v="

rem prepare for test

set t1=%time%

rem popolate evil i_%%L , i[%%L], i.%%L
for /L %%L in (1,1,5000) do (
	set "i_%%L=!random!!random!!random!!random!!random!!random!!random!!random!!random!!random!!random!!random!!random!!random!!random!"
	title %%L
)
set t2=%time%
for /F "tokens=1-8 delims=:.," %%a in ("%t1: =0%:%t2: =0%") do set /a "td=(((1%%e-1%%a)*60)+1%%f-1%%b)*6000+1%%g%%h-1%%c%%d, td+=(td>>31) & 8640000"

echo !t1!-!t2! td:!td! centiseconds

for /F "Tokens=1 delims==" %%v in ('set') do set "%%v="

set t1=%time%

rem popolate
for /L %%L in (1,1,5000) do (
	set "i%%L=!random!!random!!random!!random!!random!!random!!random!!random!!random!!random!!random!!random!!random!!random!!random!"
	title %%L
)
set t2=%time%
for /F "tokens=1-8 delims=:.," %%a in ("%t1: =0%:%t2: =0%") do set /a "td=(((1%%e-1%%a)*60)+1%%f-1%%b)*6000+1%%g%%h-1%%c%%d, td+=(td>>31) & 8640000"

echo !t1!-!t2! td:!td! centiseconds

pause

result:

Code: Select all

 8:45:02,99- 8:45:23,25 td:2026 centiseconds
 8:45:27,82- 8:45:33,41 td:559 centiseconds