Page 1 of 1
Capture variable in parallel
Posted: 12 Jan 2022 09:09
by atfon
Hello Folks. I've been reading various threads, but I haven't been able to achieve what I'm trying to do. I'm capturing various system properties as variables in a script. In this process, I am attempting to capture the level of fragmentation of the system drive. However, the defrag command is a bit slow. So, I was attempting to see if there was a way to run the command in parallel with the main script. Mainly, I was working with the start command to spawn the parallel process like this:
Code: Select all
start /b for /f "tokens=2 delims==" %%g in ('%__APPDIR__%defrag.exe %systemdrive% /a ^^^|%__APPDIR__%findstr.exe /r /C:"Total fragmented"') do for /f "tokens=*" %%h in ("%%g") do set "fragSpace=%%h"
However, when I echo the %fragSpace% variable to file, it is blank. Should I be using a different methodology to achieve my goal here?
Re: Capture variable in parallel
Posted: 12 Jan 2022 11:06
by aGerman
You are using the START command which means that your command runs in a child process. Updates of the child environment have no influence to the parent environment.
Steffen
Re: Capture variable in parallel
Posted: 12 Jan 2022 11:15
by atfon
aGerman wrote: ↑12 Jan 2022 11:06
You are using the START command which means that your command runs in a child process. Updates of the child environment have no influence to the parent environment.
Steffen
Got it. Thank you. Do you have an alternate recommended method to run a command to capture a variable in parallel to the main script or perhaps a more efficient way to capture the fragmentation information? Thanks.
Re: Capture variable in parallel
Posted: 12 Jan 2022 11:26
by aGerman
You can make them all write into separate text files and read the files after all processes have finished their task. Multi-processing is not that easy though. You have to join the processes (synchronize processes). You could also write to the same file with a write lock. Several people already wrote some examples in this forum. Another example:
Code: Select all
@echo off &setlocal
REM The code demonstrates how to install and use a mutex using a lock file.
REM This script is automatically executed twice. The second time using START and argument "secondProc" passed in order to tag it.
REM In our example, this file is what we want to protect from getting concurrently accessed while both processes write to it.
set "outfile=out.txt"
REM The mutex implementation needs a temporary file to install a lock. Also this file must be known to both processes. The local
REM %temp% directory is good enough for this example where both processes run in the same account and on the same PC. In other
REM cases, however, ensure that the path of the lock file is also a shared folder where all involved processes have write access.
set "lockfile=%temp%\mutex_implementation_detail.~lock"
REM Differentiate between the first and second process.
if "%~1"=="secondProc" (
title 2nd process
set "proc=2nd"
) else (
title 1st process
set "proc=1st"
2>nul del "%outfile%"
REM Ensure that the lock file exists with zero size before any of the processes uses it.
call :initMutex "%lockfile%"
REM Run the second process *ASYNCHRONOUSLY*.
start cmd /c "%~fs0" secondProc
)
REM This loop is executed in both processes. Thus, it tries to write into the same file concurrently.
REM Writing fails if the file is accessed by the other process at the same time.
REM For that reason the write process is critical and must be synchronized using a mutex.
REM Only this way all lines (600 in total) are written to "%outfile%" without any error.
REM - Resulting arguments in :execCriticalSection - %0 %1 %2 %3
for /l %%i in (1 1 300) do call :execCriticalSection "%lockfile%" call :criticalSection %%i
REM Comment the above line out, and uncomment the line below in order to see what happens if the mutex is bypassed.
:: for /l %%i in (1 1 300) do call :criticalSection %%i
REM Signal that the current process doesn't access the mutex anymore.
call :finishLocalMutex "%lockfile%"
REM Quit the second process. Only one process continues with the next lines.
if not "%proc%"=="1st" exit /b
REM Wait until both processes called :finishLocalMutex.
call :waitForMultipleProcesses "%lockfile%" 2
REM Clean up because no process uses the lock file anymore.
call :deleteMutex "%lockfile%"
REM Tell how many lines the two processes have been able to write.
for /f %%i in ('type "%outfile%"^|find /c /v ""') do echo %%i of 600 lines written
pause
exit /b
REM ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
REM The :criticalSection subroutine contains only the piece of code which requires mutual exclusion.
:criticalSection
>>"%outfile%" echo %proc% %1
exit /b
REM *** The below subroutines are generic and used to control working with the mutex. ***
REM ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
REM Ensure that the lock file exists with zero size. This is critical to make both :finishLocalMutex and :waitForMultipleProcesses
REM work properly.
REM - lockFile The mutex is established upon a write lock of a unique, temporary file. All involved processes need to use the same
REM same lock file. The owners of the processes need to have write permissions granted to the path of the lock file.
:initMutex lockFile
>%1 type nul
exit /b
REM ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
REM This subroutine runs a polling loop until it is able to get ownership of the mutex.
REM The mutex is owned as long as it takes to process the passed command, which should be only the critical section that needs synchronization.
REM Once the code of the critical section has been executed, the mutex is automatically released to make it available for another process.
REM Note that the mutex is unfair. Many processes may try to acquire it, but which of them gets it next is pretty much arbitrary. No FIFO.
REM - lockFile Lock file, created by :initMutex.
REM - criticalCommand Command which needs to be synchronized to prevent race conditions.
REM - [arg1 [..arg8]] Up to 8 arguments, passed to the critical command.
:execCriticalSection lockFile criticalCommand [arg1 [..arg8]]
shift
:__spin_lock__
8>&2 2>nul (9>>%0 ((2>&8 %1 %2 %3 %4 %5 %6 %7 %8 %9)&(call )))||goto __spin_lock__
exit /b
REM ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
REM This subroutine increments the size of the lock file by one byte in order to signal that the calling process doesn't intent to access the
REM critical section any longer. A process must call it exactly once, and must not use the mutex anymore after :finishLocalMutex has
REM been called.
REM - lockFile Lock file, created by :initMutex.
:finishLocalMutex lockFile
2>nul (>>%1 ((<nul set /p "=#")&(call )))||goto finishLocalMutex
exit /b
REM ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
REM This subroutine runs a polling loop until all processes called :finishLocalMutex. Only in this case the size of the lock file equals
REM the number of involved processes.
REM - lockFile Lock file, created by :initMutex.
REM - procCount Number of involved processes.
:waitForMultipleProcesses lockFile procCount
if %~z1 lss %~2 >nul pathping.exe 127.0.0.1 -n -q 1 -p 100&goto waitForMultipleProcesses
exit /b
REM ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
REM This subroutine deletes the lock file. None of the involved processees is allowed to use the mutex after deleteMutex has been called.
REM - lockFile Lock file, created by :initMutex.
:deleteMutex lockFile
>nul 2>&1 del %1
exit /b
REM *** ... more verbose ... ***
REM The code of the :execCriticalSection subroutine is both generic and cryptic. So I guess I should go through step by step
REM in order to explain what's going on here ...
REM
REM shift
REM We don't use the initial %0, it just contains :execCriticalSection. Thus, we make room for one more optional argument.
REM
REM :__spin_lock__
REM Label to establish the polling loop.
REM
REM 8>&2 2>nul (
REM This needs to be read from right to left. First discard the error message we get if the lock file can't be accessed.
REM Then we redirect stream 8 to the error stream. This enables getting error messages from the critical section which are
REM redirected to stream 8. (See further explanation below.)
REM
REM 9>%0 (
REM After SHIFT is executed in the first line, %0 contains the name of the lock file. We try to redirect stream 9 (which is
REM supposed to be unused) to the lock file. This will fail if the lock file is still used by another process. In this case
REM the program flow is conditionally continued after the ||. Otherwise the current process will lock the file, execute the
REM critical command, and unlock the file. This only works properly because there is no transitional state between a file
REM being locked and unlocked.
REM
REM (2>&8 %1 %2 %3 %4 %5 %6 %7 %8 %9)
REM This executes the critical command which is in %1, while %2..%9 are optional arguments, passed to the critical command.
REM Any error message thrown by the critical command will get redirected to stream 8. (Complementary redirection to 8>&2.)
REM Side note: Not only that the number of arguments is limited, it might also be risky to use them. Calling a subroutine
REM alters percent signs and carets in the arguments. Rather define variables and use them in a subroutine that represents
REM the critical section.
REM
REM &(call )
REM Reset the errorlevel to 0 in order to ensure that we leave the polling loop even if the critical command failed. (The
REM space after the CALL command is critical.)
REM
REM ))||goto __spin_lock__
REM If the redirection to the lock file failed, we will jump back to the :__spin_lock__ label and try again. This is repeated
REM until the lock file isn't accessed by another process anymore.
- Schema.png (36.57 KiB) Viewed 3395 times
Steffen
Re: Capture variable in parallel
Posted: 12 Jan 2022 12:49
by atfon
Thank you, Steffen! Helpful as always.