Capture variable in parallel

[atfon]

Hello Folks. I've been reading various threads, but I haven't been able to achieve what I'm trying to do. I'm capturing various system properties as variables in a script. In this process, I am attempting to capture the level of fragmentation of the system drive. However, the defrag command is a bit slow. So, I was attempting to see if there was a way to run the command in parallel with the main script. Mainly, I was working with the start command to spawn the parallel process like this:

Code: Select all

start /b for /f "tokens=2 delims==" %%g in ('%__APPDIR__%defrag.exe %systemdrive% /a ^^^|%__APPDIR__%findstr.exe /r /C:"Total fragmented"') do for /f "tokens=*" %%h in ("%%g") do set "fragSpace=%%h"

However, when I echo the %fragSpace% variable to file, it is blank. Should I be using a different methodology to achieve my goal here?

[aGerman]

You are using the START command which means that your command runs in a child process. Updates of the child environment have no influence to the parent environment.

Steffen

[atfon]

You are using the START command which means that your command runs in a child process. Updates of the child environment have no influence to the parent environment.

Steffen

Got it. Thank you. Do you have an alternate recommended method to run a command to capture a variable in parallel to the main script or perhaps a more efficient way to capture the fragmentation information? Thanks.

[aGerman]

You can make them all write into separate text files and read the files after all processes have finished their task. Multi-processing is not that easy though. You have to join the processes (synchronize processes). You could also write to the same file with a write lock. Several people already wrote some examples in this forum. Another example:

Code: Select all

@echo off &setlocal
REM The code demonstrates how to install and use a mutex using a lock file.
REM This script is automatically executed twice. The second time using START and argument "secondProc" passed in order to tag it.

REM In our example, this file is what we want to protect from getting concurrently accessed while both processes write to it.
set "outfile=out.txt"
REM The mutex implementation needs a temporary file to install a lock. Also this file must be known to both processes. The local
REM %temp% directory is good enough for this example where both processes run in the same account and on the same PC. In other
REM cases, however, ensure that the path of the lock file is also a shared folder where all involved processes have write access.
set "lockfile=%temp%\mutex_implementation_detail.~lock"


REM Differentiate between the first and second process.
if "%~1"=="secondProc" (
  title 2nd process
  set "proc=2nd"
) else (
  title 1st process
  set "proc=1st"
  2>nul del "%outfile%"
  REM Ensure that the lock file exists with zero size before any of the processes uses it.
  call :initMutex "%lockfile%"
  REM Run the second process *ASYNCHRONOUSLY*.
  start cmd /c "%~fs0" secondProc
)


REM This loop is executed in both processes. Thus, it tries to write into the same file concurrently.
REM Writing fails if the file is accessed by the other process at the same time.
REM For that reason the write process is critical and must be synchronized using a mutex.
REM Only this way all lines (600 in total) are written to "%outfile%" without any error.
REM - Resulting arguments in :execCriticalSection -      %0        %1         %2        %3
for /l %%i in (1 1 300) do call :execCriticalSection "%lockfile%" call :criticalSection %%i
REM Comment the above line out, and uncomment the line below in order to see what happens if the mutex is bypassed.
:: for /l %%i in (1 1 300) do call :criticalSection %%i

REM Signal that the current process doesn't access the mutex anymore.
call :finishLocalMutex "%lockfile%"

REM Quit the second process. Only one process continues with the next lines.
if not "%proc%"=="1st" exit /b

REM Wait until both processes called :finishLocalMutex.
call :waitForMultipleProcesses "%lockfile%" 2
REM Clean up because no process uses the lock file anymore.
call :deleteMutex "%lockfile%"
REM Tell how many lines the two processes have been able to write.
for /f %%i in ('type "%outfile%"^|find /c /v ""') do echo %%i of 600 lines written
pause
exit /b


REM ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
REM The :criticalSection subroutine contains only the piece of code which requires mutual exclusion.
:criticalSection
>>"%outfile%" echo %proc% %1
exit /b


REM *** The below subroutines are generic and used to control working with the mutex. ***

REM ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
REM Ensure that the lock file exists with zero size. This is critical to make both :finishLocalMutex and :waitForMultipleProcesses
REM work properly.
REM - lockFile  The mutex is established upon a write lock of a unique, temporary file. All involved processes need to use the same
REM             same lock file. The owners of the processes need to have write permissions granted to the path of the lock file.
:initMutex  lockFile
>%1 type nul
exit /b


REM ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
REM This subroutine runs a polling loop until it is able to get ownership of the mutex.
REM The mutex is owned as long as it takes to process the passed command, which should be only the critical section that needs synchronization.
REM Once the code of the critical section has been executed, the mutex is automatically released to make it available for another process.
REM Note that the mutex is unfair. Many processes may try to acquire it, but which of them gets it next is pretty much arbitrary. No FIFO.

REM - lockFile         Lock file, created by :initMutex.
REM - criticalCommand  Command which needs to be synchronized to prevent race conditions.
REM - [arg1 [..arg8]]  Up to 8 arguments, passed to the critical command.
:execCriticalSection  lockFile  criticalCommand  [arg1 [..arg8]]
shift
:__spin_lock__
8>&2 2>nul (9>>%0 ((2>&8 %1 %2 %3 %4 %5 %6 %7 %8 %9)&(call )))||goto __spin_lock__
exit /b


REM ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
REM This subroutine increments the size of the lock file by one byte in order to signal that the calling process doesn't intent to access the
REM critical section any longer. A process must call it exactly once, and must not use the mutex anymore after :finishLocalMutex has
REM been called.
REM - lockFile  Lock file, created by :initMutex.
:finishLocalMutex  lockFile
2>nul (>>%1 ((<nul set /p "=#")&(call )))||goto finishLocalMutex
exit /b


REM ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
REM This subroutine runs a polling loop until all processes called :finishLocalMutex. Only in this case the size of the lock file equals
REM the number of involved processes.
REM - lockFile   Lock file, created by :initMutex.
REM - procCount  Number of involved processes.
:waitForMultipleProcesses  lockFile  procCount
if %~z1 lss %~2 >nul pathping.exe 127.0.0.1 -n -q 1 -p 100&goto waitForMultipleProcesses
exit /b


REM ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
REM This subroutine deletes the lock file. None of the involved processees is allowed to use the mutex after deleteMutex has been called.
REM - lockFile  Lock file, created by :initMutex.
:deleteMutex  lockFile
>nul 2>&1 del %1
exit /b


REM *** ... more verbose ... ***
REM The code of the :execCriticalSection subroutine is both generic and cryptic. So I guess I should go through step by step
REM in order to explain what's going on here ...
REM
REM shift
REM   We don't use the initial %0, it just contains :execCriticalSection. Thus, we make room for one more optional argument.
REM
REM :__spin_lock__
REM   Label to establish the polling loop.
REM
REM 8>&2 2>nul (
REM   This needs to be read from right to left. First discard the error message we get if the lock file can't be accessed.
REM   Then we redirect stream 8 to the error stream. This enables getting error messages from the critical section which are
REM   redirected to stream 8. (See further explanation below.)
REM
REM 9>%0 (
REM   After SHIFT is executed in the first line, %0 contains the name of the lock file. We try to redirect stream 9 (which is
REM   supposed to be unused) to the lock file. This will fail if the lock file is still used by another process. In this case
REM   the program flow is conditionally continued after the ||. Otherwise the current process will lock the file, execute the
REM   critical command, and unlock the file. This only works properly because there is no transitional state between a file
REM   being locked and unlocked.
REM
REM (2>&8 %1 %2 %3 %4 %5 %6 %7 %8 %9)
REM   This executes the critical command which is in %1, while %2..%9 are optional arguments, passed to the critical command.
REM   Any error message thrown by the critical command will get redirected to stream 8. (Complementary redirection to 8>&2.)
REM   Side note: Not only that the number of arguments is limited, it might also be risky to use them. Calling a subroutine
REM   alters percent signs and carets in the arguments. Rather define variables and use them in a subroutine that represents
REM   the critical section.
REM
REM &(call )
REM   Reset the errorlevel to 0 in order to ensure that we leave the polling loop even if the critical command failed. (The
REM   space after the CALL command is critical.)
REM
REM ))||goto __spin_lock__
REM   If the redirection to the lock file failed, we will jump back to the :__spin_lock__ label and try again. This is repeated
REM   until the lock file isn't accessed by another process anymore.

Schema.png (36.57 KiB) Viewed 3385 times

Steffen

[atfon]

Thank you, Steffen! Helpful as always.