creating a true mutex for exclusive acces

[Aacini]

ok, cool but now I have a different problem. I need to clean the DOS environment before I start a new instance of CMD. The batchfile I use to develop and test uses 250MB of memory when it reaches this point. Actual memory is only 5MB but in DOS, once memory is taken, it's never released even if the environment is cleared. It takes a long time to prepare the environment for a new instance:

Code: Select all

for /F "tokens=1 delims==" %%? in (

       'set'

) do   if defined %%? set "%%?="

There is no secret switch to make the new cmd instance NOT inherit it's parent environment ?

Maybe there is a better way ?

A couple points here:

- The "if defined %%?" part in your code is not needed because the "set" command list defined variables only. Perhaps whitout the "if", the clean memory process run faster.

- Run the clean memory process this way; it should run faster:

Code: Select all

for /F "tokens=1 delims==" %%? in (

       'set ^| sort /R'

) do   set "%%?="

- You said "actual memory is only 5MB", but the test batchfile fills it up to 250MB. If the "actual memory" is complete before the develop and test part begins, you may execute a "setlocal" command at that point. This way, if you execute an "endlocal" before start the new instance of CMD, all the additional memory taken from the test part will be released.

- Your "secret switch" is "start"'s command /I switch. Try to start the new instance of CMD this way:

Code: Select all

START /I CMD.EXE same switches and parameters...

Antonio

[aGerman]

Your "secret switch" is "start"'s command /I switch.

Thanks Antonio.

The German help message of this switch is ambiguous. I never understood it and unfortunately I never tried to investigate its actual meaning.

[Aacini]

Yes, the description shown by START /? help is confusing, but the description at SS64.com is clearer:

Code: Select all

   /I         Ignore any changes to the current environment.
              Use the original environment passed to cmd.exe

Antonio

[Ed Dyreen]

- The "if defined %%?" part in your code is not needed because the "set" command list defined variables only. Perhaps whitout the "if", the clean memory process run faster.

Not quite;

Code: Select all

@echo off &setlocal enableDelayedExpansion &set $lf=^


::need line

set "$=echo.hi!$lf! =notDefined"

for /F "tokens=1 delims==" %%? in ( 'set $' ) do echo.set "%%?=" &set "%%?="

pause
exit

Code: Select all

set "$="
set " ="
De syntaxis van de opdracht is onjuist.
set "$lf="
Druk op een toets om door te gaan. . .

Now I need to redirect stream2 to nul, which explains the presence of the if statement. Though redirecting stream2 to nul and leaving out 'if' is probably faster.

- Run the clean memory process this way; it should run faster:

Code: Select all

for /F "tokens=1 delims==" %%? in (

       'set ^| sort /R'

) do   set "%%?="

It does avoid the 'De syntaxis van de opdracht is onjuist.' error thrown by set, but why sorting the variables before processing them by for would be faster ?

- You said "actual memory is only 5MB", but the test batchfile fills it up to 250MB. If the "actual memory" is complete before the develop and test part begins, you may execute a "setlocal" command at that point. This way, if you execute an "endlocal" before start the new instance of CMD, all the additional memory taken from the test part will be released.

The parent process uses 250MB and so does the child when it is spawned, ( if left uncleaned, even if it really only uses about 5MB, that tells me reserved memory is also copied even if unset. this particular test does not uses a significant amount of memory, but at this point the batchfile does since this test is near the end of the batchfile and by now pretty much all macro's have been defined. The 250MB cannot be endlocalled because the parent is in global environment so it can pass it's environment to it's caller batch when it ends.

- Your "secret switch" is "start"'s command /I switch. Try to start the new instance of CMD this way:

Code: Select all

START /I CMD.EXE same switches and parameters...

Antonio

Ok that helps, but what /I actually does is return me the DOS environment of the batch that started a batch that called this batch that starts this new instance.

Code: Select all

aBatchFile:start "bBatchFile" // this environment is returned when using the /I switch on next start command
bBatchFile:call "cBatchFile"
cBatchFile:start /I "dCommand"

Didn't know you could create a process with WMIC that uses the default environment, it's much more elegant and efficient way to default back to original environment.

Well, at least no direct child processes.
To work around it you could use a scheduled task that you could call via SCHTASKS /RUN.
Also WMIC could help. E.g.

Code: Select all

@echo off &setlocal
if "%~1"=="test" (
  set xyz
  pause
  exit /b
)

set "xyz=Hello!"
set xyz
>nul wmic process call create "cmd /c %~fs0 test"
pause

cool, I will see if I can use this for cmdContext macro's with delayed linefeeds in it, I fear for it though..

The thing is that in a cmdLineContext a linefeed has to be entered like this

Code: Select all

set "$cmd=echo.hello%%$lf%%echo.there"
cmd /C "!$cmd!"

So the linefeed is expanded after the new instance has spawned, this is the only variable that should NOT be undefined. Now I'd have to edit the registry to see if I can make it part of the default environment. ugh

I probably better just convert my variable so it won't contain any linefeeds in the first place.

Thanx aGerman

[Aacini]

- The "if defined %%?" part in your code is not needed because the "set" command list defined variables only. Perhaps whitout the "if", the clean memory process run faster.

Not quite;

Code: Select all

Now I need to redirect stream2 to nul, which explains the presence of the if statement. Though redirecting stream2 to nul and leaving out 'if' is probably faster.

I got it. You want to manage the particular case when a few variables have an embedded <LF>.

... why sorting the variables before processing them by for would be faster ?

When one variable is deleted, the rest of variables below it must be moved up in order to fill the empty space. The variables in the environment are stored in alphabeticall order, so if all variables were deleted in the same order, a large amount of data movements would be required. If the variables are deleted from the last one to the first one, all these data movements are avoided.

The parent process uses 250MB and so does the child when it is spawned, ( if left uncleaned, even if it really only uses about 5MB, that tells me reserved memory is also copied even if unset.

No. Just the occupied space is copied to the environment of the new cmd.exe. See this post:

Note that the reserved environment space will be lost if a SETLOCAL command or any external file is executed, like CMD.EXE. Only the occupied environment space will be copied to the new local environment.

- Your "secret switch" is "start"'s command /I switch. Try to start the new instance of CMD this way:

Code: Select all

START /I CMD.EXE same switches and parameters...

Ok that helps, but what /I actually does is return me the DOS environment of the batch that started a batch that called this batch that starts this new instance.

When a new instance of cmd.exe is executed, it receives its original environment. When "START /I anyprog.exe" command is executed in this cmd.exe instance, anyprog.exe receives a copy of the original cmd.exe environment instead of the current environment. This behavior also apply if a new instance of cmd.exe was executed from another running cmd.exe. The program below is an example of this mechanism; run it via a doble-click from the explorer.

Code: Select all

@echo off
if "%1" neq "" goto %1

cls
echo *ORIGINAL* (first time) environment:
set a
echo/

set "A New Variable=new value"
cmd /C echo Inherited environment (plus one var): ^& set a
echo/

pause
start /I cmd /C echo Original environment (again): ^& set a ^& echo/^& pause
pause

set "A Second Variable=second value"
cmd /C "%~F0" secondTime
goto :EOF

:secondTime
echo/
echo "Inherited original" (second time) environment:
set a
echo/

set "A Third Variable=third value"
cmd /C echo Inherited (second time) environment (plus one var): ^& set a
echo/

pause
start /I cmd /C echo "Inherited original" environment (again): ^& set a ^& echo/^& pause
pause

Antonio

[Ed Dyreen]

The parent process uses 250MB and so does the child when it is spawned, ( if left uncleaned, even if it really only uses about 5MB, that tells me reserved memory is also copied even if unset.

No. Just the occupied space is copied to the environment of the new cmd.exe. See this post:

Note that the reserved environment space will be lost if a SETLOCAL command or any external file is executed, like CMD.EXE. Only the occupied environment space will be copied to the new local environment.

Are you suggesting that I may have 250MB of memory defined in my parent session ? Cause if I start a new instance I get a second cmd process that occupies about the same space as my parent session. If I dump the variables loaded at that particular point to a file and then reload them I get an environment of only 5MB. I haven't done a complete memory dump, maybe I should ?

- Your "secret switch" is "start"'s command /I switch. Try to start the new instance of CMD this way:

Code: Select all

START /I CMD.EXE same switches and parameters...

Ok that helps, but what /I actually does is return me the DOS environment of the batch that started a batch that called this batch that starts this new instance.

When a new instance of cmd.exe is executed, it receives its original environment. When "START /I anyprog.exe" command is executed in this cmd.exe instance, anyprog.exe receives a copy of the original cmd.exe environment instead of the current environment. This behavior also apply if a new instance of cmd.exe was executed from another running cmd.exe. The program below is an example of this mechanism; run it via a doble-click from the explorer.

Code: Select all

@echo off
if "%1" neq "" goto %1

cls
echo *ORIGINAL* (first time) environment:
set a
echo/

set "A New Variable=new value"
cmd /C echo Inherited environment (plus one var): ^& set a
echo/

pause
start /I cmd /C echo Original environment (again): ^& set a ^& echo/^& pause
pause

set "A Second Variable=second value"
cmd /C "%~F0" secondTime
goto :EOF

:secondTime
echo/
echo "Inherited original" (second time) environment:
set a
echo/

set "A Third Variable=third value"
cmd /C echo Inherited (second time) environment (plus one var): ^& set a
echo/

pause
start /I cmd /C echo "Inherited original" environment (again): ^& set a ^& echo/^& pause
pause

Antonio

I don't get the original environment, I get the environment of tsttst.CMD

tsttst.CMD

Code: Select all

@echo off &setlocal disableDelayedExpansion &set "$=%~n0"
:: (
   start "" /LOW "%comspec%" /K ^""%$:~3%.CMD" /Hello: "wo^rld !"^"
:: )
endlocal

tst.CMD

Code: Select all

@echo off &setlocal enableDelayedExpansion

set $
set x
set x=something
pause
start "" /I "%comspec%" /K "%~nx0"

pause
exit

parent window after one press ENTER:

Code: Select all

$=tsttst
Omgevingsvariabele x is niet gedefinieerd
Druk op een toets om door te gaan. . .
Druk op een toets om door te gaan. . .

child window spawned by 'start /I':

Code: Select all

$=tsttst
Omgevingsvariabele x is niet gedefinieerd
Druk op een toets om door te gaan. . .

With original environment, I mean the initial default environment that cmd.EXE gives me when I run it from the start+R window. Not the environment of the parent session which I get with 'start /I'

[Ed Dyreen]

- Run the clean memory process this way; it should run faster:

Code: Select all

for /F "tokens=1 delims==" %%? in (

       'set ^| sort /R'

) do   set "%%?="

Piping the output of 'set' is insecure ( the process tried to write to a nonexistent pipe ).

Code: Select all

@echo off &setlocal enableDelayedExpansion

for /F %%? in (

   '^(for /F "tokens=1 delims==" %%? in ^( 'set' ^) do @echo.%%^^^?^)^|sort /R'

) do (
   echo.set "%%?="
   set "%%?="
)

pause
exit

It kinda works if I prepare the variables before sorting. I say kinda because I still should need to test 'if defined' before piping to sort but '@if defined %%^^^? echo.%%^^^?' didn't work and I didn't find an easy fix. But even if it would, I doubt an extra 'for' loop and a 'sort /R' would be any faster.

I guess I just better leave this part of the code as I wrote it initially or use aGerman's solution. But his solution requires me to either eliminate all linefeeds which I don't intend to, or create yet another file as a destination for wmic create process...

[Ed Dyreen]

Here is a more impressive, and easier to run demonstration. It creates two test processes within the same window that alternate control via the asynchronous mutex. I've eliminated the sleep within the mutex code, so it is a CPU hog. But it shows the nice, orderly synchronization between the two processes.

Each test process can have as many as two sub-processes active at any given time - one for the lock, and another to wait for the lock to be established.

Code: Select all

@echo off
if "%~1" neq "" goto :test
setlocal

:: initialize
set "sleep=pathping /p 500 /q 1 localhost >nul 2>nul"

start /b "" "%~f0" 1
cmd /c "%~f0" 2
exit /b

:test
for /l %%N in (1 1 10) do (
  call :lockMutex %~1
  echo mutex lock established for %~1
  %sleep%
  echo releasing mutex lock for %~1
  call :releaseMutex %~1
)
exit


:lockMutex
setlocal
:: request mutex lock
start /b cmd /c "2>nul >nul (for /l %%A in () do 9>mutex.lock ((call )>mutex%~1.active&for /l %%B in () do if not exist mutex%~1.active exit))"
:: wait for lock
cmd /c "for /l %%A in () do @if exist mutex%~1.active (exit)"
exit /b 0


:releaseMutex
del mutex%~1.active
exit /b

-- Sample Output--

Code: Select all

mutex lock established for 1
releasing mutex lock for 1
mutex lock established for 2
releasing mutex lock for 2
mutex lock established for 1
releasing mutex lock for 1
mutex lock established for 2
releasing mutex lock for 2
mutex lock established for 1
releasing mutex lock for 1
mutex lock established for 2
releasing mutex lock for 2
mutex lock established for 1
releasing mutex lock for 1
mutex lock established for 2
releasing mutex lock for 2
mutex lock established for 1
releasing mutex lock for 1
mutex lock established for 2
releasing mutex lock for 2
mutex lock established for 1
releasing mutex lock for 1
mutex lock established for 2
releasing mutex lock for 2
mutex lock established for 1
releasing mutex lock for 1
mutex lock established for 2
releasing mutex lock for 2
mutex lock established for 1
releasing mutex lock for 1
mutex lock established for 2
releasing mutex lock for 2
mutex lock established for 1
releasing mutex lock for 1
mutex lock established for 2
releasing mutex lock for 2
mutex lock established for 1
releasing mutex lock for 1
mutex lock established for 2
releasing mutex lock for 2

Dave Benham

But I also need to know whether the mutex was created successfully or not. I can't return a value, because I'm in a different cmd instance, nor can I use a FOR or a pipe which I would use to read the return value, because in that case it'll be waiting for output it'll only get if the mutex process closes. I could write the errorlevel to a file, but then how can I be sure it was not created by a mutex of another script's process or the same script running parallel. In autoIT when I create a mutex it will return a handle to the mutex or an empty value if the mutex could not be created.

[OperatorGK]

But you can just wrap your asynchronous mutex like so:

Code: Select all

:safeLockAsyncMutex [name]
if exist mutex%~1.active (
  rem Mutex is already locked
  exit /b 1
)
%lockMutex% %~1
if not exist mutex%~1.active (
  rem Mutex wasn't locked successfully
  exit /b -1
)
rem Mutex was successfully locked
exit /b 0

[Ed Dyreen]

But you can just wrap your asynchronous mutex like so:

Code: Select all

:safeLockAsyncMutex [name]
if exist mutex%~1.active (
  rem Mutex is already locked
  exit /b 1
)
%lockMutex% %~1
if not exist mutex%~1.active (
  rem Mutex wasn't locked successfully
  exit /b -1
)
rem Mutex was successfully locked
exit /b 0

Yes you can identify whether a mutex is locked this way, but there is no way to tell whether the lock was put in place by this instance or another parallel instance. Therefor it still can't tell whether it's safe to continue.

Code: Select all

function mutex() 
  2>nul (9>mutex.try (
    9>mutex.lock (
...
    ) || lock failed
  ) || try failed )

To make sure parallel functions use the function mutex in serial I added a try,
The thought behind this is that if the try succeeds I should be confident no other instance is trying to lock anything.

[OperatorGK]

I think you can give unique IDs to your parallel instances and write ID of instance in your mutex file.

[dbenham]

Bingo

[OperatorGK]

You've forgotten "Dave Benham" in the end .

[Ed Dyreen]

The only uniqueID I know of is processID.

looking up information using wmic reliably involves making sure WMIC is initialized for first time use and it's result is always in ANSI. Having to lookup the childProcessID and maybe having the child have to lookup it's PID reliably is going to add many more seconds.

creating a mutex now takes a few seconds but when adding above to a function that opens a lock will now take over 30 seconds and is unacceptable.

[OperatorGK]

By "Unique ID" I mean:

Code: Select all

set UID=%random%%random%%random%%random%%random%%random%

, not the process id. Only the batch itself knows it's ID.