creating a true mutex for exclusive acces

[Ed Dyreen]

By "Unique ID" I mean:

Code: Select all

set UID=%random%%random%%random%%random%%random%%random%

You say uniqueID but your code creates a randomID, a RID is not a UID.

Using correct terminology is important. If I'd find that code running on a Boeing 747 I'd get off the plane immediately.

I think I'll use doskey to set up a communication link to the parent. I also won't need a UID this way.

[OperatorGK]

I think I'll use doskey to set up a communication link to the parent

How ? Can you please write the code? And will it work parallel?
Also, for WHAT you need such a parallel scripting? Only file I/O heavy tasks like AES ciphering/ MD5 hashing require it.

[Ed Dyreen]

I think I'll use doskey to set up a communication link to the parent

How ? Can you please write the code? And will it work parallel?

I define macro's in delayed expansion, the advantage is that macro's can be used inside other macro's. When a change needs to be made only a single macro needs to change, it affects all other macro's where it is embedded.

The disadvantage is that a nested macro takes a parameter and produces a result like a function. This is not as efficient as if the macro was not embedded at all and it's body is used directly as then you don't need to pass it any parameter.

Dbenham's macro's you see on the forum are defined with delayedExpansion disabled. The advantage is that he can post a macro and it will work because with delayedExpansion disabled it is not possible to nest macro's. The downside is that there is no longer a single point of definition. And the body needs to be updated everywhere it occurs.

I could post the code, but if I do that I would also need to post all macro's it uses, and because most of them have other macro's embedded. I would need to post my entire library which is unpractical.

Doskey based communication between a parent and it's child processes is described in detail here, and using doskey is quite straightforward. You will need to encode your variables for doskey as it does not accept all characters.
http://www.dostips.com/forum/viewtopic.php?f=3&t=1937.

Also, for WHAT you need such a parallel scripting? Only file I/O heavy tasks like AES ciphering/ MD5 hashing require it.

For heavy tasks that lend themselves for parallelism yes. For the moment I use parallelism only for caching macro's to disk. Caching a macro involves a lot of math; special characters need to be escaped, tokens need to be protected and the cached file needs to be written out twice. One version that should be loaded with delayedExpansion enabled and another for loading with delayedExpansion disabled.

When the [parent] caching function detects a cached macro is different from it's definition due to an update, it places the macroName and macroValue in doskey memory. Another parallel [child] instance checks doskey for macroNames to be recached. It will keep doing this until it encounters a quit entry.

[OperatorGK]

Doskey used in the function which caches macro to disk ! Looks like the heavy task is so heavy that it requires a couple of uneasy tasks like macro caching to enhance it's speed. WHAT IS IT?

[Ed Dyreen]

I do not understand your question. What is what ?

The only reason I paralleled the task is to allow the library to not have to wait for a function to be cached, so that I can keep debugging and developing it while the caching occurs in the background.

[OperatorGK]

I understand that you need macro caching, but for what you need macro caching? Except some mega-heavy tasks, heavier than AES, macro caching is over-engineering and actually decreases performance, cause it'll run more slowly that all other code. Maybe you are factorizing RSA2048 ?

[Ed Dyreen]

I understand that you need macro caching, but for what you need macro caching? Except some mega-heavy tasks, heavier than AES, macro caching is over-engineering and actually decreases performance, cause it'll run more slowly that all other code. Maybe you are factorizing RSA2048 ?

When a program that will use the library calls out to the library directly the library must eventually return to the callers environment otherwise all definitions will be lost.

Code: Select all

@echo off

call library

do stuff

The process of pushing variables over endlocal is slow and memory intensive. Once memory has been used by a variable it is never released, not even if the variable is cleared.

This is why when I create a program that will use my library I do not want to call out to the library directly. Instead I call the cached macro's by enumerating the directory. By doing this I reduce the time it takes to include the entire library from 2 minutes to 20 seconds and memory from +-190MB to +-5MB. Also macro's that are never used can be deleted from the program's cache.

BTW; The US intelligence agency's were involved in the development of RSA. I'd rather use AES, at stake is our freedom of choice to keep our secrets secret. The way I see it, criminals are often not the bad guys, but civilians criminalized by money hungry power seeking demons elected by an ignorant society. Resulting in a war on the established elite and their facade they call democracy.

[OperatorGK]

Library including itself by changing the original environment?
I'm developing a similar library, but I'm doing it the other way. My solution is remove the string "call library" and place the actual function library text at the end of batch file using (goto) 2>nul GetCaller trick + on-fly batch file modification. But now it don't work with each other really well . This way has it own pros and cons:
+ Fast including
+ Allows using call :function syntax instead of macros
+ Easily modifiable
- Updating it to all batch files is pain
- General call slowness

BTW. I'm from Russia, so I'll use GOST R 34.10-2012 if I'll need to cipher something really important and government-related.

[dbenham]

Dbenham's macro's you see on the forum are defined with delayedExpansion disabled. The advantage is that he can post a macro and it will work because with delayedExpansion disabled it is not possible to nest macro's. The downside is that there is no longer a single point of definition. And the body needs to be updated everywhere it occurs.

Your memory is failing Ed

From my batch macro post dated 2011-06-23

New strategy for "calling" a macro within a macro:I have abandoned the ${calledMacroName} syntax and the relatively slow drefMacro function. I am now using delayed expansion to very quickly include a macro within a macro at definition time. I now realize it is not so difficult to escape ! and ^ as needed. (Ed - Sorry I was so pig headed before. I think you have been on the right path in this regard all along). My concern about having the macro definition persist after the batch file terminates is handled by the next developement.

New macros to return any string across the ENDLOCAL border: See the macroLib_Return.bat file later in this post. These macros are useful for both macro and function development. They not only allow macros to return any value across the ENDLOCAL border, they also allow us to preserve the macro definitions themselves across the ENDLOCAL border!

If I am doing a quick project that does not need to "call" a macro from within another macro, then yes, I defined the macros while delayed expansion is disabled. But if I am developing a batch library with macros "calling" other macros, then I indeed define them while delayed expansion is enabled.


Dave Benham

[Ed Dyreen]

It works but, how am I gonna close the mutex now ?

I can't communicate through a for loop as that would wait for the new thread to finish, that will never happen obviously.
Actually at this point I lost all means to communicate to my child process at all :/

Why do you say that

Your mutex lock process does not need to be static. It can be in a loop that checks for the existence of a signal file and either exit, or sleep before it checks again.

Here is a crude demonstration of asynchronous mutex locks:

testMutex.bat

Code: Select all

@echo off
setlocal

:: initialize
set "sleep=pathping /p 100 /q 1 localhost >nul 2>nul"

:: test
echo requesting lock for %~1
call :lockMutex %~1
echo mutex lock established for %~1
for /l %%N in (1 1 100) do %sleep%
call :releaseMutex %~1
echo mutex lock released for %~1
exit /b


:lockMutex
setlocal
:: request mutex lock
start /b cmd /c "2>nul >nul (for /l %%A in () do 9>mutex.lock ((call )>mutex%~1.active&for /l %%B in () do if exist mutex%~1.active (%sleep%) else exit)&%sleep%)"
:: wait for lock
cmd /c "for /l %%A in () do @if exist mutex%~1.active (exit) else %sleep%"
exit /b 0


:releaseMutex
del mutex%~1.active
exit /b

To run the test, open two cmd.exe console windows and run the script in each, passing 1 as an argument for the first, and 2 for the 2nd.

The PATHPING delay is used to prevent the mutex from sucking up CPU cycles, but it limits the responsiveness of the mutex. If you need "instantaneous" response, then of course you can eliminate the delay, but then you will be abusing your poor CPU.

Dave Benham

funny, I just realize there is no reason at all to create the lock inside the child process.

Code: Select all

::--------------------------------------------------------------------------------------------------------------------------
( %macroStart_% $fileMutexOpen_, enableDelayedExpansion )
::--------------------------------------------------------------------------------------------------------------------------
:: last updated       : 2015^06^21
:: support            : naDelayed, delayedChars, call-/file- macro
::
set ^"$usage.%$macro%=^
%=       =% Create a mutex.%$n1c%

%=       =% use: ( %%fileMutex_%% mutex )%$n1c%
%=       =% ret: void%$n1c%
%=       =% err: 0 for succes, 1 for error"
:: (
%=       =%for %%# in ("") do set ^"%$macro%=!forQ_! (1,2) do if %%?==2 (setlocal enableDelayedExpansion%$n1c%
%=              =%!forQ_! (^^^!_:~,1^^^!) do set "m=%%~?"^^^!%$n1c%
%=              =%(!necho_! !$macro!%e7%(^^^!m^^^!%e7%))^>^&3%$n1c%
%$c1%
%=              =%!necho_!  fileMutex: '^^^!systemDrive^^^!\ED\mutex\^^^!m^^^!.active'%$n1c%
%=              =%!necho_! [ok:0]%$n1c%
%=              =%!forI_! () do ping;2^>nul^>^&2^&if not exist "^!systemDrive^!\ED\mutex\^!m^!.active" exit%$n1c%
%$c1%
%=       =%endlocal^&endlocal)else setlocal disableDelayedExpansion^&set _=%e15%!"
:: )

This means there is also no need to check wheter the lock succeeded nor is there a need to wait for the mutex to be locked.

Code: Select all

::--------------------------------------------------------------------------------------------------------------------------
:§fileMutexOpen_ ( mutex, $wait,true, setsError,true )
::--------------------------------------------------------------------------------------------------------------------------
:: last updated   : 2015^06^21
:: support        : naDelayed
:: languages      : N/A
::
rem ( %debugOn_% "!$trace!" )
::
( %debug_% "", "inside !$trace!(!%$trace%.p!)", /EQU )
::
( %forP_% '$mutex,%%a', '$wait,%%b,true', '$sE,%%c,true', '$mutexPath,"!systemDrive!\ED\mutex"', '$time,"1"' )
:: (
   ( %icho_% !$trace!^(!$mutex!;!$wait!;!$sE!^))>&3 &( %debug_% "valid()", /EQU )
   :: (
      if not defined $fileMutexOpen_ ( %fileGetMacro_% $fileMutexOpen_, $fileMutexOpen_ )
      set "$delayedMutex=!$mutex!" &>nul ( %reDelay_% $delayedMutex, $delayedMutex )
      set "$cmd=start "" /LOW /B "C:\WINDOWS\system32\cmd.exe" /V:on /E:on /T:0B /Q /C !$fileMutexOpen_! "!$delayedMutex!""
      >nul ( %reDelay_% $cmd )
   :: )
   ( %deq_% %necho_%$cmd=!$cmd!_ )

<  :§fileMutexOpen_.loop %( null )% <nul rem^ &( %debug_% "perform()", /EQU )
   :: (
      3>nul >&3 ( %dirTouch_% "!$mutexPath!\" ) &&2>nul (

         9>"!$mutexPath!\!$mutex!.lock" (

            >"!$mutexPath!\!$mutex!.active" (call;)

            for /F "delims=" %%? in (

               "!$cmd!^!"

            ) do (
               setlocal%=                                     clean environment         =%
               rem (
                  for /F "tokens=1 delims==" %%? in (

                     'set'

                  ) do if defined %%? set "?=%%?" &if /I "!?:~0,8!" NEQ "$sysVar." if /I "!?:~0,6!" NEQ "£error" if /I "!?!" NEQ "$lf" (

                     set "%%?="
                  )
                  for /F "tokens=1 delims==" %%? in (

                     'set $sysVar.'

                  ) do set "?=%%?" &set "!?:~8!=!%%?!"

                  ( %%? )
               rem )
               endlocal
            )

             ( %exception.clear% )
         )|| ( %exception.set_% 29 )%=                        acces denied              =%
      )%=                                                     insufficient privileges   =%

      if !£e! EQU 29 if !$wait! NEQ 0 (%=                     acces denied              =%
         ::
         3>nul >&3 ( %sleep_% $time, false )
         goto :§fileMutexOpen_.loop "()"
      )

      ( %ifSetsErr% )%=               insufficient privileges OR start process failed   =%
   :: )
:: )
set "$r=£e,$mutex" &set "$rA=" &( %exception.nprint% ) &( %debug_% "outside !$trace!()", /EQU, /PAUSE ) &exit /B !£e!
::--------------------------------------------------------------------------------------------------------------------------

Because the redirection is attached to the childprocess and is released only after the childprocess is terminated!

[Ed Dyreen]

Library including itself by changing the original environment?
I'm developing a similar library, but I'm doing it the other way. My solution is remove the string "call library" and place the actual function library text at the end of batch file using (goto) 2>nul GetCaller trick + on-fly batch file modification. But now it don't work with each other really well . This way has it own pros and cons:
+ Fast including
+ Allows using call :function syntax instead of macros
+ Easily modifiable
- Updating it to all batch files is pain
- General call slowness

BTW. I'm from Russia, so I'll use GOST R 34.10-2012 if I'll need to cipher something really important and government-related.

What I didn't mention is that when the macro definition is updated, the library detects the cache is different from the definition and performs a unitTest. So manual unitTesting is no longer required. The unitTest must succeed or script execution is suspended until the error is resolved.

I can see what u mean, the good news is that the (goto) trick is not required to get it to work like that. I've done similar in the past by scanning for macro's and functions and appending them.

[einstein1969]

Hi to all,

it is possible create a test for measuring the average time of waiting for the process/processes?
This is important for fast concurrent access. You can measure the latency of the process.
And should be measured in function of the number of concurrent processes.


Einstein1969

[Ed Dyreen]

Hi to all,

it is possible create a test for measuring the average time of waiting for the process/processes?
This is important for fast concurrent access. You can measure the latency of the process.
And should be measured in function of the number of concurrent processes.


Einstein1969

Hu ?, A mutex is intended to dissalow concurrent acces. I don't understand what do you mean with latency exactly ?

[einstein1969]

I mean latency to the time between when a process asks access to the shared resource and when he gets it .

[Ed Dyreen]

I mean latency to the time between when a process asks access to the shared resource and when he gets it .

Ah ok, I simply invoke a sleep function that multiplies the wait until a certain maximum in between requests.

Code: Select all

if !£e! EQU 29 if !$wait! NEQ 0 (%=                     acces denied              =%
    ::
    3>nul >&3 ( %sleep_% $time, false )
    goto :§fileMutexOpen_.loop "()"
)

I admire your willingness to do what you do in batch, but I don't feel like writing that code, I would not enjoy writing it, feels like an unnecessary overhead. If you get stuck somewhere, I may help