Howto avoid parallel I/O conflicts, setSerial ?

[Ed Dyreen]

'
This is a more general question, not really related to any programming language.
I ask it here because I like it here, out of the box thinking is required

I often find myself running multiple different programs or multiple instances of the same program.
Often, they access the same registry keys/files remember them or not, restore them later or not...

What's important is that they don't collide.
( Let's leave out the possibility of 3th party software I didn't wrote ).

One way is by using some kind of parent/child hierarchy.
Imagine I want to write to a file but I need to make sure no other program writes to it.
For files I use a trick where I let the parent write an initial key to the registry, let's say fileOpen.
The children will try to delete this key, however only one of them can succeed at any given time and...

The second way is more complex but doesn't requires the parent/child hierarchy.
Now I simply write a key to registry it's name is PID, it's initial value is waiting.
Then wait for the CPU to drop and then enumerate all the PID's of all other processes found by a regread to the same path.
It will find out which process has the lowest PID and if that is this process change all values to locking.
The other instances or this instance will change this value again to detected.
When the program with the lowest PID finds everyone has changed to detected it changes them to freezed.
Finally the instance that initiated the locking can run and...

Because the 1st way isn't perfect because it needs a parent and the 2nd way is so complex.
I was wondering if there was an easier way to serialize multiple processes.
So I was hoping that opening a file could only be done once, but then I discovered that two unrelated programs could open a file for writing simultaneously and still collide ( I used autoIT but assume the same is true in any language ).

So far I only know two things that work, deleting a file or deleting a regkey because this can only be done once obviously.
Unfortunately they both have the same problem which is you can't delete a file or key that doesn't exist already.
Maybe I delete a system file or key, but that is a dangerous practice.
I was thinking of a default file in the driver backup cache or something, but the OS should always restore it by itself after a reboot, I mean what if my program crashes, then the whole mechanism would be broken !

Am I really asking too much, can't believe I'm the only one strugling with these kind of issues.
How do the experts do it then, there must be a simpler way to serialize a process.
like set /a $err = rundll.exe, serialize/lock filename

I hope this long post is going to help, and we can exchange knowledge.

Code: Select all

:setSerial ( $var = 'default' )
:: (
   %write% $var %= unlock =%
   %= ...we could examine the CPU to see if we need to wait longer or some other variant :? =%
   set $time = 1
   %while% not delete $var
      set $time = %sleep% ( $time ) %= $time multiplies =%
   %wend%
   %= ...We are serial/locked yippie :P =%
:: )
exit /b $err

Impatiently waiting for your input,
Ed

[dbenham]

I've already written about using redirection to implement a locking mechanism that can be used to serialize parallel batch processes.

Re: parallel process with batch
How do you have shared log files under Windows? (this in turn has links to additional posts)

All the solutions rely on the fact that only one process can have a file open for write access at a time. It takes very little code to implement a general purpose lock, you just have to make sure all processes use the lock appropriately. If you have multiple locks, then you must ensure that all processes establish locks in a consistent order, other-wise you run the risk of a deadlock.

Here is a the bare bones of a general lock. The routine endlessly loops until it is able to establish the lock. Once established, it calls the task and then releases the lock once the task completes.

Code: Select all

:getLock lockName task [args...]
:: lockName = path of a temporary file that is used as a lock.
:: task = command / :label / program to call once the lock is established
:: [args...] = optional args (up to 8) for the task
2>nul (
  >%1 (
    shift /1
    call %1 %2 %3 %4 %5 %6 %7 %8 %9
  ) || goto :append
)

Under most circumstances you want to minimize the time that any one process holds the lock. So the task usually has a small scope.

The above does NOT implement a FIFO queue for the lock - in fact it is chaos. It is random as to which process succeeds in getting the lock if multiple processes are waiting. If really needed I'm sure you could extend the code to support a FIFO queue.

Read all the links at the top, there are detailed explanations of how it works, as well as additional ideas for improvements.

Dave Benham

[Ed Dyreen]

'
Weird , I remember situations where I appended data to the same file from two different processes and the line was written only half ?!, Maybe I imagined it, it has been a while

It seems to work, and I finally found a case where a user defined stream could be handy + Jeb's command labeling

Code: Select all

@echo off &setlocal enableDelayedExpansion

<:return_ <nul echo. &call :setSerial_ fileLock return_ &goto :proceed_
echo.setSerial ok
pause
exit /b

:proceed_
echo.setParallel ok
pause
exit

:setSerial_ ( fileLock, return )
:: (
   2>nul (
      3>"%~1" (
         call :%~2
         exit /b
      ) || (
         >nul 2>&1 ping 0.0.0.0
         goto :setSerial_ "()"
      )
   )
:: )

I'm impressed

Now the only thing left for me to find out is how I can simulate the same behavior in other languages.
I tried using fileOpen in write mode, but I get two different handles ? ( I only tested in autoIT ).
Is that normal ? would that also happen in C or java ?
I was hoping for a more cross language/general solution, but I guess maybe that's just too much to ask


Thanx, Dave !

[dbenham]

'
Weird , I remember situations where I appended data to the same file from two different processes and the line was written only half ?!, Maybe I imagined it, it has been a while

You probably were working in in OS other than Windows, perhaps Unix?

Unix doesn't prevent multiple processes from writing to the same file at the same time (at least not by default). But Windows normally does. There are low level routines available on Windows that allow multiple processes on Windows to have the same file open for write access. But the default is to only allow one process at a time. (not a very technical explanation, but it has been a long time since I used a compiler)

Now the only thing left for me to find out is how I can simulate the same behavior in other languages.
I tried using fileOpen in write mode, but I get two different handles ? ( I only tested in autoIT ).
Is that normal ? would that also happen in C or java ?
I was hoping for a more cross language/general solution, but I guess maybe that's just too much to ask

I'm assuming you are looking for a least common denominator solution, something that can be used by "any" language. Many language libraries provide locking and semaphore routines. But they would be difficult to access from something like a batch file. So I think the file based solution may be your best bet. I believe it should work regardless of the language (on Windows), since pretty much any language can open a file and they all rely on the same OS routines. As long as the file isn't opened for write access in share mode, then the file should be locked.

Dave Benham

[Ed Dyreen]

'
You are right ben, I just discover it too, autoIT opens files in share mode by default
But got it working finally, I learn a lot today

Code: Select all

DllCall('kernel32.dll', 'ptr', 'CreateFileW', 'wstr', $sFile, 'dword', $iAccess, 'dword', $iShare, 'ptr', DllStructGetPtr($tSecurity), 'dword', $iCreation, 'dword', $iFlagsAndAttributes, 'ptr', $hTemplate)

http://www.autoitscript.com/forum/topic/138909-howto-avoid-parallel-io-conflicts-setserial/page__view__findpost__p__973690