parallel process with batch

[colargol]

Hello guys and happy new year

I need help again with parallel processing, I hope you'll understand my english

I use batch scripts for video and audio encoding.
My small application generates all batch commands for my encoding jobs.

It can also work with a queue list which is stored in a text file like this:
Each line is the path of a .bat that contains script for an encoding job.

D:\HD\Tests HD\queue tests\Pub 01.avs
D:\HD\Tests HD\queue tests\Pub 02.avs
D:\HD\Tests HD\queue tests\6canaux.ts.avs

(for each line, there is a corresponding batch file)

Code: Select all

for /f "usebackq tokens=* delims=" %%a in (`type "%queue_list%"`) do (
   set "bat_file=%%~dpna.bat"
   
   setlocal EnableDelayedExpansion
   call "%%bat_file%%"
   
   :: some code...
   
   endlocal
)

>"%queue_list%" type nul

Now, I'd like to run several jobs simultaneously (2 is enough for now) because sometimes, encoding jobs uses less than half cpu.
So I had a menu item to my app: "run a new worker" which calls the same batch script.

The 2 process have to share a counter %bat_cnt% which represent the last line number that is being processed.
Each time a job is finished, the process calls the counter and increments its value.

I only know 2 solutions for this: using environnement variables or using a temporary counter.bat
I choose the second one because I think it is faster:

Code: Select all

:: read counter
call "%queue_cnt_path%"

:: update counter
>"%queue_cnt_path%" echo set bat_cnt=!num_bat!

Now the problem:
I don't know how many chance I have it happens, but 2 process can read the value of the counter exactly at the same time (especially with fast tasks) and I don't know how to avoid this.
If this happens, the 2 process will do the same encoding job at the same time

I know batch is not the best for multitask but if you have ideas that could help, please share.
Thanks you very much!

[dbenham]

You are correct, you have a problem if both processes read your counter at the same time because any number of processes can read the same file simultaneously.

But only one process can have the file open for write access at a time. You can use this to make sure only one process accesses the counter at a time.

You should be able to start as many processes as you want to execute this batch script:

Code: Select all

@echo off
cls
setlocal disableDelayedExpansion

set queue="%~1"
set lock="%~dpn1.lock"
set counter="%~dpn1.counter.bat"
set "cnt="
set "skip="

:loop
call :NextFile
if not defined bat_file exit /b
call "%bat_file%"
:: some code...
goto loop


:NextFile
2>nul (
  >%lock% call :NextFile2 || goto :NextFile
)
exit /b


:NextFile2
set "file="
set "bat_file="
if exist %counter% call %counter%
if defined cnt set "skip=skip=%cnt%"
set /a cnt+=1
>%counter% echo set cnt=%cnt%
for /f "usebackq %skip% eol=: delims=" %%F in (%queue%) do (
  set file="%%~fF"
  set bat_file="%%~dpnF.bat"
  goto :break
)
:break
if not defined bat_file (
  >%queue% type nul
  del %counter%
)
exit /b 0

You should even be able to have one or more processes appending work to the queue while others are processing the queue as long as the appenders also use the lock:

Code: Select all

:append
2>nul (
  >%lock% (
    >>%queue% echo "%~1"
  ) || goto :append
)

Dave Benham

[colargol]

Fantastic !

I'll try tomorrow

Thank you Dave

[!k]

Code: Select all

@echo off &setlocal enableextensions

set /a num = 2

for /f "delims=" %%a in ('type "list.txt"') do (
   call :loop
   start /min cmd.exe /c my_batch.cmd %%a
)
goto :eof

:loop
for /f %%p in (
'"tasklist /nh /fi "windowtitle eq My batch" 2>nul |find /c "cmd.exe""'
) do set /a proc = %%p
if .%proc% geq .%num% (ping -n 5 localhost >nul &goto :loop)
goto :eof

my_batch.cmd sample

Code: Select all

@echo off &title My batch

echo Parameters = %*
echo Processing... ;)
ping -n 120 localhost >nul

[colargol]

!k, thanks for help
My first attempts were quite similar, unfortunately I can't do like this because many encoders CLI change windows title.
For exemple, x264.exe use the title to display work progress and other infos: x264 [40%]...
2 other reasons: it requires an additional batch windows and a encoding process could by executed separately which can create confusions with queue processes.

[colargol]

Dave, the file.lock is a very clever solution.
Just made some tests and it works perfectly
Nothing to say! Thx for your help.

And you are right, files can even be added to the queue during queue processing. But I'm not sure it is necessary to use the lock for this.

[dbenham]

Glad it worked

You need the lock to append because the append might happen while a processor is determining that the queue is empty. The appender could append the file and the processor could then wipe out the queue before it is processed.

Also, the lock enables multiple processes to safely append to the queue.

Dave Benham

[colargol]

I just had a power outage and I lost my queue.bat
All chars has been replaced by NUL. I didn't made a copy since 3 days.

Is my file really dead?

edit: I rewrite the file - I'm not lucky these days
Thank again to Dave.