wait for all before continue

[zpimp]

i have a script wich downloads some pages with CURL

Code: Select all

for /l %%i in (1,1,12) do (
start /wait /min curl !_url%%i! -m 10 -o %%i.htm
)

is there a way to download at the same time, and also wait for all before continuing the script?
if i use start /wait they run one after another
if i use start it doesent wait for them to finish

[Squashman]

Untested. Just the first thing that came to my head.

Code: Select all

set cnt=0
for /l %%i in (1,1,12) do (
start /min curl !_url%%i! -m 10 -o %%i.htm && SET /a cnt+=%%I
)
:LOOP
IF %cnt% LSS 12 GOTO LOOP

[dbenham]

@squashman - That can't work. It is simply incrementing the count if the curl program was launched. It does not wait for the curl program to finish. And if it did wait, then the processes would run seraially, which defeats the whole purpose

@zpimp - Have a look at Parallel execution of shell processes on StackOverflow. It is doing more than you ask for, but you can extract ideas from the script to do exactly what you want.

Dave Benham

[Squashman]

@Dave - I suppose your right. I assumed it would not execute until the command finished completely.

How about using the TITLE as part of the start command. Call it some arbitrary title for all of them like CURLBATCH. Then use Tasklist to see if they are still running and check the errorlevel.

Code: Select all

for /l %%i in (1,1,12) do (
start "CURLBATCH %%i" /min curl !_url%%i! -m 10 -o %%i.htm
)
:LOOP
tasklist /v | find "CURLBATCH"
IF "%ERRORLEVEL%"=="0" GOTO LOOP

[Aacini]

Code: Select all

for /l %%i in (1,1,12) do (
echo X > curlRunning.%%i
start /min curl !_url%%i! -m 10 -o %%i.htm ^& del curlRunning.%%i
)
:waitForNone
if exist curlRunning.* goto waitForNone

[Squashman]

Code: Select all

for /l %%i in (1,1,12) do (
echo X > curlRunning.%%i
start /min curl !_url%%i! -m 10 -o %%i.htm ^& del curlRunning.%%i
)
:waitForNone
if exist curlRunning.* goto waitForNone

Is that not just the same kind of logic as my 1st attempt that Dave said would not work?

[dbenham]

@Squashman - No, Aacini's strategy works well. The & is escaped, so the DEL command is executed in the new process after the curl command finishes. That strategy can't work with SET because your parent process cannot see the environment changes that the child process(es) make.

@Aacini - Nice simple solution. You might want to throw a TIMEOUT or PING delay in your tight file check loop to conserve CPU resources. One risk of your approach is that your parent process will hang indefinitely if the child process is terminated before it gets a chance to delete the signal file. The link I provided shows a more robust solution. But the robust solution is much more complicated

Here is another relatively simple approach. WMIC can be used to launch a process and return the PID of the new process. Then TASKLIST can be monitored to see if any of the launched processes are still running.

Code: Select all

for /l %%i in (1 1 12) do (
  for /f "tokens=2 delims==; " %%A in (
    'wmic process call create "c:\curlPath\curl.exe !url%%i! -m 10 -o c:\outpuPath\%%i.htm"^|findstr ProcessId'
  ) do set "$checkPID%%A=1"
)

:wait
timeout 1 /nobreak >nul
for /f "tokens=2" %%A in (
  'tasklist /fi "imagename eq curl.exe" /nh'
) do if defined $checkPID%%A goto :wait

endlocal
echo Done!

Dave Benham

[zpimp]

i would normally use tasklist, but i have encountered problems with it
@Aacini i did something similar but your code is better

i found a program that can do what i want (written in pure c, no dependencies, 9kb, source code included)
not made by me
http://rghost.net/47175763

so now i only have to do this in my script "wfc /ee=curl.exe exit"

[Squashman]

Code: Select all

for /l %%i in (1,1,12) do (
echo X > curlRunning.%%i
start /min curl !_url%%i! -m 10 -o %%i.htm ^& del curlRunning.%%i
)
:waitForNone
if exist curlRunning.* goto waitForNone

Linked your post to this question on SO.