non-buffering FOR /F alternative

[aGerman]

I developed a little C tool to observe a folder for changes. It is permanently running and writes a line for every event. (Maybe I publish this utility later on if someone finds it useful).
If I try to process the lines in a FOR /F loop it won't work because FOR /F buffers the outputted lines until the utility terminates. Of course the tool is not intended to terminate because it shall continue waiting for the next event. I already figured out how to work around this limitation but still I'm not happy with it.
That's what I have so far:

Code: Select all

@echo off &setlocal
for /f "delims=" %%i in ('cmd /c "echo hello&echo world&set /p =stop"') do echo %%i
pause

cls
cmd /c "echo hello&echo world&set /p =stop" | (cmd /von /q /d /c ^"for /l %%# in (^) do set "rec="^&set /p "rec="^&(if not defined rec exit^)^&^
  echo !rec!^
")
pause

cls
cmd /c "echo hello&>nul timeout 1&echo world&>nul timeout 1&set /p =stop" | (cmd /von /q /d /c ^"for /l %%# in (^) do set "rec="^&set /p "rec="^&(if not defined rec exit^)^&^
  echo !rec!^
")
pause

The code consists of three parts which all process the same output of the three lines

hello
world
stop

Because of the SET /P statement that displays "stop", the cmd process doesn't terminate unless you press Enter.

The first part displays nothing until you press Enter. Then it outputs all three lines at once. This is the unwanted behavior I try to work around.

The second part solves the problem partially. I pipe the output to another cmd process containing an infinite loop, and within the body of this loop I read the lines using SET /P. This technique is well-known for reading files. Fun fact: It displays "hello" and then waits for the Enter input. "world" and "stop" are never displayed. Something ate them

In the third part I inserted a delay of one second after each line. This works. And my tests using the C tool indicate that a few milliseconds would be sufficient already. But this is ugly.


Does anyone out there know of a better workaround?

Steffen

[dbenham]

SET /P does not play nicely with pipes. However it does work as you want with files.

So simply redirect your utility output to a temp file, and read the temp file with SET /P in an infinite loop.

SET /P will return an empty string if there is no data waiting, so you need to ignore empty lines. This also means you will need some other stop signal. I introduced a TIMEOUT to test the empty line handling.

You can still use a pipe to get two asynchronous processes, but the pipe is no longer conducting any data.

Code: Select all

@echo off
copy nul temp.txt >nul
cmd /c "(echo hello&timeout 5 >nul&echo world&set /p =stop<nul)>>temp.txt" | cmd /von /q /d /c "(for /l %%# in () do set "rec="&set /p "rec="&(if defined rec echo(!rec!)&if !rec!==stop exit)<temp.txt"
del temp.txt

Dave Benham

[aGerman]

This works great. Thanks Dave!

The problem is that the temporary file can become really huge. The C tool I'm talking about is based on a little program that I wrote which is already in use. It may run unattended over weeks or months until someone restarts the computer. It copies local files to a server which a software of a measurement device writes, in order to enable monitoring the measuring results on the shop floor simultaneously. WSH scripts using WMI don't work because you can't observe a subfolder tree using WMI. However, for this particular purpose I wrote the program in a manner which internally processes the events and copies the files without any script involved. Now I thought about writing a tool that could be used in a more generic way.

Steffen

[aGerman]

Finally I used a 10 ms delay in my tool. This is about three times as long as needed on the machines I tested it, and seems to be sufficient to make SET /P waiting again even if the processing of the output takes much more time than 10 ms.
In case you are interested:
https://sourceforge.net/projects/direvents/

Steffen

[Squashman]

So is reading the file similar to what you were doing here?

Essentially tailing a file?

Code: Select all

@echo off
call :Loop <"tailme.txt"

exit

:Loop
set "line="
set /p "line="
if defined line (
  echo %line%
) else (
  pathping -q 1 -p 300 localhost >nul
)
goto :loop

[aGerman]

That's pretty much the same. Unfortunately this leaves the polling for new data in the script code and still buffers in a growing file because already read data is not removed. That's the reason why I hoped to find a way to make SET /P read from the pipe in a proper way.

Steffen

[jfl]

Hi Steffen

Your DirEvents.exe tool is impressively small, and it works nicely, except for one detail:
How do you stop it?
I tried Enter, ESC, Ctrl-C, and many other key combinations, without success.
And I could not find any clue in the source.
Eventually I had to kill it in Task Manager, which is not very convenient!

Jean-François

[aGerman]

I think I know the reason but didn't try yet.
I registered the BeforeClosing routine in order to release memory and handles even if you close the window.

Code: Select all

  SetConsoleCtrlHandler(BeforeClosing, TRUE);

I call it actively if some errors occur, elsewise it is a callback function which was sub-classed from the console host callback.
See https://docs.microsoft.com/en-us/window ... lerroutine
It is called for several events including Ctrl+C. In that case I guess it should return FALSE rather than TRUE. I didn't care but I'll consider to change it in an upcoming version.

Steffen

[einstein1969]

steffen, if your problem is reading the pipe directly you have to pack the output of your program up to 1023 chars and you don't need the delay and you save cpu.

[aGerman]

I don't think this is applicable in my specific case. The tool I've been talking about is event-driven. The lines it writes contain the relative path of the object in the file system that triggered the event. So, each line it outputs has a another length and it's quite unlikely they are even close to 1023 characters.

Thanks anyway
Steffen