Directly reading from pipe by the parent CMD process

[sst]

I've found a way to directly reading from pipe by the parent CMD process that initiated the pipe connection between two child processes.
So it is possible to directly store a command/program output in a variable without using FOR /F
It has it's own limitations and may not be suitable for every day use, but who knows, somebody may find it useful for some real world scenario.
It is written with the assumption that handles &3 and &4 are free an not in use by prior redirection of standard handles.

I'm not sure if this has been discussed before, please don't blame me if that is the case

Code: Select all

:: Directly reading from pipe by the parent CMD process

@echo off
setlocal EnableDelayedExpansion
set "LF="

:: Permanently redirect stdin to pipe input
cmd /u /c "echo(&echo((" 0>&3 4>&0 | break

:: Now reading from pipe
pause>nul
pause>nul
set /p "LF="

echo LF test:
echo First Line!LF!Second Line


:: Restore stdin
break 0>&4 3>&0
pause
echo findstr test:

:: The file handles &3 and &4 are now pointing to original stdin &0
:: If we want to use this method one more time, we have to use handles &5 and &6
:: So eventually we will run out of handles and this method can not be used more than 3 times.

cmd /c type "%~f0" 0>&5 6>&0 | break
findstr "^::"
break 0>&6 5>&0
pause

[dbenham]

Fascinating

I need to think more about this when I get free time. I'd like to trace the technique to better understand the underlying mechanics, and I'm curious if this might be of some practical use.

The first thing that comes to mind is it functions somewhat like a temporary file, without actually creating a file.

But there is a significant limitation that you may not be aware of - the script will hang indefinitely if it tries to write > ~1000 bytes before reading from the pipe. The pipe buffer has a limited size, and if the buffer gets full, then the left side will wait until the right side frees buffer space by reading. But your ingenious script has the parent process functioning as the reader, and it cannot begin reading until the pipe has completed, so the script can easily hang.

I suggest modifying your script to launch itself in a new cmd process and then EXIT when done so that the file handles are restored to a healthy state when the script terminates. In this way the script can be run multiple times within the same session. Your original script hangs if it is run a second time in the same session.

Code: Select all

@echo off
echo %0 | findstr :start: >nul && goto :start
cmd /d /c ^""%~dp0.\:start:\..\%~nx0" %*"
exit /b

:start
setlocal EnableDelayedExpansion
set "LF="

:: Permanently redirect stdin to pipe input
cmd /u /c "echo(&echo((" 0>&3 4>&0 | break

:: Now reading from pipe
pause>nul
pause>nul
set /p "LF="

echo LF test:
echo First Line!LF!Second Line
echo --
findstr /n "^"
echo =====


:: Restore stdin
break 0>&4 3>&0
pause
echo findstr test:

:: The file handles &3 and &4 are now pointing to original stdin &0
:: If we want to use this method one more time, we have to use handles &5 and &6
:: So eventually we will run out of handles and this method can not be used more than 3 times.

cmd /c type "%~f0" 0>&5 6>&0 | break
findstr "^::"
break 0>&6 5>&0
pause

exit

Dave Benham

[dbenham]

I had gotten the ~1000 byte buffer limit from jeb at the bottom of this StackOverflow answer about pipe behavior. But I am seeing something different.

I did some testing on my Windows 10 machine, and I see a pipe buffer of exactly 4096 bytes. I used the following script to probe the buffer size. Each run tests one string length (up to 7168 bytes). Simply pass the desired test length as the one and only parameter.

Code: Select all

@echo off
echo %0 | findstr :start: >nul && goto :start
cmd /d /c ^""%~dp0.\:start:\..\%~nx0" %*"
exit /b

:start

:: Build string with length 7168
setlocal EnableDelayedExpansion
set "str=."
for /l %%N in (1 1 10) do set "str=!str!!str!"
set "str=!str!!str!!str!!str!!str!!str!!str!"

:: Permanently redirect stdin to pipe input and write %1 number of characters to pipe
cmd /v:on /c "<nul set /p "=!str:~0,%1!"" 0>&3 4>&0 | break

:: Now reading from pipe
findstr "^" >out.txt
for %%F in (out.txt) do echo %%~zF

exit

The script prints out the number of bytes read from the pipe, plus 2. The extra 2 bytes are an artifact of FINDSTR adding <CR><LF> when the piped input does not end with <LF>, as described at What are the undocumented features and limitations of the Windows FINDSTR command?

If the script does not hang, then the specified string must fit within the pipe buffer. The script hangs when the buffer size is exceeded.

On my Windows 10 machine the script succeeds with length 4096, but fails at 4097.


Dave Benham

[jeb]

Hi Dave,

I tested your code
Windows 7 x64: 4096
Windows XP x32: 4096, but FINDSTR doesn't work reliable

And I retested the blocking limit of pipe blocks.
First a testString of the configured size is send.
The a "second line" is send.
If the blocking is active the second line is send only when thread2 read the first line

PipeBlocking.bat 4095 --- Works without blocking
PipeBlocking.bat 4096 --- Blocks

Code: Select all

@echo off

Setlocal EnableDelayedExpansion
set "S=."
FOR /L %%n in ( 1 1 13) DO set "S=!S:~-4000!!S:~-4000!"

set "xtime=%%t^ime%%"
set "testLen=%~1"

set "testStr=!S:~0,%testLen%!"
set "testStr=!testStr:~0,-2!" # Remove 2 chars, to account for CR/LF of the ECHO command
(
	(call echo %%xtime%% ### Thread1 Test with a length of %testLen%) > con
	echo %testStr%
	call echo %%xtime%% ### Thread1 Send second line > con
	echo Send second line
    echo ### Thread1 end > con
) | (
	call echo %%xtime%% ### Thread2 ..... Delay of 5 seconds
	ping -n 5 localhost > nul
	call echo %%xtime%% ### Thread2 Read Data
	set /p x=
	call echo %%xtime%% ### Thread2 End
)

[dbenham]

Cool

So you mostly confirm my finding, but there is one thing that surprises me.

My script shows that the buffer is 4096 bytes long. But your script shows that if the writer performs an operation that exactly fills the buffer, then subsequent writer commands are blocked until the reader reads the buffer.

I expected that the writer would not be blocked until the buffer was exceeded (attempt to write byte 4097).


Dave Benham

[sst]

PipeBlocking.bat 4095 --- Works without blocking
PipeBlocking.bat 4096 --- Blocks

Hi Jeb,
Except one thing: you forgot that an extra space follows by echo %%testStr%% which will be inserted by parser. You have to remove one more char from the testStr set "testStr=!testStr:~0,-3!", PipeBlocking.bat 4096 is actually blocks at 4097

I can also confirm that pipe block buffer is 4096 bytes long on Windows XP and Windows 7. The buffer size is hard coded at OS kernel, which have not changed at least from XP.

Hi Dave,
in your script at this line

Code: Select all

cmd /v:on /c "<nul set /p "=!str:~0,%1!"" 0>&3 4>&0 | break

There is no need to use /v:on because !str:~0,%1! will be expanded in batch context. Delayed expansion will not be affected by pipe.

There is a difference between

Code: Select all

echo !str! | break

and

Code: Select all

(cmd /c "echo !str!) | break

and

Code: Select all

cmd /c "echo !str!" | break

In the first two cases delayed expansion will not work but in the third case it will work
in the first two cases, each side of the pipe will be wrapped in another cmd instance (cmd /d /s /c ....) which will happen before execution phase so delayed expansion will not work whereas in the third case the left side will executed directly without wrapping the left side in another cmd, so delayed expansion will work.

Actually this is the key point that even this trick of using pipe by pipe maker will work.
It is even possible for parent CMD to act as the pipe writer or even act as the pipe writer and reader at the same time.

I will provide examples of these additional tricks in a future post. If time permits I will write an explanation for internals and mechanics of this tricks and more details about how cmd actually builds and handles the pipe creation.

[dbenham]

Hi Dave,
in your script at this line

Code: Select all

cmd /v:on /c "<nul set /p "=!str:~0,%1!"" 0>&3 4>&0 | break

There is no need to use /v:on because !str:~0,%1! will be expanded in batch context. Delayed expansion will not be affected by pipe.

You are correct, but only because I had accidentally dropped parentheses. It wasn't necessary, but I wanted to delay the expansion until within the child CMD process. I intended to write the following:

Code: Select all

(cmd /v:on /c "<nul set /p "=!str:~0,%1!"" 0>&3 4>&0) | break

But the the above won't work. The parentheses break your technique for some reason.

There is a difference between

Code: Select all

echo !str! | break

and

Code: Select all

(cmd /c "echo !str!) | break

and

Code: Select all

cmd /c "echo !str!" | break

In the first two cases delayed expansion will not work but in the third case it will work
in the first two cases, each side of the pipe will be wrapped in another cmd instance (cmd /d /s /c ....) which will happen before execution phase so delayed expansion will not work whereas in the third case the left side will executed directly without wrapping the left side in another cmd, so delayed expansion will work.

I know there are differences, but you got the outcomes incorrect. Assuming the parent script has delayed expansion enabled, then the 1st and 3rd expand in the parent, and the parentheses in the 2nd prevent the delayed expansion. The delayed expansion fails entirely in the 2nd because the child process does not inherit the delayed expansion state.

I posted a StackOverflow question about the differences at Why does delayed expansion fail when inside a piped block of code?, and jeb did a great job explaining the situation in his answer (I already included that link in my first post).

Below is a demonstration showing that 1 and 3 expand, but 2 does not:

Code: Select all

@echo off
setlocal enableDelayedExpansion
set "str=Expanded Successfully"

echo Test 1: !str! | findstr "^"

(cmd /c "echo Test 2: !str!") | findstr "^"

cmd /c "echo Test 3: !str!" | findstr "^"

-- OUTPUT --

Code: Select all

Test 1: Expanded Successfully
Test 2: !str!
Test 3: Expanded Successfully

Actually this is the key point that even this trick of using pipe by pipe maker will work.
It is even possible for parent CMD to act as the pipe writer or even act as the pipe writer and reader at the same time.

I will provide examples of these additional tricks in a future post. If time permits I will write an explanation for internals and mechanics of this tricks and more details about how cmd actually builds and handles the pipe creation.

I would very much like to see that. I am aware of how the Windows implementation of redirection can lead to permanent redirection. But your addition of pipes to the mix adds a new wrinkle that I do not yet understand, and I am very intrigued.


Dave Benham

[sst]

I know there are differences, but you got the outcomes incorrect.

Yes you are correct that was my mistake. I know echo !str! | break works (assuming delayed expansion was enabled before of course). I was intended to use (echo !str! )| break, and my description was not precise for why it is not working when I said: because it will be wrapped by (cmd /d /s /c...). As echo !str! | break will wrapped by another cmd too. I was focused more on the difference between (cmd /c command n>&m ) | break and cmd /c command n>&m | break but at the same time I was talking about delayed expansion so I doubled my mistake.

I'm aware that you know how delayed expansion or handle redirection works because I just learned them(along with other aspects CMD/Batch like parser, ...) from you and jeb. I brought delayed expansion to your attention as an attempt to pinpoint the difference between those two types of redirection in pipes but I failed.

... I intended to write the following:

Code: Select all

(cmd /v:on /c "<nul set /p "=!str:~0,%1!"" 0>&3 4>&0) | break

But the the above won't work. The parentheses break your technique for some reason.

Now this is what I was intended to talk about. It won't work because be entire left hand command will be wrapped in another layer of cmd

Code: Select all

cmd /d /s /c "cmd /v:on /c "<nul set /p "=!str:~0,%1!"" 0>&3 4>&0"

So the redirection will take place in the child CMD not in the main CMD.
The following illustrates the difference

Code: Select all

MAIN_CMD
|    executing: (cmd /v:on /c "<nul set /p "=!str:~0,%1!"" 0>&3 4>&0) | break
|
|--Child1---cmd /d /s /c "cmd /v:on /c "<nul set /p "=!str:~0,%1!"" 0>&3 4>&0"
|           |    executing: cmd /v:on /c "<nul set /p "=!str:~0,%1!"" 0>&3 4>&0
|           |    0>&3 4>&0 -------> The redirection occurs here
|           |
|           |--Child3---cmd /v:on /c "<nul set /p "=!str:~0,%1!
|
|--Child2---cmd /d /s /c "break"


MAIN_CMD
|    executing: cmd /v:on /c "<nul set /p "=!str:~0,%1!"" 0>&3 4>&0 | break
|    0>&3 4>&0 -------> The redirection occurs here
|
|--Child1---cmd /v:on /c "<nul set /p "=expanded""
|
|
|--Child2---cmd /d /s /c "break"

This explains why wrapping the pipe operand in parenthesis or directly using internal commands, breaks the technique.
This technique only works with explicit external processes.

How It works:
The above information is the key to understand how the technique works.
Before that I would like to briefly describe how cmd redirects standard handles of its child processes.
Generally any process can redirect the handles of its child processes in two ways:

1. Implicitly by inheritance, The process redirects its own standard handle(s) then creates the child process. The child process automatically inherits the parent process standard handles(stdin , stdout, stderr). The handles must be flagged as inheritable and the parent process must allow handle inheritance when creating the child process.

2. Explicitly by specifying the standard handle(s) in the STARTUPINFO structure when calling the CreateProcess API function.

CMD uses the first method.
So how CMD creates pipes and how it redirects the child processes input/output?
Assuming a typical case without any explicit redirections:

Code: Select all

LeftProcess | RightProcess

The ordered steps that CMD will perform to create the pipe connection between the two processes are as follows: (The values for internal handles 3-9 are based on the assumption that they are all undefined and clean)

1. It creates an anonymous pipe by calling CreatePipe function. It receives two handles, one handle to the write side of the pipe which I will refer to it as PipeOut and another handle to the read side of the pipe which I will refer to it as PipeIn

2. It then saves PipeIn to handle slot &3 and PipeOut to handle slot &4. At this time the state of the CMD internal handles are as follows:

Code: Select all

&0 : Current stdin
&1 : Current stdout
&2 : Current stderr
&3 : PipeIn
&4 : PipeOut
&5-&9 : UNDEFINED

3. It then redirects its own stdout to the handle which is pointed by &4. In this process the original value of &1 will saved to &5 and the value of &4 will be cleared. At this time the state of the CMD internal handles are as follows:

Code: Select all

&0 : Unaltered
&1 : PipeOut
&2 : Unaltered
&3 : PipeIn
&4 : UNDEFINED
&5 : Original value of &1
&6-&9 : UNDEFINED

4. It creates the left hand process in suspended state. The left hand process have already inherited the standard handles from CMD. This is the state of the standard handles of the left hand process:

Code: Select all

stdin  : CMD's current stdin
stdout : CMD's current stdout (PipeOut)
stderr : CMD's current stderr

5. CMD restores its internal handles to their original state. and updates its own process OS level standard handles accordingly. In this process it closes the handle to PipeOut by calling CloseHandle API

Code: Select all

&0 : Unaltered
&1 : Restored original value from &5
&2 : Unaltered
&3 : PipeIn
&4-&9 : UNDEFINED

6. This time It redirects its own stdin to the handle which is pointed by &3. In this process the original value of &0 will saved to &4 and the value of &3 will be cleared. At this time the state of the CMD internal handles are as follows:

Code: Select all

&0 : PipeIn
&1 : Unaltered
&2 : Unaltered
&3 : UNDEFINED
&4 : Original value of &0
&5-&9 : UNDEFINED

7. It then creates the right hand process in suspended state. The right hand process have already inherited the standard handles from CMD. This is the state of the standard handles of the right hand process:

Code: Select all

stdin  : CMD's current stdin (PipeIn)
stdout : CMD's current stdout
stderr : CMD's current stderr

8. Then restores its internal handles to their original state. and updates its own process OS level standard handles accordingly. In this process it closes the handle to PipeOut by calling CloseHandle API

Code: Select all

&0 : Restored original value from &4
&1 : Unaltered
&2 : Unaltered
&3-&9 : UNDEFINED

9. It resumes both processes to start execution
10. It waits for both processes to terminate then continues its own execution flow.


---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Now It should be clear how the technique works. But for completeness I will describe the details of the technique used in the first post.

Code: Select all

cmd /d /c "echo test" 0>&3 4>&0 | break
findstr "^"

1. CMD obtains PipeIn and PipeOut and sets its own internal handles as below:

Code: Select all

&0 : Unaltered
&1 : PipeOut
&2 : Unaltered
&3 : PipeIn
&4 : UNDEFINED
&5 : Original value of &1
&6-&9 : UNDEFINED

2. Then comes the redirection: 0>&3 4>&0.
2-1. The first one is 0>&3 It firsts saves the current value of &0 in the first UNDEFINED slot which is &4 then sets the value of &0 to what is pointed by &3 which is PipeIn. But one additional important thing will be done: The handle to the PipeIn will be duplicated by calling DuplicateHandle API. At this point the state of the internal CMD handles are as follows

Code: Select all

&0 : PipeIn (Duplicated_1)
&1 : PipeOut
&2 : Unaltered
&3 : PipeIn (Original)
&4 : Original value of &0
&5 : Original value of &1
&6-&9 : UNDEFINED

2-2. The next one is 4>&0 It firsts saves the current value of &4 in the first UNDEFINED slot which is now &6 then then sets the value of &4 to what is pointed by &0 which is Duplicated PipeIn. The handle to the Duplicated PipeIn again will be duplicated. At this point the state of the internal CMD handles are as follows

Code: Select all

&0 : PipeIn (Duplicated_1)
&1 : PipeOut
&2 : Unaltered
&3 : PipeIn (Original)
&4 : PipeIn (Duplicate_2)
&5 : Original value of &1
&6 : Original value of &0
&7-&9 : UNDEFINED

3. It creates the left hand process (cmd /d /c "echo test") in suspended state. The state of the left hand child standard handles are:

Code: Select all

&0 : CMD's current stdin (PipeIn_Duplicated_1)  |  This is actually a self pipe. The child process can potentially write to pipe and read it back
&1 : CMD's current stdeout (PipeOut)            |  But this will happen in the child process. It can not only do this by pause(eat) or set /p but not with findstr
&2 : CMD's current stderr                       | It cause a dead lock because findstr will not terminate until it reads EOF and but it is also the writer of pipe.

4. CMD attempts to restores its own handles: Closes PipeOut handle and restores original value of &1 from &5 and clears &5. Closes the handle that is pointed by &0 which is PipeIn (Duplicated_1). Copies the value of handle &4 which is PipeIn (Duplicated_2) to &0. Then copies the value of handle &6 to &4 and clears the value of handle &6 because it was UNDEFINED before. The final state of CMD internal handles will be

Code: Select all

&0 : PipeIn (Duplicated_2)
&1 : Restored original value from &5
&2 : Unaltered
&3 : PipeIn (Original)
&4 : Original value of &0
&5-&9 : UNDEFINED

5. This time It redirects its own stdin to the handle which is pointed by &3. In this process the original value of &0 will saved to &5 and the value of &3 will be cleared. At this time the state of the CMD internal handles are as follows:

Code: Select all

&0 : PipeIn (Original)
&1 : Unaltered
&2 : Unaltered
&3 : UNDEFINED
&4 : Original value of &0
&5 : PipeIn (Duplicated_2)
&6-&9 : UNDEFINED

6. Creates the right hand process (cmd /d /s /c "break") in suspended state.

7. Closes the handle to PipeIn (Original) and restores value of &0 from &5 which is PipeIn (Duplicated_2) and clears value of &5

Code: Select all

&0 : Restored from &5 (PipeIn_Duplicated_2)
&1 : Unaltered
&2 : Unaltered
&3 : UNDEFINED
&4 : Original value of &0
&5-&9 : UNDEFINED

So the above will be state of the CMD after pipe processes have finished. It still have access to pipe input (PipeIn_Duplicated_2) and the original stdin is saved in handle &4

[sst]

This is the script that demonstrates all three kinds of pipe stealing by reading from pipe, writing to, or writing to and reading from pipe from the same process (self pipe) within CMD/batch scripts.

This script is just intended to demonstrate the concept, not to promote using the techniques in production scripts. As this can easily lead to dead locks or other unintended behaviors.

Code: Select all

@echo off
setlocal
if /i "%~1"=="/WorkerPipe" goto :/WorkerPipe
if /i "%~1"=="/ReadPipe" goto :/ReadPipe
if /i "%~1"=="/WritePipe" goto :/WritePipe
if /i "%~1"=="/PipeToSelf" goto :/PipeToSelf
cmd /d /c @"%~f0%" /ReadPipe
cmd /d /c @"%~f0%" /WritePipe
cmd /d /c @"%~f0%" /PipeToSelf
pause
exit /b


:: Access pipe by the parent cmd process
:: This can be done only with explicit external processes
:: So CMD commands must be wrapped by cmd /c

:: Process1 | Process2
:: At the time of creating left hand process, the state of cmd handles are as follows
:: &0=Unaltered
:: &1=Pipe Output
:: &2=Unaltered
:: &3=Pipe Input
:: &4=UNDEFINED
:: &5=Original value of &1

:: The left hand process will be created in suspended state
:: It inherit it standard handles (&0 &1 &2) from CMD

:: After creating left hand process in suspended state,
:: The hanldes will be restored to their original value,
:: Or remain redirected if permanent redirection was applied when creating left hand process
:: this can affect the handles which the right hand process inherits.
:: Then the right hand process will be created in suspended state

:: At the time of creating righ hand process, the state of cmd handles are as follows
:: &0=Pipe Input
:: &1=Unaltered
:: &2=Unaltered
:: &3=UNDEFINED
:: &4=Original value of &0

:: The right hand process will be created in suspended state
:: It inherit it standard handles (&0 &1 &2) from CMD

:: CMD itself closes the handles to the both sides of the pipes which have been inherited by its childs.
:: Then both processes will be resumed


:: Examples

:: <Reading from pipe>

:/ReadPipe
echo ---------------------------
echo Read from pipe demo
echo ---------------------------

setlocal EnableDelayedExpansion
set "LF="

:: Permanently redirect stdin to pipe input
cmd /d /u /c "echo(&echo(" 0>&3 4>&0 | break

:: Now reading from pipe
pause>nul
pause>nul
set /p "LF="

echo LF test:
echo First Line!LF!Second Line

:: Restore stdin
break 0>&4 3>&0
pause
echo findstr test:

:: The file handles &3 and &4 are now pointing to original stdin &0

cmd /d /c start /b cmd /d /c type "%~f0" 0>&5 6>&0 | break
findstr "^::"
break 0>&6 5>&0
pause
endlocal
exit /b
:: </Reading from pipe>



:: <Writing to pipe>

:/WritePipe

echo ---------------------------
echo Write to pipe demo
echo ---------------------------

:: Because permanent redirecton of stdout (&1) the right hand pipe will inherit the pipe output handle in its process
:: This creates a loop to itself so another stdout must be explicitly specified for it
:: But the risk of dead lock remains. if some program holds stdin until the other side is signaled. (findstr for example)
:: Because it has inherited the handle to read pipe too.

cmd /c "break" 5>&1 4>&5 | cmd /c start /b cmd /c "%~f0" /WorkerPipe 1>&4


timeout /t 1 /nobreak >nul
echo echo Comming directly from parent process (The pipe initiator)
timeout /t 1 /nobreak >nul
echo echo waiting for 5 seconds...
timeout /t 1 /nobreak >nul
echo cmd /c ping -n 6 127.0.0.1 ^>nul
timeout /t 1 /nobreak >nul
echo exit

ping -n 6 127.0.0.1

exit /b
:: </Writing to pipe>


exit /b
:/WorkerPipe
setlocal EnableDelayedExpansion
echo Worker Thread listening for commands from pipe
for /L %%A in (0,0,1) do (
    echo,
    echo Waiting for command...
    set /p "cmd="
    echo Recieved command: !cmd!
    echo Executing...
    !cmd!
)


:: <Pipe to self>

:/PipeToSelf

echo ---------------------------
echo Pipe to self demo
echo ---------------------------

cmd /d /c break 0>&3 4>&0 5>&1 7>&5 | break

set "MyEcho=echo This will be saved in to the variable 'MyVar' just by echoing it!"

echo executing: %MyEcho% >&7
%MyEcho%
set /p "MyVar="
set MyVar >&7
exit /b

:: </Pipe to self>

[dbenham]

Thanks sst.

Your explanation of how the child processes inherit the file handles is really helpful, and steps 1 and 2 of the pipe setup makes enough sense that I can envision how your technique could work. But the details after step 2 confuse me.

I can't tell if there is a problem with your write up, or with your analysis, or if my poor brain is being dense. Perhaps some of the steps you describe are so counter-intuitive that my mind just refuses to accept it.

Regardless, it is still very interesting.


Dave

[sst]

Hi Dave,
Thanks for your feedback.

Well I can't deny the fact that my write up is far from perfect, so, at least it could be one of the reasons behind your confusion. As it is obvious, English is not my native language. That's not an excuse, but anyway, It's not an easy task for me to write in English. I tried my best, but still...

Aside from the flaws in my English and/or presentation, there is also possibility for minor errors/mistakes in the technical parts but not that much that invalidates the whole analysis. I wrote the last code sample based on that analysis and it perfectly works and supports the theory. But if necessary I can share details of my analysis and the steps I took.(With my poor English )

Perhaps if you can point me to at-least one the descriptions that seems counter-intuitive, I would have more insight on what could be responsible for the confusion, so I can try to fix any errors and/or describe it more clearly, I hope.

[jfl]

Thanks @sst, that's a really great trick!!!

Like @dbenham, it took me a long time to understand it.
The critical point is a bug in cmd.exe, which undoes multiple redirections in the same order it created them, instead of the inverse order as it should.
So by using TWO redirections that step on each other, the handles end up restored incorrectly.

I've simplified the script with comments explaining the handle changes at each stage:

[2019-01-03 edit: Corrected an error in the last comment of the first two code samples]

Code: Select all

@echo off
:# Rerun self in a sub-shell, to avoid breaking the original shell file handles
echo %0 | findstr :: >nul || (cmd /d /c ^""%~dp0\::\..\%~nx0" %*^" & exit /b)

:# Pipe creation: PipeIn->&3, PipeOut->&4, save &1->&5, move PipeOut from &4->&1
:# Redirections: 0>&3 saves &0->&4, then 4>&3 saves &4->&6 (Orig&0)
cmd /c "echo From Child" 0>&3 4>&3 | break
:# Cleanup: [0>&3] restores &4->&0 (PipeIn), then [4>&3] restores &6->&4 (Orig&0)
:# This leaves behind: &0=PipeIn, &1=Orig&1, &2=Orig&2, &4=Orig&0

set /p "READ="				&:# Read value from stdin
echo Read on stdin: %READ%

:# Redirections: &0->&4 saves &0->&3, then 3>&4 saves &3->&5
break 0>&4 3>&4		&:# Restore stdin
:# Cleanup: [&0->&4] restores &3->&0 (Orig&0), then [3>&4] restores &5->&3 (PipeIn)
:# This leaves behind: &0=Orig&0, &1=Orig&1, &2=Orig&2, &3=PipeIn, &4=Orig&0
set /p "READ=Please enter something: "	&:# Read value from stdin
echo Read on stdin: %READ%

Note that for both lines with double redirections, the target of the first redirection does not matter.
(I've changed it from what sst used, because I think things are sligthly clearer this way, but the effect is exactly the same as in his original code.)
It's the second one that redirects again the handle saved by the first one that really matters!
For example, if I change the first redirection to NUL in the above script, everything works just as well:

Code: Select all

@echo off
:# Rerun self in a sub-shell, to avoid breaking the original shell file handles
echo %0 | findstr :: >nul || (cmd /d /c ^""%~dp0\::\..\%~nx0" %*^" & exit /b)

:# Pipe creation: PipeIn->&3, PipeOut->&4, save &1->&5, move PipeOut from &4->&1
:# Redirections: 0>&3 saves &0->&4, then 4>&3 saves &4->&6 (Orig&0)
cmd /c "echo From Child" 0>NUL 4>&3 | break
:# Cleanup: [0>&3] restores &4->&0 (PipeIn), then [4>&3] restores &6->&4 (Orig&0)
:# This leaves behind: &0=PipeIn, &1=Orig&1, &2=Orig&2, &4=Orig&0

set /p "READ="				&:# Read value from stdin
echo Read on stdin: %READ%

:# Redirections: &0->&4 saves &0->&3, then 3>&4 saves &3->&5
break 0>NUL 3>&4		&:# Restore stdin
:# Cleanup: [&0->&4] restores &3->&0 (Orig&0), then [3>&4] restores &5->&3 (PipeIn)
:# This leaves behind: &0=Orig&0, &1=Orig&1, &2=Orig&2, &3=PipeIn, &4=Orig&0
set /p "READ=Please enter something: "	&:# Read value from stdin
echo Read on stdin: %READ%

At this stage, it's now easy to add a third redirection that allows preserving the PipeOut handle.
This way, the pipe can be reused again as many times as we want, without exhausting file handles:

Code: Select all

@echo off
:# Rerun self in a sub-shell, to avoid breaking the original shell file handles
echo %0 | findstr :: >nul || (cmd /d /c ^""%~dp0\::\..\%~nx0" %*^" & exit /b)

:# Pipe creation: PipeIn->&3, PipeOut->&4, save &1->&5, move PipeOut from &4->&1
:# Redirections: 0>&3 saves &0->&4, then 4>&3 saves &4->&6, then 6>&1 saves &6->&7 (Orig&0)
cmd /c "echo From Child" 0>&3 4>&3 6>&1 | break
:# Cleanup: [0>&3] restores &4->&0 (PipeIn), then [4>&3] restores &6->&4 (PipeOut), then [6>&1] restores &7->&6 (Orig&0)
:# This leaves behind: &0=PipeIn, &1=Orig&1, &2=Orig&2, &4=PipeOut, &6=Orig&0

set /p "READ="				&:# Read value from stdin
echo Read on stdin: %READ%

:# Redirections: &0->&6 saves &0->&3, then 3>&6 saves &3->&5
break 0>&6 3>&6		&:# Restore stdin
:# Cleanup: [&0->&4] restores &3->&0 (Orig&0), then [3>&4] restores &5->&3 (PipeIn)
:# This leaves behind: &0=Orig&0, &1=Orig&1, &2=Orig&2, &3=PipeIn, &4=PipeOut, &6=Orig&0
set /p "RESULT=Please enter something: "
echo You entered on stdin: %RESULT%

>&4 echo From another child		&:# Write to pipe
<&3 set /p "READ="			&:# Read value from pipe
echo Read on pipe: %READ%

And everything can be simplified even further, so as to create a pipe without writing anything into it:

Code: Select all

@echo off
:# Rerun self in a sub-shell, to avoid breaking the original shell file handles
echo %0 | findstr :: >nul || (cmd /d /c ^""%~dp0\::\..\%~nx0" %*^" & exit /b)

:# Create pipe
cmd /c "exit 0" 0>&3 4>&3 6>&1 | break
break 0>&6 3>&6
:# This leaves behind: &0=Orig&0, &1=Orig&1, &2=Orig&2, &3=PipeIn, &4=PipeOut, &6=Orig&0

>&4 echo From another child		&:# Write to pipe
<&3 set /p "READ="			&:# Read value from pipe
echo Read on pipe: %READ%

I'm pretty sure the pipe handles creation can be reduced even further to a single line, without duplicating &0.
The race is on

Jean-François

[jfl]

Hi,
I found the solution, and it's surprisingly simple.
This allows to write a reusable :CreatePipe routine:

Code: Select all

@echo off
:# Rerun self in a sub-shell, to avoid breaking the original shell file handles
echo %0 | findstr :: >nul || (cmd /d /c ^""%~dp0\::\..\%~nx0" %*^" & exit /b)
goto :start

:# Pipe creation routine. Returns with &3=PipeIn, &4=PipeOut
:CreatePipe
cmd /c break >&4 4>&6 | cmd /c break 0>&3 3>&6
exit /b

:start
call :CreatePipe

>&4 echo From first child		&:# Write to pipe
<&3 set /p "READ="			&:# Read value from pipe
echo Read on pipe: %READ%

How this works:

• As explained in the previous posts, all this relies on the fact that multiple redirections are undone in the same order they were created, instead of the reverse order as they should. So when the second redirection redirects the backup handle created by the first redirection, the handles are restored incorrectly:
  • The first handle is restored from the target of the second redirection.
  • The second handle remains, with the copy of the initial value of the first handle.

• A second finding is that a handle can be duplicated by redirecting it to its own backup handle! Ex: (echo Hello >&3) outputs text normally.
  This is because the redirection can be decomposed this way: Move &1->&3; Duplicate &3 into &1.

The sequence of handle operations in the single line in the :CreatePipe routine is thus:

• Handles initially: &0=Orig&0, &1=Orig&1, &2=Orig&2

• Pipe creation: PipeIn->&3, PipeOut->&4, save &1->&5, move PipeOut from &4->&1
  Handles now: &0=Orig&0, &1=PipeOut, &2=Orig&2, &3=PipeIn, &5=Orig&1

• Left redirections: >&4 saves &1->&4; 4>&6 saves &4->&6 (PipeOut)
  Handles now: &0=Orig&0, &1=PipeOut, &2=Orig&2, &3=PipeIn, &4=PipeOut, &5=Orig&1, &6=PipeOut

• Spawn left cmd with the above handles: cmd /c break >&4 4>&6

• Cleanup left redirs: [>&4] restores &4->&1; [4>&6] restores &6->&4
  Handles now: &0=Orig&0, &1=PipeOut, &2=Orig&2, &3=PipeIn, &4=PipeOut, &5=Orig&1

• Restore stdout: Delete &1, Restore &5->&1
  Handles now: &0=Orig&0, &1=Orig&1, &2=Orig&2, &3=PipeIn, &4=PipeOut

• Setup stdin for right pipe: Save &0-> &5; move &3->0
  Handles now: &0=PipeIn, &1=Orig&1, &2=Orig&2, &4=PipeOut, &5=Orig&0

• Right redirections: 0>&3 saves &0->&3; 3>&6 saves &3->&6 (PipeIn)
  Handles now: &0=PipeIn, &1=Orig&1, &2=Orig&2, &3=PipeIn, &4=PipeOut, &5=Orig&0, &3=PipeIn

• Spawn right cmd with the above handles: cmd /c break 0>&3 3>&6

• Cleanup right redirs: [0>&3] restores &3->&0; [3>&6] restores &6->&3
  Handles now: &0=PipeIn, &1=Orig&1, &2=Orig&2, &3=PipeIn, &4=PipeOut, &5=Orig&0

• Restore stdin: Delete &0, Restore &5->&1
  Handles now: &0=Orig&0, &1=Orig&1, &2=Orig&2, &3=PipeIn, &4=PipeOut

I tried using just break, instead of cmd /c break, on each side of the pipe, but this does not work. Apparently, when it's an internal command on one side of the pipe, the handles are managed differently. Is there any other executable that we could use, that writes nothing, and that's lighter and faster than cmd.exe?

Also this :CreatePipe routine is relatively fragile: It'll break if any other handle already exists beyond &2. To make it more resilient, we'd need to add a way to detect that. Any idea on how to do this?

Ideally we'd need to detect all existing handles, and dynamically adapt the routine to select the output handle numbers to use. (Or use handle numbers chosen by the caller, and passed as arguments?)

Anyway, have fun with the :CreatePipe routine!

Jean-François

[carlos]

Very interesting topic.

Is there any other executable that we could use, that writes nothing, and that's lighter and faster than cmd.exe?

I found doskey.exe (19 KB) print anything and is lightweight than cmd.exe (268 KB)
Other option can be rundll32.exe (68 KB)

this works okay.

Code: Select all

:CreatePipe
doskey 1>&4 4>&6 | doskey 0>&3 3>&6
exit /b

[jfl]

I found doskey.exe (19 KB) print anything and is lightweight than cmd.exe (268 KB)
Other option can be rundll32.exe (68 KB)

Good ideas!

I tested the three versions on 1000 loops creating pipes, and got these results on my laptop:

• cmd /c break: 25 ms

• doskey: 23 ms

• rundll32: 21 ms

The difference is small, but not negligible. I'll use rundll32 now on.

Code: Select all

:CreatePipe
rundll32 1>&4 4>&6 | rundll32 0>&3 3>&6
exit /b