Newly discovered pipe behavior - and a fun challenge!

[dbenham]

Over at https://stackoverflow.com/a/57492041/1012053 jeb deduced a newly discovered behavior of Windows pipes.

We had all thought that each side of a pipe is always executed in a new cmd.exe process via CMD /S /D /C. Well it turns out cmd.exe is only used if the command is an internal command, or if the command is enclosed within parentheses. External commands that are not enclosed in parentheses are launched in a new process without invoking cmd.exe.

test.txt

Code: Select all

%var%
OK
&

test.bat

Code: Select all

@echo off
setlocal
set "var=OK"

<nul set /p "=test 1: " & break | echo %%var%%
<nul set /p "=test 2: " & break | findstr %%var%% test.txt
<nul set /p "=test 3: " & break | (findstr %%var%% test.txt)
<nul set /p "=test 4: " & break | findstr ^& test.txt
<nul set /p "=test 5: " & break | (findstr ^^^& test.txt)

Output of test.bat

Code: Select all

test 1: OK
test 2: %var%
test 3: OK
test 4: &
test 5: &

Here is the challenge:
So I thought it would be fun to further probe / prove the behavior via %CMDCMDLINE%. Boy was I surprised by how quickly the results can become insanely complicated.

Code: Select all

@echo off
echo(&echo 1) break ^| cmd /v:on /c echo %%%%cmdcmdline%%%%
break | cmd /v:on /c echo %%cmdcmdline%%

echo(&echo 2) break ^| cmd /v:on /c echo !cmdcmdline!
break | cmd /v:on /c echo !cmdcmdline!

echo(&echo 3) break ^| cmd /v:on /c echo %%%%cmdcmdline%% %%%%cmdcmdline%%%%
break | cmd /v:on /c echo %%cmdcmdline%% %%cmdcmdline%%

echo(&echo 4) break ^| cmd /v:on /c echo !cmdcmdline! !cmdcmdline!
break | cmd /v:on /c echo !cmdcmdline! !cmdcmdline!

echo(&echo 5) break ^| cmd /v:on /c echo %%%%cmdcmdline%%%% !cmdcmdline!
break | cmd /v:on /c echo %%cmdcmdline%% !cmdcmdline!

echo(&echo 6) break ^| (cmd /v:on /c echo !cmdcmdline!)
break | (cmd /v:on /c echo !cmdcmdline!)

echo(&echo 7) break ^| (cmd /v:on /c echo %%%%cmdcmdline%%%%)
break | (cmd /v:on /c echo %%cmdcmdline%%)

echo(&echo 8) break ^| (cmd /v:on /c echo !cmdcmdline! !cmdcmdline!)
break | (cmd /v:on /c echo !cmdcmdline! !cmdcmdline!)

echo(&echo 9) break ^| (cmd /v:on /c echo %%%%cmdcmdline%%%% %%%%^^^^cmdcmdline%%%%)
break | (cmd /v:on /c echo %%cmdcmdline%% %%^^cmdcmdline%%)

echo(&echo 10) break ^| (cmd /v:on /c echo %%%%cmdcmdline%%%% !cmdcmdline!)
break | (cmd /v:on /c echo %%cmdcmdline%% !cmdcmdline!)

echo(&echo 11) break ^| (cmd /v:on /c echo %%%%cmdcmdline%%%% %%%%cmdcmdline%%%%)
break | (cmd /v:on /c echo %%cmdcmdline%% %%cmdcmdline%%)

Tests 1 - 8 are all trivial and easy to trace. But my jaw dropped when I saw the results of 9, 10, and 11. The output for test 11 is 907 bytes
I have yet to take the time to trace the logic of the last three tests.

Code: Select all

1) break | cmd /v:on /c echo %%cmdcmdline%%
cmd  /v:on /c echo %cmdcmdline%

2) break | cmd /v:on /c echo !cmdcmdline!
cmd  /v:on /c echo !cmdcmdline!

3) break | cmd /v:on /c echo %%cmdcmdline% %%cmdcmdline%%
cmd  /v:on /c echo %cmdcmdline% %cmdcmdline% cmd  /v:on /c echo %cmdcmdline% %cmdcmdline%

4) break | cmd /v:on /c echo !cmdcmdline! !cmdcmdline!
cmd  /v:on /c echo !cmdcmdline! !cmdcmdline! cmd  /v:on /c echo !cmdcmdline! !cmdcmdline!

5) break | cmd /v:on /c echo %%cmdcmdline%% !cmdcmdline!
cmd  /v:on /c echo %cmdcmdline% cmd  /v:on /c echo %cmdcmdline% !cmdcmdline! cmd  /v:on /c echo %cmdcmdline% !cmdcmdline!

6) break | (cmd /v:on /c echo !cmdcmdline!)
cmd  /v:on /c echo !cmdcmdline!  

7) break | (cmd /v:on /c echo %%cmdcmdline%%)
C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c echo cmd  /v:on /c echo C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c echo %cmdcmdline% )"  )" 

8) break | (cmd /v:on /c echo !cmdcmdline! !cmdcmdline!)
cmd  /v:on /c echo !cmdcmdline! !cmdcmdline!  cmd  /v:on /c echo !cmdcmdline! !cmdcmdline!  

9) break | (cmd /v:on /c echo %%cmdcmdline%% %%^^cmdcmdline%%)
C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c echo cmd  /v:on /c echo C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c echo %cmdcmdline% %cmdcmdline% )" %cmdcmdline%  %^cmdcmdline% )" cmd  /v:on /c echo C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c echo %cmdcmdline% %^cmdcmdline% )" %cmdcmdline%  

10) break | (cmd /v:on /c echo %%cmdcmdline%% !cmdcmdline!)
C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c echo cmd  /v:on /c echo C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c echo %cmdcmdline% cmd  /v:on /c echo C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c echo %cmdcmdline% !cmdcmdline! )" !cmdcmdline!  )" cmd  /v:on /c echo C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c echo %cmdcmdline% !cmdcmdline! )" !cmdcmdline!   cmd  /v:on /c echo C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c echo %cmdcmdline% !cmdcmdline! )" !cmdcmdline!  )" cmd  /v:on /c echo C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c echo %cmdcmdline% !cmdcmdline! )" !cmdcmdline!  

11) break | (cmd /v:on /c echo %%cmdcmdline%% %%cmdcmdline%%)
C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c echo cmd  /v:on /c echo C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c echo %cmdcmdline% %cmdcmdline% )" C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c echo %cmdcmdline% %cmdcmdline% )"  cmd  /v:on /c echo C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c echo %cmdcmdline% %cmdcmdline% )" C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c echo %cmdcmdline% %cmdcmdline% )"  )" C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c echo cmd  /v:on /c echo C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c echo %cmdcmdline% %cmdcmdline% )" C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c echo %cmdcmdline% %cmdcmdline% )"  cmd  /v:on /c echo C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c echo %cmdcmdline% %cmdcmdline% )" C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c echo %cmdcmdline% %cmdcmdline% )"  )" 

Dave Benham

[dbenham]

OMG! Adding one measly little CALL causes the output to balloon to 4145 bytes!

Code: Select all

echo(&echo 12) break ^| (cmd /v:on /c call echo %%%%cmdcmdline%%%% %%%%cmdcmdline%%%%)
break | (cmd /v:on /c call echo %%cmdcmdline%% %%cmdcmdline%%)

--OUTPUT--

Code: Select all

12) break | (cmd /v:on /c call echo %%cmdcmdline%% %%cmdcmdline%%)
C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c call echo cmd  /v:on /c call echo C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c call echo cmd  /v:on /c call echo C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c call echo %cmdcmdline% %cmdcmdline% )" C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c call echo %cmdcmdline% %cmdcmdline% )"  cmd  /v:on /c call echo C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c call echo %cmdcmdline% %cmdcmdline% )" C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c call echo %cmdcmdline% %cmdcmdline% )"  )" C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c call echo cmd  /v:on /c call echo C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c call echo %cmdcmdline% %cmdcmdline% )" C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c call echo %cmdcmdline% %cmdcmdline% )"  cmd  /v:on /c call echo C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c call echo %cmdcmdline% %cmdcmdline% )" C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c call echo %cmdcmdline% %cmdcmdline% )"  )"  cmd  /v:on /c call echo C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c call echo cmd  /v:on /c call echo C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c call echo %cmdcmdline% %cmdcmdline% )" C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c call echo %cmdcmdline% %cmdcmdline% )"  cmd  /v:on /c call echo C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c call echo %cmdcmdline% %cmdcmdline% )" C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c call echo %cmdcmdline% %cmdcmdline% )"  )" C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c call echo cmd  /v:on /c call echo C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c call echo %cmdcmdline% %cmdcmdline% )" C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c call echo %cmdcmdline% %cmdcmdline% )"  cmd  /v:on /c call echo C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c call echo %cmdcmdline% %cmdcmdline% )" C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c call echo %cmdcmdline% %cmdcmdline% )"  )"  )" C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c call echo cmd  /v:on /c call echo C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c call echo cmd  /v:on /c call echo C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c call echo %cmdcmdline% %cmdcmdline% )" C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c call echo %cmdcmdline% %cmdcmdline% )"  cmd  /v:on /c call echo C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c call echo %cmdcmdline% %cmdcmdline% )" C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c call echo %cmdcmdline% %cmdcmdline% )"  )" C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c call echo cmd  /v:on /c call echo C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c call echo %cmdcmdline% %cmdcmdline% )" C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c call echo %cmdcmdline% %cmdcmdline% )"  cmd  /v:on /c call echo C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c call echo %cmdcmdline% %cmdcmdline% )" C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c call echo %cmdcmdline% %cmdcmdline% )"  )"  cmd  /v:on /c call echo C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c call echo cmd  /v:on /c call echo C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c call echo %cmdcmdline% %cmdcmdline% )" C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c call echo %cmdcmdline% %cmdcmdline% )"  cmd  /v:on /c call echo C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c call echo %cmdcmdline% %cmdcmdline% )" C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c call echo %cmdcmdline% %cmdcmdline% )"  )" C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c call echo cmd  /v:on /c call echo C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c call echo %cmdcmdline% %cmdcmdline% )" C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c call echo %cmdcmdline% %cmdcmdline% )"  cmd  /v:on /c call echo C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c call echo %cmdcmdline% %cmdcmdline% )" C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c call echo %cmdcmdline% %cmdcmdline% )"  )"  )"

Dave Benham

[sst]

I explained this before to describe the inner workings of pipe creation in CMD. (In the graph that demonstrates the difference between the parenthesized and naked external processes in the pipes)

This behavior is the main reason that the technique for accessing the pipe by the parent CMD is working in the first place. But it seems that no one paid attention to it.

[dbenham]

And so you did

You introduced a number of new concepts, and my brain was a bit overwhelmed. There are still a number of points I don't fully understand. I hope to revisit your posts and see if I can better absorb the info.

For me, this is the key post where you point out the difference. I even used that information in some of my responses. But I was struggling to maintain some level of understanding, and the significance as it relates to the command parsing rules escaped me at the time.


Dave Benham

[penpen]

But my jaw dropped when I saw the results of 9, 10, and 11. The output for test 11 is 907 bytes
I have yet to take the time to trace the logic of the last three tests.

I probably miss something...
but i don't see anything else but expected recursive environment variable expansion?!

It was a little bit hard to read (to differ between the recursion levels, but you could add some "syntaktik sugar":

Code: Select all

:: 9
break | (cmd /v:on /c echo {%%cmdcmdline%%} else {%%^^cmdcmdline%%})

:: 10
break | (cmd /v:on /c echo {%%cmdcmdline%%} else {!cmdcmdline!})

:: 11
break | (cmd /v:on /c echo {%%cmdcmdline%%} else {%%cmdcmdline%%})

... and rewrite the result using C++/Java-style syntax (example 11 only):

Code: Select all

{
	C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c echo {
		cmd  /v:on /c echo {
			C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c echo {
				%cmdcmdline%
			} else {
				%cmdcmdline%
			} )"
		} else {
			C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c echo {
				%cmdcmdline%
			} else {
				%cmdcmdline%
			} )"
		} 
	} else {
		cmd  /v:on /c echo {
			C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c echo {
				%cmdcmdline%
			} else {
				%cmdcmdline%
			} )"
		} else {
			C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c echo {
				%cmdcmdline%
			} else {
				%cmdcmdline%
			} )"
		} 
	} )"
} else {
	C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c echo {
		cmd  /v:on /c echo {
			C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c echo {
				%cmdcmdline%
			} else {
				%cmdcmdline%
			} )"
		} else {
			C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c echo {
				%cmdcmdline%
			} else {
				%cmdcmdline%
			} )"
		} 
	} else {
		cmd  /v:on /c echo {
			C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c echo {
				%cmdcmdline%
			} else {
				%cmdcmdline%
			} )"
		} else {
			C:\WINDOWS\system32\cmd.exe  /S /D /c" ( cmd /v:on /c echo {
				%cmdcmdline%
			} else {
				%cmdcmdline%
			} )"
		} 
	} )"
} 

penpen

[sst]

But my jaw dropped when I saw the results of 9, 10, and 11. The output for test 11 is 907 bytes
I have yet to take the time to trace the logic of the last three tests.

I probably miss something...
but i don't see anything else but expected recursive environment variable expansion?!

Yes of course it is recursive, but I guess what was interesting to Dave is the growth rate of expansion by just adding one more %cmdcmdline%
For instance, comparing the test cases 7 and 11

Code: Select all

REM case 7
break | (cmd /v:on /c echo %%cmdcmdline%%)

REM case 11
break | (cmd /v:on /c echo %%cmdcmdline%% %%cmdcmdline%%)

The case 11 has only one more %cmdcmdline% but the length of the output jumps from 156 bytes in case 7 to 907 bytes in case 11

And here is the mathematical reason:

For the test case 11, the command line of the implicit CMD instance(The first level CMD) contains two instances of %cmdcmdline% :

C:\Windows\system32\cmd.exe /S /D /c" ( cmd /v:on /c echo %cmdcmdline% %cmdcmdline% )" )"

For this very same reason, each of the %cmdcmdline% variables themselves contains 2 instances of %cmdcmdline%, so after the expansion by the first level CMD, we have 4 instances %cmdcmdline%

Therefor the second level CMD will have 4 instances of %cmdcmdline% in its command line:

cmd /v:on /c echo .... %cmdcmdline% .... %cmdcmdline% .... %cmdcmdline% .... %cmdcmdline% ....

Now for the second level CMD, each of the %cmdcmdline% variables themselves, contains 4 instances of %cmdcmdline% so after expansion we have 16 instances of %cmdcmdline% (which will not be expanded further)

So after expansion by the second level CMD we have:

echo .... %cmdcmdline% .... %cmdcmdline% .... %cmdcmdline% .... %cmdcmdline% .... %cmdcmdline% .... %cmdcmdline% .... %cmdcmdline% .... %cmdcmdline% .... %cmdcmdline% .... %cmdcmdline% .... %cmdcmdline% .... %cmdcmdline% .... %cmdcmdline% .... %cmdcmdline% .... %cmdcmdline% .... %cmdcmdline% ....

Adding another level of CMD will theoretically yields an output with 256 ( 2 to the power of 8 ) unexpanded instances of %cmdcmdline%

And the theoretical output from a case by adding one more %cmdcmdline%

Code: Select all

break | (cmd /v:on /c echo %%cmdcmdline%% %%cmdcmdline%% %%cmdcmdline%%)

will be something with 81 ( 3 to the power of 4 ) unexpanded instances of %cmdcmdline%

The number of unexpanded %cmdcmdline% can be used to have a rough estimate for length of the output.
For (L) levels of nested CMDs and (C) number of initial %cmdcmdlines%s, the number of unexpanded %cmdcmdline%s in the final output can be calculated using:

Code: Select all

f(L, C) = C ^ (2 ^ L)

Here ^ means the power function not the xor operator in CMD