DelayedExpansion fails inside For /F pipe mode ( bug ?)

[carlos]

Hello.
Maybe this will be mentioned before. But I never post something that I read before.

For /F fails, maybe this is a bug, when you expand delayed a variable inside a pipe command.
This permits a command code execution.

Code: Select all

@Echo Off
SetLocal EnableDelayedExpansion

Set "Text=My Text&Calc.exe"

Rem Works:
Echo !Text!
Pause

Rem Fails:
For /F "delims=" %%# in ('Echo !Text!') do Echo %%#
Pause

The batch run the next command:

Code: Select all

Echo !Text!

Outside the For /F it print:

Code: Select all

My Text&Calc.exe

But, inside the For /F using pipe mode, it print:

Code: Select all

My Text

and run Calc.exe.

Bug?

[foxidrive]

A for command outputs most poison characters in the metavariable just fine, so that is expected.

The fact that a !var! doesn't encapsulate the characters could be an issue in some cases... another 'gotcha' to remember, but it is the same behaviour that you get when echoing %var%

[carlos]

echoing %var% is not the same that echoing !var! inside for /f in pipe mode:

Code: Select all

Set "Text=My Text&Calc.exe"

Rem Print: My Text&Calc.exe
Echo !Text!

Rem Print: My Text and run Calc.exe
For /F "delims=" %%# in ('Echo !Text!') do Echo %%#

Rem Abort the batch file, because the & character is not escaped.
For /F "delims=" %%# in ('Echo %Text%') do Echo %%#

[foxidrive]

You really should add delayedexpansion to your snippet but yes, I see the difference.

[dbenham]

It fails because the commands within IN('command') are run in a new CMD.EXE session with delayed expansion turned off, even if the parent script has delayed expansion on.

The Windows pipe and FOR command use similar techniques to execute the commands in a new CMD.EXE context.

See jeb's excellent answer to my StackOverflow Question: Why does delayed expansion fail when inside a piped block of code?. I recommend reading the entire thread, as the other answers help give context to jeb's answer.


Dave Benham

[carlos]

Thanks Dave.
But this also fail:

Code: Select all

SetLocal EnableDelayedExpansion
Set "Text=My Text&Calc.exe"

Rem Fails, print: My Text and run calc.exe
For /F "delims=" %%# in (
'SetLocal EnableDelayedExpansion ^&Echo !Text!'
) do Echo %%#

But this works:

Code: Select all

SetLocal EnableDelayedExpansion
Set "Text=My Text&Calc.exe"
Rem Works:
For /F "delims=" %%# in (
'SetLocal EnableDelayedExpansion ^&Echo ^^!Text^^!'
) do Echo %%#

I have the explanation of this.

[carlos]

But this next print the help of echo command, even using echo(

Code: Select all

SetLocal EnableDelayedExpansion
Set "Text=/?"

Rem Fails, print: My Text and run calc.exe
For /F "delims=" %%# in (
'SetLocal EnableDelayedExpansion ^&Echo(^^!Text^^!'
) do Echo %%#

Maybe, the use of pipe mode in for /f in some scenarios is unreliable.

[Ed Dyreen]

But this next print the help of echo command, even using echo(

Code: Select all

SetLocal EnableDelayedExpansion
Set "Text=/?"

Rem Fails, print: My Text and run calc.exe
For /F "delims=" %%# in (
'SetLocal EnableDelayedExpansion ^&Echo(^^!Text^^!'
) do Echo %%#

Maybe, the use of pipe mode in for /f in some scenarios is unreliable.

The problem is 'Echo %%#' which results in 'Echo /?' which displays the help.

Code: Select all

@echo off &SetLocal EnableDelayedExpansion
Set "Text=/?"

echo
echo /?
echo./?
echo.test
For /F "delims=" %%# in (

   'echo.^^^!Text^^^!'

) do echo.%%#

pause
exit

The for obfuscates the fact that you used 'echo' without a dot, simply add a dot and your problem is solved.

Code: Select all

ECHO is off (uit).
Meldingen weergeven of de opdracht ECHO aan- of uitschakelen.

  ECHO [ON | OFF]
  ECHO [melding]

ECHO zonder parameters geeft de huidige instelling voor de opdracht ECHO weer.
/?
test
/?
Druk op een toets om door te gaan. . .

ed

[dbenham]

@Ed - Not quite the full story. :;

!Text! is not getting expanded within the IN() clause CMD.EXE session. It is getting expanded by the parent batch within the DO clause. Disable delayed expansion and you can see what I mean.

Code: Select all

@echo off
SetLocal EnableDelayedExpansion
Set "Text=/?"
For /F "delims=" %%# in (
  'SetLocal EnableDelayedExpansion ^&Echo(^^!Text^^!'
) do (
  setlocal disableDelayedExpansion
  Echo(%%#
  endlocal
)

result

Code: Select all

!Text!

The problem is that the IN() clause is executed under a command line context, not a batch file context. SETLOCAL does not work under a command line context.

To get delayed expansion within the IN() clause requires using CMD /V:ON. It is easiest if quotes are used. But it is also possible without quotes.

Code: Select all

@echo off
SetLocal EnableDelayedExpansion
Set "Text=/?"

For /F "delims=" %%# in (
  'cmd /v:on /c "echo(^!Text^!"'
) do (
  setlocal disableDelayedExpansion
  Echo(%%#
  endlocal
)
echo(

For /F "delims=" %%# in (
  'cmd /v:on /c echo(^^^!Text^^^!'
) do (
  setlocal disableDelayedExpansion
  Echo(%%#
  endlocal
)
exit /b

But, how to prove that the expansion is happening within the IN() clause CMD session, and not within the parent batch? Here I provide 2 proofs within one set of examples.

I'm going to use !ERRORLEVEL! instead of !Text!. I want to echo the ERRORLEVEL that results from a command that executed within the IN() clause. I'll use a command (HELP HELP) that generates an error, and print the ERRORLEVEL both before and after.

The other proof is the use of !CMDCMDLINE! to show the command line that was used to launch the currently executing CMD.EXE session. This gets very confusing because there are actually two CMD sessions in the example. One is created implicitly by the FOR /F command, and the second is created explicitly by our command. This also makes the escape rules horrificly complicated.

My first test uses quotes around everything. I show both CMD contexts.

My 2nd and 3rd tests require more escaping, so I simplified by dropping the printout of the initial CMD context.

But my 4th example attempts to put it all together, showing both contexts without any quotes, and it gives really screwy results. The CMD /V:ON... command gets executed twice
Edit - removed extra ^ from escaped (not quoted) delayed expansion: ^^^!var^^^! --> ^^!var^^!
Edit 2 - removed some additional extra ^: ^^^= --> ^= and ^^^^^^^& --> ^^^^^& and ^^^^^^^> --> ^^^^^>

Code: Select all

@echo off
setLocal enableDelayedExpansion
for /f "delims=" %%# in (
  '"echo 1st cmdLine: %%cmdcmdline%%&cmd /v:on /c echo 2nd cmdLine: ^!cmdcmdline^!^^&echo err=^!errorlevel^!^^&^^>nul help help^^&echo err=^!errorlevel^!"'
) do (
  setlocal disableDelayedExpansion
  echo test1: %%#
  endlocal
)
echo(

for /f "delims=" %%# in (
  'cmd /v:on /c "echo 2nd cmdLine: ^!cmdcmdline^!&echo err=^!errorlevel^!&>nul help help&echo err=^!errorlevel^!"'
) do (
  setlocal disableDelayedExpansion
  echo test2: %%#
  endlocal
)
echo(

for /f "delims=" %%# in (
  'cmd /v:on /c echo 2nd cmdLine: ^^!cmdcmdline^^!^^^^^&echo err^=^^!errorlevel^^!^^^^^&^^^^^>nul help help^^^^^&echo err^=^^!errorlevel^^!'
) do (
  setlocal disableDelayedExpansion
  echo test3: %%#
  endlocal
)
echo(

for /f "delims=" %%# in (
  'echo 1st cmdLine: %%cmdcmdline%%^&cmd /v:on /c echo 2nd cmdLine: ^^!cmdcmdline^^!^^^^^&echo err^=^^!errorlevel^^!^^^^^&^^^^^>nul help help^^^^^&echo err^=^^!errorlevel^^!'
) do (
  setlocal disableDelayedExpansion
  echo test4: %%#
  endlocal
)

results

Code: Select all

test1: 1st cmdLine: C:\Windows\system32\cmd.exe /c "echo 1st cmdLine: %cmdcmdline%&cmd /v:on /c echo 2nd cmdLine: !cmdcmdline!^&echo err=!errorlevel!^&^>nul help help^&echo err=!errorlevel!"
test1: 2nd cmdLine: cmd  /v:on /c echo 2nd cmdLine: !cmdcmdline!&echo err=!errorlevel!&>nul help help&echo err=!errorlevel!
test1: err=0
test1: err=1

test2: 2nd cmdLine: cmd  /v:on /c "echo 2nd cmdLine: !cmdcmdline!&echo err=!errorlevel!&>nul help help&echo err=!errorlevel!"
test2: err=0
test2: err=1

test3: 2nd cmdLine: cmd  /v:on /c echo 2nd cmdLine: !cmdcmdline!&echo err=!errorlevel!&>nul help help&echo err=!errorlevel!
test3: err=0
test3: err=1

test4: 1st cmdLine: C:\Windows\system32\cmd.exe /c echo 1st cmdLine: %cmdcmdline%
test4: 2nd cmdLine: cmd  /v:on /c echo 2nd cmdLine: !cmdcmdline!&echo err=!errorlevel!&>nul help help&echo err=!errorlevel!
test4: err=0
test4: err=1
test4: 2nd cmdLine: cmd  /v:on /c echo 2nd cmdLine: !cmdcmdline!&echo err=!errorlevel!&>nul help help&echo err=!errorlevel!
test4: err=0
test4: err=1

Dave Benham

[Ed Dyreen]

'
O I just responded to his last post, where he make an obvious mistake.

That last test4 is quite weird indeed, don't understand the logic which would require such constructs, couldn't explain it,
don't care and leave it fully up to you

[carlos]

@Ed - Not quite the full story. :;

!Text! is not getting expanded within the IN() clause CMD.EXE session. It is getting expanded by the parent batch within the DO clause. Disable delayed expansion and you can see what I mean.

Code: Select all

@echo off
SetLocal EnableDelayedExpansion
Set "Text=/?"
For /F "delims=" %%# in (
  'SetLocal EnableDelayedExpansion ^&Echo(^^!Text^^!'
) do (
  setlocal disableDelayedExpansion
  Echo(%%#
  endlocal
)

result

Code: Select all

!Text!

The problem is that the IN() clause is executed under a command line context, not a batch file context. SETLOCAL does not work under a command line context.

@Ed Dyreen: Thanks for the fix.
@Dave: Thanks for the example. Now, the problem is more clear.

Also I found that run Setlocal EnableDelayedExpansion have none effect if you run it inside the pipe command of for /f.
But also, I found that If you do the delayed expansion inside of the pipe command of a variable with LF characters, it will not be splitted with for /F.
Check this example of code:

Code: Select all

@Echo Off
SetLocal EnableDelayedExpansion
(Set \n=^

)

Set "text=text1!\n!text2!\n!text3"

Set /a cnt=0

For /F %%# in (
'Echo(^^!text^^!'
) Do (
Set /a cnt+=1
Echo Loop !cnt!:%%#
)

It print:

Code: Select all

Loop 1:text1
text2
text3

(For /F don't split the variable in lines. The echo command do it. Bug?)

[Ed Dyreen]

(For /F don't split the variable in lines. The echo command do it. Bug?)

http://www.dostips.com/forum/viewtopic.php?p=11528#p11528

Code: Select all

@echo off
setlocal enableDelayedExpansion
set LF=^


set "pLF=%%LF%%"

set ^"@macro=(^
   echo text1 %pLF%^
   echo text2 %pLF%^
   echo text3 %pLF%^
)"

set @macro

setlocal enableExtensions enableDelayedExpansion
echo --- Output is ---
for /f %%1 in ( '!@macro!' ) do (
   Set /a cnt+=1
   Echo Loop !cnt!:%%1
)

pause
exit

Code: Select all

@macro=(   echo text1 %LF%   echo text2 %LF%   echo text3 %LF%)
--- Output is ---
Loop 1:text1
Loop 2:text2
Loop 3:text3
Druk op een toets om door te gaan. . .

like that

Code: Select all

@echo off &setlocal enableDelayedExpansion &set $lf=^


::
set "$=text1!$lf!text2!$lf!text3!$lf!"
set "$"
echo --- Output is ---
for /f delims^=^ eol^= %%r in ("!$!") do set /a §+= 1 &echo.loop !§!:%%r

pause
exit

Code: Select all

$=text1
text2
text3

$lf=

--- Output is ---
loop 1:text1
loop 2:text2
loop 3:text3
Druk op een toets om door te gaan. . .

[dbenham]

@Carlos - You have made the same mistake. Your !text! is getting expanded by the parent batch in the DO() clause. You want it to be expanded by the child process in the IN() clause, so you need to enable delayed expansion with CMD /V:ON /C

Code: Select all

@Echo Off
SetLocal EnableDelayedExpansion
(Set \n=^

)

Set "text=text1!\n!text2!\n!text3"

Set /a cnt=0

For /F %%# in ('cmd /v:on /c Echo(^^!text^^!') Do (
  Set /a cnt+=1
  Echo Loop !cnt!:%%#
)

result

Code: Select all

Loop 1:text1
Loop 2:text2
Loop 3:text3

But my 4th example attempts to put it all together, showing both contexts without any quotes, and it gives really screwy results. The CMD /V:ON... command gets executed twice

Achh - it is so obvious
%cmdcmdline% is not quoted and the contents are not escaped, so it functions like a macro. The first unescaped, unquoted & terminates the ECHO and then the CMD /V:ON /C command is executed. I simply surround the variable in quotes and all is well.

Code: Select all

@echo off
setLocal enableDelayedExpansion
for /f "delims=" %%# in (
  'echo 1st cmdLine: "%%cmdcmdline%%"^&cmd /v:on /c echo 2nd cmdLine: ^^!cmdcmdline^^!^^^^^&echo err^=^^!errorlevel^^!^^^^^&^^^^^>nul help help^^^^^&echo err^=^^!errorlevel^^!'
) do (
  setlocal disableDelayedExpansion
  echo test4: %%#
  endlocal
)

results

Code: Select all

test4: 1st cmdLine: "C:\Windows\system32\cmd.exe /c echo 1st cmdLine: "%cmdcmdline%"&cmd /v:on /c echo 2nd cmdLine: !cmdcmdline!^&echo err=!errorlevel!^&^>nul help help^&echo err=!errorlevel!"
test4: 2nd cmdLine: cmd  /v:on /c echo 2nd cmdLine: !cmdcmdline!&echo err=!errorlevel!&>nul help help&echo err=!errorlevel!
test4: err=0
test4: err=1

Dave Benham

[Ed Dyreen]

'
Now I look stupid

[dbenham]

No MicroSoft FOR bug involved within this thread. But the escape rules are crazy

I think one can make an argument that there is a design flaw in how pipes and FOR /F ... IN('command') are executed. It shouldn't have to be this difficult.


Dave Benham