Delayed expansion fails in some cases

[jeb]

Hi,

while evaluating a question from SO: Im passing a multi line text as argument which will be saved in a variable and then the file created has only 1 Lline I found a curious behaviour.

Code: Select all

@echo off
setlocal EnableDelayedExpansion
(set \n=^
%=empty=%
)
receveiver.bat "Line1!\n!line2"

And Receiver.bat

Code: Select all

@echo off
setlocal DisableDelayedExpansion
prompt :
echo on
(
@exit /b
REM # %~1 #
)

:(REM # Line1!\n!line2 # )

Absolutly NOT the expected output, because I expected that the !\n! would be expanded in the first batch before starting the receiver.bat.

If the line is in a code block, it works as expected.

Code: Select all

(receveiver.bat "Line1!\n!line2")

:(

REM # Line
line2 #
)

Curious

jeb

[Thor]

This works, but I don't know why

Code: Select all

@echo off
setlocal EnableDelayedExpansion
(set \n=^
%=empty=%
)&receiver.bat "Line1!\n!line2"

[dbenham]

The newline is a bit of a red herring.

This single simple script demonstrates the problem.

test.bat

Code: Select all

@echo off
setlocal disableDelayedExpansion
if "%~1" neq "" echo %*&exit /b

setlocal EnableDelayedExpansion
set a=OK
"%~f0" !a!

--OUTPUT--

Code: Select all

!a!

But CALLing the script works:

Code: Select all

@echo off
setlocal disableDelayedExpansion
if "%~1" neq "" echo %*&exit /b

setlocal EnableDelayedExpansion
set a=OK
call "%~f0" !a!

--OUTPUT--

Code: Select all

OK

Extending Thors discovery, command concatenation with any command before the script works:

Code: Select all

@echo off
setlocal disableDelayedExpansion
if "%~1" neq "" echo %*&exit /b

setlocal EnableDelayedExpansion
set a=OK
rem. & "%~f0" !a!

--OUTPUT--

Code: Select all

OK

Concatenation with an appended command works:

Code: Select all

@echo off
setlocal disableDelayedExpansion
if "%~1" neq "" echo %*&exit /b

setlocal EnableDelayedExpansion
set a=OK
"%~f0" !a! & rem

--OUTPUT--

Code: Select all

OK

Putting the command within parentheses works:

Code: Select all

@echo off
setlocal disableDelayedExpansion
if "%~1" neq "" echo %*&exit /b

setlocal EnableDelayedExpansion
set a=OK
("%~f0" !a!)

--OUTPUT--

Code: Select all

OK

Executing the same script from the command line with enabled delayed expansion works:

Code: Select all

C:\test>cmd /v:on
Microsoft Windows [Version 10.0.14393]
(c) 2016 Microsoft Corporation. All rights reserved.

C:\test>set a=OK

C:\test>test.bat !a!
OK

Very odd that the only failure is when a batch script executes another script, if and only if the command is isolated on a stand-alone line, with no other concatenated commands or parentheses.


Dave Benham

[Aacini]

@Dave: CALL not works in the way expected by jeb (and I):

First, Thor's method that do work:

Code: Select all

@echo off
setlocal EnableDelayedExpansion
(set \n=^
%=empty=%
)&receiver.bat "Line1!\n!line2"

This is Receiver.bat:

Code: Select all

@echo off
setlocal DisableDelayedExpansion

echo on
(
@exit /b
REM # %~1 #
)

Output:

Code: Select all

(

 REM # Line1
 line2 #
)

That is, the expected output must show two lines!

CALLing the script not works:

Code: Select all

@echo off
setlocal EnableDelayedExpansion
(set \n=^
%=empty=%
)
call receiver.bat "Line1!\n!line2"

Output:

Code: Select all

(

 REM # Line1 #
)

However, there is not a single case where the multi-line value received in the parameter may be used to generate a multi-line output in Receiver.bat:

Code: Select all

@echo off
setlocal DisableDelayedExpansion

echo %~1

Output:

Code: Select all

Line1

If I use a for /F "delims=" %%a in ("%~1") do echo %%a command in Receiver.bat, nothing is shown, not even the first line!

For this reason, IMHO jeb is wrong when he states that "is possible to pass a multi-line value in a Batch file parameter".

Antonio

[penpen]

It seems that the command and the parameterstring are not parsed at the same time:
First the command is "delayed parsed" and executed. The "command process" then acesses the parameterstring expanding it (then).

I think i've read before that delayed Expansion "differs" between command and parameter string (although the consequences were different), but i cannot remember where;
if i had to guess, i would say it was a post/code from jeb/dbenham/both - something like this:

Code: Select all

@echo off
setlocal enableExtensions enableDelayedExpansion
set "a=1"
set "o =o 2"

ech%o %A% %o %  [1
ech!o !A! !o ! [2
call ech!o !A! !o !  [3
(ech!o !A! !o ! [4)
ech!o !A! !o ! [5&echo(

echo:^^^^ ^^^^%A% [1
echo:^^^^ ^^^^!A!  [2
call echo:^^^^ ^^^^!A!   [3
(echo:^^^^ ^^^^!A!  [4)
echo:^^^^ ^^^^!A!  [5&echo(

endlocal
goto :eof

In case the above is true, then the nitpick probably is, that normally you cannot turn off delayed expansion between executing the command and accessing the parameter from within the command (process).

The compound statement probably works, because there is another level of parsing included:
- command == "()" parameter all within ()
- command == "&" surrounded by two parameter;
The execution of "()" or "&" causes the parameter to expand, before executing the enclosed commands.

Analoguous for "call" - but "call" (contrary to the compound statement) treats all characters behind itself as one parameter string, which is expanded before the commands "within" are executed by call.

(But currently its late so maybe it's just my fantasy .)


penpen

[dbenham]

@Dave: CALL not works in the way expected by jeb (and I)

There are two totally separate mysteries going on.

1) Why does delayed expansion fail when a batch executes (without CALL) another script?

To me, this is the more shocking and important question.

2) Why does CALL strip all content after the first newline?

This is partially explained by phase 2, 3rd bullet, and the fact that CALL re-executes phases 1, 1.5, and 2, all described at http://stackoverflow.com/a/4095133/1012053. But I think there is more to the story, as I show below:

IMHO jeb is wrong when he states that "is possible to pass a multi-line value in a Batch file parameter".

You forget that jeb has a working example at http://stackoverflow.com/a/42880266/1012053

And here is my version that also works

Code: Select all

@echo off
setlocal
(set \n=^
%= This defines a newline character =%
)

set "hat=^"
set "q=""

call receiver %%hat%%%%q%%Line 1%%hat%%%%\n%%%%\n%%Line 2%%hat%%%%q%%

--OUTPUT--

Code: Select all

"Line 1

:(

 REM # Line 1
 Line 2 #
)

The thing that bothers me is that it seems I should be able to expand the newlines above with delayed expansion, and the late appearing caret should protect the newline during the 2nd round of phase 2, thus resulting in the newline being passed. But it doesn't work

Code: Select all

@echo off
setlocal enableDelayedExpansion
cls

(set \n=^
%= This defines a newline character =%
)

set "hat=^"
set "q=""

call receiver %%hat%%%%q%%Line 1%%hat%%!\n!!\n!Line 2%%hat%%%%q%%

--OUTPUT--

Code: Select all

"Line 1

:(

 REM # "Line 1 #
)

Dave Benham

[Aacini]

IMHO jeb is wrong when he states that "is possible to pass a multi-line value in a Batch file parameter".

You forget that jeb has a working example at http://stackoverflow.com/a/42880266/1012053

And here is my version that also works

Code: Select all

@echo off
setlocal
(set \n=^
%= This defines a newline character =%
)

set "hat=^"
set "q=""

call receiver %%hat%%%%q%%Line 1%%hat%%%%\n%%%%\n%%Line 2%%hat%%%%q%%

--OUTPUT--

Code: Select all

"Line 1

:(

 REM # Line 1
 Line 2 #
)

Dave Benham

I think there is a confusion here...

If you read the SO question that jeb refers to in the first post of this thread, you'll realize that the problem stated there is create a disk file with three lines when such lines are stored in a multi-line variable and such value is passed in the parameter of a Batch file (executed via cmd /C batch.bat).

I stated that "there is no way to pass a value with multi-line text in a Batch file parameter", but only if the name of the variable is passed. This is a working example based on the original specifications:

Code: Select all

echo off
setlocal EnableDelayedExpansion

rem Create a variable with just LineFeed
set LF=^
%Don't remove%
%these two lines%

rem Create the variable with multi-line text
set "MultiLineString=Line1: Test.!LF!Line2: Testing.!LF!Line3: Tested."

rem Call the Batch script passing *THE NAME* of the variable
cmd /C batch.bat MultiLineString

This is batch.bat:

Code: Select all

@echo off
setlocal EnableDelayedExpansion

Set a=%~1
echo !%a%!> file.txt

echo File created:
type file.txt

And this is the output:

Code: Select all

File created:
Line1: Test.
Line2: Testing.
Line3: Tested.

On the other hand, jeb stated that "Obviously, there is a way to pass a value with multi-line text in a Batch file parameter". I reviewed his answer and followed the links on it, but I can't find any working code that create such a disk file with three lines as requested in the original question, and your "version that also works" do not create such a file either...

Where is a working example of this method?

Antonio

[dbenham]

Now I understand the source of confusion. It all depends on how you interpret the language.

Jeb correctly states that the parameter with newline is properly passed to the receiving batch script. That is not intended to imply that the receiving batch script can effectively use the entire parameter. Jeb's script (as well as my version) proves that the newline is indeed received by the script, as evidenced by the REM output. But all normal attempts to subsequently use the parameter truncate the value at the first newline.

I believe you are interpreting "pass" to mean that the receiver must be able to interpret and use the content, which is not the intent.

Here is an analogy - An alien from mars might successfully pass me a message (speak to me) in martian. I could prove that I received the message by parroting the exact phrase, sound by sound. But I could not do anything useful with the message because I do not understand martian.

Jeb goes on to explain that the REM output with newline could be parsed, but such parsing would not be reliable if the value also includes < or > (or any other poison character?) Jeb does not actually show the code that can parse the REM value with newline, presumably because it is not always reliable, therefor it is not worth posting.

Here are the complete relevant statements from jeb:

Obviously, there is a way to to pass a value with multi-line text in a Batch file parameter.
But it is a little bit complex and currently there isn't a way to get the content out of the parameter in a safe way for any possible content.

...

To fetch this, the solution from How to receive even the strangest command line parameters? can be extended.

The problem is, that it only works with simple content, but there exists content which can't be received without producing parser errors.
Like: "Line1<line feed>Line2"

Therefore I would recommend Aacini's solution

So jeb's point is an academic one that helps elucidate the underpinnings of the batch parser. But it is not a practical point that can be used in a useful program (at least no one has discovered a use yet). However, understanding the fact that the newline is actually being passed might lead someone to discover a use in the future.


Dave Benham

[jeb]

Thanks Dave,

you get my points.
I'm sorry to choose a really bad example for
1) Why does delayed expansion fail when a batch executes (without CALL) another script?

I should have choose an example with a plain simple variable, not with a line feed

2) Why does CALL strip all content after the first newline?
Some strange behaviour, but I will open a new thread for that.
Examination of Linefeeds with CALL

3) Transfering and accessing line feeds to another batch in a parameter

I think there is a confusion here...

If you read the SO question that jeb refers to in the first post of this thread, you'll realize that the problem stated there is create a disk file with three lines when such lines are stored in a multi-line variable and such value is passed in the parameter of a Batch file (executed via cmd /C batch.bat).

Yes, I'm still not sure, that I understand you.
But I added a working solution to bring a multi line variable into a parameter with a helper function (as I'm understand the problem of the OP was to receive the result not to get the multi line variable into the parameter).

And I added a simple parser

Code: Select all

@echo off
setlocal EnableExtensions DisableDelayedExpansion
prompt :
(
    @echo on
    for %%a in (4) do (
      @goto :next
      rem # %~1#
   )
) > param.tmp
:next
@echo off
endlocal
setlocal DisableDelayedExpansion
set lineNo=0
(
for /F "skip=3 delims=" %%L in (param.tmp) do (
   set /a lineNo+=1
   set "line=%%L"
   setlocal EnableDelayedExpansion
   if !lineNo! EQU 1 (
      set "line=!line:*# =!"
      set "line=!line:~,-2!"
   ) ELSE IF "!line!"==") " (
      goto :exit
   ) ELSE (
      set "line=!line:* =!"
      set "line=!line:~,-1!"
   )
   (echo(!line!)
   endlocal
)
) > file.txt
:exit
type file.txt

But back again to my main issue: Delayed expansion fails in some cases
I made some more experiments

Code: Select all

@echo off
setlocal EnableDelayedExpansion
set "var=content"
echo.bat ---!var!---

bat ---content---

But when echo.bat exists

Code: Select all

@echo on
REM # %~1 #

Then the delayed expansion fails

REM # ---!var!--- #

I think i've read before that delayed Expansion "differs" between command and parameter string (although the consequences were different), but i cannot remember where;
if i had to guess, i would say it was a post/code from jeb/dbenham/both - something like this:

Yes, I described somewhere that the delayed expansion is completly independent for the command token and the remaining parameter tokens.

But your comment inspired me to test another case

Code: Select all

@echo off
setlocal EnableDelayedExpansion
set "var=content"
receiver.bat 1--!var!--

versus
set cmd=receiver.bat
!cmd! 2--!var!--

REM # 1--!var!-- #
versus
REM # 2--content-- #

Strange
jeb

[penpen]

REM # 1--!var!-- #
versus
REM # 2--content-- #

Strange

One or more exclamation marks present within the command string is probably a corner case, where also the complete parameter string is "delayed expanded" immediately (before executing the command):
The delayed variable could easily contain more than just a command token. If you don't expand the rest of the line, than you have to track how much you have "delayed expanded", to proceed from there - in case of a subprocess (external exe); you also have to implement some way to inform the subprocess of such an "half delayed state", which could be messy if some programmer ignore such possibility and so on ... .


penpen

[dbenham]

That makes sense penpen for that one case.

But there is still a mystery as to why the delayed expansion should ever fail

I can't come up with any rational reason for the failed delayed expansion when a script executes another script without using CALL. It seems like the parser must be going out of its way to recognize that the command is another batch script, and it prevents the delayed expansion. I would love to hear the justification from the developer(s) of this "feature".

I've come up with another case where the delayed expansion succeeds:

Code: Select all

@echo off
setlocal disableDelayedExpansion
if "%~1" neq "" echo %*&exit /b

setlocal EnableDelayedExpansion
set a=OK
"%~f0" !a! | findstr "^"

Dave Benham

[penpen]

Your example is nice, although one could argue against that (for example "execution of |"; but i don't do this because of the following).
Your example can be extended to:

Code: Select all

@echo off
setlocal disableDelayedExpansion
if "%~1" == "out" echo %*&exit /b

setlocal EnableDelayedExpansion
set a=OK
"%~f0" out !a! | findstr "^"
("%~f0" out !a!) | findstr "^"

%~1
("%~f0" out !a!)
"%~f0" out !a!

Result:

Code: Select all

Z:\>check.bat ":: First "
out OK
out !a!
out OK

Z:\>check.bat ":: Second ^"
out OK
out !a!
out !a!

The funny thing:

Code: Select all

Not expanded:
("%~f0" out !a!) | findstr "^"
"%~f0" out !a!

Expanded:
"%~f0" out !a! | findstr "^"
("%~f0" out !a!)

Currrently i have no idea how to explain that behaviour... .
Maybe it is a bug that under specific circumstances uses a wrong buffer... - but i have to think more about that;
it (theoretically) also could be some kind of order of operations ("|" versus "()")).


penpen

[penpen]

There could also be either an internal switch to avoid delayed expansion or the exclamation marks must be escaped, see "test.bat":

Code: Select all

@echo off
setlocal enableExtensions disableDelayedExpansion
set "var=#"
set "var2=!var!"

setlocal enableExtensions enableDelayedExpansion
echo !var2!
call echo !var2! ^^^!= #
call call echo !var2! ^^^!= #
endlocal
endlocal
goto :eof

Result:

Code: Select all

Z:\>test.bat
!var!
!var! != #
!var! != #

Maybe one of these is (also) responsible for the above "unexpected" results.


penpen

[dbenham]

The funny thing:

Code: Select all

Not expanded:
("%~f0" out !a!) | findstr "^"
"%~f0" out !a!

Expanded:
"%~f0" out !a! | findstr "^"
("%~f0" out !a!)

Currrently i have no idea how to explain that behaviour... .

Yes, I have been aware of that first case of failure for quite some time.
I thought about bringing that example up, but opted not to because I believe it is an unrelated mechanism.

But there is a perverse beauty in the inverse symmetry between the two expansion failures vs the expansion success

As far as I know, those top two test cases are the only known forms where delayed expansion fails unexpectedly:
- Delayed expansion within parentheses on either side of a pipe
- batch "nakedly" executing another batch with delayed expansion (no call, no parens, no command concatenation (&, &&, or ||), no pipe


I am not seeing the point of your subsequent post


Dave Benham

[jeb]

There could also be either an internal switch to avoid delayed expansion or the exclamation marks must be escaped, see "test.bat":

Your results are the expected ones, as delayed expansion is done only once, in the phases of the CALL there isn't any delayed expansion at all.

jeb