Examination of Linefeeds with CALL

[jeb]

Hi,

this is a split of the thread Delayed expansion fails in some cases.
As there are multiple different strange things to examine.

Here I want to examine the strange behaviour of linefeeds with CALL statements.

There exists the rule, that an unescaped line feed in the special charater phase will skip the remaining charaters of a line.
That's explains the following behaviour

@echo off
setlocal EnableDelayedExpansion
(set \n=^
%=empty=%
)

echo Test1 fail: %\n% line2
echo Test2 fail: ^%\n% line2
echo Test3 okay: ^%\n%%\n% line2
echo Test4 okay: !\n! line2

When using CALL, we know that the parser do a second round up to the special charater phase.

The previous tests can be simply modified for CALL by doubling the percent signs.
The caret needs a simple trick to transfer it to the second parse round by defining a variable.

Code: Select all

@echo off
(set \n=^
%=empty=%
)

set "caret=^"
call echo Test5 fail: %%\n%% line2
call echo Test6 fail: %%caret%%%%\n%% line2
call echo Test7 okay: %%caret%%%%\n%%%%\n%% line2

This is all known for a long time.

The thing that bothers me is that it seems I should be able to expand the newlines above with delayed expansion, and the late appearing caret should protect the newline during the 2nd round of phase 2, thus resulting in the newline being passed. But it doesn't work

Code: Select all

call echo Test8 fail: %%caret%%!\n!!\n! line2

The line feeds shouldn't stop the parser, as they are escaped in the second round.
First, I thought that prior to the percent expansion there could be "reader phase" that stops at line feeds.
That would be invisible without CALL, as there a line feed is the definition of the end of line.

0.5) Phase(Read line):
Read the line up to the first line feed or end of file

1) Phase(Percent):
A double %% is replaced by a single %
Expansion of argument variables (%1, %2, etc.)
Expansion of %var%, if var does not exists replace it with nothing
For a complete explanation read this from dbenham Same thread: percent expansion

But then I made some more tests, they contradict the idea of phase 0.5

Code: Select all

set "var=#"
call echo Test9 okay: %%var:X=!\n!%% remaing

The parser in the second round should see something like this

echo Test9 okay: %var:X=<linefeed>% remaing

This should also stops the parser in phase 0.5 at the linefeed.

But it's even possible to build something like this

Code: Select all

set "var=#"
call echo Test10 okay: %%caret%%%%var:#=!\n!%%%%var:#=!\n!%% line2
call echo Test11 okay: %%caret%%%%\n:!\n!=!\n!%%%%\n:!\n!=!\n!%% line2

A phase 0.5 can't exists, but my best guess is that the percent phase must be combined with phase 0.5

1) Phase(Percent):
S1: Read next character from input stream (File or CALL-Buffer)
S2: If the character is a line feed stop reading
S3: If the character is a percent goto percent expansion parser
S4: If a carriage return is found goto S1
S5: For any other character put it into the line buffer

(Percent parser): See the explanation of Dave
But additional: Even line feeds can be used here.
But there must be a little difference for the source of characters, as linefeeds are only allowed when they are read from the CALL-line-buffer.

Any other ideas?

jeb

[penpen]

Code: Select all

call echo Test8 fail: %%caret%%!\n!!\n! line2

I might error, but shouldn't this line be processed as follows:

Code: Select all

call echo Test8 fail: %%caret%%!\n!!\n! line2
:: percentage expansion =>
call echo Test8 fail: %caret%!\n!!\n! line2
:: delayed expansion =>
call echo Test8 fail: %caret%

 line2
:: execute "call" => (caret not present at this time, so drop the rest of the line)
echo Test8 fail: %caret%
:: percentage expansion =>
echo Test8 fail: ^
:: execute "echo" =>
Test8 fail: 

(The possible seperation of the delayed expansion of command "call" and its argument string shouldn't effect the result, because the caret isn't present when needed.)

penpen

[dbenham]

I assume you are implying that the old rules posted on SO explain the behavior - I don't see how.

Even if there are multiple mini phases of delayed expansion (one for the command, and another for the arguments) My understanding was both should be completed before the CALL execution of phases 1, 1.5, and 2. The linefeed stripping doesn't occur until phase 2, after the percent expansion. So no, I don't think the old rules can explain that behavior.

[jeb]

(The possible seperation of the delayed expansion of command "call" and its argument string shouldn't effect the result, because the caret isn't present when needed.)

First, I thought this is the cause, too.

But as shown in Test9, 10 and 11 this can't be true.

Code: Select all

@echo off
(set \n=^
%=empty=%
)

set "caret=^"
set "var=#"
call echo Test9 okay: %%var:X=!\n!%% remaing
call echo Test10 okay: %%caret%%%%var:#=!\n!%%%%var:#=!\n!%% line2
call echo Test11 okay: %%caret%%%%\n:!\n!=!\n!%%%%\n:!\n!=!\n!%% line2

Examin Test9

call echo Test9 okay: %%var:X=!\n!%% remaing
:: percentage expansion =>
call echo Test9 okay: %var:X=!\n!% remaing
:: delayed expansion =>
call echo Test9 okay: %var:X=

% remaing
:: execute "call" => (caret not present but the rest of the line isn't dropped !)
call echo Test9 okay: %var:X=

% remaing
:: percentage expansion =>
echo Test9 okay: # remaing
:: execute "echo" =>
Test9 okay: # remaing

The same can be done for Test10 and 11.
It looks like that line feeds are only valid inside percents

jeb

[penpen]

I assume you are implying that the old rules posted on SO explain the behavior - I don't see how.

No, it should be an improvement to the rules at ":: execute "call" => (caret not present at this time, so drop the rest of the line)" (confirming jebs idea with a specific implementation)- in fact that is the only case of the above that contradicts the rules.

Code: Select all

But as shown in Test9, 10 and 11 this can't be true.

Well it still could be true, if the phases are not seperated clearly:
If phase 1-2 could read single characters (from a buffer) it could be that phase 1 allows "\n" within percentage expansion, while phase 2 handles it as a "no go".
So it could be that the interpreter switches between these phases (1, 1, 1, 1, 2, 1.5, 2, 1, 1, ...; sidenote: This is a typical result of implementation parser described as deterministic finite state machines (DFSM)/ pushdown automaton (PDA), so this is very likely.)
Also the sequence of processing within phase 6 might be different - for example "doubling of carets" is done before expanding variables (i didn't test this theory on other examples - just an idea).

I hope i've made no error, and the following example shows, that phase 6 should be incomplete/incorrect:

Code: Select all

@echo off
setlocal enableExtensions disableDelayedExpansion
set "caret=^"

call echo abc ^^^"%%accent%%^

def
endlocal
goto :eof

(Hopefully the) expected result accoring to the actual parser rules (using c++ like notation): "abc ^\"^\r\n"

Code: Select all

call echo abc ^^^"%%accent%%^

def
0)   => "call echo abc ^^^\"%%accent%%^\n\ndef\r\n"
1)   => "call echo abc ^^^\"%accent%^\n\ndef\r\n"
1.5) => "call echo abc ^^^\"%accent%^\n\ndef\n"
2)   => "call echo abc ^\"%accent%\ndef"
6)
   i 1)   => "call echo abc ^\"^\ndef"
     2)   => "call echo abc \"def"
   ii       => " echo abc \"def"
7)   =>  "abc \"def\r\n"
abc "def

Real result ("abc ^\"^\r\n"):

Code: Select all

abc ^"^

penpen

[penpen]

Well i think a modified phase 6 (6*) of the actual parser rules could explain (at least) the given examples:

Code: Select all

if (istreamPhase5.PeekToken("call")) {
   for (char c = istreamPhase5.skip(5).get (); c != '\n' && c != '\0'; c = istreamPhase5.get ()) switch (c) {
      case  '^': line.append("^^"); break;
      case  '%': line.append(expandVariable(istreamP5)); break;
      case '\r': break;
      default:   line.append(c); break;
   };
   ostreamPhase6 << escape(line);
} else ...

Equivalent to:

Code: Select all

Phase 6*) only if command token != "call" skip to 7), else do (i) - (v)
(i)   Remove "call" head and "plain" '\n' tail (out of variable).
(ii)  Double all carets
(iii) Phase 1) Percentage Expansion
(iv)  Phase 1.5) -0x0D (actually i didn't check this)
(v)   Phase 2) Special Chars

Expected Result (using "call"): (I hope the notation i use is obvious.)

Code: Select all

call echo Test5 fail: %%\n%% line2
0)   => "call echo Test5 fail: %%\\n%% line2\r\n"
1)   => "call echo Test5 fail: %\\n% line2\r\n"
1.5) => "call echo Test5 fail: %\\n% line2\n"
2)   => "call echo Test5 fail: %\\n% line2"
6*)
   (i)   => "echo Test5 fail: %\\n% line2"
   (iii) => "echo Test5 fail: \n line2"
   (v)   => "echo Test5 fail: "
7)   => "Test5 fail: \r\n"
===================
call echo Test6 fail: %%caret%%%%\n%% line2
0)   => "call echo Test6 fail: %%caret%%%%\\n%% line2\r\n"
1)   => "call echo Test6 fail: %caret%%\\n% line2\r\n"
1.5) => "call echo Test6 fail: %caret%%\\n% line2\n"
2)   => "call echo Test6 fail: %caret%%\\n% line2"
6*)
   (i)   => "echo Test6 fail: %caret%%\\n% line2"
   (iii) => "echo Test6 fail: ^\n line2"
   (v)   => "echo Test6 fail:  line2"
7)   => "Test6 fail:  line2\r\n"
===================
call echo Test7 okay: %%caret%%%%\n%%%%\n%% line2
0)   => "call echo Test7 okay: %%caret%%%%\\n%%%%\\n%% line2\r\n"
1)   => "call echo Test7 okay: %caret%%\\n%%\\n% line2\r\n"
1.5) => "call echo Test7 okay: %caret%%\\n%%\\n% line2\n"
2)   => "call echo Test7 okay: %caret%%\\n%%\\n% line2"
6*)
   (i)   => "echo Test7 okay: %caret%%\\n%%\\n% line2"
   (iii) => "echo Test7 okay: ^\n\n line2"
   (v)   => "echo Test7 okay: \n line2"
7)   => "Test7 okay: \n line2\r\n"
===================
call echo Test8 fail: %%caret%%!\n!!\n! line2
0)   => "call echo Test8 fail: %%caret%%!\\n!!\\n! line2\r\n"
1)   => "call echo Test8 fail: %caret%!\\n!!\\n! line2\r\n"
1.5) => "call echo Test8 fail: %caret%!\\n!!\\n! line2\n"
2)   => "call echo Test8 fail: %caret%!\\n!!\\n! line2"
5)   => "call echo Test8 fail: %caret%\n\n line2"
6*)
   (i)   "echo Test8 fail: %caret%"
   (iii) "echo Test8 fail: ^"
   (v)   "echo Test8 fail: "
7)   => "Test8 fail: \r\n"
===================
call echo Test9 okay: %%var:X=!\n!%% remaing
0)   => "call echo Test9 okay: %%var:X=!\\n!%% remaing\r\n"
1)   => "call echo Test9 okay: %var:X=!\\n!% remaing\r\n"
1.5) => "call echo Test9 okay: %var:X=!\\n!% remaing\n"
2)   => "call echo Test9 okay: %var:X=!\\n!% remaing"
5)   => "call echo Test9 okay: %var:X=\n% remaing"
6*)
   (i)   "echo Test9 okay: %var:X=\n% remaing"
   (iii) "echo Test9 okay: # remaing"
7)   => "Test9 okay: # remaing\r\n"
===================
call echo Test10 okay: %%caret%%%%var:#=!\n!%%%%var:#=!\n!%% line2
0)   => "call echo Test10 okay: %%caret%%%%var:#=!\\n!%%%%var:#=!\\n!%% line2\r\n"
1)   => "call echo Test10 okay: %caret%%var:#=!\\n!%%var:#=!\\n!% line2\r\n"
1.5) => "call echo Test10 okay: %caret%%var:#=!\\n!%%var:#=!\\n!% line2\n"
2)   => "call echo Test10 okay: %caret%%var:#=!\\n!%%var:#=!\\n!% line2"
5)   => "call echo Test10 okay: %caret%%var:#=\n%%var:#=\n% line2"
6*)
   (i)   "echo Test10 okay: %caret%%var:#=\n%%var:#=\n% line2"
   (iii) "echo Test10 okay: ^\n\n line2"
   (v)   "echo Test10 okay: \n line2"
7)   => "echo Test10 okay: \n line2\r\n"
===================
call echo Test11 okay: %%caret%%%%\n:!\n!=!\n!%%%%\n:!\n!=!\n!%% line2
0)   => "call echo Test11 okay: %%caret%%%%\\n:!\\n!=!\\n!%%%%\\n:!\\n!=!\\n!%% line2\r\n"
1)   => "call echo Test11 okay: %caret%%\\n:!\\n!=!\\n!%%\\n:!\\n!=!\\n!% line2\r\n"
1.5) => "call echo Test11 okay: %caret%%\\n:!\\n!=!\\n!%%\\n:!\\n!=!\\n!% line2\n"
2)   => "call echo Test11 okay: %caret%%\\n:!\\n!=!\\n!%%\\n:!\\n!=!\\n!% line2"
5)   => "call echo Test11 okay: %caret%%\\n:\n=\n%%\\n:\n=\n% line2"
6*)
   (i)   "echo Test11 okay: %caret%%\\n:\n=\n%%\\n:\n=\n% line2"
   (iii) "echo Test11 okay: ^\n\n line2"
   (v)   "echo Test11 okay: \n line2"
7)   => "Test11 okay: \n line2\r\n"
===================
call echo abc ^^^"%%accent%%^

def
0)   => "call echo abc ^^^\"%%accent%%^\n\ndef\r\n"
1)   => "call echo abc ^^^\"%accent%^\n\ndef\r\n"
1.5) => "call echo abc ^^^\"%accent%^\n\ndef\n"
2)   => "call echo abc ^\"%accent%\ndef"
6*)
   (i)   "echo abc ^\"%accent%"
   (ii)  "echo abc ^^\"%accent%"
   (iii) "echo abc ^^\"^"
   (v)   "echo abc ^\"^"
7)   =>  "abc ^\"^\r\n"

penpen

Edit: Replaced this (wrong) implementation with the right above: (differs in step (i): "(i) Remove "call" token (and spacer)").
Removed a now unneccessary info.
Corrected the wrong expansion (example .

[dbenham]

Unless I am missing something, the central problem is still the different results of tests 7 and 8.

If you disregard the opening "testN okay/fail" strings, then your traces of tests 7 and 8 are identical starting at 6-iii, but the final outcomes are different. I don't see anything in what you have written that can explain the difference.

I also don't see what you have changed, other than being more precise with your listing of the order of operations in phase 6. jeb's SO write-up talks about looping back to phases 1 and 2 prior to doubling the carets, but obviously that is simply imprecise notation, because he also talks about how the doubled carets are reduced back to single carets in phase 2. That can't happen unless the doubling happens before re-execution of phases 1 and 2.


Dave Benham

[penpen]

@dbenham:
I've tricked myself... or more precise:
I've jumbled up my different expansion examples... written examples to files not matching the rules at the top.

I had to redo these examples again, until i've found the (last - what else) one producing the desired results... (if there is still an error, i will review tomorrow...), scetched (ugly version of) phase 6:

Code: Select all

if (istreamPhase5.PeekToken("call")) {
   for (char c = istreamPhase5.skip(5).get (); c != '\n' && c != \0'; c = istreamPhase5.get ()) switch (c) {
      case  '^': line.append("^^"); break;
      case  '%': line.append(expandVariable(istreamP5)); break;
      case '\r': break;
      default:   line.append(c); break;
   };
   ostreamPhase6 << escape(line);
} else ...

Cuts off at first '\n' character outside a variable.
(Edit:) Or as jeb said: "It looks like that line feeds are only valid inside percents - should have re-read this sentence 3 hours earlier. (End of edit.)
I have correct the above post.


penpen

[jeb]

I'm still thinking that this is true.

A phase 0.5 can't exists, but my best guess is that the percent phase must be combined with phase 0.5

1) Phase(Percent):
S1: Read next character from input stream (File or CALL-Buffer)
S2: If the character is a line feed stop reading
S3: If the character is a percent goto percent expansion parser
S4: If a carriage return is found goto S1
S5: For any other character put it into the line buffer

(Percent parser): See the explanation of Dave
But additional: Even line feeds can be used here.
But there must be a little difference for the source of characters, as linefeeds are only allowed when they are read from the CALL-line-buffer.

I build some test cases to check if the parser stops at a line feed or if it only drops the rest of the line but parses anyway the line.

Code: Select all

@echo off
setlocal EnableDelayedExpansion

REM *** Prepare the cmdcmdline full lineor
set "ccl=!cmdcmdline!"            
set "replace=stop     -"
for /L %%A in (9,1,30) do if "!ccl:~%%A,1!" NEQ "" set "replace=!replace! "
echo !cmdcmdline:%ccl%=%replace%! > nul

(set \n=^
%=empty=%
)

set "undef="
set "def=DEFINED"
set "var=VAR"

echo  Test1: Check full lineor %cmdcmdline:*-=full line-%
echo Parser: !cmdcmdline!!\n!
REM Reset the full lineor %cmdcmdline:*-=stop     -%

echo  Test2: Remove line2, but still parsing %\n% removed %cmdcmdline:*-=full line-%
echo Parser: !cmdcmdline!!\n!
REM Reset the full lineor %cmdcmdline:*-=stop     -%

call echo  Test3: Remove line2, but still parsing %%\n%% removed  %%cmdcmdline:*-=full line-%%
echo Parser: !cmdcmdline!!\n!
REM Reset the full lineor %cmdcmdline:*-=stop     -%

call echo  Test4: Delayed \n, Parser stops !\n! removed %%cmdcmdline:*-=full line-%%
echo Parser: !cmdcmdline!!\n!
REM Reset the full lineor %cmdcmdline:*-=stop     -%

call echo  Test5: undef, parser stops -   %%undef:string=!\n! removed  %%cmdcmdline:~6%%
echo Parser: !cmdcmdline!!\n!
REM Reset the full lineor %cmdcmdline:*-=stop     -%

call echo  Test6: def, parser stops -   %%def:invalid!\n!%%var%% removed  %%cmdcmdline:~6%%
echo Parser: !cmdcmdline!!\n!
REM Reset the full lineor %cmdcmdline:*-=stop     -%

Test1: Check full lineor full line-
Parser: full line-

Test2: Remove line2, but still parsing
Parser: full line-

Test3: Remove line2, but still parsing
Parser: full line-

Test4: Delayed \n, Parser stops
Parser: stop -

Test5: undef, parser stops - string=
Parser: stop -

Test6: def, parser stops - def:invalid
Parser: stop -

Linefeeds stops the parser, but when they are the result from a percent expansion they only remove the rest of the line, but doesn't stop the parser.

I found a new or more less not explained behaviour in the percent/delayed expansion rules

Code: Select all

@echo off
setlocal EnableDelayedExpansion
set "def=DEFINED"
set "var=VAR"

echo Test1: %def:invalid %var% =X
echo Test2: %def:invalid %var% =X%

echo Test3: !def:invalid !var! =X
echo Test4: !def:invalid !var! =X!

Test1/3 shows unexpected behaviour (for me), the parser must have searched the full line for the closing %/!, but after not finding it, it seems the parser jumps back to the first possible break point and restarts the expansion again

@Dave I suppose you should be do one more edit in SO: Percent expansion rules
jeb

[dbenham]

Test1/3 shows unexpected behaviour (for me), the parser must have searched the full line for the closing %/!, but after not finding it, it seems the parser jumps back to the first possible break point and restarts the expansion again

@Dave I suppose you should be do one more edit in SO: Percent expansion rules

Actually, results 1 and 3 are exactly what I expect. Maybe the language could be improved, but I don't think the Percent expansion logic needs to change.

Side note - I never saw any feedback from you regarding my attempted extension of the SO answer to support command line % expansion. It looks good to me, but I would like your seal of approval

I'll just look at Test 1. I've added one other variable, just to make sure the logic is sound:

Code: Select all

@echo off
setlocal
set "def=DEFINED"
set "def:invalid =ODD"
set "var=VAR"
echo Test1: %def:invalid %var% =X

--OUTPUT--

Code: Select all

Test1: def:invalid VAR =X

1)(Percent) Starting from left, scan each character for %. If found then...

For the remainder of the logic, the parser always remembers the position of this first %. I never state this explicitly, but my logic is dependent on this fact

Step 1 scans until it finds the first %.

From there, we obviously get to 1.3 (expand variable)

Command extensions are disabled, so we skip the first bullet.

- Look at next string of characters, breaking before % : or <LF>, and call them VAR (may be an empty list). If VAR breaks before : and the subsequent character is % then include : as the last character in VAR and break before %.

This looking ahead is a provisional scan, it does not necessarily advance the pointer of the main scan at the beginning of step 1.

At this point, we have tentatively identified the VAR as "def", and the provisional scan position is after the :

- If next character is % then ...

Nope

- Else if next character is : then ...

Nope

- Else if next character is ~ then ...

Nope

- Else if followed by = or *= then ...

Nope

- Else if next string of characters matches pattern of [*]search=[replace]%, where search may include any set of characters except = and <LF>, and replace may include any set of characters except % and <LF>, then replace
%VAR:[*]search=[replace]% with value of VAR after performing search and replace (possibly resulting in empty string) and continue scan

Nope The = is nowhere to be found

- Else goto 1.4

Here is where the rubber hits the road. The entire variable expansion failed, so we resort to the following default behavior of %

1.4 (strip %)
Else if batch mode then Remove % and continue scan

We are back at the original %, it gets stripped, and we resume with the main scan in step 1.

The final result should be obvious at this point.


Dave Benham

[jeb]

Actually, results 1 and 3 are exactly what I expect. Maybe the language could be improved, but I don't think the Percent expansion logic needs to change.

Okay, after reading it again, I see your point.

Side note - I never saw any feedback from you regarding my attempted extension of the SO answer to support command line % expansion. It looks good to me, but I would like your seal of approval
I

Sorry for the long delay
Yes, I suppose it's correct now, for a beginner the text is hard to understand, but I don't know a better concept for a complete description of this part.

- Else if next string of characters matches pattern of [*]search=[replace]%, where search may include any set of characters except = and <LF>, and replace may include any set of characters except % and <LF>, then replace
%VAR:[*]search=[replace]% with value of VAR after performing search and replace (possibly resulting in empty string) and continue scan

Nope The = is nowhere to be found

Nit picking: There is a = but the closing % after the = is missing, but you conclusion is still right.

jeb

[dbenham]

Not sure how this fits into existing theories, but I investigated what happens if \n follows immediately after CALL.

Code: Select all

@echo off
setlocal EnableDelayedExpansion
(set \n=^
%=empty=%
)
set "caret=^"

prompt batch$g
echo on
rem This works
call %%caret%%%%\n%%echo OK

rem The rest do nothing
call %%caret%%!\n!echo OK
call %%caret%%%%\n%%%%\n%%echo OK
call %%caret%%!\n!!\n!echo OK
call %%caret%%%%\n%%%%\n%%
call %%caret%%!\n!!\n!
call %%caret%%%%\n%%
call %%caret%%!\n!

-- OUTPUT --

Code: Select all

batch>rem This works

batch>call %caret%%\n%echo OK
OK

batch>rem The rest do nothing

batch>call %caret%!\n!echo OK

batch>call %caret%%\n%%\n%echo OK

batch>call %caret%!\n!!\n!echo OK

batch>call %caret%%\n%%\n%

batch>call %caret%!\n!!\n!

batch>call %caret%%\n%

batch>call %caret%!\n!

If the \n is not escaped, then it will crash cmd.exe
Each of the following will crash cmd.exe:

Code: Select all

call !\n! Crashes cmd.exe
call %%\n%% Crashes cmd.exe

I get the same results from the command line:

Code: Select all

prompt>cmd /v:on
Microsoft Windows [Version 10.0.15063]
(c) 2017 Microsoft Corporation. All rights reserved.

prompt>set \n=^
More?
More?

prompt>set "caret=^"

prompt>rem This works

prompt>call %^caret%%^\n%echo OK
OK

prompt>rem The rest do nothing

prompt>call %^caret%!\n!echo OK

prompt>call %^caret%%^\n%%^\n%echo OK

prompt>call %^caret%!\n!!\n!echo OK

prompt>call %^caret%%^\n%%^\n%

prompt>call %^caret%!\n!!\n!

prompt>call %^caret%%^\n%

prompt>call %^caret%!\n!

prompt>

Each of the following will crash cmd.exe

Code: Select all

prompt>call !\n! Crashes cmd.exe

Code: Select all

prompt>call %^\n% Crashes cmd.exe

Dave Benham

[dbenham]

I re-examined the behaviors and proposed theories and I think I have a simple set of rule changes that account for the behaviors. It is very similar to what jeb proposed, but just fleshed out a bit more.

Currently phase 1 is defined as:

Phase 1) Percent Expansion
Starting from left, scan each character for %. If found then

• 1.1 (escape %) skipped if command line mode
  • If batch mode and followed by another % then
    Replace %% with single % and continue scan
• 1.2 (expand argument) skipped if command line mode
  • Else if batch mode then
    • If followed by * and command extensions are enabled then
      Replace %* with the text of all command line arguments (Replace with nothing if there are no arguments) and continue scan.
    • Else if followed by <digit> then
      Replace %<digit> with argument value (replace with nothing if undefined) and continue scan.
    • Else if followed by ~ and command extensions are enabled then
      • If followed by optional valid list of argument modifiers followed by required <digit> then
        Replace %~[modifiers]<digit> with modified argument value (replace with nothing if not defined or if specified $PATH: modifier is not defined) and continue scan.
        Note: modifiers are case insensitive and can appear multiple times in any order, except $PATH: modifier can only appear once and must be the last modifier before the <digit>
      • Else invalid modified argument syntax raises fatal error: All parsed commands are aborted, and batch processing aborts if in batch mode!
• 1.3 (expand variable)
  • Else if command extensions are disabled then
    Look at next string of characters, breaking before % or <LF>, and call them VAR (may be an empty list)
    • If next character is % then
      • If VAR is defined then
        Replace %VAR% with value of VAR and continue scan
      • Else if batch mode then
        Remove %VAR% and continue scan
      • Else goto 1.4
    • Else goto 1.4
  • Else if command extensions are enabled then
    Look at next string of characters, breaking before % : or <LF>, and call them VAR (may be an empty list). If VAR breaks before : and the subsequent character is % then include : as the last character in VAR and break before %.
    • If next character is % then
      • If VAR is defined then
        Replace %VAR% with value of VAR and continue scan
      • Else if batch mode then
        Remove %VAR% and continue scan
      • Else goto 1.4
    • Else if next character is : then
      • If VAR is undefined then
        • If batch mode then
          Remove %VAR: and continue scan.
        • Else goto 1.4
      • Else if next character is ~ then
        • If next string of characters matches pattern of [integer][,[integer]]% then
          Replace %VAR:~[integer][,[integer]]% with substring of value of VAR (possibly resulting in empty string) and continue scan.
        • Else goto 1.4
      • Else if followed by = or *= then
        Invalid variable search and replace syntax raises fatal error: All parsed commands are aborted, and batch processing aborts if in batch mode!
      • Else if next string of characters matches pattern of
      • search=[replace]%, where search may include any set of characters except = and <LF>, and replace may include any set of characters except % and <LF>, then replace
        %VAR:
      • search=[replace]% with value of VAR after performing search and replace (possibly resulting in empty string) and continue scan
      • Else goto 1.4
• 1.4 (strip %)
  • Else If batch mode then
    Remove % and continue scan starting with the next character after the %
  • Else preserve % and continue scan starting with the next character after the %

Just two changes are needed to account for all behavior:
1 - The top of Phase 1 needs an extra 1.05 step to truncate at <LF>
2 - Step 1.3 stops at end of buffer rather than <LF> when scanning for variable name.

Phase 1) Percent Expansion
Starting from left, scan each character for % or <LF>. If found then

• 1.05 Terminate line at <LF>
  • If the character is <LF> then
    • Drop (ignore) the remainder of the line from the <LF> onward
    • Goto Phase 1.5 (Strip <CR>)
  • Else the character must be %, so proceed to 1.1
• 1.1 ... No changes
• 1.2 ... No changes
• 1.3 (expand variable)
  • Else if command extensions are disabled then
    Look at next string of characters, breaking before % or end of buffer, and call them VAR (may be an empty list)
    • ... (No change)
  • Else if command extensions are enabled then
    Look at next string of characters, breaking before % : or end of buffer, and call them VAR (may be an empty list). If VAR breaks before : and the subsequent character is % then include : as the last character in VAR and break before %.
    • ... (No change)
• 1.4 ... '(No change)

There is no need to differentiate between the source of the characters because when originally reading from the file, the Phase 0 will already have stopped at the first encountered <LF> (end of line). It doesn't matter if phase 0 strips the <LF> or not, because 1.05 will do it if phase 0 didn't.

The only way for 1.05 to see an internal <LF> is if it is the result of a CALL phase 1 restart.

The <LF> within percents is protected because 1.05 does not apply to the 1.3 percent variable expansion scanning.

jeb or penpen - if you are there - what do you think?


Dave Benham

[dbenham]

I updated the Phase 1 parsing rules on StackOverflow here and here to account for the newline behavior after CALL.

Dave Benham

[penpen]

Somehow i can't memorize the parser rules for long - so i have to familiarize myself with that topic (again), so it might take a while.
But wouldn't rule 1.05 mean that the "def"-part in the following batch file should be dropped (which doesn't happen)?
"test.bat"

Code: Select all

@echo off
setlocal
(set \n=^
%=empty=%
)
echo(abc^%\n%%\n%def

goto :eof

Result (Win 10, x64):

Code: Select all

Z:\>test.bat
abc
def

penpen