Discussion about jeb's batch parsing rules on StackOverflow

[dbenham]

I suppose we are too old to remember all the things we already know.
Do you can remember the batch macros? They are using exactly this effect, when the macro is defined, the LF is injected in an escaped form, but when executing a macro it uses an unescaped LF.

Once I "discovered" the "new" LF behavior, I immediately wondered if I could use it to simplify creation of macros.
I quickly abandoned the thought, and never traced it through enough to realize the macro technique was already using it

I updated the SO post with the new <LF> rules. It wasn't easy - the answer is bumping against the 30,000 character limit.

Dave Benham

[dbenham]

Your !var:~,7! expansion is broken because you forgot to escape the comma, so the expression is split between the command token and the arguments token in phase 2.

Exactly that was my intention, to simply detect, if the character between ECHO and !var is a phase 2 delimiter.

Very nice

I've got two variants of an improved form that somewhat self documents, and it is safe to use additional delayed expansion afterwards.

Code: Select all

@echo off
setlocal enableDelayedExpansion
set "###CMD= "

echo].Variant.1.!###CMD: =ARG^^!......!###CMD: =ARG^^!
echo(.Variant.1.!###CMD: =ARG^^!......!###CMD: =ARG^^!
echo].Variant.2.!###CMD: =ARG!^^!......!###CMD: =ARG!^^!
echo(.Variant.2.!###CMD: =ARG!^^!......!###CMD: =ARG!^^!

--OUTPUT--

Code: Select all

.Variant.1.###CMD: =ARG!......ARG^
.Variant.1.ARG^......ARG^
.Variant.2.###CMD: =ARG......ARG!
.Variant.2.ARG!......ARG!

If the expression is split between the command and argument tokens, then "###CMD:" is the end of the phase 2 command token, and " =ARG!" (or " =ARG") is the beginning of the phase 2 argument token
If entirely within the phase 2 argument token, then it simply expands to "ARG^" (or "ARG!")


Dave Benham

[dbenham]

Back to jeb's phase 7 token delimiter tests. I wondered if <LF> and <CR> might have any effect on any other ECHO white space characters besides <space> and <tab>. I also wanted to cleanup the rules a little bit.

So I did some more tests - What a can of worms this turned out to be

First I will state my new discoveries / rules. Later on I will provide the evidence in the form of scripts and output.

1) I discovered two additional phase 2 token delimiters

• <VT> 0x0B Vertical tab
• <FF> 0x0C Form Feed

So the complete list of standard phase 2 token delimiters is <space> <tab> , ; = <VT> <FF> <NBSP> (Non-Breaking Space 0xFF)
In addition, there is the special command token delimiter ( that only serves as the stop character for a command token.
After discovering these, I decided to test all control characters from 0x01 - 0x1F (test not shown), and I did not find any additional ones.

2) I refined the list of phase 7 command token delimiters

• Class 1 (rarely fails): <space> <tab> , ; = + / [ ]
• Class 2 (sometimes fails): \ . :

Note that the following phase 2 token delimiters are not included: ( <VT> <FF> <LF> <CR> <NBSP>
The command token delimiter is always included in the arguments list for the internal command. How it is interpreted is up to the individual command.

3) The odd phase 7 ECHO behavior that jeb discovered regarding <LF> <CR> and <NBSP> does not work on all machines
For this discussion, I will call the odd behavior "Extended ECHO"
I tested jeb's script on 3 machines, and I was only able to reproduce his Extended ECHO results on one of them.
I cannot figure out any predictor as to when Extended ECHO works

Windows 7 Desktop: Extended ECHO works

Code: Select all

 INFO.BAT version 1.4
--------------------------------------------------------------------------------
Windows version        :  Microsoft Windows [Version 6.1.7601]
Product name           :  Windows 7 Enterprise, 32 bit
Performance indicators :  Processor Cores: 4      Visible RAM: 3659760 kilobytes

Date/Time format       :  (mm/dd/yy)  02/05/2018  12:46:47.06
__APPDIR__             :  C:\Windows\System32\
ComSpec                :  C:\Windows\system32\cmd.exe
PathExt                :  .COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC;.sh;.ksh;.csh;.sed;.awk;.pl
Extensions             :  system: Enabled   user: Enabled 
Delayed expansion      :  system: Disabled  user: Disabled
Locale name            :  en-US       Code Pages: OEM  437    ANSI 1252
DIR  format            :  02/02/2018  09:44 PM     3,747,594,240 pagefile.sys
Permissions            :  Elevated Admin=No, Admin group=No

Windows 7 Laptop: Extended ECHO fails

Code: Select all

 INFO.BAT version 1.4
--------------------------------------------------------------------------------
Windows version        :  Microsoft Windows [Version 6.1.7601]
Product name           :  Windows 7 Enterprise, 32 bit
Performance indicators :  Processor Cores: 8      Visible RAM: 3058776 kilobytes

Date/Time format       :  (mm/dd/yy)  Mon 02/05/2018  12:48:42.19
__APPDIR__             :  C:\windows\system32\
ComSpec                :  C:\windows\system32\cmd.exe
PathExt                :  .COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC
Extensions             :  system: Enabled   user: Enabled 
Delayed expansion      :  system: Disabled  user: Disabled
Locale name            :  en-US       Code Pages: OEM  437    ANSI 1252
DIR  format            :  02/05/2018  07:17 AM     3,132,186,624 pagefile.sys
Permissions            :  Elevated Admin=No, Admin group=No

Windows 10 Desktop: Extended ECHO fails

Code: Select all

 INFO.BAT version 1.4
--------------------------------------------------------------------------------
Windows version        :  Microsoft Windows [Version 10.0.16299.125]
Product name           :  Windows 10 Pro, 64 bit
Performance indicators :  Processor Cores: 4      Visible RAM: 4192432 kilobytes

Date/Time format       :  (mm/dd/yy)  Mon 02/05/2018  23:47:16.85
__APPDIR__             :  C:\WINDOWS\system32\
ComSpec                :  C:\WINDOWS\system32\cmd.exe
PathExt                :  .COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC
Extensions             :  system: Enabled   user: Enabled 
Delayed expansion      :  system: Disabled  user: Disabled
Locale name            :  en-US       Code Pages: OEM  437    ANSI 1252
DIR  format            :  02/04/2018  07:20 PM     4,445,007,872 pagefile.sys
Permissions            :  Elevated Admin=No, Admin group=Yes

                          Missing from the tool collection:  debug

4) I refined the phase 7 rules for Extended ECHO behavior

• The list of phase 7 Extended ECHO delimiters is <LF> <CR> <VT> <FF> <NBSP>
• The Extended ECHO delimiters work only for the ECHO command, and only in phase 7
• If the ECHO command is delimited by an Extended ECHO delimiter in phase 7, then:
  • The Extended ECHO delimiter is output normally - it is not stripped like most command delimiters
  • Each contiguous string of <space> and/or <tab> is collapsed into a single <space>

And below are my "proofs" for the above:

TEST1.bat

Code: Select all

@echo off
setlocal disableDelayedExpansion
cls
for /f "tokens=1-6 delims= " %%A in (
  'forfiles /p "%~dp0." /m "%~nx0" /c "cmd /c echo(0x08 0x09 0x0B 0x0C 0x1A 0xFF"'
) do (
  set "BS=%%A
  set "TAB=%%B"
  set "VT=%%C"
  set "FF=%%D"
  set "SUB=%%E"
  set "NBSP=%%F"
  set "SP= "
  set "SC=;"
  set "COM=,"
  set "EQ=="
  set "LP=("
)
(set LF=^
%= empty line creates <LF> character =%
)
for /F %%# in ('copy /Z "%~dpf0" NUL') do set "CR=%%#"
set "###CMD= "

call :test "%%%%NBSP%%%% = 0xFF" "%NBSP%"
call :test "!NBSP! = 0xFF" !NBSP!
call :test !LF! !LF!
call :test !CR! !CR!
call :test ( (
call :test "!LP! = (" !LP!
call :test "%%%%BS%%%% = 0x08" "%BS%"
call :test "%%%%VT%%%% = 0x0B" "%VT%"
call :test "%%%%FF%%%% = 0x0C" "%FF%"
call :test "%%%%SUB%%%% = 0x1A" "%SUB%"
call :test "!BS! = 0x08" !BS!
call :test "!VT! = 0x0B" !VT!
call :test "!FF! = 0x0C" !FF!
call :test "!SUB! = 0x1A" !SUB!
call :test !SP! !SP!
call :test !TAB! !TAB!
call :test "!SC! = ;" !SC!
call :test "!COM! = ," !COM!
call :test "!EQ! = =" !EQ!
call :test [ [
call :test ] ]
call :test + +
call :test / /
call :test . .
call :test \ \
call :test : :

exit /b
:test
echo(&echo(Test %~1  %~2
setlocal enableDelayedExpansion
<nul set/p"=---"
echo%~2#^ ^ #!TAB!!TAB!!VT!!VT!!FF!!FF!!NBSP!!NBSP!^,^,^;^;^=^=!LF!---!CR!!###CMD: =ARG^^!^
 ^ #!TAB!!TAB!!VT!!VT!!FF!!FF!!NBSP!!NBSP!^,^,^;^;^=^=!LF!---!CR!!###CMD: =ARG^^!
exit /b

test1 output on machine with Extended ECHO support:

Code: Select all

Test %NBSP% = 0xFF   
---#  #         ♂♂♀♀  ,,;;==
ARG^  #         ♂♂♀♀  ,,;;==
ARG^

Test !NBSP! = 0xFF  !NBSP!
--- # # ♂♂♀♀  ,,;;==
###CMD: =ARG! # ♂♂♀♀  ,,;;==
ARG^

Test !LF!  !LF!
---
# # ♂♂♀♀  ,,;;==
###CMD: =ARG! # ♂♂♀♀  ,,;;==
ARG^

Test !CR!  !CR!
# # ♂♂♀♀  ,,;;==
###CMD: =ARG! # ♂♂♀♀  ,,;;==
ARG^

Test (  (
---#  #         ♂♂♀♀  ,,;;==
ARG^  #         ♂♂♀♀  ,,;;==
ARG^

Test !LP! = (  !LP!
---'echo(#' is not recognized as an internal or external command,
operable program or batch file.

Test %BS% = 0x08
---'ech#' is not recognized as an internal or external command,
operable program or batch file.

Test %VT% = 0x0B  ♂
---#  #         ♂♂♀♀  ,,;;==
ARG^  #         ♂♂♀♀  ,,;;==
ARG^

Test %FF% = 0x0C  ♀
---#  #         ♂♂♀♀  ,,;;==
ARG^  #         ♂♂♀♀  ,,;;==
ARG^

Test %SUB% = 0x1A  →
---'echo→#' is not recognized as an internal or external command,
operable program or batch file.

Test !BS! = 0x08  !BS!
---'ech#' is not recognized as an internal or external command,
operable program or batch file.

Test !VT! = 0x0B  !VT!
---♂# # ♂♂♀♀  ,,;;==
###CMD: =ARG! # ♂♂♀♀  ,,;;==
ARG^

Test !FF! = 0x0C  !FF!
---♀# # ♂♂♀♀  ,,;;==
###CMD: =ARG! # ♂♂♀♀  ,,;;==
ARG^

Test !SUB! = 0x1A  !SUB!
---'echo→#' is not recognized as an internal or external command,
operable program or batch file.

Test !SP!  !SP!
---#  #         ♂♂♀♀  ,,;;==
###CMD: =ARG!  #                ♂♂♀♀  ,,;;==
ARG^

Test !TAB!  !TAB!
---#  #         ♂♂♀♀  ,,;;==
###CMD: =ARG!  #                ♂♂♀♀  ,,;;==
ARG^

Test !SC! = ;  !SC!
---#  #         ♂♂♀♀  ,,;;==
###CMD: =ARG!  #                ♂♂♀♀  ,,;;==
ARG^

Test !COM! = ,  !COM!
---#  #         ♂♂♀♀  ,,;;==
###CMD: =ARG!  #                ♂♂♀♀  ,,;;==
ARG^

Test !EQ! = =  !EQ!
---#  #         ♂♂♀♀  ,,;;==
###CMD: =ARG!  #                ♂♂♀♀  ,,;;==
ARG^

Test [  [
---#  #         ♂♂♀♀  ,,;;==
###CMD: =ARG!  #                ♂♂♀♀  ,,;;==
ARG^

Test ]  ]
---#  #         ♂♂♀♀  ,,;;==
###CMD: =ARG!  #                ♂♂♀♀  ,,;;==
ARG^

Test +  +
---#  #         ♂♂♀♀  ,,;;==
###CMD: =ARG!  #                ♂♂♀♀  ,,;;==
ARG^

Test /  /
---#  #         ♂♂♀♀  ,,;;==
###CMD: =ARG!  #                ♂♂♀♀  ,,;;==
ARG^

Test .  .
---#  #         ♂♂♀♀  ,,;;==
###CMD: =ARG!  #                ♂♂♀♀  ,,;;==
ARG^

Test \  \
---#  #         ♂♂♀♀  ,,;;==
###CMD: =ARG!  #                ♂♂♀♀  ,,;;==
ARG^

Test :  :
---#  #         ♂♂♀♀  ,,;;==
###CMD: =ARG!  #                ♂♂♀♀  ,,;;==
ARG^

test1.bat output on a machine without Extended ECHO support:

Code: Select all

Test %NBSP% = 0xFF   
---#  #         ♂♂♀♀  ,,;;==
ARG^  #         ♂♂♀♀  ,,;;==
ARG^

Test !NBSP! = 0xFF  !NBSP!
---'echo' is not recognized as an internal or external command,
operable program or batch file.

Test !LF!  !LF!
---'echo' is not recognized as an internal or external command,
operable program or batch file.

Test !CR!  !CR!
---'echo' is not recognized as an internal or external command,
operable program or batch file.

Test (  (
---#  #         ♂♂♀♀  ,,;;==
ARG^  #         ♂♂♀♀  ,,;;==
ARG^

Test !LP! = (  !LP!
---'echo(#' is not recognized as an internal or external command,
operable program or batch file.

Test %BS% = 0x08
---'ech#' is not recognized as an internal or external command,
operable program or batch file.

Test %VT% = 0x0B  ♂
---#  #         ♂♂♀♀  ,,;;==
ARG^  #         ♂♂♀♀  ,,;;==
ARG^

Test %FF% = 0x0C  ♀
---#  #         ♂♂♀♀  ,,;;==
ARG^  #         ♂♂♀♀  ,,;;==
ARG^

Test %SUB% = 0x1A  →
---'echo→#' is not recognized as an internal or external command,
operable program or batch file.

Test !BS! = 0x08  !BS!
---'ech#' is not recognized as an internal or external command,
operable program or batch file.

Test !VT! = 0x0B  !VT!
---'echo' is not recognized as an internal or external command,
operable program or batch file.

Test !FF! = 0x0C  !FF!
---'echo' is not recognized as an internal or external command,
operable program or batch file.

Test !SUB! = 0x1A  !SUB!
---'echo→#' is not recognized as an internal or external command,
operable program or batch file.

Test !SP!  !SP!
---#  #         ♂♂♀♀  ,,;;==
###CMD: =ARG!  #                ♂♂♀♀  ,,;;==
ARG^

Test !TAB!  !TAB!
---#  #         ♂♂♀♀  ,,;;==
###CMD: =ARG!  #                ♂♂♀♀  ,,;;==
ARG^

Test !SC! = ;  !SC!
---#  #         ♂♂♀♀  ,,;;==
###CMD: =ARG!  #                ♂♂♀♀  ,,;;==
ARG^

Test !COM! = ,  !COM!
---#  #         ♂♂♀♀  ,,;;==
###CMD: =ARG!  #                ♂♂♀♀  ,,;;==
ARG^

Test !EQ! = =  !EQ!
---#  #         ♂♂♀♀  ,,;;==
###CMD: =ARG!  #                ♂♂♀♀  ,,;;==
ARG^

Test [  [
---#  #         ♂♂♀♀  ,,;;==
###CMD: =ARG!  #                ♂♂♀♀  ,,;;==
ARG^

Test ]  ]
---#  #         ♂♂♀♀  ,,;;==
###CMD: =ARG!  #                ♂♂♀♀  ,,;;==
ARG^

Test +  +
---#  #         ♂♂♀♀  ,,;;==
###CMD: =ARG!  #                ♂♂♀♀  ,,;;==
ARG^

Test /  /
---#  #         ♂♂♀♀  ,,;;==
###CMD: =ARG!  #                ♂♂♀♀  ,,;;==
ARG^

Test .  .
---#  #         ♂♂♀♀  ,,;;==
###CMD: =ARG!  #                ♂♂♀♀  ,,;;==
ARG^

Test \  \
---#  #         ♂♂♀♀  ,,;;==
###CMD: =ARG!  #                ♂♂♀♀  ,,;;==
ARG^

Test :  :
---#  #         ♂♂♀♀  ,,;;==
###CMD: =ARG!  #                ♂♂♀♀  ,,;;==
ARG^

TEST2.BAT

Code: Select all

@echo off
setlocal disableDelayedExpansion
cls
for /f "tokens=1-6 delims= " %%A in (
  'forfiles /p "%~dp0." /m "%~nx0" /c "cmd /c echo(0x08 0x09 0x0B 0x0C 0x1A 0xFF"'
) do (
  set "BS=%%A
  set "TAB=%%B"
  set "VT=%%C"
  set "FF=%%D"
  set "SUB=%%E"
  set "NBSP=%%F"
  set "SP= "
  set "SC=;"
  set "COM=,"
  set "EQ=="
  set "LP=("
)
(set LF=^
%= empty line creates <LF> character =%
)
for /F %%# in ('copy /Z "%~dpf0" NUL') do set "CR=%%#"
set "###CMD= "

call :test [
call :test ]
call :test +
call :test !SP!
call :test !TAB!
call :test !EQ!
call :test !COM!
call :test !SC!
call :test !EQ!
call :test !LF!
call :test !CR!
call :test !VT!
call :test !FF!
call :test !NBSP!
exit /b

:test
setlocal enableDelayedExpansion
echo(
echo on
echo%~1 ECHO works
type%~1
set%~1xxx
@echo off
exit /b

test2.bat output on machine with Extended ECHO support

Code: Select all

P:\test>echo[ ECHO works
 ECHO works

P:\test>type[
The system cannot find the file specified.

P:\test>set[xxx
Environment variable [xxx not defined


P:\test>echo] ECHO works
 ECHO works

P:\test>type]
The system cannot find the file specified.

P:\test>set]xxx
Environment variable ]xxx not defined


P:\test>echo+ ECHO works
 ECHO works

P:\test>type+
The system cannot find the file specified.

P:\test>set+xxx
Environment variable +xxx not defined


P:\test>echo!SP! ECHO works
 ECHO works

P:\test>type!SP!
The syntax of the command is incorrect.

P:\test>set!SP!xxx
Environment variable xxx not defined


P:\test>echo!TAB! ECHO works
 ECHO works

P:\test>type!TAB!
The syntax of the command is incorrect.

P:\test>set!TAB!xxx
Environment variable xxx not defined


P:\test>echo!EQ! ECHO works
 ECHO works

P:\test>type!EQ!
The syntax of the command is incorrect.

P:\test>set!EQ!xxx
The syntax of the command is incorrect.


P:\test>echo!COM! ECHO works
 ECHO works

P:\test>type!COM!
The syntax of the command is incorrect.

P:\test>set!COM!xxx
Environment variable ,xxx not defined


P:\test>echo!SC! ECHO works
 ECHO works

P:\test>type!SC!
The syntax of the command is incorrect.

P:\test>set!SC!xxx
Environment variable ;xxx not defined


P:\test>echo!EQ! ECHO works
 ECHO works

P:\test>type!EQ!
The syntax of the command is incorrect.

P:\test>set!EQ!xxx
The syntax of the command is incorrect.


P:\test>echo!LF! ECHO works

 ECHO works

P:\test>type!LF!
'type' is not recognized as an internal or external command,
operable program or batch file.

P:\test>set!LF!xxx
'set' is not recognized as an internal or external command,
operable program or batch file.


P:\test>echo!CR! ECHO works
 ECHO works

P:\test>type!CR!
'type' is not recognized as an internal or external command,
operable program or batch file.

P:\test>set!CR!xxx
'set' is not recognized as an internal or external command,
operable program or batch file.


P:\test>echo!VT! ECHO works
♂ ECHO works

P:\test>type!VT!
'type' is not recognized as an internal or external command,
operable program or batch file.

P:\test>set!VT!xxx
'set' is not recognized as an internal or external command,
operable program or batch file.


P:\test>echo!FF! ECHO works
♀ ECHO works

P:\test>type!FF!
'type' is not recognized as an internal or external command,
operable program or batch file.

P:\test>set!FF!xxx
'set' is not recognized as an internal or external command,
operable program or batch file.


P:\test>echo!NBSP! ECHO works
  ECHO works

P:\test>type!NBSP!
'type' is not recognized as an internal or external command,
operable program or batch file.

P:\test>set!NBSP!xxx
'set' is not recognized as an internal or external command,
operable program or batch file.

TEST3.BAT

Code: Select all

@echo off
setlocal disableDelayedExpansion
cls
for /f "tokens=1-2 delims= " %%A in (
  'forfiles /p "%~dp0." /m "%~nx0" /c "cmd /c echo(0x0B 0x0C"'
) do (
  set "VT=%%A"
  set "FF=%%B"
)
echo on

cmd /c for %%. in (.) do if a%VT%b==a%VT%b echo VT OK
cmd /c for %%. in (.) do if a%FF%b==a%FF%b echo FF OK
cmd /v:on /c for %%. in (.) do if a!VT!b==a!VT!b echo VT OK
cmd /v:on /c for %%. in (.) do if a!FF!b==a!FF!b echo FF OK

test3.bat output

Code: Select all

P:\test>cmd /c for %. in (.) do if a♂b==a♂b echo VT OK
b==a was unexpected at this time.

P:\test>cmd /c for %. in (.) do if a♀b==a♀b echo FF OK
b==a was unexpected at this time.

P:\test>cmd /v:on /c for %. in (.) do if a!VT!b==a!VT!b echo VT OK

P:\test>if a!VT!b == a!VT!b echo VT OK
VT OK

P:\test>cmd /v:on /c for %. in (.) do if a!FF!b==a!FF!b echo FF OK

P:\test>if a!FF!b == a!FF!b echo FF OK
FF OK

Dave Benham

[jeb]

3) The odd phase 7 ECHO behavior that jeb discovered regarding <LF> <CR> and <NBSP> does not work on all machines
For this discussion, I will call the odd behavior "Extended ECHO"
I tested jeb's script on 3 machines, and I was only able to reproduce his Extended ECHO results on one of them.
I cannot figure out any predictor as to when Extended ECHO works

Sorry for that
First I checked it on another machine and it fails there.
Then I created an empty echo.bat and it starts that batch file, then I got the enlightenment

Check your "working" machine with

Code: Select all

where echo

I got
C:\MinGW\msys\1.0\bin\echo.exe
That explains the "odd" effects
Therefore the new rules are obsolete

4) I refined the phase 7 rules for Extended ECHO behavior

[dbenham]

That is a relief
I think I ran into C:\MinGW\msys\1.0\bin\echo.exe before on that same machine when I was doing some unrelated tests years ago. No chance in hell that I would have remembered it and applied it to this situation.
EDIT - On my machine it is "D:\Program Files\Integrity\Toolkit\mksnt\echo.exe" from our code management installation (formerly MKS)

I didn't plan on including "Extended ECHO" rules on StackOverflow anyway, especially with the inconsistencies.

Thanks for figuring this out.

The whole episode does point out something interesting though - The command delimiters for phase 7 internal commands must be different than the command delimiters for phase 7 external command searches
I kind of had a clue to this when I looked at the error messages, but I foolishly thought it was some special behavior of the error message formatter.

How about those new phase 2 delimiters
I had hoped they would be safe with ECHO, but alas, ECHO%VT%/? and ECHO%FF%/? both print ECHO help


Dave Benham

[Sponge Belly]

Hi Dave and Jeb!

How about this for strange behaviour?

multiLine.cmd:

Code: Select all

@echo off & setLocal enableExtensions disableDelayedExpansion

prompt @
for /f "delims=" %%A in ('
    for /f %%B in ("1"^) do rem # %1 #
') do @echo(%%A

echo on
@(
    @for /f "delims=" %%A in ("1") do rem: # %1 #
) | @findStr "^"
@echo off

endLocal & goto :EOF

Input:

Code: Select all

Prompt> multiLine ^"hello^
More?
More? world!"

Output:

Code: Select all

@rem # "hello world!" #

@rem: # "hello & world!" #

Please note that the colon after the second REM is required. Just don’t ask me why!

- SB

[dbenham]

Nothing mysterious about any of that
The existing phase rules already predict all the observed behavior.

I've modified the script so that it is completely self-contained - no need to pass any odd line continuation arguments to the script.
I also added an example that uses ECHO instead of REM

Code: Select all

@echo off & setlocal enableExtensions disableDelayedExpansion & prompt @
(set LF=^
%= Empty line creates linefeed 0x0A character =%
)
for /f "delims=" %%A in ('
  for /f %%B in ("1"^) do rem #1 "Hello%LF%world!" #
') do echo(%%A
(
  for %%A in (.) do rem: #2 "Hello%LF%world!" #
) | findStr "^"
(
  echo #3 "Hello%LF%world!" #
) | findstr "^"

--OUTPUT--

Code: Select all

@rem #1 "Hello world!" #

@rem: #2 "Hello & world!" #
#3 "Hello & world!" #

Explanation:

#1 - This one is quite simple.

• Unescaped <LF> within a FOR IN parenthesized block
  • <LF> is converted into a <space>
  • If at the end of the line buffer, then the next line is read and appended to the current one.

#2 - This one is a bit trickier, as it involves multiple phases.

• Unescaped <LF> within a parenthesized command block
  • <LF> is converted into <LF><space>, and the <space> is treated as part of the next line of the command block.
  • If at the end of line buffer, then the next line is read and appended to the space.

At this point, the code block looks like

Code: Select all

(<LF> for %%A in (.) do rem: #2 "Hello<LF> world!" #<LF> )

Phase 5.3) Pipe processing: Only if commands are on either side of a pipe
Each side of the pipe is processed independently.

• If dealing with a parenthesized command block, then all <LF> with a command before and after are converted to <space>&. Other <LF> are stripped.
• The command (or command block) is executed asynchronously in a new cmd.exe thread via %comspec% /S /D /c" commandBlock".
  This means the command block gets a phase restart, but this time in command line mode.

The parser sees rem: #2 "Hello as one command, and on the next line it sees <space>world!" # as another command. So the intervening <LF> is converted into <space><LF>

The linefeeds after the opening parenthesis and before the closing parenthesis are not between commands, so they each get stripped.

The command then becomes:

Code: Select all

C:\WINDOWS\system32\cmd.exe /S /D /c" ( for %%A in (.) do rem: #2 "Hello & world!" # )"

The REM: command is not recognized as REM in phase 2, so the entire line gets parsed normally during Phase 2 of the command line parsing.
But if you remove the colon, then REM is recognized in phase 2, and the closing parenthesis is treated as part of the remark. So the parenthesized block is never closed, and the block never executes.

You don't have to use REM:
According to the rules in Phase 7.1, you could use any of the following to delay recognition of the internal REM command until phase 7

Code: Select all

rem:
rem.
rem\
rem+
rem/
rem[
rem]

The explanation for #3 is basically the same as #2, except you don't have to worry about ECHO hiding the closing parenthesis


Dave Benham

[Sponge Belly]

Hi Dave,

Thanks for the detailed explanation.

I noticed consecutive LFs were being converted into a single space when I was trying to capture an input parameter inside the in (…) clause of a FOR /F loop using the REM tecchnique.

I tried all kinds of crazy things, including piping the output from the REM command through FindStr, which is when I noticed LFs were being replaced with “ & ”.

My efforts were unsuccessful because I couldn’t find a way to deal with unbalanced quotes.

- SB

[dbenham]

My efforts were unsuccessful because I couldn’t find a way to deal with unbalanced quotes.

One point from Phase 2 that is critical to understanding the behavior is the following:

<LF> always turns off the quote flag.

It is possible to put each block on a single line if you escape the last quote in #2 and #3.

Code: Select all

@echo off & setlocal enableExtensions disableDelayedExpansion & prompt @
(set LF=^
%= Empty line creates linefeed 0x0A character =%
)
(for %%A in (.) do rem: #2 "Hello%LF%world!^" #) | findStr "^"
(echo #3 "Hello%LF%world!^" #) | findstr "^"

The <LF> closes the first quote, so if you don't escape the 2nd quote, then the right parenthesis is quoted and the block never closes.

Code: Select all

@echo off & setlocal enableExtensions disableDelayedExpansion & prompt @
(set LF=^
%= Empty line creates linefeed 0x0A character =%
)
(for %%A in (.) do rem: #2 "This %LF% FAILS" #) | findStr "^"
(echo #3 "This %LF% FAILS" #) | findstr "^"

Dave Benham

[pieh-ejdsch]

I had the function viewtopic.php?f=3&t=6496&p=56356#p56356return in the do. I first tried to replace an exclamation point with percent signs. which is NOT possible anyway:

Code: Select all

set "return.var=%return.var:!=%%5%"

then with delayedexpansion I replaced the variable at the time of execution.

Code: Select all

set "delayed=%%5"
...
set "return.var=%return.var:!=!delayed!%"

That worked well.

Now I wanted to do the whole in a row with call.
the countless attempts with escape characters in several variants have always failed.

Code: Select all

 ...
 call set ^"return.var=%%%%return.var:^!=!delayed!%%%%" ^!
...
call set ^^^"return.var=%%return.var:^^!^=^!delayed^!=%%" ^!

and so on ...
in all variants, which I have tried with call (also nested), the percent sign from the Delayed Variable has been expanded BEFORE executing the command line.
this has led to - that the replacement has NOT taken place.

changing the variable in advance did not change anything:

Code: Select all

 rem set "A=B"
  rem set "X=^!A^!"
  rem call echo !X!
  rem X => B
set "delayedA=%%5"
set "delayed=^!delayedA^!"
...

If delayedExpansion only takes effect shortly before the command is executed - it still resolves after the call but before the variable is resolved in double percent signs???

[dbenham]

If delayedExpansion only takes effect shortly before the command is executed - it still resolves after the call but before the variable is resolved in double percent signs???

I'm not sure of the source of your confusion.

Delayed expansion occurs in phase 5. Then the CALL is processed in phase 6, which kicks of another round of phase 1 percent expansion. Finally in phase 7 the SET command is executed.

So yes, the CALL percent expansion will always occurs after the delayed expansion.

Carets are never used to escape or modify percent expansion. The only way to escape a percent is to double it. But it is impossible to include a percent within the replace string when doing percent expansion find/replace.

You can use % within the find string when doing percent expansion find/replace, but unfortunately the percent cannot be the first character of the find string. The capabilities for delayed expansion are symmetric. You can use ! within the find string when doing delayed expansion find/replace, but the first character of the find string cannot be !. Also, the ! cannot be used in the replace string.

You want to replace ! with %5, and you believe you need to use CALL percent expansion, presumably because you want to do percent expansion within a parenthesized block. But there is no way to add an additional replace after the CALL percent expansion.

The only way forward that I see is to do your replacement in two steps by using some escape sequence that does not involve % or !. I'll arbitrarily use @. But @ could already be a character within your string, so you need a way to escape the @. So now it will require 4 steps. I will use @A for @ and @P for percent.

Code: Select all

set "return.var=!return.var:@=@A!"
call set "return.var=%%return.var:!=@P5%%"
set "return.var=!return.var:@P=%%!"
set "return.var=!return.var:@A=@!"

Dave Benham

[jeb]

Related to a comment from aschipf at stackoverflow:

Phase 7.1: I think it would be worth mentioning that for the special set semantics set "name=content" ignored the command extensions need to be enabled, otherwise a syntax error arises: what do you think, @dbenham or @jeb?

In my opinion, the SET-syntax it's not related to the batch parser phases, more to a special parser of the SET-command.

But while playing with Extensions I was suprised of the following difference

Code: Select all

setlocal DisableExtensions
set "varC"=ccc
setlocal EnableExtensions
set varC
echo The next one fails
set "varD"=ddd

But that set "varD"=ddd fails is an effect of the extended SET syntax, when the assign starts with a quote, then all characters after the last quote are removed.
Therefore the line is equivalent to set "varD" only.
set "varD"=ddd" works again.

With Enabled Extensions it's possible to embedd quotes into the variable name, without extensions this seems not to be possible,
but it's still possible to expand them

Code: Select all

@echo off
setlocal
set "varA="
set "varB="
set "varC="
set "var"D=ddd"

setlocal DisableExtensions
set varA=aaa
for %%V in ("varB""") DO (
	echo %%V
	set %%V=bbb
)
set varC=ccc
echo It's possible to access var^<quote^>D: %var"D%
setlocal EnableExtensions
set var

[penpen]

In my opinion, the SET-syntax it's not related to the batch parser phases, more to a special parser of the SET-command.

I'm unsure, because the SET-statement is an internal command", which in my opinion makes it unlikely that it has an own parser.
Because of that i prefer to view "SET" as a keyword recognized by the batch parser.


penpen