Successfully escape a token delimiter in a fully qualified command path!

[dbenham]

I had always understood that you cannot escape token delimiters - only quotes will work. I suppose I knew that the token delimiter is initially escaped in phase 2, but in a subsequent phase (execution phase 7, possibly others) the token delimiter cannot be escaped. So even if you escape a space in phase 2, it does no good.

But then I saw Stephan's comments at https://stackoverflow.com/q/55978586/1012053.

I did some tests with the following

c:\test\abc xyz.bat

Code: Select all

@echo OK

I was shocked that the following works:

Code: Select all

c:\test\>c:\test\abc^ xyz.bat
OK

But if you remove the leading c:, I get the result I expect:

Code: Select all

C:\test\>\test\abc^ xyz.bat
'\test\abc' is not recognized as an internal or external command,
operable program or batch file.

Note also that the space must be escaped in phase 2. Even a fully qualified path does not work without the escape

Code: Select all

C:\test>c:\test\abc xyz.bat
'c:\test\abc' is not recognized as an internal or external command,
operable program or batch file.

Dave Benham

[dbenham]

Update - The path need not be fully qualified. As long as the volume is specified, you can escape the spaces in the command string - quotes are not needed.

Code: Select all

C:\test>c:abc^ xyz.bat
OK

I suppose I shouldn't be too shocked at the inconsistent behavior (quotes not needed with volume, needed without). The rules for phase 7 are filled with inconsistencies - each internal command can have its own set of parsing rules, and I have never seen a full accounting of the precise rules for determining an external command.

But I am surprised I had never seen this behavior described before.

Now that I think on this a bit more, I realize I have seen similar behavior before - the PATH environment variable can include spaces within a path string without quotes as long as the path does not contain a semicolon. For example, if I have a folder "c:\a b c", then my PATH can be SET "PATH=somePath;c:\a b c;anotherPath". The spaces need not be escaped within PATH because there is no phase 2 when the individuals paths are iterated.

So I guess the behavior should not be a surprise. I had just never thought to try and directly execute a command using such a path, for example c:\a^ b^ c\test.bat.

The only other situations I am aware of that allow spaces within a phase 7 token to be unquoted are the commands CD and PUSHD. The following work just fine without quotes:

Code: Select all

cd a b c
pushd a b c

The difference here is that the spaces need not be escaped for phase 2 because the phase 2 argument tokenization is ignored by the time we get to phase 7. Both CD and PUSHD accept only one argument, so I suppose it makes sense that quotes are not needed.


Dave Benham

[penpen]

I had always understood that you cannot escape token delimiters - only quotes will work.

I'm unsure if i misread something here... although i read the batch parsing rules multiple times, i never noticed that and i always thought the excact opposite.
Therefore i repeatedly used something like that ("abc xyz.bat" == "test this.bat"):

Code: Select all

Z:\test>set test=

Z:\test>>test^ this.txt echo(Hello world!

Z:\test><test^ this.txt set /p "test="

Z:\test>dir /b "test this.txt"
test this.txt

Z:\test>set test
test=Hello world!

Z:\>type "test.bat"
test\test^ this.bat

Z:\>test.bat

Z:\>test\test this.bat
OK

But after testing some other examples, i guess i was lucky on that;

Code: Select all

test\test^ this.bat
Der Befehl "test\test" ist entweder falsch geschrieben oder
konnte nicht gefunden werden.

Z:\test>dir /b test^ this.txt
Datei nicht gefunden

Z:\test>if not exist does^ not^ exist.ext echo ok
ok

Z:\test>if exist test^ this.txt echo ok

Z:\test>if not exist test^ this.txt echo ok
ok

Z:\test>type test^ this.txt
Das System kann die angegebene Datei nicht finden.
Folgender Fehler trat auf: test.
Das System kann die angegebene Datei nicht finden.
Folgender Fehler trat auf: this.txt.

I somehow suspect (can't imagine the above would happen otherwise) that cmd.exe uses (fast and dirty programmed) "setjmp" and "longjmp" with bad side-effects instead of properly handling states with loops and "if" or "switch":
In that case we could end up with tons of special cases, that we even haven't noticed. (Hopefully i am wrong with both.)


penpen

Edit: PS: Actually i can't read the parser rules online, so i'm not sure if my above examples match the issue:
I always get "Oops! Something Bad Happened! ..." page.

[dbenham]

I had always understood that you cannot escape token delimiters - only quotes will work.

I'm unsure if i misread something here... although i read the batch parsing rules multiple times, i never noticed that and i always thought the excact opposite.

No, you did not misread anything. The phase rules definitely imply that you can escape token delimiters in phase 2. I just never thought it could do any good by the time you got to phase 7.


Dave Benham

[jeb]

Hi Dave,

yes, in phase 2 the command token and the parameters are determined, therefore the escaping plays a role. but not for the splitting of the parameters (perhaps in pahse7 ?)
It's also important for the delayed expansion phase, as this phase is executed indipendent for the command and the parameter token.

Code: Select all

setlocal EnableDelayedExpansion
echo^ Remove^ caret"^"-^ by^ delayed^ expansion^ !   But not in the parameter "^"
echo^ Remove^ caret"^"-^ by^ delayed^ expansion !   NOW in the parameter "^"

Remove caret""- by delayed expansion But not in the parameter "^"
Remove caret"^"- by delayed expansion NOW in the parameter ""

The command token in phase2 is used to detect the special internal commands IF, FOR, REM. ( REM is special special!)
Therefore this fails

Code: Select all

IF^ 1==1 echo TRUE
FOR^ %%A in (*) do echo
REM^  /?

But there has to be another splitting in phase 7 (Phase 7-B), else it wouldn't be possible to execute/detect executables or internal commands.
This splitting is done if no external command with the "full name" can't be found.
But even then a second search is done.

Code: Select all

@echo off
setlocal EnableDelayedExpansion
del "echo .bat"
del "echo.bat .bat"
echo^ .bat #1 hello
echo echo #2 THIS IS %%~0 > "echo .bat"
REM echo echo #3 THIS IS %%~0 > "echo.bat .bat"
echo.bat^ .bat #4 hello

.bat #1 hello
bat .bat #4 hello

Code: Select all

@echo off
setlocal EnableDelayedExpansion
del "echo .bat"
del "echo.bat .bat"
echo^ .bat #1 hello
echo echo #2 THIS IS %%~0 > "echo .bat"
echo echo #3 THIS IS %%~0 > "echo.bat .bat"
echo.bat^ .bat #4 hello

.bat #1 hello
#3 THIS IS echo.bat .bat

With your other post, I assume that the delayed expansion is only executed in the Phase7-B.

Code: Select all

setlocal DisableDelayedExpansion
set "myFile=other"
(
echo setlocal DisableDelayedExpansion
echo echo THIS is %%~f0 args=%%*
) > other.bat

setlocal EnableDelayedExpansion
!myFile!.bat Hello"^"!

THIS is c:\temp\empty\other.bat args=Hello""

But with only phase 7-A

Code: Select all

@echo off
setlocal DisableDelayedExpansion
set "myFile=other"
del !myFile!.bat 2>nul
(
echo setlocal DisableDelayedExpansion
echo echo THIS is %%~f0 args=%%*
)  > !myFile!.bat 
(
echo setlocal DisableDelayedExpansion
echo echo THIS is %%~f0 args=%%*
) > other.bat

setlocal EnableDelayedExpansion
!myFile!.bat Hello"^"!

THIS is c:\temp\empty\!myFile!.bat args=Hello"^"!

If the command tokens wasn't handled by delayed expansion (Phase 7-B), then the parameters are also not handled

[jeb]

I tested the behaviour described by Dave.

Code: Select all

c:\temp\with^ space.bat   -  WORKS
\temp\with^ space.bat   -  FAILS
with^ space.bat   -  FAILS

But only when you try it from the command line

Tested in a batch file the results are different

Code: Select all

c:\temp\with^ space.bat   -  WORKS
\temp\with^ space.bat   -  WORKS
with^ space.bat   -  WORKS

[dbenham]

I tested the behaviour described by Dave.

Code: Select all

c:\temp\with^ space.bat   -  WORKS
\temp\with^ space.bat   -  FAILS
with^ space.bat   -  FAILS

But only when you try it from the command line

Tested in a batch file the results are different

Code: Select all

c:\temp\with^ space.bat   -  WORKS
\temp\with^ space.bat   -  WORKS
with^ space.bat   -  WORKS

Yes - I looked at your previous post, and saw that the escape was working for you within a batch script. I then realized we had another difference between batch and command line.

I then reacquainted myself with the phase 6 and 7 rules - I was amazed at how much I had forgotten. And then I spent a number of hours developing a bunch of experiments to try to determine the minimum set of additional rules that accounts for "everything".

I believe all we need is the following addition at the beginning of phase 7.3:

• 7.3 - Execute external command - Else try to treat the command as an external command
  • If in command line mode and the command is not quoted and does not begin with a volume specification, then break the command token at the first occurrence of <space> <comma> <semicolon> or <equal> and prepend the remainder to the argument token(s)
  • ... (the rest of the 7.3 rules go here)

Here is my evidence:

Code: Select all

@echo off
setlocal enableDelayedExpansion

:: Define file names with token delimiter
set files=^
  "echo .bat"^
  "echo.bat .bat"^
  "echo this.bat"^
  "a b.bat"^
  "echo,.bat"^
  "echo;.bat"

:: Defined filenames with escaped token delimiter
set escFiles=%files: =^ %
set escFiles=%escFiles:,=^,%
set escFiles=%escFiles:;=^;%

:: Add = token delimiter to files and escFiles
set files=%files% "echo=.bat"
set escFiles=%escFiles% "echo^=.bat"

:: Create Batch scripts that execute first argument
>exec.bat echo @%%~1
>exec.cmd echo @%%~1

:: Create batch scripts with token delimiter in name
for %%F in (%files%) do >%%F echo @echo #This is "%%~f0" : %%*

echo on

:: Test command line
@for %%F in (%escFiles%) do cmd /c "%%~F CommandLine"

:: Test command line with volume
@for %%F in (%escFiles%) do cmd /c "c:%%~F CommandLineWithVolume"

:: Test .BAT batch
@for %%F in (%escFiles%) do cmd /c exec.bat "%%~F .BAT Batch"

:: Test .CMD batch
@for %%F in (%escFiles%) do cmd /c exec.cmd "%%~F .CMD Batch"

:: Test .BAT batch with parenthesized cmd
@for %%F in (%escFiles%) do cmd /c exec.bat "(%%~F .BAT Batch parenthesized command)"

:: Test .BAT batch with concatenated command
@for %%F in (%escFiles%) do cmd /c exec.bat "verify off&%%~F .BAT Batch concatenated command"

:: Test .BAT batch with IF
@for %%F in (%escFiles%) do cmd /c exec.bat "if 1==1 %%~F .BAT Batch IF"

:: Test .BAT batch with FOR
@for %%F in (%escFiles%) do cmd /c exec.bat "for %%A in (.) do @%%~F .BAT Batch FOR"

:: Test batch call
@set "c=^"
@set escFiles=!escFiles:^^=%%c%%!
@for %%F in (%escFiles%) do call %%~F Batch CALL

:: Test command line call
@set "cc=%%c%%"
@set escFiles=!escFiles:%%c%%=%%cc%%!
@for %%F in (%escFiles%) do cmd /c call %%~F CommandLine CALL

:: Clean up
@del %files% exec.bat exec.cmd

-- OUTPUT --

Code: Select all

C:\test>cmd /c "echo^ .bat CommandLine" 
.bat CommandLine

C:\test>cmd /c "echo.bat^ .bat CommandLine" 
bat .bat CommandLine

C:\test>cmd /c "echo^ this.bat CommandLine" 
this.bat CommandLine

C:\test>cmd /c "a^ b.bat CommandLine" 
'a' is not recognized as an internal or external command,
operable program or batch file.

C:\test>cmd /c "echo^,.bat CommandLine" 
.bat CommandLine

C:\test>cmd /c "echo^;.bat CommandLine" 
.bat CommandLine

C:\test>cmd /c "echo^=.bat CommandLine" 
.bat CommandLine

C:\test>cmd /c "c:echo^ .bat CommandLineWithVolume" 
#This is "C:\test\echo .bat" : CommandLineWithVolume

C:\test>cmd /c "c:echo.bat^ .bat CommandLineWithVolume" 
#This is "C:\test\echo.bat .bat" : CommandLineWithVolume

C:\test>cmd /c "c:echo^ this.bat CommandLineWithVolume" 
#This is "C:\test\echo this.bat" : CommandLineWithVolume

C:\test>cmd /c "c:a^ b.bat CommandLineWithVolume" 
#This is "C:\test\a b.bat" : CommandLineWithVolume

C:\test>cmd /c "c:echo^,.bat CommandLineWithVolume" 
#This is "C:\test\echo,.bat" : CommandLineWithVolume

C:\test>cmd /c "c:echo^;.bat CommandLineWithVolume" 
#This is "C:\test\echo;.bat" : CommandLineWithVolume

C:\test>cmd /c "c:echo^=.bat CommandLineWithVolume" 
#This is "C:\test\echo=.bat" : CommandLineWithVolume

C:\test>cmd /c exec.bat "echo^ .bat .BAT Batch" 
#This is "C:\test\echo .bat" : .BAT Batch

C:\test>cmd /c exec.bat "echo.bat^ .bat .BAT Batch" 
#This is "C:\test\echo.bat .bat" : .BAT Batch

C:\test>cmd /c exec.bat "echo^ this.bat .BAT Batch" 
#This is "C:\test\echo this.bat" : .BAT Batch

C:\test>cmd /c exec.bat "a^ b.bat .BAT Batch" 
#This is "C:\test\a b.bat" : .BAT Batch

C:\test>cmd /c exec.bat "echo^,.bat .BAT Batch" 
#This is "C:\test\echo,.bat" : .BAT Batch

C:\test>cmd /c exec.bat "echo^;.bat .BAT Batch" 
#This is "C:\test\echo;.bat" : .BAT Batch

C:\test>cmd /c exec.bat "echo^=.bat .BAT Batch" 
#This is "C:\test\echo=.bat" : .BAT Batch

C:\test>cmd /c exec.cmd "echo^ .bat .CMD Batch" 
#This is "C:\test\echo .bat" : .CMD Batch

C:\test>cmd /c exec.cmd "echo.bat^ .bat .CMD Batch" 
#This is "C:\test\echo.bat .bat" : .CMD Batch

C:\test>cmd /c exec.cmd "echo^ this.bat .CMD Batch" 
#This is "C:\test\echo this.bat" : .CMD Batch

C:\test>cmd /c exec.cmd "a^ b.bat .CMD Batch" 
#This is "C:\test\a b.bat" : .CMD Batch

C:\test>cmd /c exec.cmd "echo^,.bat .CMD Batch" 
#This is "C:\test\echo,.bat" : .CMD Batch

C:\test>cmd /c exec.cmd "echo^;.bat .CMD Batch" 
#This is "C:\test\echo;.bat" : .CMD Batch

C:\test>cmd /c exec.cmd "echo^=.bat .CMD Batch" 
#This is "C:\test\echo=.bat" : .CMD Batch

C:\test>cmd /c exec.bat "(echo^ .bat .BAT Batch parenthesized command)" 
.bat .BAT Batch parenthesized command

C:\test>cmd /c exec.bat "(echo.bat^ .bat .BAT Batch parenthesized command)" 
bat .bat .BAT Batch parenthesized command

C:\test>cmd /c exec.bat "(echo^ this.bat .BAT Batch parenthesized command)" 
this.bat .BAT Batch parenthesized command

C:\test>cmd /c exec.bat "(a^ b.bat .BAT Batch parenthesized command)" 
'a' is not recognized as an internal or external command,
operable program or batch file.

C:\test>cmd /c exec.bat "(echo^,.bat .BAT Batch parenthesized command)" 
.bat .BAT Batch parenthesized command

C:\test>cmd /c exec.bat "(echo^;.bat .BAT Batch parenthesized command)" 
.bat .BAT Batch parenthesized command

C:\test>cmd /c exec.bat "(echo^=.bat .BAT Batch parenthesized command)" 
.bat .BAT Batch parenthesized command

C:\test>cmd /c exec.bat "verify off&echo^ .bat .BAT Batch concatenated command" 
.bat .BAT Batch concatenated command

C:\test>cmd /c exec.bat "verify off&echo.bat^ .bat .BAT Batch concatenated command" 
bat .bat .BAT Batch concatenated command

C:\test>cmd /c exec.bat "verify off&echo^ this.bat .BAT Batch concatenated command" 
this.bat .BAT Batch concatenated command

C:\test>cmd /c exec.bat "verify off&a^ b.bat .BAT Batch concatenated command" 
'a' is not recognized as an internal or external command,
operable program or batch file.

C:\test>cmd /c exec.bat "verify off&echo^,.bat .BAT Batch concatenated command" 
.bat .BAT Batch concatenated command

C:\test>cmd /c exec.bat "verify off&echo^;.bat .BAT Batch concatenated command" 
.bat .BAT Batch concatenated command

C:\test>cmd /c exec.bat "verify off&echo^=.bat .BAT Batch concatenated command" 
.bat .BAT Batch concatenated command

C:\test>cmd /c exec.bat "if 1==1 echo^ .bat .BAT Batch IF" 
.bat .BAT Batch IF

C:\test>cmd /c exec.bat "if 1==1 echo.bat^ .bat .BAT Batch IF" 
bat .bat .BAT Batch IF

C:\test>cmd /c exec.bat "if 1==1 echo^ this.bat .BAT Batch IF" 
this.bat .BAT Batch IF

C:\test>cmd /c exec.bat "if 1==1 a^ b.bat .BAT Batch IF" 
'a' is not recognized as an internal or external command,
operable program or batch file.

C:\test>cmd /c exec.bat "if 1==1 echo^,.bat .BAT Batch IF" 
.bat .BAT Batch IF

C:\test>cmd /c exec.bat "if 1==1 echo^;.bat .BAT Batch IF" 
.bat .BAT Batch IF

C:\test>cmd /c exec.bat "if 1==1 echo^=.bat .BAT Batch IF" 
.bat .BAT Batch IF

C:\test>cmd /c exec.bat "for %A in (.) do @echo^ .bat .BAT Batch FOR" 
.bat .BAT Batch FOR

C:\test>cmd /c exec.bat "for %A in (.) do @echo.bat^ .bat .BAT Batch FOR" 
bat .bat .BAT Batch FOR

C:\test>cmd /c exec.bat "for %A in (.) do @echo^ this.bat .BAT Batch FOR" 
this.bat .BAT Batch FOR

C:\test>cmd /c exec.bat "for %A in (.) do @a^ b.bat .BAT Batch FOR" 
'a' is not recognized as an internal or external command,
operable program or batch file.

C:\test>cmd /c exec.bat "for %A in (.) do @echo^,.bat .BAT Batch FOR" 
.bat .BAT Batch FOR

C:\test>cmd /c exec.bat "for %A in (.) do @echo^;.bat .BAT Batch FOR" 
.bat .BAT Batch FOR

C:\test>cmd /c exec.bat "for %A in (.) do @echo^=.bat .BAT Batch FOR" 
.bat .BAT Batch FOR

C:\test>call echo%c% .bat Batch CALL 
#This is "C:\test\echo .bat" : Batch CALL

C:\test>call echo.bat%c% .bat Batch CALL 
#This is "C:\test\echo.bat .bat" : Batch CALL

C:\test>call echo%c% this.bat Batch CALL 
#This is "C:\test\echo this.bat" : Batch CALL

C:\test>call a%c% b.bat Batch CALL 
#This is "C:\test\a b.bat" : Batch CALL

C:\test>call echo%c%,.bat Batch CALL 
#This is "C:\test\echo,.bat" : Batch CALL

C:\test>call echo%c%;.bat Batch CALL 
#This is "C:\test\echo;.bat" : Batch CALL

C:\test>call echo%c%=.bat Batch CALL 
#This is "C:\test\echo=.bat" : Batch CALL

C:\test>cmd /c call echo%cc% .bat CommandLine CALL 
#This is "C:\test\echo .bat" : CommandLine CALL

C:\test>cmd /c call echo.bat%cc% .bat CommandLine CALL 
#This is "C:\test\echo.bat .bat" : CommandLine CALL

C:\test>cmd /c call echo%cc% this.bat CommandLine CALL 
#This is "C:\test\echo this.bat" : CommandLine CALL

C:\test>cmd /c call a%cc% b.bat CommandLine CALL 
#This is "C:\test\a b.bat" : CommandLine CALL

C:\test>cmd /c call echo%cc%,.bat CommandLine CALL 
#This is "C:\test\echo,.bat" : CommandLine CALL

C:\test>cmd /c call echo%cc%;.bat CommandLine CALL 
#This is "C:\test\echo;.bat" : CommandLine CALL

C:\test>cmd /c call echo%cc%=.bat CommandLine CALL 
#This is "C:\test\echo=.bat" : CommandLine CALL

The batch parenthesized command, IF command, FOR command, concatenated command results were already predicted by the following in Phase 7.1 - Execute internal command

• Else break the command token before the first occurrence of + / [ ] <space> <tab> , ; or =
  If the preceding text is an internal command, then remember that command
  • If in command line mode, or if the command is from a parenthesized block, IF true or false command block, FOR DO command block, or involved with command concatenation, then execute the internal command

The fact that the escaped token delimiters always work after CALL in both batch and command line was already predicted by Phase 6) CALL processing/Caret doubling

• If the resultant command token is a batch script or a :label, then execution of the CALL is fully handled by the remainder of Phase 6.
  ... Phase 7 is not executed for Called scripts or :labels

I think the modified rules do a good job of predicting behavior, but I hate them. It seems way too complex. I wish we could come up with a simpler set of rules.


Dave Benham

[dbenham]

I updated step 7.3 in the parsing rules on StackOverflow to account for the discussed behavior.

Dave Benham