Closing parenthesis prevents escape of subsequent special character operator

[dbenham]

I am not able to escape a special character operator token if it appears immediately after a closing parenthesis. Not sure how this is useful, but I find it interesting. I would expect the escaped forms to generate a syntax error, but rather the "escaped" operator is fully functional

Anyone have any ideas as to the mechanism
Do the phase rules need to be augmented

Code: Select all

@echo on

REM This pair with | is equivalent
(echo Hello) ^| findstr /n .
(echo Hello) | findstr /n .

REM This pair with & is equivalent
(echo Hello) ^& echo World
(echo Hello) & echo World

del test.txt >nul 2>nul
REM This pair with >> is equivalent
(echo Hello) ^>^> test.txt
(echo World) >> test.txt
:: show result
type test.txt


REM This pair with && is equivalent
(echo Hello) ^&^& echo World
(echo Hello) &&  echo World

REM This pair with || is equivalent
(call) ^|^| echo Error
(call) || echo Error

REM But escaped operator that does not immediately follow ) does not work
(call) ^&^& echo OK ^|^| echo Error

--RESULTS--

Code: Select all

D:\test>REM This pair with | is equivalent

D:\test>(echo Hello )  | findstr /n .
1:Hello

D:\test>(echo Hello )  | findstr /n .
1:Hello

D:\test>REM This pair with & is equivalent

D:\test>(echo Hello )  & echo World
Hello
World

D:\test>(echo Hello )  & echo World
Hello
World

D:\test>del test.txt  1>nul 2>nul

D:\test>REM This pair with >> is equivalent

D:\test>(echo Hello ) 1>>test.txt

D:\test>(echo World ) 1>>test.txt

D:\test>type test.txt
Hello
World

D:\test>REM This pair with && is equivalent

D:\test>(echo Hello )  && echo World
Hello
World

D:\test>(echo Hello )  && echo World
Hello
World

D:\test>REM This pair with || is equivalent

D:\test>(call)  || echo Error
Error

D:\test>(call)  || echo Error
Error

D:\test>REM But escaped operator that does not immediately follow ) does not work

D:\test>(call)  && echo OK || echo Error

D:\test>

Dave Benham

[Aacini]

Interesting! It seems that the parsing rules are incomplete...

Code: Select all

@echo on & setlocal & set "prompt=Prompt$G "

(echo Hello) ^| findstr /n .
(echo Hello) | findstr /n .

echo Hello1 ^| findstr /n .
echo Hello1 | findstr /n .

echo Hello2) ^| findstr /n .
echo Hello2) | findstr /n .

(echo Hello3^) ^| findstr /n . )
(echo Hello3^) | findstr /n . )

Output:

Code: Select all

Prompt> (echo Hello )  | findstr /n .
1:Hello

Prompt> (echo Hello )  | findstr /n .
1:Hello

Prompt> echo Hello1 | findstr /n .
Hello1 | findstr /n .

Prompt> echo Hello1   | findstr /n .
1:Hello1

Prompt> echo Hello2) | findstr /n .
Hello2) | findstr /n .

Prompt> echo Hello2)   | findstr /n .
1:Hello2)

Prompt> (echo Hello3) | findstr /n .  )
Hello3) | findstr /n .

Prompt> (echo Hello3)   | findstr /n .  )
1:Hello3)

Antonio

[penpen]

How did you stumble about that?

If i see it right, then the only special character tokens that will be escaped in such a situation are the right parenthesis (')') and the space character (' '):

Code: Select all

Z:\>( (echo some) ^)
")" kann syntaktisch an dieser Stelle nicht verarbeitet werden.

Z:\>(echo some)^ 
" " kann syntaktisch an dieser Stelle nicht verarbeitet werden.

penpen

Edit: Added the space character in the second example manually; wasn't copied when marking and copying it.

[dbenham]

How did you stumble about that?

Ach, I forgot to post a link.

It comes from a StackOverflow Q&A. I almost missed the significance, until AFH posted a comment to my answer.


Dave Benham

[jeb]

wow

This seems to be a new interesting effect of command blocks.
Perhaps there are more effects of the following code.
Does it affect only the directly following characters or also characters later in the line?
(I would like to test it right now, but I'm sitting on a swimming pool the next two weeks)
For me, it looks like an additional parser execution.

jeb

[penpen]

Does it affect only the directly following characters or also characters later in the line?

It only affectes the next token:

Code: Select all

Z:\>(echo Hello) ^&^& echo ^^^world ^&^& echo !
Hello
^world && echo !

penpen

[dbenham]

Does it affect only the directly following characters or also characters later in the line?

I tried to investigate that a bit in my original post, especially with my last example.

It appears to affect only the immediately subsequent token (except for the right parenthesis exception that penpen points out)

It is not just a single character, as ^&^& and ^|^| are not escaped if they follow a closing paren.

But later tokens are escaped. Here are two additional examples that demonstrate this

Code: Select all

Prompt>(echo Hello) ^& echo World ^&^& echo OK ^|^| echo ERR
Hello
World && echo OK || echo ERR

Prompt>
Prompt>(echo Hello) ^& echo World ^> test.txt
Hello
World > test.txt

Prompt>

I've found another oddity - It is not possible to escape redirection if it is at the beginning of a line, no parentheses needed.

Code: Select all

Prompt>del test.txt

Prompt> ^> test.txt echo Hello world

Prompt>type test.txt
Hello world

Prompt>type test.txt

But removing the space after the redirection creates some havoc.

Code: Select all

Prompt> ^>test.txt echo Hello world
'Hello' is not recognized as an internal or external command,
operable program or batch file.

Prompt>dir echo
 Volume in drive D is Local_D_Drive
 Volume Serial Number is 544C-8EF2

 Directory of D:\test

10/24/2017  11:43 AM                 0 echo
               1 File(s)              0 bytes
               0 Dir(s)  81,266,401,280 bytes free

Prompt>

The "test.txt" is swallowed up, with no effect, and echo is treated as the file name. I think I've seen you (jeb) post something similar way back when, but my memory is hazy.

It is possible (but nearly pointless) to escape other operators at the beginning of a line:

Code: Select all

Prompt> ^& echo expected result
'&' is not recognized as an internal or external command,
operable program or batch file.

Prompt> ^| echo expected result
'|' is not recognized as an internal or external command,
operable program or batch file.

Prompt>

Those error messages are very different than the "normal" unescaped forms, as is expected

Code: Select all

Prompt>& echo expected result
& was unexpected at this time.

Prompt>| echo expected result
| was unexpected at this time.

Prompt>

Dave Benham

[penpen]

I've done some tests with circumflex accent at the end of lines (under Win10, 32 bit, Version 10.0.14393); i copied the following test string to the shell:

Code: Select all

(echo test1) ^
&^
& echo success
(echo test2) ^

&^

& echo success
(echo test3) ^

&^
& echo success
(echo test4) ^
&^

& echo success

The result in the shell:

Code: Select all

Z:\>(echo test1) ^
Mehr? &^
Mehr? & echo success
test1
success

Z:\>(echo test2) ^
Mehr?
Mehr? &^
Mehr?
Mehr? & echo success
test2
success

Z:\>(echo test3) ^
Mehr?
Mehr? &^
Mehr? & echo success
test3
Der Befehl "&" ist entweder falsch geschrieben oder
konnte nicht gefunden werden.

Z:\>(echo test4) ^
Mehr? &^
Mehr?
Mehr? & echo success
test4
success

Z:\>

If Running from a batch file (test.bat) the result is:

Code: Select all

Z:\>test.bat

Z:\>(echo test1 )  & echo success
test1
success

Z:\>(echo test2 )  &
 & echo success
test2
success

Z:\>(echo test3 )  & & echo success
test3
Der Befehl "&" ist entweder falsch geschrieben oder
konnte nicht gefunden werden.

Z:\>(echo test4 )  & echo success
test4
success

Z:\>

I'm unsure of what to think of the result from test3.

penpen

[Squashman]

Seems like you could use it as a comment character as long as you put a space after the parentheses but does not work inside a FOR code block escaping it.

Code: Select all

@ECHO OFF
echo (hello
REM comment
:: comment 
) where outfile.txt
echo hello

FOR /L %%G IN (1,1,1) DO (
 ^) comment
echo comment %%G
)

PAUSE

output

Code: Select all

(hello
hello
')' is not recognized as an internal or external command,
operable program or batch file.
comment 1
Press any key to continue . . .

[penpen]

I found a way to make this type of comment work from within a for loop (only tested under win10, 32 bit, version 10.0.14393):

Code: Select all

@ECHO OFF
FOR /L %%G IN (1,1,1) DO (
 (@^) comment
 )
 echo comment %%G
)
PAUSE

The debug output in this case is pretty interesting (CRLF is recognized as a command token - whyever it appears there):

Code: Select all

Z:\>test.bat
Cmd: test.bat  Type: 0
@
  Cmd: ECHO  Type: 0 Args: ` OFF'
FOR %G IN (1 1 1) DO
  (
    CRLF
    (
      @
    Cmd: echo  Type: 0 Args: ` comment %G'
comment 1
Cmd: PAUSE  Type: 0
Drücken Sie eine beliebige Taste . . .

Z:\>

penpen

[dbenham]

Seems like you could use it as a comment character as long as you put a space after the parentheses but does not work inside a FOR code block escaping it.

Yes, back in 2012 I documented most of that behavior when I edited the Phase 2 batch parsing rules to better describe the parentheses behavior.

2) Phase(Special chars)
...

• Process parenthesis (provides for compound statements across multiple lines):
  
  • If the parser is not looking for a command token, then ( is not special.
  • If the parser is looking for a command token and finds (, then start a new compound statement and increment the parenthesis counter
  • If the parenthesis counter is > 0 then ) terminates the compound statement and decrements the parenthesis counter.
  • If the line end is reached and the parenthesis counter is > 0 then the next line will be appended to the compound statement (starts again with phase 1)
  • If the parenthesis counter is = 0, and the parser is looking for a commmand, then ) functions like a REM statement, and all remaining characters on the line are ignored

However, I missed the fact that there must be at least one token delimiter after the ) in order for the comment "feature" to work. I'll edit the SO answer to add that detail.

The debug output in this case is pretty interesting (CRLF is recognized as a command token - whyever it appears there):

We are drifting a bit from the original topic, but the CRLF is simply an indicator that the next line does not terminate the open left parenthesis. The first line without CRLF terminates the preceding left parenthesis.

Code: Select all

@echo off
(
  echo line 1
  echo line 2
  echo line 3
  (
    echo inner 1
    echo inner 2
    echo inner 3
  )
  (
    echo 2nd inner 1
    echo 2nd inner 2
    echo 2nd inner 3
  )
)

-- DEBUG OUTPUT --

Code: Select all

Cmd: test  Type: 0
@
  Cmd: echo  Type: 0 Args: ` off'
(
  CRLF
  Cmd: echo  Type: 0 Args: ` line 1'
  CRLF
  Cmd: echo  Type: 0 Args: ` line 2'
  CRLF
  Cmd: echo  Type: 0 Args: ` line 3'
  CRLF
  (
    CRLF
    Cmd: echo  Type: 0 Args: ` inner 1'
    CRLF
    Cmd: echo  Type: 0 Args: ` inner 2'
    Cmd: echo  Type: 0 Args: ` inner 3'
  (
    CRLF
    Cmd: echo  Type: 0 Args: ` 2nd inner 1'
    CRLF
    Cmd: echo  Type: 0 Args: ` 2nd inner 2'
    Cmd: echo  Type: 0 Args: ` 2nd inner 3'
line 1
line 2
line 3
inner 1
inner 2
inner 3
2nd inner 1
2nd inner 2
2nd inner 3

Dave Benham

[penpen]

We are drifting a bit from the original topic, but the CRLF is simply an indicator that the next line does not terminate the open left parenthesis. The first line without CRLF terminates the preceding left parenthesis.

I first should have explained "whyever it appears there" in more detail, because i think you have misunderstood that part:
Sorry for that ambiguous formulation in my last post.

I don't wanted to say it were surprising that the CRLF appears at all in the parse tree:
The reason probably is that the parse tree is not stored within a pointer structure but in an one dimensional field or array (this information is nearly uninteresting, because it must be stored somehow and i'm sure they have good reasons for doing it that way).

I wanted to say that this CRLF is misplaced:
If you build an interpreter, then it is highly unusual to store information for the instruction after next, because the interpreter has to store such an information two (or more) times (within the parse tree, and when executing the code), which is simply a waste of RAM.
So the expected placement of the CRLF token is after the "not-last-command" instead of in front of it.

If a programmer hasn't done this by purpuse, then it might be a bug and lead the interpreter into an invalid state.
The behaviour described in this topic might be caused by this misplaced CRLF because it could be that the token after the closing parenthesis is scanned twice (first time as a command line parameter and the second time as a special command token).
The misplaced CRLF doesn't seem to harm much more - but who knows.

For example i think it is possible that the behaviour of a label inside a compound statement might also be caused by this misplaced CRLF:

Code: Select all

Z:\>(
Mehr? echo some
Mehr? :label
Mehr? echo text
Mehr? )
(
  CRLF
  Cmd: echo  Type: 0 Args: ` some'
  Cmd: echo  Type: 0 Args: ` text'
some
text

Z:\>(
Mehr?
Mehr?
Mehr? :label
Mehr?
Syntaxfehler.

Z:\>(
Mehr? echo some
Mehr? :label1
Mehr? :label2
Mehr? echo text
Mehr? )
(
  CRLF
  Cmd: echo  Type: 0 Args: ` some'
  CRLF
  Cmd: :label2  Type: 0
  Cmd: echo  Type: 0 Args: ` text'
some
Die Syntax für den Dateinamen, Verzeichnisnamen oder die Datenträgerbezeichnung ist falsch.
text

Z:\>

A right placed CRLF would have avoided the last two errors (as you could see in the first example).

So i'm not sure if we are really drifting away from the topic:
If you "only" want to describe the actual situation, then i agree the misplaced CRLF is unneeded information.
But if you also want to explain why this happens (which might help detecting or excluding additional side effects) then the misplaced CRLF might be important.

penpen

Edit:
Added missing info.

[dbenham]

@penpen - I'm not able to follow the logic of your theory enough to draw any conclusion.

But I have stumbled on some new interesting variants.

Code: Select all

prompt>(rem.) ^& echo escape failed
escape failed

prompt>(rem.) ^&echo escape succeeded
&echo was unexpected at this time.

prompt>(rem.) ^&^& echo escape failed
escape failed

prompt>(rem.) ^&^&echo escape succeeded
&&echo was unexpected at this time.

prompt>(rem.) ^&& escape failed, treated as & &
& was unexpected at this time.

prompt>(rem.) ^&^ escape^ succeeded
& escape succeeded was unexpected at this time.

I'm not sure how to describe the parsing behavior. I have a feel for how it works, but I struggle putting it into words.

[pieh-ejdsch]

Hello Dave,

oh I find it very interesting that the REM lines can now be included in a variable to provide a continuous line in a macro with contained comments, which can also be read and can be entered in any line.

Some characters that allow the second token after REM are escaped whitespaces and following characters, which also work normally.

Code: Select all

rem^ if the second Token will begin with escapeded whitespace the LF will be processed
rem    = , ; space
rem:this second tokens will work normal and escaped not escaped woking tokens:
for %%i in (1) do rem^. : [ ] + \ /&echo next line
set LF=^


set ^"\n=^^^%LF%%LF%^%LF%%LF%^^"
set macro.with.comment=(%\n%rem+this is commented%\n%echo testline%\n%)
for %%i in (1) do %macro.with.comment%
for %%i in (1) do @%macro.with.comment%

Phil

[dbenham]

There is nothing new about REM.
or REM^<space>
or any of the other variants.
viewtopic.php?p=3500#p3500

If you want to put a remark within a macro, then I much prefer the %= REMARK HERE =% syntax, as it only appears in the batch code, and is not reflected in the macro definition in memory. (saves memory)

The point of my previous post was an investigation into what constitutes the first token after the closing parenthesis, and how does that impact whether the special character is escaped or not.

I was just looking for a no-op to go between the parentheses. My examples would work just as well with (CALL ) or (BREAK) or ...


Dave Benham