Call and goto may fail when the batch file has Unix line endings

[jfl]

Hi,

I had a crazy problem this morning after doing a minor fix of my AutoRun.cmd script for XP.
The fix was working fine. Before committing the updated version, I added a comment in the file header. One last test running 'AutoRun -l' and... BOOM
Of course, I first thought that AutoRun.cmd itself was the culprit, as it's a new and relatively untested program. The problem had to come from one of the AutoRun scripts, like the pid script I had added late yesterday evening.
But removing the AutoRun.cmd.d directory did not help. Uninstalling AutoRun.cmd with 'AutoRun.cmd -u' (using the latest version that still worked!) did not help either.
So I started removing the changes between the last version in GitHub that worked, and the broken version... Until I removed ALL code changes, but 'AutoRun -l' still exploded
The only remaining thing that still differed between the working and failing versions was the comment in the header. I removed it, and suddenly 'AutoRun -l' worked fine again (on Windows 10, but not on XP!).
I reintroduced the code changes for XP. Everything still worked fine on both systems.
Adding back the comment reintroduced the problem.

Then things became really crazy: I retyped the comment in a copy of the old script that worked, and the updated version worked fine.
So I now had two scripts that both fc.exe and WinMerge told me were identical, one that worked, and one that did not!!!!

Then I did a binary comparison, and found the difference:
The failing version had TABs in the fateful comment, whereas the working version only had spaces.
Trying to add or remove spaces in the header comment, I noticed that the first function call ended up in the middle of another line!
And the exact position it arrived at depended on the number of spaces added in the comment, hence the inconsistent error messages I had had until then.

By chance, I eventually noticed that my script had incorrect line endings, with Unix LFs instead of Windows normal CRLFs.
Searching on the Internet, I found a number of people reporting similar symptoms for batch files having Unix line endings.
Changing the line endings indeed fixed the problem for all versions of AutoRun.cmd, whatever the number of spaces in the header comments.
So, good for AutoRun.cmd, I quickly pushed on GitHub an updated version with the right line endings.

Still, I think we have here an opportunity to investigate this issue further.
Contrary to what other people have reported on the Internet, a batch file may work perfectly well with Unix line endings. My AutoRun.cmd script worked fine for days on several machines, running various versions of Windows from 32-bits XP to 64-bits Windows 10.
I'm attaching below the most stripped down version I have now, that still exhibits the problem.
The remaining code is very simple: It sets a few variables, then jumps to the :list function, which in turns call twice the :query function.
Extract AutoRun3.cmd from the attached zip file, and run it in a new cmd.exe window without any argument. (Don't do it in a cmd.exe window with important information displayed, as this may blow up that window!) This displays error:

Code: Select all

C:\JFL\Temp>autorun3
The system cannot find the batch label specified - list

C:\JFL\Temp>

Yet the :list label is present in the file!

If you carefully append one space to one of the header comment lines, you'll see a very different error message:

Code: Select all

C:\JFL\Temp>autorun3
AutoRun scripts:

C:\JFL\Temp>for %h in (HKLM HKCU) do call :query %h

C:\JFL\Temp>call :query HKLM

C:\JFL\Temp>putVarName
'putVarName' is not recognized as an internal or external command,
operable program or batch file.
...

So this time, it finds the :list label, but it's the call to the :query routine that ends up in the middle of the comment line preceding the :query label!

The challenge now is to further simplify the script while still seeing the problem... Until maybe we can understand the root cause.
And maybe we can even use that knowledge to do crazier things

[siberia-man]

Yes. I can confirm this issue with labels. Batch files stored with EOL in Unix style still continue working but fail when jump to a label is needed. I encountered this trouble when stored a batch file to git. (offtop: the line ending is very annoying issue I have ever faced in my practice with git).

[dbenham]

I'm very interested.

I recently saw that old StackOverflow post and was intrigued, since it is an old issue, but I had never run across it before. I tried to investigate, but could not reproduce the problem on my Win 10 machine.

I'm too busy now to investigate myself, but I will monitor this thread. When I get some free time I'll try to contribute.


Dave Benham

[jeb]

Hi jfl,

like Dave I'm very interessted.

And I simplified your code to a minimal, failing example:

Code: Select all

@echo off
goto :zwei
:##############################################################################################################################################################################################################################################################
:##############################################################################################################################################################################################################################################################
:##############################################################################################################################################################################################################################################################
:#######################################################################################################################################################################################################################################

:eins
ECHO eins
goto :eof

:zwei
echo zwei
goto :eins

zwei
Das Sprungziel - eins wurde nicht gefunden.

I suppose I discovered three necessary points to see the effect.

1. line endings have to be unix style, LF only
   • The failing goto has to use lable before the current line
   • The colon of the label has to be on a specific file position, a multiple of 1024

I can't find any relation to TAB characters.

I don't understand, why it should be necessary to be a back jump but my experiments shows exaclty that.

The next sample shows the strange effect, the first time the label :eins is found, but not the second time

Code: Select all

@echo off
goto :eins
:##############################################################################################################################
:##############################################################################################################################
:##############################################################################################################################
:##############################################################################################################################
:##############################################################################################################################
:##############################################################################################################################
:##############################################################################################################################
:#######################################################################################################

:eins
ECHO eins
goto :zwei

:##############################################################################################################################
:##############################################################################################################################
:##############################################################################################################################
:##############################################################################################################################
:##############################################################################################################################
:##############################################################################################################################
:##############################################################################################################################
:#################################################################################################

:zwei
echo zwei
goto :eins

:drei
echo drei
goto :zwei

[dbenham]

I suppose I discovered three necessary points to see the effect.

1. line endings have to be unix style, LF only
   • The failing goto has to use lable before the current line
   • The colon of the label has to be on a specific file position, a multiple of 1024

I don't understand, why it should be necessary to be a back jump but my experiments shows exaclty that.

Wonderful You reproduced the behavior in a reliable way. But thankfully, your rules are not quite right.
However, your rules did enable me to investigate further

The CALL or GOTO can fail on a forward label, but the 1024 boundary condition is relative to the current batch file position, not the start of file. Assuming there is no code block involved, then the file position starts with the character immediately after the CALL or GOTO line.

When jumping backward, the scan has to restart from the beginning of the file. This resets the boundary condition relative to the start of file, which is why the problem is more predictable when jumping backward.

Here is a simple script that demonstrates a failure jumping forward. I used CALL instead of GOTO so I could easily demonstrate how the boundary is relative to the end of the CALL statement. Only the 2nd of three CALLs fails.

test.zip

(185 Bytes) Downloaded 869 times

Code: Select all

@echo off
call :eins A
call :eins B
call :eins C
exit /b
:##############################################################################################################################
:##############################################################################################################################
:##############################################################################################################################
:##############################################################################################################################
:##############################################################################################################################
:##############################################################################################################################
:##############################################################################################################################
:####################################################################################################

:eins
echo %1
exit /b

--OUTPUT--

Code: Select all

A
The system cannot find the batch label specified - eins
C

Dave Benham

[dbenham]

Actually, the boundary condition is any multiple of 512, as reported at http://help.wugnet.com/windows/system-f ... 15555.html.

The rules for CALL or GOTO failure are fairly straightforward, and now make more sense (I feel dirty saying that)

• The file must use Unix style line endings (\n), not Windows (\r\n)
• The target label must span a 512 byte boundary
  • When jumping backward, the boundary is relative to the start of file
  • When jumping forward, the boundary is relative to the current file position when the CALL or GOTO executes

Here is a test demonstrating the 512 byte boundary, jumping forward.

test.zip

(175 Bytes) Downloaded 855 times

Code: Select all

@echo off
call :eins A
call :eins B
call :eins C
exit /b
:##############################################################################################################################
:##############################################################################################################################
:##############################################################################################################################
:####################################################################################################

:eins
echo %1
exit /b

--OUTPUT--

Code: Select all

A
The system cannot find the batch label specified - eins
C

Dave Benham

[dbenham]

I achieved a failure with a proper Windows file with \r\n line endings

This is achieved by placing the label straddling position 512 on a line, with preceding spaces.

Code: Select all

@echo off
call :test505
call :test510
call :test515
exit /b

:: The next line has a label at position 505
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        :test505
echo 505 OK
exit /b

:: The next line has a label at position 510
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             :test510
echo 510 OK
exit /b

:: The next line has a label at position 515
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  :test515
echo 515 OK
exit /b

--OUTPUT--

Code: Select all

505 OK
The system cannot find the batch label specified - test510
515 OK

So now there is only one rule:

- The CALL or GOTO fails if the label spans a 512 byte boundary

The boundary starts off relative to the file pointer when the CALL or GOTO is executed.
The boundary starting position is reset whenever \r\n end-of-line is detected, and also when the scan loops back to the top of file.

Since people are not in the habit of putting labels 500 bytes after the start of a line, this is not an issue with "normal" batch files with \r\n line endings.
But with unix style files, the problem crops up from time to time in real life.


Dave Benham

[jeb]

Thanks Dave,

your findings are impressive .

I didn't see the idea of relative counting the characters
I played with a file with unix style endings and a single CR/LF and thought the problem was gone as I didn't see the reset behaviour.
But your analysis is really good.

And your rule is really simple and make sense (More or less reasonable).

But one more remark:
I suppose the label search function reads the lines in ~512 byte chunks.
Therefore it's possible to place a label after any text

Code: Select all

@echo off

goto :next

:: The next line has a label at position 513 AND it works!
ABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCD:next
echo end

PS: The label can start at position 513 OR 514, to be valid

PPS: Unexpected code obfuscation is possible with this technic

Code: Select all

@echo off

call :test

:: The next line has a label at position 513 AND it works!
ECHO This is a valid command line AND a valid label & REM CDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCD:test
echo It works

jeb

[jeb]

One more observation.

As you already said, the scan for labels is reset at the file beginning or after a CR/LF.
The label scanner reads chunks of ~512 bytes, therefore we see the described effect.
BUT the scanner is able to set the new file position to the wrong position after a label is found

Code: Select all

@echo off
goto :test2
:test2  CDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDAB    ECHO ###########
echo end

It should move the file position to the line "echo end", but it executes the "ECHO #######" at the end of the label line

[jfl]

@jeb and @dbenham: Bravo, you've progressed beyond my wildest dreams, while I was busy on other things. The magic of the Internet

the scanner is able to set the new file position to the wrong position after a label is found

Yes, that's actually the initial failure mode I had, with a call ending up in the middle of another line.
As reported in the initial post, I noticed that by adding or removing spaces in the header comment, I moved the (crash) landing position by the same number of characters.
The failure to reach a label only happened after I started seriously tinkering with the code, adding or removing large blocks of seemingly irrelevant code!

[jfl]

BUT the scanner is able to set the new file position to the wrong position after a label is found

Here, it seems that the line scanner simply assumes that a line cannot be longer than 512 bytes.
And whatever follows is just another line.

[dbenham]

I suppose the label search function reads the lines in ~512 byte chunks.
Therefore it's possible to place a label after any text

Code: Select all

@echo off

goto :next

:: The next line has a label at position 513 AND it works!
ABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCD:next
echo end

PS: The label can start at position 513 OR 514, to be valid

PPS: Unexpected code obfuscation is possible with this technic

Code: Select all

@echo off

call :test

:: The next line has a label at position 513 AND it works!
ECHO This is a valid command line AND a valid label & REM CDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCD:test
echo It works

One more observation.

As you already said, the scan for labels is reset at the file beginning or after a CR/LF.
The label scanner reads chunks of ~512 bytes, therefore we see the described effect.
BUT the scanner is able to set the new file position to the wrong position after a label is found

Code: Select all

@echo off
goto :test2
:test2  CDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDAB    ECHO ###########
echo end

It should move the file position to the line "echo end", but it executes the "ECHO #######" at the end of the label line

Oooh, brilliant
... and wicked

And thanks jfl for bringing this to our attention and providing initial examples.


I'm thinking that these new discoveries should be refined a bit more and incorporated into the Batch Parsing Rules.

I don't have time to do further investigation now, but I hope that these discoveries may help resolve one or both of these unexplained edge cases:

• Delayed expansion fails in some cases Nope, totally unrelated
• Examination of Linefeeds with CALL Nope again

Below is a simple obfuscated batch script with unix \n line endings that demonstrates 4 of the 5 seemingly odd behaviors:

• A seemingly valid label at the beginning of a line that is missed or "invisible"
• A working label that appears in the middle of a line after arbitrary text
• A code line that begins in the middle of a line after arbitrary text
• A single line that functions as both a label and code (actually one label and two lines of code in this case!)

The only behavior missing is an example of seemingly identical CALLs that yield different results

test.zip

(262 Bytes) Downloaded 877 times

Code: Select all

@echo off & goto :test ABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDA:test BCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCECHO How did I get here?!&exit /b
ABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDAB
:test
echo Why am I invisible?
exit /b

--OUTPUT--

Code: Select all

How did I get here?!

Dave Benham

[dbenham]

Just for yucks, I decided to provide one more simple demonstration, and it is a doozy

To casually look at it, you would think there is no way to get the observed behavior. But I used unix format coupled with a strategic number of trailing spaces on lines to bring the 512 byte label boundary into play. Three seemingly identical calls give totally different results.

test.zip

(214 Bytes) Downloaded 894 times

Code: Select all

@echo off
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     
call :MyLabel                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
call :MyLabel                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 
call :MyLabel                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 
exit /b                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       

:MyLabel
echo Label 1         
exit /b
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         
:MyLabel
echo Label 2         
exit /b
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         
:MyLabel
echo Label 3         
exit /b

-- OUTPUT --

Code: Select all

Label 3
Label 2
Label 1

Dave Benham

[penpen]

Offtopic:
It could be usefull to add a couple of test batches to your parser rules (or link them),
just to see if the parser rules have changed between window versions.

I wonder what the stackoverflow users say when you use that example as the first one (with the expected output ).


penpen

[dbenham]

Ooooh, a cmd.exe regression test plan

That would be nice, but I can't imagine actually doing it:
1) It would probably take thousands of individual tests
2) Documenting each test and showing how it relates to the rules would be difficult - there are lots of many to many relationships.

I simply don't have the energy or incentive to undertake such a monumental task.
But I would be thrilled if such a resource existed.


Dave Benham