HOW TO: FOR /F Disabling EOF or using a quote as delim

[jeb]

Hi,

I repeat here some description from other posts, as it's a bit hard to find them anymore, and the corresponding threads are quite long.

I will try to explain some advanced FOR /F tricks/usages

1. Disabling of the EOL character
If the first character of a line is the EOL-character, the line will be ignored.
Sometimes you didn't want any EOL-character, but it's not obvious how to disable it.

As this only sets the EOL-Standard semicolon to a quote character

Code: Select all

FOR /F "EOL=" %%A in ("test") do echo %%A

In my opinion the EOL option reads always the next character and doesn't regards any special characters.

But dbenham found a good trick how to disable it, if you use one ore more delim-characters,
if the EOL character is also a delim character the EOL is disabled at all.

Code: Select all

FOR /F "EOL=x delims=xyz"

But how to disable it, if the delims list is empty?
You could use a line feed character, as this is always the line termination character, the effect is like the EOL is disabled.
But it's not so obvious how to set the EOL to the line feed character, but it's possible.

Code: Select all

FOR /F ^"tokens^=1^,2^ EOL^=^

^" %%A in ...

This isn't a typo, the empty line is necessary, also the carets, as the key is the multiline caret at the end of line1,
but this can only work without quoting, therefore the first quote is escaped too.
But as the quote is escaped all batch line delimiters (not the FOR/F delims-option) are comma, semicolon, tab,space and the equal sign have to be escaped now with a caret.

And then there is the old question: "Is it possible to use the quote itself as a delim-character?"
Yes, but it's much more worse than the EOL-line feed trick.

FOR /F tokens^=1^,2^,3^ delims^=^" %%A in ("one"two"three") do echo %%A--%%B--%%C

This trick is found by pieh-ejdsch at adminstrator.de: FOR loop double quotes as Delimiter

jeb

[dbenham]

And then there is the old question: "Is it possible to use the quote itself as a delim-character?"
Yes, but it's much more worse than the EOL-line feed trick.

FOR /F tokens^=1^,2^,3^ delims^=^" %%A in ("one"two"three") do echo %%A--%%B--%%C

Thanks for finding and sharing that one jeb. I was wondering how to do that.

Dave Benham

[dbenham]

But it's not so obvious how to set the EOL to the line feed character, but it's possible.

Code: Select all

FOR /F ^"tokens^=1^,2^ EOL^=^

^" %%A in ...

This isn't a typo, the empty line is necessary, also the carets, as the key is the multiline caret at the end of line1, but this can only work without quoting, therefore the first quote is escaped too. But as the quote is escaped all batch line delimiters (not the FOR/F delims-option) are comma, semicolon, tab,space and the equal sign have to be escaped now with a caret.

Since this only works without quoting, the quotes aren't needed at all

The EOL=<LF> can be written similarly to the DELIMS=" solution

Code: Select all

for /f tokens^=1^,2^ eol^=^

  %%a in ...

It's even possible to set both EOL=" and DELIMS="

Code: Select all

for /f tokens^=1^,2^ eol^=^"^ delims^=^" %%a in ...

Dave Benham

[Ed Dyreen]

'
I compile a help file

Code: Select all

:: -------------------------------------------------------------------------------------------------------------
:: - The delimiter
:: -------------------------------------------------------------------------------------------------------------
::
:: http://www.dostips.com/forum/viewtopic.php?f=3&t=2385
::
:: (
   @echo off &setlocal DisableDelayedExpansion

   echo.
   echo.sets " as delimiter
   for /f tokens^=1-3^ delims^=^" %%a in ("one"two"three") do echo %%a--%%b--%%c

   echo.
   echo.sets delims to null
   for /f "tokens=1-3 delims=" %%a in ("one"two"three") do echo %%a--%%b--%%c

   echo.
   echo.disables eol because it matches the delim
   for /f "tokens=1-3 eol=: delims=:" %%a in (":one"two"three") do echo %%a--%%b--%%c

   echo.
   echo.sets eol to LF and delims to null
   for /f tokens^=1-3^ eol^=^

   delims^= %%a in (";Please <preserve=^"this line"!>") do echo %%a--%%b--%%c

   for /f ^"tokens^=1-3^ eol^=^

   delims^=^" %%a in (";Please <preserve=^"this line"!>") do echo %%a--%%b--%%c

   pause
   exit

   -- output --
   sets " as delimiter
   one--two--three

   sets delims to null
   one"two"three----

   disables eol because it matches the delim
   one"two"three----

   sets eol to LF and delims to null
   ;Please <preserve=^"this line"!>----
   ;Please <preserve=^"this line"!>----
   -- output --
:: )
:: -------------------------------------------------------------------------------------------------------------

Tags; quote as delim, disable delims, disable eol, eol to linefeed, preserve special characters.

[dbenham]

SS64 forum user carlos published a much simpler method to disable both EOL and DELIMS

Code: Select all

    for /f delims^=^ eol^= %%A in ...

We are all so conditioned to put DELIMS at the end so that we can include space as a delimiter that we never considered putting EOL at the end when DELIMS is disabled. But it is so logical if you think about it.


Dave Benham

[jeb]

Hmm, but this isn't really new, I suppose, I saw this posted before on Dostips.

I thought that Aacini, Ed or you posted this

We are all so conditioned to put DELIMS at the end so that we can include space as a delimiter that we never considered putting EOL at the end when DELIMS is disabled. But it is so logical if you think about it.

It supprised me, that delims isn't set to space in this case, but apperntly it sets it really to nothing.

jeb

[dbenham]

Hmm, but this isn't really new, I suppose, I saw this posted before on Dostips.

I thought that Aacini, Ed or you posted this

We are all so conditioned to put DELIMS at the end so that we can include space as a delimiter that we never considered putting EOL at the end when DELIMS is disabled. But it is so logical if you think about it.

It supprised me, that delims isn't set to space in this case, but apperntly it sets it really to nothing.

jeb

It doesn't surprise me at all. EOL will always take the next character in the option string. But there is no next character because the space is not quoted or escaped.

This works - EOL is disabled (undefined or set to nothing)

Code: Select all

for /f delims^=^ eol^= %a in (" hello world") do @echo %a

None of these work - EOL is set to space or quote

Code: Select all

for /f delims^=^ eol^=^  %a in (" hello world") do @echo %a
for /f "delims= eol= " %a in (" hello world") do @echo %a
for /f "delims= eol=" %a in (""hello world"") do @echo %a

Perhaps someone posted this before, but if so, then I certainly missed it.

Either way, all I know is I am switching to this "new" syntax to disable EOL


Dave Benham

[Ed Dyreen]

It doesn't saves any bytes but it surely is less error prone than ^%LF%%LF%.

Either way, all I know is I am switching to this "new" syntax to disable EOL

I only had to change one location in a single file

Code: Select all

set  ^"forLineR_=for /f delims^^=^^ eol^^= %%r in"
set ^"mForLineR_=for /f delims^^^^=^^^^ eol^^^^= %%r in"

Thanks for sharing

[sst]

Hi Everyone
This is my first post here.

Sorry for bringing up this 6 years old thread but I would like to mention that there is no need to unquote and escape the entire FOR /F parameters.
It is sufficient to put just the eol out of the quote with escaped syntax.

Code: Select all

FOR /F "skip=n tokens=1,2,3 delims=xyz"eol^= %%a in ...

needless to say that the eol must come immediately after closing quote with no spaces between.
This will disable EOL and also avoids the ugly and confusing escaped syntax

Test Script:

Code: Select all

@echo off

setlocal EnableExtensions DisableDelayedExpansion

set "TextBeginningWithSemicolon=;abc xyz"
set "TextBeginningWithSpace= ;abc xyz"

echo,
echo TextBeginningWithSemicolon=%TextBeginningWithSemicolon%
echo Assigning 'TextBeginningWithSemicolon' value to MyVar via FOR /F...

echo,
echo Testing with default EOL...
for /F "tokens=1 delims=" %%a in ("%TextBeginningWithSemicolon%") do set "MyVar=%%a"
if not defined MyVar (
   echo MyVar not defined. This was expected.
) else (
   echo MyVar defined! WTF
)

echo,
echo Testing with supposedly disabled EOL...
for /F "tokens=1 delims="eol^= %%a in ("%TextBeginningWithSemicolon%") do set "MyVar=%%a"
if defined MyVar (
    echo It works! EOL was disabled.
    echo MyVar=%MyVar%

    echo,
    echo I'm Paranoid. Let's make sure EOL was not set to SPACE...
    for /F "tokens=1 delims="eol^= %%a in ("%TextBeginningWithSpace%") do set "MyVar=%%a"
    if defined MyVar (
        echo Yes It really works.
        call echo MyVar=%%MyVar%%
    ) else (
        echo Something is wrong.
    )
    
) else (
    echo It doesnt work :(
)

Output:

Code: Select all

TextBeginningWithSemicolon=;abc xyz
Assigning 'TextBeginningWithSemicolon' value to MyVar via FOR /F...

Testing with default EOL...
MyVar not defined. This was expected.

Testing with supposedly disabled EOL...
It works! EOL was disabled.
MyVar=;abc xyz

I'm Paranoid. Let's make sure EOL was not set to SPACE...
Yes It really works.
MyVar= ;abc xyz

[dbenham]

Very nice
And perfectly logical

Although I still think that looks pretty ugly, and is confusing upon a casual glance. One has to understand how both quotes and escapes work with FOR /F, whereas the other form only requires understanding of escapes.

Perhaps the biggest advantage of this new syntax is that it is easier to type (for me anyway)

Thanks for sharing


Dave Benham

[dbenham]

So here is a test case for the new discovery that gives the expected behavior

Code: Select all

for /f  "tokens=1,2 delims=xyz"eol^= %a in (";abcxyz"^"123^") do @echo [%a] [%b]

-- OUTPUT --

Code: Select all

[;abc] ["123]

Putting an extra character before the eol gives the expected error

Code: Select all

for /f  "tokens=1,2 delims=xyz".eol^= %a in (";abcxyz"^"123^") do @echo [%a] [%b]

-- OUTPUT --

Code: Select all

.eol= was unexpected at this time.

But inserting an escaped quote avoids any error, yet the EOL remains defined as the default semicolon

Code: Select all

for /f "tokens=1,2 delims=xyz"^"eol^= %a in (";abcxyz"123^") do @echo [%a] [%b]

NO OUTPUT


In fact, you get the same result with an escaped quote tacked on the end - EOL does not treat the trailing quote as a value

Code: Select all

for /f "tokens=1,2 delims=xyz"^"eol^=^" %a in (";abcxyz"123^") do @echo [%a] [%b]

NO OUTPUT


In fact, the 2nd quoted section need not use any escapes. The quoted EOL assignment is simply ignored without error

Code: Select all

for /f "tokens=1,2 delims=xyz""eol=a" %a in ("abcxyz"123^") do @echo [%a] [%b]

--OUTPUT--

Code: Select all

[abc] ["123]

But revert to the escaped form and remove the first escaped quote, and EOL is set to a quote

Code: Select all

for /f "tokens=1,2 delims=xyz"eol^=^" %a in (""abcxyz"123") do @echo [%a] [%b]

NO OUTPUT


Dave Benham