FOR /F with Line Feed

[Squashman]

I thought I remember seeing this type of syntax in the past but have not been able to get it to work.

Code: Select all

@echo off
setlocal enabledelayedexpansion
set LF=^


REM Do Not remove empty lines above
for /F "tokens=1,2 delims=," %%G in (
"1,Martini Shaker!LF!"
"2,Wine Bottle!LF!"
"3,Beer Glass"
) do (
echo T1=%%~G T2=%%~H
)
pause

My expectation is to have this as the output

Code: Select all

T1=1 T2=Martini Shaker
T1=2 T2=Wine Bottle
T1=3 T2=Beer Glass

But I end up with this.

Code: Select all

T1=1 T2=Martini Shaker
T1= "2 T2=Wine Bottle
T1= "3 T2=Beer Glass

I know this has been done on the forums in the past, I just can't find the thread.

[aGerman]

I think several pairs of quotes won't work. A string literal has only one pair of surrounding quotes.

Code: Select all

@echo off &setlocal
set ^"LF=^

^" Do Not remove the empty line above
set ^"\n=^^^%LF%%LF%^%LF%%LF%^^"
for /F "tokens=1,2 delims=," %%G in (^"^
1^,Martini Shaker%\n%
2^,Wine Bottle%\n%
3^,Beer Glass%\n%
") do (
echo T1=%%~G T2=%%~H
)
pause

Steffen

[npocmaka_]

there was an easier way to the the LF with set and brackets but I cant find the link. If I'm not wrong it was found by Aacini.

[dbenham]

I currently don't have access to a Windows machine at the moment, so I can't test or verify. But I think I see a couple problems with aGerman's code.

1) The last quote must be escaped, else the closing parenthesis is treated as a string literal. I don't think that code will run at all. My mistake. The code should run just fine because the preceding %\n% escapes the end of the line, which is stripped, and then the next character is also escaped, which happens to be the quote.

2) The commas should not require escaping, unless you are worried about the XP FOR /F bug. If you do want the code to be safe on XP, then all token delimiters must be escaped. In this case that would be commas and spaces. Disregarding the XP bug, FOR /F should not require token delimiters to be escaped or quoted. It is only the simple FOR command that uses token delimiters as the iteration delimiter.

If I were to write aGerman's code, I would use USEBACKQ and substitute ' for " so that I don't have to worry about escaping the quotes.

I generally only use the FOR /F <LF> multi line string trick if I have a delimited string that I want to iterate, in which case I would use expansion find/replace to substitute <LF> for the delimiter, all within the IN() clause.

For example (again, untested):

Code: Select all

@echo off
setlocal
set "str=1,Martini Shaker|2,Wine Bottle|3,Beer Glass"

(set LF=^
%= This creates a Line Feed (0x0A) character - DO NOT ALTER =%
)

setlocal enableDelayedExpansion
for %%L in ("!LF!") do for /f "tokens=1,2 delims=," %%A in ("!str:|=%%~L!") do (
  if "!!" == "" endlocal  %= pop SETLOCAL stack only if delayed expansion is enabled to preserve any ! that may be in the strings =%
  echo T1=%%A  T2=%%B
)

If I want to iterate a series of string literals embedded within my source code, then I generally combine the simple FOR with FOR /F. But of course this only works if none of the strings contain * or ?

Code: Select all

@echo off
for %%S in (
  "1,Martini Shaker"
  "2,Wine Bottle"
  "3,Beer Glass"
) do for /f "tokens=1,2 delims=," %%A in ("%%~S") do echo T1=%%A  T2=%%B

Dave Benham

[aGerman]

2) The commas should not require escaping

That was surprising for me, too. And for whatever reason the unescaped spaces work flawlessly even if the commas do not.
Output with unescaped commas (Win10, 32bit):

Code: Select all

T1=1 Martini Shaker T2=
T1=2 Wine Bottle T2=
T1=3 Beer Glass T2=
Drücken Sie eine beliebige Taste . . .

Steffen

[dbenham]

Hmmm. That means the parser must be substituting space for the comma, in which case I suspect that any contiguous string of unquoted, unescaped token delimiters will be replaced by a single space.

I knew this was done when FOR /F iterates the output of a command. I thought this was part of packaging up the IN clause for insertion into a CMD /C command. I didn't think it applied to string processing. But apparently it is universal to all FOR /F IN() clauses.

I don't think the SO command parsing rules address this point properly. I'll have to wade through them again.


Dave Benham

[dbenham]

Well it is embedded in phase 2 after all - and I wrote that portion

FOR is split in two after the DO. A syntax error in the FOR construction will result in a fatal syntax error.

• The portion through DO is the actual FOR iteration command that flows all the way through phase 7
  • All FOR options are fully parsed in phase 2.
  • The IN parenthesized clause treats <LF> as <space>. After the IN clause is parsed, all tokens are concatenated together to form a single token.
  • Consecutive token delimiters collapse into a single space throughout the FOR command through DO.
• The portion after DO is a command block that is parsed normally. Subsequent processing of the DO command block is controled by the iteration in phase 7.

I think the bolded line should be modified to state that only unquoted, unescaped token delimiters are affected


Dave Benham

[Squashman]

there was an easier way to the the LF with set and brackets but I cant find the link. If I'm not wrong it was found by Aacini.

Yes. I that is what I thought as well. I kept trying to find it in his previous posts last night.

[Squashman]

If I want to iterate a series of string literals embedded within my source code, then I generally combine the simple FOR with FOR /F. But of course this only works if none of the strings contain * or ?

Code: Select all

@echo off
for %%S in (
  "1,Martini Shaker"
  "2,Wine Bottle"
  "3,Beer Glass"
) do for /f "tokens=1,2 delims=," %%A in ("%%~S") do echo T1=%%A  T2=%%B

Dave Benham

Yes Dave, that is what I am doing already. I should have put that in my original post. For some reason I thought it was possible to do it with a single FOR /F command. I could have sworn Antonio had posted something like I was trying a few years ago. I even remember posting in the thread.

I did get this to work.

Code: Select all

@echo off
setLocal EnableDelayedExpansion
set lf=^


for /F ^"eol^=^

tokens^=^1^,^2^ delims^=^,^" %%G in ("1,A B!lf!2,C D!lf!3,E F") do echo T1=%%G T2=%%H
pause

But I was hoping to list each grouping on its own line because they will be larger then this example and was hoping for some readability in the code.

[dbenham]

I also vaguely remember Aacini posting something, but I don't remember where, or what it looked like.

Try this. I still can't test, but I think it should work:

Code: Select all

@echo off
setlocal

(set ;=^^^
%= This creates an escaped Line Feed - DO NOT ALTER =%
)

For /f "usebackq tokens=1,2 delims=," %%A in ('^
1^,line 1%;%
2^,line 2%;%
3^,line 3'
) do echo T1=%%A  T2=%%B

The ; variable should contain ^<LF>

So the IN clause should effectively look like ('^string1^<LF><LF>string2^<LF><LF>string3')

[aGerman]

I'm afraid it still doesn't work without escaping the commas, Dave. As you wrote above you don't need to worry about escaping double quotes using USEBACKQ but besides of that it's still basically the same proposal than mine.

Steffen

[dbenham]

Doh! Of course. I edited the post.

The other main difference is the use of a simple definition for ; as opposed to a fairly complex definition for \n

Code: Select all

(set LF=^
%= This creates a Line Feed - DO NOT ALTER =%
)
set ^"\n=^^^%LF%%LF%^%LF%%LF%^^"
REM \n = ^<LF><LF>^

vs.

(set ;=^^^
%= This creates an escaped Line Feed - DO NOT ALTER =%
)
REM ; = ^<LF>

But the end result is the same - the IN() clause is parsed as one string with multiple <LF> terminated lines.

One potential advantage is the use of \n escapes the line feed and the first character of the next line, but the use of ; only escapes the line feed.

I'm thinking this simpler syntax could be used for macro definition as well, though I would probably use \n for the variable name instead of ;

[aGerman]

Your idea of using your technique for macro definitions sounds great. I remember the original explanation by jeb. He told that the \n variable expands as you said but that it would be required because in the end the two line feeds generate a single line feed the same way as you originally define the LF variable. That seems to be wrong or at least not necessary

Hmm, but we are about to hijack Squashman's thread now ...

[Squashman]

Hmm, but we are about to hijack Squashman's thread now ...

All for the greater good.

[dbenham]

I'm finally at a PC, and no, it does not work for macros.

This

Code: Select all

(set \n=^
%= This defines an escaped line feed - DO NOT ALTER =%
)

echo Hello%\n%
world!

Is just a more pleasing way of writing

Code: Select all

echo Hello^

world

Both the above only get parsed once. The same is true with the IN() clause in earlier posts. But the macro must be parsed twice, once to define the macro, and again to execute the macro. So the more complicated definition is required to define a macro.

Dave Benham