Newline character as delimiter in for-loop

[Reino]

I have the following situation:

Code: Select all

youtube-dl.exe -gf 135+141 https://www.youtube.com/watch?v=KpvLMyC4a8o
https://r3---sn-mn4vg5aa-5hns.googlevideo.com/...longurl...
https://r3---sn-mn4vg5aa-5hns.googlevideo.com/...longurl...

When you want youtube-dl to return the videolink of 2 formats of a certain video, it will return the 2nd videolink on a new line.

I have the following test-script:

Code: Select all

@ECHO off

SET youtube-dl="youtube-dl.exe"
SET "url=https://www.youtube.com/watch?v=KpvLMyC4a8o"
SET format=135+141
set LF=^


FOR /F "tokens=1,2" delims=%LF%" %%A IN ('^"%youtube-dl% --no-warnings -gf %format% "%url%"^"') DO (
   SET "_url1=%%A"
   SET "_url2=%%B"
)

ECHO "%_url1%"
ECHO.
ECHO "%_url2%"

PAUSE

I want to put each videolink in a variable, but the linefeed is posing a problem.
Some weeks ago while searching for another issue I stumbled upon a website which showed that with a trick the linefeed character could be used as a delimiter in a for-loop. The problem is, that despite lots of Google-queries, I just can't find that website anymore.
What I do still remember from that trick was that the double quotes surrounding "tokens= delims=" were replaced by lots of carets.
Does anyone here know how to use a newline character as a delimiter in a for-loop?

[Ed Dyreen]

Setting delims to a linefeed has no advantage over simply disabling eol and delims because 'For' can't handle linefeeds.

Code: Select all

@echo off &setlocal enableDelayedExpansion

for /F delims^=^ eol^= %%r in ('ping') DO set /A line[0] += 1 &set "line[!line[0]!]=%%r"

set line

pause
exit

[Reino]

This isn't about ignoring lines with eol=x, but the other way around, so I can't use your code. I want to create a singleline of youtube-dl's multiline output, by using CRLF as a delimiter, so I can "tokenize" it into 2 variables. You're saying I can't?

I've tried...

Code: Select all

FOR /F "tokens=1,2 delims=¶" %%A IN ('^"%youtube-dl% --no-warnings -gf %format% "%url%"^"') DO (

and...

Code: Select all

set LF=^


FOR /F tokens^=1^,2^ delims^=^%LF%^ %%A IN ('^"%youtube-dl% --no-warnings -gf %format% "%url%"^"') DO (

but both to no avail.

In the meantime the following temporary solution would have to do:

Code: Select all

@ECHO off

SET youtube-dl="youtube-dl.exe"
SET jq="jq.exe"
SET "url=https://www.youtube.com/watch?v=KpvLMyC4a8o"
SET format=135+141

FOR /F %%A IN ('^"%youtube-dl% --no-warnings -gf %format% "%url%" ^| %jq% -R -s .^"') DO SET "_url=%%A"
FOR /F "tokens=1,2" %%A IN (%_url:\n= %) DO SET "_url1=%%A" & SET "_url2=%%B"

ECHO "%_url1%"
ECHO.
ECHO "%_url2%"

[ShadowThief]

Why can't you send the output of the command to a temp file and then process the output from there?

Code: Select all

%youtube-dl% --nowarnings -gf %format "%url%" >output.txt

(
    set /p ignore_line=
    set /p _url1=
    set /p _url2=
)<output.txt
del output.txt

[Reino]

Sorry, but I refuse to use temp files.

[Ed Dyreen]

This isn't about ignoring lines with eol=x, but the other way around, so I can't use your code. I want to create a singleline of youtube-dl's multiline output, by using CRLF as a delimiter, so I can "tokenize" it into 2 variables. You're saying I can't?

My code does not ignore any lines with eol=x, in fact both eol and delims are disabled. For always splits when it encounters a linefeed, therefore the effect is exactly the same as what you get when you set 'delims=^%$lf%%$lf%'. And because there is no difference, setting delims to a lineFeed is depriciated. Because for can't handle lineFeeds, your attempt to tokenize multiLine output inevitably fails.

Code: Select all

@echo off &setlocal enableDelayedExpansion &set $lf=^


::
set "$=echo.https://longurlA...&echo.https://longurlB..."

echo. &echo.with eol and delims disabled;
for /F delims^=^ eol^= %%r in ( '!$!' ) DO set /A url[0] += 1 &set "url[!url[0]!]=%%r"
for /L %%i in ( 1, 1, !url[0]! ) do echo.url[%%i]=!url[%%i]!_ &set "url[%%i]="
set /A url[0] = 0

echo. &echo.with eol disabled and delims=lineFeed;
for /F delims^=^%$lf%%$lf%^ eol^= %%r in ( '!$!' ) DO set /A url[0] += 1 &set "url[!url[0]!]=%%r"
for /L %%i in ( 1, 1, !url[0]! ) do echo.url[%%i]=!url[%%i]!_ &set "url[%%i]="
set /A url[0] = 0

pause
exit

Each line is added to an array and you can create a single variable out of it by iterating the array.

Code: Select all

with eol and delims disabled;
url[1]=https://longurlA..._
url[2]=https://longurlB..._

with eol disabled and delims=lineFeed;
url[1]=https://longurlA..._
url[2]=https://longurlB..._
Druk op een toets om door te gaan. . .

you can also use a temp file as ShadowThief suggest.

[Aacini]

Try this:

Code: Select all

SET "_url1="
FOR /F "delims=" %%A IN ('^"%youtube-dl% --no-warnings -gf %format% "%url%"^"') DO (
   IF NOT DEFINED _url1 (
      SET "_url1=%%A"
   ) ELSE (
      SET "_url2=%%A"
   )
)

Antonio

[Reino]

I'm sorry, Ed Dyreen. I didn't understand your code at first.

Code: Select all

@ECHO off

SET youtube-dl="youtube-dl.exe"
SET "url=https://www.youtube.com/watch?v=KpvLMyC4a8o"
SET format=135+141

SETLOCAL ENABLEDELAYEDEXPANSION
FOR /F %%A IN ('^"%youtube-dl% --no-warnings -gf %format% "%url%"^"') DO (
   SET /A _url[0] += 1
   SET "_url[!_url[0]!]=%%A"
)

ECHO "%_url[1]%"
IF DEFINED _url[2] (
   ECHO.
   ECHO "%_url[2]%"
)
ENDLOCAL

This works. BUT, I'm going for Antonio's more straight forward code:

Code: Select all

@ECHO off

SET youtube-dl="youtube-dl.exe"
SET "url=https://www.youtube.com/watch?v=KpvLMyC4a8o"
SET format=135+141

SET "_url1="
FOR /F %%A IN ('^"%youtube-dl% --no-warnings -gf %format% "%url%"^"') DO (
   IF NOT DEFINED _url1 (
      SET "_url1=%%A"
   ) ELSE (
      SET "_url2=%%A"
   )
)

ECHO "%_url1%"
IF DEFINED _url2 (
   ECHO.
   ECHO "%_url2%"
)

It makes perfect use of the for-loop principle; During the first iteration the first line of youtube-dl's output is processed and assigned to _url1. During the second iteration the second line is processed, and assigned to _url2, because _url1 is already defined. Code I can better understand.

Thanks a lot for your help you two.

[foxidrive]

Sorry, but I refuse to use temp files.

Back in the dark ages of batch files temp files were used so very often, because they were absolutely necessary.

Many people say "no temp files" these days but they seem to miss the point that Windows uses
temporary files all day long.

I think a thank you to ShadowThief should be in your post, also.

[Reino]

Perhaps that was a little bit harsh, yes.
ShadowThief, thanks for your help too, but I still won't be using temp files.

[foxidrive]

ShadowThief, thanks for your help too, but I still won't be using temp files.

One point wrt temp files is that when they are in an inner loop and speed is paramount, and it is badly affected by temp files,
then there is a case for writing extra code to get around it.

However there are cases where a temp file will speed up the batch file enormously - in particular a bug with for /f where a huge
number of files being parsed,or with long names, can lead to 30 or more minutes before it starts to do anything.

[dbenham]

Sorry, but I refuse to use temp files.

That position can lead to a world of hurt. There are multiple scenarios where the use of one or more temp files is MUCH faster than any other batch alternative. One example is processing a command that produces lots of output. It is much faster to redirect the output to a temp file and process the file with FOR /F. Processing the command output directly with FOR /F is painfully slow if the output is really large.


Dave Benham