Pipe in for loop breaks double quoted variables

[Reino]

Situation: Using batchscript to retrieve certain values from a JSON.
I've got the following batchscript:

Code: Select all

@ECHO off

SET url=http://www.npo.nl/buitenhof/03-05-2015/VPWON_1232766/POMS_VPRO_850040

FOR /F "tokens=6 delims=/" %%A IN ("%url%") DO (
   FOR /F "delims=" %%B IN ('curl.exe -s http://e.omroep.nl/metadata/aflevering/%%A ^| jq.exe -R -r -s ".[1+index(\"(\"): rindex(\"^)\")]"') DO (
      FOR /F "tokens=1-3" %%C IN ('ECHO %%B ^| jq.exe -r "[.tijdsduur,.start,.eind] | \"\(.[0]^) \(.[1]^) \(.[2]^)\""') DO (
         ECHO Duration: %%C
         ECHO Start:    %%D
         ECHO End:      %%E
      )
   )
)
PAUSE

What does it do?
Upon having entered a npo.nl program-url, the first for-loop strips the url down to the prid: POMS_VPRO_850040. In the 2nd for-loop curl.exe retrieves the JSON...:

Code: Select all

parseMetadata({"STATUS":"OK","VERSION":"1.11.12","prid":"VPWON_1232766",[.........],"sko_cl":"3284"}})
//epc

...and sends it through a pipe to jq.exe which removes the non-JSON-data parseMetadata( and ) //epc and leaves the single line intact. This is for 2 reasons: 1) with non-JSON-data present jq.exe can't process anything, and 2) for-loops process only 1 line at a time.
The 3rd for-loop retrieves and echoes values for the specified objects without double quotes.
As long as curl.exe and jq.exe are in the same directory as the batchscript, or in the %path%-variable, this is working all fine:

Code: Select all

Duration: 00:05:27
Start:    00:23:13
End:      00:28:40

Now I want to call curl.exe and jq.exe from another map. One with spaces in it:

Code: Select all

@ECHO off

SET curl="C:\map with spaces\curl.exe"
SET jq="C:\map with spaces\jq.exe"
SET url=http://www.npo.nl/buitenhof/03-05-2015/VPWON_1232766/POMS_VPRO_850040

FOR /F "tokens=6 delims=/" %%A IN ("%url%") DO (
   FOR /F "delims=" %%B IN ('%curl% -s http://e.omroep.nl/metadata/aflevering/%%A ^| %jq% -R -r -s ".[1+index(\"(\"): rindex(\"^)\")]"') DO (
      FOR /F "tokens=1-3" %%C IN ('ECHO %%B ^| %jq% -r "[.tijdsduur,.start,.eind] | \"\(.[0]^) \(.[1]^) \(.[2]^)\""') DO (
         ECHO Duration: %%C
         ECHO Start:    %%D
         ECHO End:      %%E
      )
   )
)
PAUSE

For the 2nd for-loop this causes problems:

Code: Select all

'C:\map' is not recognized as an internal or external command,
operable program or batch file.

While 'ECHO %%B ^| %jq%' does work, it seems '%curl% ^| %jq%' doesn't. So for some reason things go wrong as soon as 2 variables in a pipe are parsed.

Well, no more pipe then:

Code: Select all

@ECHO off

SET curl="C:\map with spaces\curl.exe"
SET jq="C:\map with spaces\jq.exe"
SET url=http://www.npo.nl/buitenhof/03-05-2015/VPWON_1232766/POMS_VPRO_850040

FOR /F "tokens=6 delims=/" %%A IN ("%url%") DO (
   FOR /F "delims=" %%B IN ('%curl% -s http://e.omroep.nl/metadata/aflevering/%%A') DO (
      FOR /F "delims=" %%C IN ('ECHO %%B ^| %jq% -R -r -s ".[1+index(\"(\"): rindex(\"^)\")]"') DO (
         FOR /F "tokens=1-3" %%D IN ('ECHO %%C ^| %jq% -r "[.tijdsduur,.start,.eind] | \"\(.[0]^) \(.[1]^) \(.[2]^)\""') DO (
            ECHO Duration: %%D
            ECHO Start:    %%E
            ECHO End:      %%F
         )
      )
   )
)
PAUSE

Now curl.exe and jq.exe each in a for-loop. At first this seems to work fine. The 3 values are echoed, but then things go wrong:

Code: Select all

Tijdsduur: 00:05:27
Start:     00:23:13
Einde:     00:28:40
parse error: Invalid numeric literal at line 1, column 4

parse error: Invalid numeric literal at line 1, column 4

Like I said before; for-loops parse and process only 1 line at a time. The non-JSON-data //epc on the 2nd line causes the for-loop to start over, which goes horribly wrong as you can see. That's the reason for the pipe between curl and jq in the code above. To output 1 single line to process. Sadly that didn't work either...sigh.

Of course using temporary files is a last resort when curl and jq are still in a map with spaces in it, but I prefer to use variables, so I'm trying to solve the pipe-issue. I've tried 'usebackq' in the for-loop using backticks around the command instead of single-quotes for instance, but to no avail.
So far I haven't found a solution. Does anyone have an explanation for this behaviour and how to solve it?

[foxidrive]

This code will simplify your first loop, but I'd also like to see the full code with sample files and a comment or two on
the problem.

jq.exe is not a default program in my version of windows.

Code: Select all

SET url=http://www.npo.nl/buitenhof/03-05-2015/VPWON_1232766/POMS_VPRO_850040

FOR %%A IN ("%url%") DO (
  echo "%%~nxA"
)

[Reino]

I didn't know %%~nxA would also work for urls. Nice!
I've further worked out my script:

Code: Select all

@ECHO off
CLS

:: NPO JSON-extractor geschreven door Reino Wijnsma, 2015 (reino@degeelebosch.nl)

SET batchname=NPO JSON-extractor
SET version=1.1
TITLE %batchname% %version%

SET curl="C:\map with spaces\curl.exe"
SET jq="C:\map with spaces\jq.exe"

:Check
IF EXIST %curl% (
   IF EXIST %jq% (
      GOTO Input
   ) ELSE (
      ECHO 'jq.exe' niet gevonden.
      ECHO.
      PAUSE
      GOTO :eof
   )
   GOTO Input
) ELSE (
   ECHO 'curl.exe' niet gevonden.
   ECHO.
   PAUSE
   GOTO :eof
)

:Input
ECHO Voer npo.nl programmalink in :
SET url=
SET /P url=
:: http://www.npo.nl/buitenhof/03-05-2015/VPWON_1232766/POMS_VPRO_850040 bijvoorbeeld
IF "%url%"=="" GOTO :eof

SETLOCAL ENABLEDELAYEDEXPANSION
FOR %%A IN ("%url%") DO (
   FOR /F "delims=" %%B IN ('curl.exe -s http://e.omroep.nl/metadata/aflevering/%%~nxA ^| jq.exe -R -r -s ".[1+index(\"(\"): rindex(\"^)\")]"') DO (
      ECHO.
      ECHO JSON:
      FOR /F "delims=" %%C IN ('ECHO %%B ^| %jq% .') DO ECHO %%C
      ECHO.
      FOR /F "tokens=1-3" %%C IN ('ECHO %%B ^| %jq% -r "[.tijdsduur,.start,.eind] | \"\(.[0]^) \(.[1]^) \(.[2]^)\""') DO (
         ECHO Tijdsduur: %%C
         IF NOT "%%D"=="null" (
            SET ss=%%D
            SET /A "_ss=((1!ss:~0,2!-100)*3600)+((1!ss:~3,2!-100)*60)+(1!ss:~6,2!-100)"
            ECHO Start:     %%D (!_ss!s^)
         )
         IF NOT "%%E"=="null" (
            SET to=%%E
            SET /A "_to=((1!to:~0,2!-100)*3600)+((1!to:~3,2!-100)*60)+(1!to:~6,2!-100)"
            ECHO Einde:     %%E (!_to!s^)
         )
      )
   )
)
ECHO.
ENDLOCAL
GOTO Input

Okay, from the beginning. Npo.nl ("Dutch Public Broadcast") is a website where you, or at least Dutch citizens, can freely watch television broadcasts.
http://www.npo.nl/buitenhof/03-05-2015/VPWON_1232766 is examplary for a typical program-url for a full-length tvshow.
http://www.npo.nl/buitenhof/03-05-2015/VPWON_1232766/POMS_VPRO_850040 on the other hand is a program-url for a fragment of the same tvshow.
Each has a different JSON and I'm trying to obtain certain metadata from within these JSONs.
Let's process http://www.npo.nl/buitenhof/03-05-2015/VPWON_1232766/POMS_VPRO_850040. The url for the JSON is http://e.omroep.nl/metadata/aflevering/POMS_VPRO_850040. By simply opening this url in your browser you'd get...

Code: Select all

parseMetadata({"STATUS":"OK","VERSION":"1.11.12","prid":"VPWON_1232766","titel":"Schuim & As","aflev
ering_titel":"","info":"Schuim & As met Jelle Brandt Corstius","ratio":"16:9","medium":"tv","gidsdat
um":"2015-05-03","tijdsduur":"00:05:27","start":"00:23:13","eind":"00:28:40","url":"","webcast":1,"i
mages":[{"size":"640x480","ratio":"4:3","url":"http:\/\/images.poms.omroep.nl\/image\/sx480\/c640x48
0\/606030.jpg"},{"size":"720x405","ratio":"16:9","url":"http:\/\/images.poms.omroep.nl\/image\/sx405
\/c720x405\/606030.jpg"}],"omroepen":[{"naam":"VPRO"}],"pubopties":["adaptive","h264_bb","h264_sb","
h264_std"],"tt888":"ja","serie":{"srid":"VPWON_1232748","serie_titel":"Buitenhof","serie_url":null},
"sitestat":{"baseurl":"http:\/\/b.scorecardresearch.com\/p?c1=2&c2=17827132&ns_site=po-totaal","prog
ramurl":"uitzendinggemist.publiekeomroep.ondemand.tv.buitenhof.20150503","programurlpost":"category=
uitzendinggemist&thema=informatief&po_source=video","baseurl_subtitle":"http:\/\/nl.sitestat
.com\/klo\/po\/s","subtitleurl":"uitzendinggemist.publiekeomroep.ondemand.tv.player.tt888.buitenhof"
,"subtitleurlpost":"category=uitzendinggemist&po_source=video&po_sitetype=webonly"},"reclame
":"http:\/\/pubads.g.doubleclick.net\/gampad\/ads?_cookie_&impl=s&gdfp_req=1&env=vp&output=xml_vast2
&unviewed_position_start=1&sz=_sz_&correlator=_correlator_&iu=\/9233\/_site_\/buitenhof&url=_url_&cu
st_params=genre%3Dinformatief%2Cnieuws%2Factualiteiten%26dur%3D3284%26prid%3DVPWON_1232766%26srid%3D
VPWON_1232748%26player%3D_player_","streamSense":{"episode":"buitenhof","program":"buitenhof","stati
on":"nederland_1","sitestatname":"uitzendinggemist.publiekeomroep.ondemand.tv.buitenhof.20150503","s
ko":"TRUE","sko_dt":"20150503","sko_pr":"buitenhof","sko_stid":"1","sko_ty":"tv.seg","sko_prid":"vpw
on1232766","sko_t":"1210","sko_cl":"3284"}})
//epc

...which is the same as curl.exe's output in the first for-loop. Everything is on a single line, except "//epc". So to prevent the for-loop from starting over, this 2nd line has to go. This is where jq (a command-line JSON processor) comes in. The jq-command after the pipe first of all puts everything on one single line and then removes parseMetadata( and ) //epc so that we end up with:

Code: Select all

{"STATUS":"OK","VERSION":"1.11.12","prid":"VPWON_1232766","titel":"Schuim & As","aflevering_titel":"
","info":"Schuim & As met Jelle Brandt Corstius","ratio":"16:9","medium":"tv","gidsdatum":"2015-05-0
3","tijdsduur":"00:05:27","start":"00:23:13","eind":"00:28:40","url":"","webcast":1,"images":[{"size
":"640x480","ratio":"4:3","url":"http:\/\/images.poms.omroep.nl\/image\/sx480\/c640x480\/606030.jpg"
},{"size":"720x405","ratio":"16:9","url":"http:\/\/images.poms.omroep.nl\/image\/sx405\/c720x405\/60
6030.jpg"}],"omroepen":[{"naam":"VPRO"}],"pubopties":["adaptive","h264_bb","h264_sb","h264_std"],"tt
888":"ja","serie":{"srid":"VPWON_1232748","serie_titel":"Buitenhof","serie_url":null},"sitestat":{"b
aseurl":"http:\/\/b.scorecardresearch.com\/p?c1=2&c2=17827132&ns_site=po-totaal","programurl":"uitze
ndinggemist.publiekeomroep.ondemand.tv.buitenhof.20150503","programurlpost":"category=uitzendinggemi
st&thema=informatief&po_source=video","baseurl_subtitle":"http:\/\/nl.sitestat.com\/klo\/po\
/s","subtitleurl":"uitzendinggemist.publiekeomroep.ondemand.tv.player.tt888.buitenhof","subtitleurlp
ost":"category=uitzendinggemist&po_source=video&po_sitetype=webonly"},"reclame":"http:\/\/pu
bads.g.doubleclick.net\/gampad\/ads?_cookie_&impl=s&gdfp_req=1&env=vp&output=xml_vast2&unviewed_posi
tion_start=1&sz=_sz_&correlator=_correlator_&iu=\/9233\/_site_\/buitenhof&url=_url_&cust_params=genr
e%3Dinformatief%2Cnieuws%2Factualiteiten%26dur%3D3284%26prid%3DVPWON_1232766%26srid%3DVPWON_1232748%
26player%3D_player_","streamSense":{"episode":"buitenhof","program":"buitenhof","station":"nederland
_1","sitestatname":"uitzendinggemist.publiekeomroep.ondemand.tv.buitenhof.20150503","sko":"TRUE","sk
o_dt":"20150503","sko_pr":"buitenhof","sko_stid":"1","sko_ty":"tv.seg","sko_prid":"vpwon1232766","sk
o_t":"1210","sko_cl":"3284"}}

In the next for-loop this line is fed to jq.exe to make the JSON more readible:

Code: Select all

{
  "STATUS": "OK",
  "VERSION": "1.11.12",
  "prid": "VPWON_1232766",
  "titel": "Schuim & As",
  "aflevering_titel": "",
  "info": "Schuim & As met Jelle Brandt Corstius",
  "ratio": "16:9",
  "medium": "tv",
  "gidsdatum": "2015-05-03",
  "tijdsduur": "00:05:27",
  "start": "00:23:13",
  "eind": "00:28:40",
  "url": "",
  "webcast": 1,
  "images": [
    {
      "size": "640x480",
      "ratio": "4:3",
      "url": "http://images.poms.omroep.nl/image/sx480/c640x480/606030.jpg"
    },
    {
      "size": "720x405",
      "ratio": "16:9",
      "url": "http://images.poms.omroep.nl/image/sx405/c720x405/606030.jpg"
    }
  ],
  "omroepen": [
    {
      "naam": "VPRO"
    }
  ],
  "pubopties": [
    "adaptive",
    "h264_bb",
    "h264_sb",
    "h264_std"
  ],
  "tt888": "ja",
  "serie": {
    "srid": "VPWON_1232748",
    "serie_titel": "Buitenhof",
    "serie_url": null
  },
  "sitestat": {
    "baseurl": "http://b.scorecardresearch.com/p?c1=2&c2=17827132&ns_site=po-totaal",
    "programurl": "uitzendinggemist.publiekeomroep.ondemand.tv.buitenhof.20150503",
    "programurlpost": "category=uitzendinggemist&thema=informatief&po_source=video",
    "baseurl_subtitle": "http://nl.sitestat.com/klo/po/s",
    "subtitleurl": "uitzendinggemist.publiekeomroep.ondemand.tv.player.tt888.buitenhof",
    "subtitleurlpost": "category=uitzendinggemist&po_source=video&po_sitetype=webonly"
  },
  "reclame": "http://pubads.g.doubleclick.net/gampad/ads?_cookie_&impl=s&gdfp_req=1&env=vp&output=xm
l_vast2&unviewed_position_start=1&sz=_sz_&correlator=_correlator_&iu=/9233/_site_/buitenhof&url=_url
_&cust_params=genre%3Dinformatief%2Cnieuws%2Factualiteiten%26dur%3D3284%26prid%3DVPWON_1232766%26sri
d%3DVPWON_1232748%26player%3D_player_",
  "streamSense": {
    "episode": "buitenhof",
    "program": "buitenhof",
    "station": "nederland_1",
    "sitestatname": "uitzendinggemist.publiekeomroep.ondemand.tv.buitenhof.20150503",
    "sko": "TRUE",
    "sko_dt": "20150503",
    "sko_pr": "buitenhof",
    "sko_stid": "1",
    "sko_ty": "tv.seg",
    "sko_prid": "vpwon1232766",
    "sko_t": "1210",
    "sko_cl": "3284"
  }
}

In the next for-loop the same single-line input is again being fed to jq.exe, but this time to retrieve the values for "tijdsduur", "start" and "eind". In this JSON all 3 objects are present, so the value for "tijdsduur" is echoed, and since the values for "start" and "eind" are not "null" these are echoed as well, with the time in seconds between brackets as a bonus:

Code: Select all

Tijdsduur: 00:05:27
Start:     00:23:13 (1393s)
Einde:     00:28:40 (1720s)

In the JSON of a full-length tvshow only "tijdsduur" is present, so only the value for "tijdsduur" is echoed.

With the path to curl.exe and jq.exe in variables this is working fine, except for the first for-loop where curl's output is piped to jq.

[Reino]

Thanks to Dave Benham's answer on a related issue I've found the solution!
It appeared to be a specific FOR /F bug in WinXP, and guess what, here I'm still on WinXP. To fix the main offender, the curl-pipe-jq-for-loop, I had to put '^"' in front of and after the entire command, like ('^"<command>^"'). Thus the entire batchscript:

Code: Select all

@ECHO off
CLS

:: NPO JSON-extractor geschreven door Reino Wijnsma, 2015 (reino@degeelebosch.nl)

SET batchname=NPO JSON-extractor
SET version=1.1
TITLE %batchname% %version%

SET curl="C:\map with spaces\curl.exe"
SET jq="C:\map with spaces\jq-1.5rc1.exe"

:Check
IF EXIST %curl% (
   IF EXIST %jq% (
      GOTO Input
   ) ELSE (
      ECHO 'jq.exe' niet gevonden.
      ECHO.
      PAUSE
      GOTO :eof
   )
   GOTO Input
) ELSE (
   ECHO 'curl.exe' niet gevonden.
   ECHO.
   PAUSE
   GOTO :eof
)

:Input
ECHO Voer npo.nl programmalink in :
SET url=
SET /P url=
:: http://www.npo.nl/buitenhof/03-05-2015/VPWON_1232766/POMS_VPRO_850040 bijvoorbeeld
IF "%url%"=="" GOTO :eof

SETLOCAL ENABLEDELAYEDEXPANSION
FOR %%A IN ("%url%") DO (
   FOR /F "delims=" %%B IN ('^"%curl% -s http://e.omroep.nl/metadata/aflevering/%%~nxA ^| %jq% -R -r -s ".[1+index(\"(\"): rindex(\"^)\")]"^"') DO (
      ECHO.
      ECHO JSON:
      FOR /F "delims=" %%C IN ('ECHO %%B ^| %jq% .') DO ECHO %%C
      ECHO.
      FOR /F "tokens=1-3" %%C IN ('ECHO %%B ^| %jq% -r "[.tijdsduur,.start,.eind] | @tsv"') DO (
         ECHO Tijdsduur: %%C
         IF NOT "%%D"=="" (
            SET ss=%%D
            SET /A "_ss=((1!ss:~0,2!-100)*3600)+((1!ss:~3,2!-100)*60)+(1!ss:~6,2!-100)"
            ECHO Start:     %%D (!_ss!s^)
         )
         IF NOT "%%E"=="" (
            SET to=%%E
            SET /A "_to=((1!to:~0,2!-100)*3600)+((1!to:~3,2!-100)*60)+(1!to:~6,2!-100)"
            ECHO Einde:     %%E (!_to!s^)
         )
      )
   )
)
ECHO.
ENDLOCAL
GOTO Input

Thanks to a new jq release I could also simplify the extraction with @tsv.

Important notes for future reference:

Code: Select all

jq-syntax:           jq -R -r -s  '.[1+index("("):   rindex(")")]'
cmd-shell:           jq -R -r -s  ".[1+index(\"(\"): rindex(\")\")]"
for-loop:           'jq -R -r -s  ".[1+index(\"(\"): rindex(\"^)\")]"
for-loop (path): '^"%jq% -R -r -s ".[1+index(\"(\"): rindex(\"^)\")]"^"'

- For the cmd-shell you have to escape the double quotes with a line-feed '\'.
- While 2 of the closing parenthesis are part of jq's syntax, the 1 between double quotes is not. So when in a for-loop, to prevent this for-loop from closing, you'd have to escape this one with a '^'.
- When jq's executable path is put in a variable with double quotes, then to circumvent a WinXP bug, you'd also have to put the entire command between '^"', because the parenthesis are now considered special characters! (This is compatible with Vista and beyond)

[dbenham]

I'm glad you found that StackOverflow post.

But this issue is a CMD /C design flaw, not a FOR /F bug. The issue arises with FOR /F because FOR /F executes commands via CMD /C.

The quoting rules for a CMD /C command on XP are exactly as documented. Your command does not meet the requirements of rule 1, so it defaults to rule 2:

Code: Select all

If /C or /K is specified, then the remainder of the command line after
the switch is processed as a command line, where the following logic is
used to process quote (") characters:

    1.  If all of the following conditions are met, then quote characters
        on the command line are preserved:

        - no /S switch
        - exactly two quote characters
        - no special characters between the two quote characters,
          where special is one of: &<>()@^|
        - there are one or more whitespace characters between the
          two quote characters
        - the string between the two quote characters is the name
          of an executable file.

    2.  Otherwise, old behavior is to see if the first character is
        a quote character and if so, strip the leading character and
        remove the last quote character on the command line, preserving
        any text after the last quote character.

The rules are unfortunate and poorly thought out, but they have been implemented on XP as designed and documented, so no bug.

The situation is a bit more nuanced on Vista and beyond. More recent versions of Windows no longer treat ( and ) as special characters, which is a good thing. So the relevant documentation should read:

Code: Select all

        - no special characters between the two quote characters,
          where special is one of: &<>@^|

I suppose you could claim this is a bug since Vista and beyond do not work as documented. But I prefer to think that the documentation is simply wrong, and Vista and beyond are working as designed.

I still think the feature is poorly designed, even after ( and ) are dropped from the special character list. I would have designed rule 1 as follows:

Code: Select all

    1.  If all of the following conditions are met, then quote characters
        on the command line are preserved:

        - no /S switch
        - the string between the very first token is quoted, and corresponds
          to an executable file.

Now that would have been useful and easy to implement. But alas, Microsoft wasn't that smart.


There actually is a nasty FOR /F bug on XP. But that is totally unrelated to the issues on this thread.


Dave Benham

[Reino]

So if I understand correctly, the issue was never about spaces in a directory name, or the presence of a pipe, but rather the presence of parenthesis in the jq-command! But why then isn't it necessary to put the entire command between '^"' when curl.exe and jq.exe are called directly? The parenthesis are here too.

Code: Select all

FOR /F "delims=" %%B IN ('curl.exe -s http://e.omroep.nl/metadata/aflevering/%%~nxA ^| jq.exe -R -r -s ".[1+index(\"(\"): rindex(\"^)\")]"') DO (

[dbenham]

Well..., in that instance, the command does not begin with a quote, so CMD /C preserves all quotes as is.

Spaces within the executable name (or executable path), can indeed wreak havoc when used with FOR /F. Quote rules quickly become extremely difficult to fathom unless you have a lot of experience. Even with all my experience, it often takes me considerable time to figure out how to handle a given situation.


Dave Benham

[Reino]

I see. Understood. Thanks a lot!

[foxidrive]

Without following all your code - is this all you are after from the JSON files?

Code: Select all

Tijdsduur: 00:05:27
Start:     00:23:13 (1393s)
Einde:     00:28:40 (1720s)

[dbenham]

EDIT - ignore the red text below - it is completely wrong
I just discovered a simpler solution to preserve quotes around the executable when using FOR /F.

CMD /C only strips the outer quotes if the very first character is a quote. So I simply put an extra space before the command.

Code: Select all

for /f %%A in (' "path with space\command.exe" "arg one" "arg two"') do ...    THIS DOES NOT WORK

But I have a simpler solution for your problem as a whole
You can use my JREPL.BAT utility to parse the CURL output directly.

Code: Select all

@echo off

set "curl=c:\path with spaces\curl.exe"
set "jrepl=c:\path with spaces\jrepl.bat"
set "url=http://www.npo.nl/buitenhof/03-05-2015/VPWON_1232766/POMS_VPRO_850040"

for /f %%U in ("%url%") do (
  "%curl%" -s "http://e.omroep.nl/metadata/aflevering/%%~nxU" ^
  | "%jrepl%" "\q(tijdsduur|start|eind)\q:\q(.*?)\q" "$1+': '+$2" /x /jmatch
)

--Output--

Code: Select all

tijdsduur: 00:05:27
start: 00:23:13
eind: 00:28:40

Using http://www.npo.nl/buitenhof/03-05-2015/VPWON_1232766 I get:

Code: Select all

tijdsduur: 00:54:44

EDIT - You might want to use a slightly altered script that allows white space before and after the colon. It is not needed in your example, but your source may not be consistent:

Code: Select all

@echo off

set "curl=c:\path with spaces\curl.exe"
set "jrepl=c:\path with spaces\jrepl.bat"
set "url=http://www.npo.nl/buitenhof/03-05-2015/VPWON_1232766/POMS_VPRO_850040"

for /f %%U in ("%url%") do (
  "%curl%" -s "http://e.omroep.nl/metadata/aflevering/%%~nxU" ^
  | "%jrepl%" "\q(tijdsduur|start|eind)\q\s*:\s*\q(.*?)\q" "$1+': '+$2" /x /jmatch
)

Dave Benham

[Reino]

CMD /C only strips the outer quotes if the very first character is a quote. So I simply put an extra space before the command.

Code: Select all

SET curl="C:\map with spaces\curl.exe"
SET jq="C:\map with spaces\jq-1.5rc1.exe"
...
FOR %%A IN ("%url%") DO (
   FOR /F "delims=" %%B IN (' %curl% -s http://e.omroep.nl/metadata/aflevering/%%~nxA ^| %jq% -R -r -s ".[1+index(\"(\"): rindex(\"^)\")]"') DO (

This didn't work.

Code: Select all

SET "curl=C:\map with spaces\curl.exe"
SET "jq=C:\map with spaces\jq-1.5rc1.exe"
...
FOR %%A IN ("%url%") DO (
   FOR /F "delims=" %%B IN (' "%curl%" -s http://e.omroep.nl/metadata/aflevering/%%~nxA ^| "%jq%" -R -r -s ".[1+index(\"(\"): rindex(\"^)\")]"') DO (

This didn't work either.
Why btw do you put the entire commandset between double quotes? Then you'd also have to put that variable everywhere in your script between double quotes.

Without following all your code - is this all you are after from the JSON files?

You can use my JREPL.BAT utility to parse the CURL output directly.

For the script I posted above I could indeed use your JREPL.bat as well, but the extraction of JSON data was only an issue I was facing when working on a larger batchscript. Here the JSON data extraction is for a slightly different purpose.
Today I've added comments. They're in Dutch, but I'm sure you can figure out what the script does. I'm still a novice programmer, and I'm sure parts of code could be written more efficiently, but hey... it's a first release and at least it's working now.

[dbenham]

... This didn't work. ...

Sorry about that. I temporarily lost my sanity, and managed to fool myself with my testing. You are correct, that idea does not work at all. I've edited my prior post.

Why btw do you put the entire commandset between double quotes? Then you'd also have to put that variable everywhere in your script between double quotes.

That is a programming style I have adopted, and I think it is generally good advice. The quotes are not really part of the value, but they are important (depending on context) to get batch to parse the command the way you want. I choose to "always" store my value without quotes within a variable, and then explicitly supply the quotes at expansion time, as needed.

This provides a few benefits:
- It makes it easier to ECHO the value without quotes.
- It makes it easier to concatenate strings without unwanted internal quotes
- It separates the value from the syntax, thus making it easier to verify syntax is correct. I loosely consider the quotes to be part of the syntax, and it puts all the syntax in one place. If the quotes are embedded in the value, then I must go back to the assignment to make sure the quotes are actually there. If I "never" put the quotes within the value, then I can rely on "what you see is what you get" when I use the expanded variable within a statement.

Bear in mind, this is a guiding principle, not a hard and fast rule.


Dave Benham