Exclamationmark in variable causes parsing-errors

[Reino]

VPWON_1226681.jsonp:

Code: Select all

parseMetadata({"STATUS":"OK","VERSION":"1.11.13\n","prid":"VPWON_1226681","titel":"De IJzeren Eeuw","aflevering_titel":"Ten oorlog!","info":"Geschiedenisserie over de negentiende eeuw.","ratio":"16:9","medium":"tv","gidsdatum":"2015-06-19","tijdsduur":"00:44:06","url":"","webcast":1,"images":[{"size":"640x480","ratio":"4:3","url":"http:\/\/images.poms.omroep.nl\/image\/sx480\/c640x480\/622080.jpg"},{"size":"720x405","ratio":"16:9","url":"http:\/\/images.poms.omroep.nl\/image\/s720\/c720x405\/622080.jpg"}],"omroepen":[{"naam":"NTR"},{"naam":"VPRO"}],"pubopties":["adaptive","h264_bb","h264_sb","h264_std"],"tt888":"ja","serie":{"srid":"VPWON_1226669","serie_titel":"De IJzeren Eeuw","serie_url":null},"sitestat":{"baseurl":"http:\/\/b.scorecardresearch.com\/p?c1=2&c2=17827132&ns_site=po-totaal","programurl":"uitzendinggemist.publiekeomroep.ondemand.tv.de_ijzeren_eeuw.20150619","programurlpost":"category=uitzendinggemist&thema=documentaire&po_source=video","baseurl_subtitle":"http:\/\/nl.sitestat.com\/klo\/po\/s","subtitleurl":"uitzendinggemist.publiekeomroep.ondemand.tv.player.tt888.de_ijzeren_eeuw","subtitleurlpost":"category=uitzendinggemist&po_source=video&po_sitetype=webonly"},"reclame":"http:\/\/pubads.g.doubleclick.net\/gampad\/ads?_cookie_&impl=s&gdfp_req=1&env=vp&output=xml_vast2&unviewed_position_start=1&sz=_sz_&correlator=_correlator_&iu=\/9233\/_site_\/deijzereneeuw&url=_url_&cust_params=genre%3Ddocumentaire%26dur%3D2646%26prid%3DVPWON_1226681%26srid%3DVPWON_1226669%26player%3D_player_","streamSense":{"episode":"de_ijzeren_eeuw","program":"de_ijzeren_eeuw","station":"nederland_2","sitestatname":"uitzendinggemist.publiekeomroep.ondemand.tv.de_ijzeren_eeuw.20150619","sko":"TRUE","sko_dt":"20150619","sko_pr":"de_ijzeren_eeuw","sko_stid":"2","sko_ty":"tv.ep","sko_prid":"vpwon1226681","sko_t":"2110","sko_cl":"2646"}})
//epc

The JSON object of concern: "aflevering_titel":"Ten oorlog!"

This batchscript works fine:

Code: Select all

@ECHO off

SET xidel="xidel.exe"
SET jq="jq.exe"

FOR /F "delims=" %%A IN ('^"%xidel% -q VPWON_1226681.jsonp -e "json(extract(.,'\((.+)\)',1))"^"') DO (
  ECHO JSON:
  FOR /F "delims=" %%B IN ('ECHO %%A ^| %jq% .') DO ECHO %%B

(Xidel strips everything outside the outer brackets. Jq makes it human-readible.)

Output:

Code: Select all

JSON:
{
  "STATUS": "OK",
  "VERSION": "1.11.13\n",
  "prid": "VPWON_1226681",
  "titel": "De IJzeren Eeuw",
  "aflevering_titel": "Ten oorlog!",
  "info": "Geschiedenisserie over de negentiende eeuw.",
  "ratio": "16:9",
  "medium": "tv",
  "gidsdatum": "2015-06-19",
  "tijdsduur": "00:44:06",
  "url": "",
  "webcast": 1,
  "images": [
    {
      "size": "640x480",
      "ratio": "4:3",
      "url": "http://images.poms.omroep.nl/image/sx480/c640x480/622080.jpg"
    },
    {
      "size": "720x405",
      "ratio": "16:9",
      "url": "http://images.poms.omroep.nl/image/s720/c720x405/622080.jpg"
    }
  ],
  "omroepen": [
    {
      "naam": "NTR"
    },
    {
      "naam": "VPRO"
    }
  ],
  "pubopties": [
    "adaptive",
    "h264_bb",
    "h264_sb",
    "h264_std"
  ],
  "tt888": "ja",
  "serie": {
    "srid": "VPWON_1226669",
    "serie_titel": "De IJzeren Eeuw",
    "serie_url": null
  },
  "sitestat": {
    "baseurl": "http://b.scorecardresearch.com/p?c1=2&c2=17827132&ns_site=po-totaal",
    "programurl": "uitzendinggemist.publiekeomroep.ondemand.tv.de_ijzeren_eeuw.20150619",
    "programurlpost": "category=uitzendinggemist&thema=documentaire&po_source=video",
    "baseurl_subtitle": "http://nl.sitestat.com/klo/po/s",
    "subtitleurl": "uitzendinggemist.publiekeomroep.ondemand.tv.player.tt888.de_ijzeren_eeuw",
    "subtitleurlpost": "category=uitzendinggemist&po_source=video&po_sitetype=webonly"
  },
  "reclame": "http://pubads.g.doubleclick.net/gampad/ads?_cookie_&impl=s&gdfp_req=1&env=vp&output=xml_vast2&unviewed_position_start=1&sz=_sz_&correlator=_correlator_&iu=/9233/_site_/deijzereneeuw&url=_url_&cust_params=genre%3Ddocumentaire%26dur%3D2646%26prid%3DVPWON_1226681%26srid%3DVPWON_1226669%26player%3D_player_",
  "streamSense": {
    "episode": "de_ijzeren_eeuw",
    "program": "de_ijzeren_eeuw",
    "station": "nederland_2",
    "sitestatname": "uitzendinggemist.publiekeomroep.ondemand.tv.de_ijzeren_eeuw.20150619",
    "sko": "TRUE",
    "sko_dt": "20150619",
    "sko_pr": "de_ijzeren_eeuw",
    "sko_stid": "2",
    "sko_ty": "tv.ep",
    "sko_prid": "vpwon1226681",
    "sko_t": "2110",
    "sko_cl": "2646"
  }
}

However, when I have to use ENABLEDELAYEDEXPANSION for further extraction of JSON objects, this causes the batchscript to parse ampersands because of the exclamationmark in the episode-title.

Batchscript:

Code: Select all

@ECHO off

SET xidel="xidel.exe"
SET jq="jq.exe"

SETLOCAL ENABLEDELAYEDEXPANSION
FOR /F "delims=" %%A IN ('^"%xidel% -q VPWON_1226681.jsonp -e "json(extract(.,'\((.+)\)',1))"^"') DO (
  ECHO JSON:
  FOR /F "delims=" %%B IN ('ECHO %%A ^| %jq% .') DO ECHO %%B
  ECHO.
  FOR /F "delims=" %%B IN ('ECHO %%A ^| %xidel% -q - -e "$json/(t:=tijdsduur,ss:=start,to:=eind)" --output-format=cmd') DO %%B
  ECHO Tijdsduur: !t!
  IF DEFINED ss (
    SET /A "_ss=((1!ss:~0,2!-100)*3600)+((1!ss:~3,2!-100)*60)+(1!ss:~6,2!-100)"
    SET /A "_to=((1!to:~0,2!-100)*3600)+((1!to:~3,2!-100)*60)+(1!to:~6,2!-100)"
    ECHO Start:     !ss! (!_ss!s^)
    ECHO Einde:     !to! (!_to!s^)
  )
)
ENDLOCAL

Output:

Code: Select all

JSON:
'c2' is not recognized as an internal or external command,
operable program or batch file.
'ns_site' is not recognized as an internal or external command,
operable program or batch file.
'thema' is not recognized as an internal or external command,
operable program or batch file.
'po_source' is not recognized as an internal or external command,
operable program or batch file.
'po_source' is not recognized as an internal or external command,
operable program or batch file.
'po_sitetype' is not recognized as an internal or external command,
...
etc...

What can I do to prevent SETLOCAL ENABLEDELAYEDEXPANSION to see the exclamationmark as syntax for a variable?

[jeb]

Hi CoRoNe,

normally delayed expansion doesn't have the effect you described here.
Delayed expansion can remove carets and exclamation marks from texts, but it will not activate ampersands inside quotes.

But in your case you activate the delayed expansion a second time by using the ECHO %%A in the line
FOR /F "delims=" %%B IN ('ECHO %%A ^| %jq% .') DO ECHO %%B

And there the delayed expansion of this part fails
"Ten oorlog!","info":"Geschiedenisseri ....

This will be expanded to
"Ten oorlog"Geschiedenisseri ....
As the parser tries to expand the expression !","info": but as it can't find a variable named ","info" it simply remove that part.

The solution to handle exclamation marks within for-loops is the toggling technic.

Code: Select all

SETLOCAL DisableDelayedExpansion
FOR /F "delims=" %%A IN ('^"%xidel% -q VPWON_1226681.jsonp -e "json(extract(.,'\((.+)\)',1))"^"') DO (
  ECHO JSON: %%A
  FOR /F "delims=" %%B IN ('ECHO %%A ^| %jq% .') DO ECHO %%B
  echo(
  FOR /F "delims=" %%B IN ('ECHO %%A ^| %xidel% -q - -e "$json/(t:=tijdsduur,ss:=start,to:=eind)" --output-format=cmd') DO %%B

  SETLOCAL EnableDelayedExpansion
  ECHO Tijdsduur: !t!
  IF DEFINED ss (
    SET /A "_ss=((1!ss:~0,2!-100)*3600)+((1!ss:~3,2!-100)*60)+(1!ss:~6,2!-100)"
    SET /A "_to=((1!to:~0,2!-100)*3600)+((1!to:~3,2!-100)*60)+(1!to:~6,2!-100)"
    ECHO Start:     !ss! (!_ss!s^)
    ECHO Einde:     !to! (!_to!s^)
  )
  ENDLOCAL
)

This ensures that only the relevant part operates with the delayed expansion mode and it doesn't destroy any variable contents.

[Reino]

Hello jeb,

But in your case you activate the delayed expansion a second time by using the ECHO %%A in the line
FOR /F "delims=" %%B IN ('ECHO %%A ^| %jq% .') DO ECHO %%B

Ah! I didn't know that. We learn everyday.

The solution to handle exclamation marks within for-loops is the toggling technic.

The delayed expansion has always been a thing I find hard to understand, but I get it now. Only apply it where it's really necessary. I was actually really close in finding the same solution just 30minutes ago.
Thanks a lot for your quick support, jeb!

[jeb]

jeb wrote:
But in your case you activate the delayed expansion a second time by using the ECHO %%A in the line
FOR /F "delims=" %%B IN ('ECHO %%A ^| %jq% .') DO ECHO %%B
Ah! I didn't know that. We learn everyday.

And I'm not quite correct here ,
the delayed expansion is only once parsed,
but the content of %%A will be parsed again,
normally without delayed expansion, as delayed expansion is not activated as default when a new cmd.exe instance will be created.