Dynamic loop code over multiple lines causes extra quotes

[lockedscope]

In following experimental code, i try to create dynamic code to loop over multiple lines.
But an extra double quotes("") appear after the last echo.
I solved it with a "rem", but not sure what causes it. So, what causes those quotes?
Also, how to write this code elegantly, with less delayed expansion variables and better escaping?

Code: Select all

@echo off
setlocal enableDelayedExpansion
(set \n=^^^
%= This creates an escaped Line Feed - DO NOT ALTER =%
)

set "quot.=""

set "quot=^!quot.^!"

(set text.=11x y z%\n%
22d e f%\n%
33q w x)

rem A simple loop
for /F "tokens=1-2 skip=1 delims= " %%G in ("!text.!") do (echo %%G -- %%H)

set "param=tokens=1-2 skip=1 delims= "

set "text=^!text.^!"


echo --- Dynamic Loop ---
rem It does not work when there are parenthesis after "do", raises error : "" was unexpected at this time.
rem call :myloop "for /F !quot!!param!!quot! %%%%G in ("!text!") do ( echo %%%%G --- %%%%H )"

rem Without parenthesis, extra double quotes("") at the end, solved by rem, set a= etc.
rem call :myloop "for /F !quot!!param!!quot! %%%%G in ("!text!") do echo %%%%G --- %%%%H && rem"

rem With parenthesis, rem also solves the double quote problem.
call :myloop "for /F !quot!!param!!quot! %%%%G in ("!text!") do (echo %%%%G --- %%%%H ) && rem"

rem Following works fine with separate parameters, so quoting of options handled in the routine
rem call :myloop2 "echo for loop && for /F" "!param!" "%%%%G in ("^!text^!") do (echo %%%%G --- %%%%H)"


endlocal

goto :eof


:myloop
setlocal

set "str=%~1"
%str%

endlocal
exit /b  
goto :eof


:myloop2
setlocal

%~1 "%~2" %~3

endlocal

exit /b  
goto :eof

[T3RRY]

As discussed on stack overflow, the escaping of variables and double %% escaping of metavariables can be avoided by defining the loop as a macro and using the subroutine to expand that macro within delayed expansion environments such as code blocks. This also avoids the need of handling doublequotes in call parameters.

Defining the macro:

Code: Select all

Set "myloop=for /F "!param!" %%G in ("!text!") do (echo %%G --- %%H )"&call :myloop

And the subroutine to expand the macro:

Code: Select all

:myloop
setlocal
%myloop%
endlocal
exit /b

[lockedscope]

Thanks T3RRY for all those efforts.

So, i could get it without those quotes at the end.
It might be because of correct escaping of quotes of param or escaping of text, i am not sure.
But i wonder, how could i get rid of multiple variables for text?
Are there any guide for multiple levels of escaping for surviving from expansions?

rem @echo off
setlocal enableDelayedExpansion
(set \n=^^^
%= This creates an escaped Line Feed - DO NOT ALTER =%
)

(set text.=11x y z%\n%
22d e f%\n%
33q w x)


rem A simple loop
for /F "tokens=1-2 skip=1 delims= " %%G in ("!text!") do (echo %%G -+- %%H)

echo --- Dynamic Loop ---
set "text=^!text.^!"
set "param="tokens=1-2 delims= ""
call :myloop "for /F ^!param^! %%%%G in ("^^^!text^^^!") do (echo %%%%G -- echo %%%%H ) "

endlocal

goto :eof


:myloop
setlocal


set "s=%~1"
echo --- %s%
echo start
%s%
echo end

endlocal

exit /b

[lockedscope]

I could finally escaped text variable to prevent from expansion without using extra variable.
It worked with 10 or 11 carets, may be with more carets too.
But i do not understand how it worked? I checked parsing rules but could not find any direct clue about this. Not sure about caret doubling phase.
So, how could that 10/11 carets works?

Code: Select all

rem @echo off
setlocal enableDelayedExpansion
(set \n=^^^
%= This creates an escaped Line Feed - DO NOT ALTER =%
)

(set text=11x y z%\n%
22d e f%\n%
33q w x)



rem A simple loop
for /F "tokens=1-2 skip=1 delims= " %%G in ("!text!") do (echo %%G -+- %%H)

echo --- Dynamic Loop ---


set "param="tokens=1-2 skip=1 delims= ""
rem Works with 11 carets each side
call :myloop "for /F ^!param^! %%%%G in ("^^^^^^^^^^^!text^^^^^^^^^^^!") do (echo %%%%G -- echo %%%%H ) "
rem Following works too with 10 carets each side
call :myloop "for /F ^!param^! %%%%G in ("^^^^^^^^^^!text^^^^^^^^^^!") do (echo %%%%G -- echo %%%%H ) " 

endlocal

goto :eof

:myloop
setlocal


set "s=%~1"
echo --- %s%
echo start

%s%
echo end

endlocal

exit /b

[lockedscope]

I could simplify those carets into a variable so i have now 5 carets
I still like to understand the caret escaping mechanism.

Code: Select all

REM @echo off
setlocal enableDelayedExpansion
(set \n=^^^
%= This creates an escaped Line Feed - DO NOT ALTER =%
)

(set text=11x y z%\n%
22d e f%\n%
33q w x)

rem A simple loop
for /F "tokens=1-2 skip=1 delims= " %%G in ("!text!") do (echo %%G -+- %%H)

echo --- Dynamic Loop ---


set "param="tokens=1-2 skip=1 delims= ""

rem With 5 carets
set "crt=^^^^^!"
call :myloop "for /F ^!param^! %%%%G in ("!crt!text!crt!") do (echo %%%%G -- echo %%%%H ) " 

endlocal

goto :eof

:myloop
setlocal

set "s=%~1"
%s%
echo end

endlocal

exit /b

[T3RRY]

Here's a simple test script to show how caret escaping occurs. A good tip when your struggling to understand whats happening with the commands your parsing is to NOT use @echo off, so you can see how each command is being parsed
I've tailored this example to the delayed expansion environment and escaping of ! expansion character you're efforts are focused on.

Code: Select all

Setlocal DisableDelayedExpansion
Set "E[re]=!"
Setlocal EnableDelayedExpansion

Set "E[0]=!"
Set "E[1]=^!"
Set "E[2]=^^!"
Set "E[3]=^^^!"
Set "E[4]=^^^^!"
Set "E[5]=^^^^^!"
Set "var=String"
@Set E[
@For /F "tokens=1,2 Delims==" %%A in ('Set E[') Do (
	Echo/%%A = %%B
	Echo/%%Bvar%%B
	Set "Var2=Echo/%%Bvar%%B"
	CALL %%Var2%%
)
Pause

[jeb]

Hi lockedscope,

So, how could that 10/11 carets works?

The problem is, in your case there are multiple escape phases!

Code: Select all

 Start: "for... "^^^^^^^^^^!text^^^^^^^^^^! ...."
phase2: "for... "^^^^^!text^^^^^! ..."
phase5: "for... "^^!text^^! ..."

CALL doubling: "for... "^^^^!text^^^^! ..."
Restart parsing phases- up to phase2
phase2: "for... "^^!text^^! ..."

:jump to myfunc
:now we are in the function at the line set "s1=%~1"
The content of %1  is "for... "^^!text^^ ..."
The content of %~1 is for... "^^!text^^ ...
phase2: for... "^!text^! ..."
phase5: for... "!text! ..."

That's all, more or less obvious

[lockedscope]

Hi jeb, thanks ,

that's very obvious now, i am already trying to understand phases. It is gorgeous how you could find out all those phases, very good job.