FOR variable %%A in macro definition gets overridden when script is called by other script within FOR loop

[Maylow]

Situation description:
I have some scripts which define macros within FOR loops and these macros are called using FOR loops.
I have some other scripts which call these scripts within FOR loops.
When doing so, i noticed that FOR variables within macro definitions are overridden.

I have a work-around, but i would like to know:
Q: Is this behavior to be expected?
Q: Is there a way to call scripts within FOR loops AND to keep %%A empty when defining macros within FOR loops?
Note: delayed expansion is not acceptable as part of the solution.

The actual used code in my project imho does not serve the topic.
I created some example scripts which reproduce the situation as described.

Script which defines macros within FOR loops using FOR loops, i named it callee.cmd:

Code: Select all

@echo off
REM @SITUATION: FOR variable %%A gets overridden when script is called by other script within FOR loop.
REM @NOTE:      %%A doesn't stay empty as expected while defining the macro, 
REM             instead it expands to the FOR variable of the caller script.
REM @TODO:      find a way to call script within FOR loop AND to keep %%A empty 
REM             when defining the macros.
REM @FIX:       no fix yet, work-around is to call script directly, 
REM             NOT within FOR loop context.

REM Define macro to call macros:
set __call=for %%A in 

for %%g in (__macro) do (
    REM Define sample macro in FOR loop:
    REM @NOTE: here, the variable "%%~A" gets overridden when this script is called by other script within FOR loop:
    set "%%~g= do (if /i "%%~A" equ "_some value" (echo("some value" detected) else (echo(incorrect value detected))"
)

REM Display contents of macros:
set __call
set __macro

REM Call macro with argument:
%__call%("_%*") %__macro%
exit /b

Execute above script by doing: callee.cmd some value
This will give expected results:
__call=for %A in
__macro= do (if /i "%~A" equ "_some value" (echo("some value" detected) else (echo(incorrect value detected))
"some value" detected


An other script, which i named caller_directly.cmd, demonstrates correct results when callee.cmd is called via this script:

Code: Select all

@echo off
REM @NOTE: calling script.
call callee.cmd %*
exit /b

Execute above script by doing: caller_directly.cmd some value
This will also give expected results:
__call=for %A in
__macro= do (if /i "%~A" equ "_some value" (echo("some value" detected) else (echo(incorrect value detected))
"some value" detected


The last script, which i named caller_FOR.cmd, invokes and reproduces the issue:

Code: Select all

@echo off
for %%A in (whatever) do (
    REM @NOTE: calling script within FOR loop context.
    call callee.cmd %*
)
exit /b

Execute above script by doing: caller_FOR.cmd some value
This will give the following results:
__call=for %A in
__macro= do (if /i "whatever" equ "_some value" (echo("some value" detected) else (echo(incorrect value detected))
incorrect value detected

Notice the "whatever" part has overridden the "%%~A" part in the macro definition.

If this behavior is to be expected, and no solution is available, i'll stick to the work-around of NOT calling such scripts within FOR loop context but always call these scripts directly.

Edit: using %%a instead of %%A doesn't really solve the issue. %%A here is just an example.

[T3RRY]

I do believe this to be a known issue, but attempted search terms failed to locate a suitably relevant topic.

From the help output for For /?

Remember, FOR variables are single-letter, case sensitive, global,
and you can't have more than 52 total active at any one time.

For variables can be repeated when calling outside of a for loop within the same script, however it's a practice that's best avoided.
If you wish to stick with this approach, reserving a non alphabetical character as the for metavariable character for your definition script is a suitable approach, however:

The principle advantage to batch macro's is in NOT calling external scripts / subroutines for the sake of improved performance by using a nested for loop to "capture" an argument. This does however require delayed expansion. If you'd care to elaborate on your actual usage situation/s and the reason why delayed expanion in not suitable, I may be able to suggest a more efficient alternative - as calling another program (or subroutine) countless times during a for loop is far slower than calling a script once to define the macro then expanding a macro command during the for loops execution

[Maylow]

I see. I know that they are global.
The external scripts are, what i call, libraries with macro definitions. That's the reason why they are called externally, to load the macro definitions.
These libraries are only called once, to load their macro definitions.
And delayed expansion is not acceptable as these macros are to be defined globally.

Most library scripts define their macros per line of code.
However, i have one or two such libraries which define a set of macros in a FOR loop.
It is in this situation that the issue occurs.
For now, i can stick to having these libraries called per line instead of in a FOR loop.
But a more elegant solution would be appreciated

[Maylow]

By the way, about:

Remember, FOR variables are single-letter, case sensitive, global,
and you can't have more than 52 total active at any one time.

The documentation is actually incorrect about itself.
The 52 characters limitation ought to be based on a-z A-Z.
However, 0-9 and other characters like ?, @ and * are also allowed.
For example:

Code: Select all

for %1 in (test) do echo(%1

works.
About 70 characters (don't know them all exactly) are allowed active at one time i believe.

[T3RRY]

The simplest solution then that wouldn't require any drastic change to your process would be to as mentioned previously use a non alphabetical for variable Ie:

Code: Select all

%%@

in the definition macro - any usable character that you would not ordinarialy otherwise use in any library that calls the macro definer

[Maylow]

Ah yea thought about that too.
But chicken-egg problem:
Macros are called with a caller macro which uses %%A.
Macro definitions use %%A as argument input.
The library scripts are called using a macro which uses %%A.
Some library scripts define macros wihtin a FOR loop (with whatever %%*, that doesn't matter).
Hence, the chicken-egg problem when i call a library using a macro within a FOR loop.

[Maylow]

Thus far, the work-around is:
Either i don't define macros within FOR loops, or have such scripts called per line instead of within FOR loops.

Thanks for thinking with me by the way, appreciated!

[T3RRY]

you cannot pass the value of a for variable to another script or subroutine in that way. The script / subroutine is proccessed in a new child environment outside the for loop, and as such does not have access to the for loops variable. You would have to assign the for variable to a "global" variable in the parent and use that global variable in the child script to proccess the value in the called script without passing it as an argument in the traditional way. Alternately, store the value in a temporary file and read it in via the called script.

[Maylow]

I don't want to pass the value of a for variable to another script or subroutine, far from that. I want the FOR variable to be left alone when defining the macro.

To resummarize, i have accomplished:

Code: Select all

set __call=for %%A in 
set "__macro= do (if /i "%%~A" equ "_some value" (echo("some value" detected) else (echo(incorrect value detected))"

This works.

When a script like this has been called once, the macro can be called any time during that command session by any script with the following statement:

Code: Select all

%__call%("_some value") %__macro%

As you can see, the macro itself does not represent a script or subroutine, it is simply a FOR loop which performs some actions.

The script / subroutine is proccessed in a new child environment outside the for loop

Ye, that's what thought as well. But this is the issue: when calling example script within a FOR loop, then the %%A in __macro gets overridden.
So, not at all that script / subroutine is proccessed in a new child environment outside the for loop, unfortunately
That's the whole problem.

[Maylow]

Interesting thing i remembered: FOR command also and even accepts unicode characters as variables.

Code: Select all

for %%È in (test) do echo(%%È

This works.
(using È character with Alt+0456 on numeric keypad)

However, it does not solve the situation.

[T3RRY]

you cannot pass the value of a for variable to another script or subroutine in that way. The script / subroutine is proccessed in a new child environment outside the for loop, and as such does not have access to the for loops variable. You would have to assign the for variable to a "global" variable in the parent and use that global variable in the child script to proccess the value in the called script without passing it as an argument in the traditional way. Alternately, store the value in a temporary file and read it in via the called script.

The above was incorrect, The value does persist

The fix is to use a non standard value for the for loop macro. As long as the For loop variable in the calling script isn't %%@, the macro will be defined correctly. Note however, that the variable __macro will be overwritten.
I'm assuming in your actual usage a different macro naming method will be used to account for this?

Code: Select all

@echo off

REM Define macro to call macros:
set "__call=for %%@ in "

for %%g in (__macro) do (
    REM Define sample macro in FOR loop:
    REM @NOTE: here, the variable "%%~A" gets overridden when this script is called by other script within FOR loop:
    Set "%%~g=do if /i "%%~@"=="some value" (echo/"some value" detected) else echo/incorrect value detected"
)

REM Display contents of macros:
set __call
set __macro

REM Call macro with argument:
%__call%(%*) %__macro%

[jeb]

Thanks Maylow for the question

I had the same problem some time ago, and it's because I built also a batch macro library

a) It's expected behaviour
b) It can be solved,
It can be solved even that you can define macros inside any FOR-Loop and independent of delayed mode
It doesn't need setlocal for the definition (and no return out of the setlocal scope), so it's fast.

But there are some conditions:

FOR-Parameter characters: Do NOT use any of these characters as a FOR parameter: $adfnpstxzADFNPSTXZ
Because the FOR-parser does not stop at these characters, if the next character is also a FOR parameter, it takes that one instead
The problem occurs as in any FOR-loop all parameters are visible from a "global" space
Use only one or a range of: bc e ghijklm o qr uvw y 0123456789 #<>"&()[]{}=?!%/\+-*,;.:_
Sample:
For %%. in (DOTDOT) DO FOR %%n in (myfile) do echo %%~n.
Output: "DOTDOT" instead of "myFile."

First, how to solve your original question?
Simple use an immutable percent sign

Code: Select all

set "$PERCENT=%%%%~$=_p1_FOR-variable_DOLLAR_is_required_while_MACRO_DEFINITION_=:$"

REM *** Always use an outer FOR-loop when defining macros
FOR %%$ in (DOLLAR) DO set myMacro=FOR %$PERCENT%Q in (1 2 3) do echo # %$PERCENT%Q

This is a simplified example (and currently I can't test, because I'm mostly using linux these days)

But you can try a look at my batchLibrary, I just pushed it to https://github.com/jeb-de/BatchLibrary
There are some explanations inside "libBase.cmd", and there could be perhaps one or two new useful techiques

jeb

[Aacini]

This "strange" behavior of FOR parameters was previously discussed here (and in a nested link there):

"The "global within a FOR context" design can lead to some nasty surprises. You might write a subroutine with a FOR loop that echos text with a percent literal. The routine works perfectly fine. Then one day you call the routine from within another FOR loop, and all hell breaks loose when the character following the percent literal happens to match a FOR variable in your calling loop."

Antonio

[OJBakker]

The problem is in the for''s.
You don't want the %%A used when you define your macro but only when you use the macro.
The cause of your problem is the for %%g loop in caller.cmd. This for loop makes the %%A available in the do command block.
In your example code you don't need this loop.

replace

Code: Select all

for %%g in (__macro) do (
    set "%%~g= do (if /i "%%~A" equ "_some value" (echo("some value" detected) else (echo(incorrect value detected))"
)

with

Code: Select all

set "__macro= do (if /i "%%~A" equ "_some value" (echo("some value" detected) else (echo(incorrect value detected))"
)

[Maylow]

@All, many thanks for responding and giving a lot of information and explanation!

@T3RRY, your hints and Jeb's explanation make me wondering, i am going to experiment with the suggestions posted here.

@Jeb ahh i see, thanks for the clear explanation.
I will try to implement the suggestions and let you know asap.
Yes, in fact, i was greatly inspired by your batchLibrary earlier! and i am trying expand on it further.
I will examine your latest version of batchLibrary to read any information i missed.

@Aacini, i did search here about the issue on this forum but could not find it with the keywords i used.
I will read more about it at the links you provided.

@OJBakker, yes i know, most of my libraries define macros that way. And the few libraries that do define macros within FOR loops actually do not pose a problem when called directly.
Only when such libraries, which define macros inside FOR loops, are called by another script within a FOR loop, the problem occurs.