New Function Template - return ANY string safely and easily

[dbenham]

OK - I've finally posted examples on how to use these macros for macro development at Batch "macros" with arguments - Major Update

-------------------------------------------------

OMG - My original post on this thread (not the link above) has two major bugs that cancel each other out and the end result is correct

The Rtn1 and RtnN macros have dead code where I am manipulating variables named var but these variables don't tie into the rest of the code! Jeb's original method that I was encapsulating required the extra code. But the code isn't needed in my macro because it uses a FOR loop to get past the ENDLOCAL in a macro, and this negates the need for the extra code! How's that for dumb luck.

The most recent cleaned up version of the library is available in the link above. But the test code here is not yet compatible with it. I'll try to update the code in this thread soon, but I am running out of steam.

Dave Benham

[Ed Dyreen]

'
I was cleaning up this function in my library. My eye fell on this line:

Code: Select all

call set "$RetVal=%%^$RetVal:^!=""^!%%" !

This is Ben's line

Code: Select all

call set "rtn=%%^rtn:^!=""^!%%" ! %\n%

I vaguely remember the first ^ was needed but I'm unsure, a simple test indicates it's unnecessary.

Code: Select all

call set "rtn=%%rtn:^!=""^!%%" ! %\n%

Anyone remembers why

[dbenham]

'
I was cleaning up this function in my library. My eye fell on this line:

Code: Select all

call set "$RetVal=%%^$RetVal:^!=""^!%%" !

This is Ben's line

Code: Select all

call set "rtn=%%^rtn:^!=""^!%%" ! %\n%

I vaguely remember the first ^ was needed but I'm unsure, a simple test indicates it's unnecessary.

Code: Select all

call set "rtn=%%rtn:^!=""^!%%" ! %\n%

Anyone remembers why

The line appears in macro.AnyRtn1
There are similar lines in macro.AnyRtnN, macro.Rtn1, and macro.RtnN

My memory is hazy, but I think it had to do with the fact that I call my macros using:

Code: Select all

for /f "tokens=1-26" %%a in

So I think %%r is detected as a FOR variable when I intended it to be part of %%rtn%%. Escaping the lead character of the variable name avoided the problem. If my variable name began with upper case then I don't think the escape is necessary. Perhaps in your modified code your variable name and FOR variables have a case mismatch so the escape is not necessary for you. (or perhaps this is all meaningless rant, I don't have time to verify right now.)

Dave Benham

[Ed Dyreen]

'
Thanx ben, I really appreciate it a lot, call me stupid, I'm just not getting it !

Code: Select all

set macro.RtnN=do (%\n%
  setlocal enableDelayedExpansion%\n%
  set /a "macro.Rtn1.ErrLvl=(%%~a), macro.Rtn1.EndLocalCnt=(%%~b)+2"%\n%
  set "cmd="%\n%
  for /l %%n in (1,1,!macro.Rtn1.EndLocalCnt!) do set "cmd=!cmd!endlocal!lf!"%\n%
  for %%c in (%%~c) do for /f "tokens=1,2 eol=: delims=:" %%c in ("%%c") do (%\n%
    set "rtn=!%%~c!"%\n%
    if defined rtn (%\n%
      if not defined NotDelayed (%\n%
        set "rtn=!rtn:^=^^!"%\n%
        set "rtn=!rtn:"=""Q!^"%\n%
        call set "rtn=%%^rtn:^!=""E^!%%" ! %\n%
        set "rtn=!rtn:""E=^!"%\n%
        set "rtn=!rtn:""Q="!^"%\n%
      )%\n%
      set ^"var=!var:^"=^^^"!^"%\n%
      set "var=!var:&=^&!"%\n%
      set "var=!var:|=^|!"%\n%
      set "var=!var:<=^<!"%\n%
      set "var=!var:>=^>!"%\n%
      set "var=!var:(=^(!"%\n%
      set "var=!var:)=^)!"%\n%
    )%\n%
    set "cmd=!cmd!set "%%~d=!rtn!"^!!lf!"%\n%
  )%\n%
  if defined macro_inFcn (%\n%
      set "cmd=!cmd!exit /b !macro.Rtn1.ErrLvl!!lf!"%\n%
  ) else if "!macro.Rtn1.ErrLvl!" == "0" (%\n%
      rem errorlevel set to 0 by default%\n%
  ) else (%\n%
      set "cmd=!cmd!cmd /c exit !macro.Rtn1.ErrLvl!!lf!"%\n%
  )%\n%
  for /f "delims=" %%v in ("!cmd!") do %%v%\n%
)

I am having problems with this code block:

Code: Select all

      set ^"var=!var:^"=^^^"!^"%\n%
      set "var=!var:&=^&!"%\n%
      set "var=!var:|=^|!"%\n%
      set "var=!var:<=^<!"%\n%
      set "var=!var:>=^>!"%\n%
      set "var=!var:(=^(!"%\n%
      set "var=!var:)=^)!"%\n%

Why are u doing that ? var isn't mentioned anywhere but in this block, and why are you escaping those symbols ?
My return seems to work without that code block, what am I missing ?

[dbenham]

I am having problems with this code block:

Code: Select all

      set ^"var=!var:^"=^^^"!^"%\n%
      set "var=!var:&=^&!"%\n%
      set "var=!var:|=^|!"%\n%
      set "var=!var:<=^<!"%\n%
      set "var=!var:>=^>!"%\n%
      set "var=!var:(=^(!"%\n%
      set "var=!var:)=^)!"%\n%

Why are u doing that ? var isn't mentioned anywhere but in this block, and why are you escaping those symbols ?
My return seems to work without that code block, have a look at my return macros, what am I missing ?

You are referring to outdated code - You missed my 23 Jun 2011 18:12 post (4 posts up on this page)
I discovered the same issue and provided a link to the corrected code.

Dave Benham

[Ed Dyreen]

'
I've tried to optimize this function ( from different threads ) .

First of all I change %%~1 to %%1, it isn't required here and saves us 1char per percent replace.
Next I split the variable in two chunks because of the risk it grows too large while replacing the '!' char, with the help of '""'.
Then I use two variables/lines to push over endlocal, because jeb explained there is an 8k limit per line

The maximum length I can push over is now 8192 - 2560 chars using this repeated test string:

Code: Select all

_##^#"q"ex!#%1#_

Here are the changes, ( please ignore the personalized code ):

Code: Select all

%=   =% set ^"@macroEnd=( set "$details=" ^&setlocal enableDelayedExpansion %$n1c%
%=      =% for %%? in ( %$n1c%
%=         =% "!$defines!" %$n1c%
%=      =% ) do    set "$=" ^&set "$=!%%~?!" ^&^&^>^&2 ( %$n1c%
%=         =% echo. ^&echo.$defines=!$defines! %$n1c%
%=         =% set  "$1=!$:~0,4096!" ^&set "$2=!$:~4096!" ^&for %%? in ( %$n1c%
%=            =% 1,2 %$n1c%
%=         =% ) do if defined $%%~? ( %$n1c%
%=            =% echo.$%%~?=!$%%~?! %$n1c%
%=            =% set  "$=!$%%~?!" %$n1c%
%=            =% set  "$=!$:%%=%%1!" %=            Replace by injection    =% %$n1c%
%=            =% set ^"$=!$:^"=%%~2!^" %$n1c%
%=            =% for %%r in ( "!$cr!" ) do set "$=!$:%%~r=%%~3!" %$n1c%
%=            =% for %%r in ( "!$lf!" ) do set "$=!$:%%~r=%%~4!" %$n1c%
%=            =% if not defined $NotDelayedFlag ( %$n1c%
%=               =% call set "$=%%^$:^!=""^!%%" ! %$n1c%
%=               =% set "$=!$:^=^^!" ^&set "$=!$:""=^!" %$n1c%
%=            =% ) %$n1c%
%=            =% set  "$%%~?=!$!" %$n1c%
%=            =% echo.$%%~?=!$%%~?! %$n1c%
%=         =% ) %$n1c%
%=      =% ) ^|^|call :endSimple "@macroEnd, not defined: '!$defines!'" %$n1c%
%=      =% set     "@GO=!preFetchRet_! endlocal &endlocal &endlocal &(" %$n1c%
%=      =% set "@COMMIT=set "!$defines!=!$1!!$2!"^!)" ^&if not defined @COMMIT call :endSimple "@macroEnd, variable too big: '!$defines!'" %$n1c%
%=      =% echo. ^&^<nul set /p "= ^<^< !$defines! [OK]" %$n1c%
%=      =% ^^^>^^^> "succes.log" echo.!$defines! %$n1c%
%=   =% )"

Code: Select all

   setLocal enableDelayedExpansion
   :: (
      %= theoretical maximum 8192, real maximum 8192 - 10 =%
      set "$defines=$8kVar"
      >nul ( %macroStart_% "enableExtensions enableDelayedExpansion" )
      :: (
         set "$8kVar=_##^^#"q"ex^!#%%1#_" %=    16 chars initially =%
         for /l %%! in ( 1, 1, 11 ) do set "$8kVar=!$8kVar!!$8kVar!"
         set "$8kVar=!$8kVar!!$8kVar:~0,-2560!"
         echo.$8kVar=!$8kVar!_
      :: )
      2>nul %@macroEnd%
      ::
      ( %@GO%
        %@COMMIT%
      )
      ::
      echo. &echo.$8kVar=!$8kVar!_
   :: )
   endlocal

The third change to spread the push over 2 variables/lines is a guess, I didn't know how to test the difference

How about it, is this good

[jeb]

Hi Ed,

the main idea seems to be good, as the return will fail, if the string contains too much substitutions.
Ex. if the string contains 6000 characters of percent,linefeed,CR, or quote.

But as some of them still are substituted by a three character sequence, you need to split it into three chunks to be absolutly sure.

jeb

[Ed Dyreen]

'
[edit 31/01/12] fixed variable overflow detection

But as some of them still are substituted by a three character sequence, you need to split it into three chunks to be absolutly sure.

I mainly return macro's, 99.9% is good enough for me...

I've been happily using the safe return macro for a while now

Code: Select all

%=   =%set ^"EndlocalR_=(set ^"$replace=%% ^"^"^" !$cr!!$cr!^"%$n1c%
%=   =%)^&for /f "tokens=1-3" %%1 in ("!$replace!") do for %%4 in ("!$lf!") do endlocal^&endlocal^&"

Using the following test string, the practical maximum is somewhere around 8192-2515 characters.

Code: Select all

set "$8kVar=_##^^#"q"ex^!#%%1#_" %=    16 chars initially =%
for /l %%! in ( 1, 1, 11 ) do set "$8kVar=!$8kVar!!$8kVar!"
set "$8kVar=!$8kVar!!$8kVar:~0,-2515!"

I already figured the %%~1 tilde replacement was unnecessary.
I wondered why ben &jeb tripled the quotes, and used the tilde during the %%~2 and %%~3 replacement.
I was baffled to discover again it's unnecessary.

Using this optimization, the practical maximum is somewhere around 8192-1965 characters.

Code: Select all

:: --------------------------------------------------------------------------------------------------------------------------
%Pre_% EndlocalR_
:: --------------------------------------------------------------------------------------------------------------------------
:: last updated       : 30/01/2012
:: support            : onDelayed
:: style              : dBenham's replace
::
:: push over endlocal using replace technique
:: (
%=   =%set ^"EndlocalR_=(set ^"$replace=%% ^" !$cr!!$cr!^"%$n1c%
%=   =%)^&for /f "tokens=1-3" %%1 in ("!$replace!") do for %%4 in ("!$lf!") do endlocal^&endlocal^&"
:: )
%Post_% EndlocalR_ [ok:loaded]
>> "succes.log" echo.EndlocalR_
::
setlocal enabledelayedexpansion
:: (
   setlocal &setlocal &%EndlocalR_% set "$cr=%%3"   !
   ::
   %@necho_% $cr failed !$cr! $cr succes

   setlocal &setlocal
   :: (
      echo.
   :: )
   %EndlocalR_% ( %=                      skip a line to max return size to 8k   =%
      ::
      set "$var=one=%%1_%%~4 two=%%2_"   !
   )
   ::
   %@necho_% !$var!
:: )
endlocal
:: --------------------------------------------------------------------------------------------------------------------------
::goto :skip "()"
%@endoftest%
:skip ()

Code: Select all

 >> EndlocalR_
 << EndlocalR_ [ok:loaded]
 $cr succes

 one=%_
 two="_
 endoftestDruk op een toets om door te gaan. . .

The Linefeed is a totally different matter though, I failed in finding a way that didn't require the ~, for /f didn't work

Code: Select all

:: --------------------------------------------------------------------------------------------------------------------------
%Pre_% @macroEnd
:: --------------------------------------------------------------------------------------------------------------------------
:: last updated       : 31/01/2012
:: support            : naDelayed, $cr, $lf, theoretical max 8192, real max 8192-10, replacer practical max 8192-+-1965
:: style              : dBenham's replace
::
:: Prepares $defined to be pushed over endlocal using immediate expansion + replace technique,
:: regardless the outer delayed setting
:: (
%=   =%set ^"@macroEnd=(setlocal enableDelayedExpansion%$n1c%

%=      =%(echo.^&if defined $NotDelayedFlag (set/p= NotDelayed^<nul) else set/p= Delayed^<nul)^>^&2%$n1c%
%=      =%%forQ_% ("!$defines!") do set $=^&set $=!%%~?!^&^&(%$n1c%
%=         =%(echo.^&set/p= ^<nul^&set "%%~?")^>^&2%$n1c%
%=         =%set $1=!$:~0,4096!^&set $2=!$:~4096!^&%forQ_% (1,2) do if defined $%%~? (%$n1c%
%=            =%set $=!$%%~?!^&set $=!$:%%=%%1!^&set ^"$=!$:^"=%%2!^"%=   Replace by injection   =%%$n1c%
%=            =%for %%r in ("!$cr!") do set $=!$:%%~r=%%3!%$n1c%
%=            =%for %%r in ("!$lf!") do set $=!$:%%~r=%%~4!%$n1c%
%=            =%if not defined $NotDelayedFlag call set "$=%%^$:^!=""^!%%"!^&set "$=!$:^=^^!"^&set "$=!$:""=^!"%$n1c%
%=            =%set $%%~?=!$!%$n1c%
%=         =%)%$n1c%
%=      =%)^|^|call :endSimple "@macroEnd, not defined: '!$defines!'"%$n1c%
%=      =%set $=^&set $=set "!$defines!=!$1!!$2!"^!^|^|call :endSimple "@macroEnd, variable overflow: '!$defines!'"%$n1c%
%=      =%(echo.^&set/p= !$!^<nul)^>^&2%$n1c%
%=      =%echo.!$defines!^>^>"succes.log"%$n1c%
%=      =%echo.^&set/p"= ^<^< !$defines! [ok:loaded]"^<nul%$n1c%

%=   =%)"
:: )
%Post_% @macroEnd [ok:loaded]
>> "succes.log" echo.@macroEnd
:: (
   setlocal enableDelayedExpansion &set "$defines=$8kVar"
   :: (
      %@n2echo_% testing replacer practical max 8192-+-1965

      2>nul ( %macroStart_% enableDelayedExpansion )
      :: (
         set "$8kVar=_##^^#"q"ex^!#%%1#_" %=    16 chars initially =%
         for /l %%! in ( 1, 1, 11 ) do set "$8kVar=!$8kVar!!$8kVar!"
         set "$8kVar=!$8kVar!!$8kVar:~0,-1965!"
      :: )
      %@macroEnd%
      %EndlocalR_% (
%=      =%%$%
      )

      %@n2echo_% $8kVar=!$8kVar!_
   :: )
   endlocal
%@endoftest%

Code: Select all

 >> @macroEnd
 << @macroEnd [ok:loaded]

 testing replacer practical max 8192-+-1965

 >> $8kVar
 Delayed
 $8kVar=_##^#"q"ex!#%1#__##^#"q"ex!#%1#__##^#"q"ex!#%1#__##^#"q"ex!#%1#__##^#"q"
...
ex!#%1#__##^#"q"ex!#%1#__##^#"q"ex!#%1#__##^#"q"ex!#%1#__##^#"q"ex!#%1

 set "$8kVar=_##^^#%2q%2ex^!#%11#__##^^#%2q%2ex^!#%11#__##^^#%2q%2ex^!#%11#__##^
...
__##^^#%2q%2ex^!#%11"!
 << $8kVar [ok:loaded]

 $8kVar=_##^#"q"ex!#%1#__##^#"q"ex!#%1#__##^#"q"ex!#%1#__##^#"q"ex!#%1#__##^#"q"
...
ex!#%1#__##^#"q"ex!#%1#__##^#"q"ex!#%1#__##^#"q"ex!#%1#__##^#"q"ex!#%1_
 endoftestDruk op een toets om door te gaan. . .

I implemented the code, it works perfectly !
If you believe you can optimize, even if it's just a single char, please let me now...

[Ed Dyreen]

just want to say that this can also be done using dosKey's ability to store and retrieve variables from dosKey memory which is unaffected by setlocal and endlocal thus always Global.

http://www.dostips.com/forum/viewtopic.php?p=41362#p41362