Observations about the tilde-dollar syntax %~$variable:I

[jeb]

I would like to share some insights about the rarely used syntax of "%~$PATH:I"

%~$PATH:I Searches the directories listed in the PATH environment variable and expands %I to the fully qualified name of the first directory found. If the environment variable name isn't defined or the file isn't found by the search, this modifier expands to the empty string.

I try to explain this a bit with an example.

Code: Select all

@echo off
mkdir subdir
echo X > subdir\dummy.tmp
setlocal EnableDelayedExpansion
set "__definedVar=%~dp0subdir"
set "undefinedVar="
set "p=%%"
FOR %%X in (dummy.tmp) do (
    FOR %%Y in (%~dp0subdir\dummy.tmp) do (
        echo X contains %%X
        echo Y contains %%Y
        echo ------ Using X -------
        echo #1 !p!!p!~$__definedVar:X = %%~$__definedVar:X
        echo #2 !p!!p!~$undefinedVar:X = %%~$undefinedVar:X
        echo #3 !p!!p!~$:X             = %%~$:X
        echo ------ Using Y -------
        echo #4 !p!!p!~$__definedVar:Y = %%~$__definedVar:Y
        echo #5 !p!!p!~$undefinedVar:Y = %%~$undefinedVar:Y
        echo #6 !p!!p!~$:Y             = %%~$:Y
        echo ------ Using Z -------
        echo #7 !p!!p!~$__definedVar:Z = %%~$__definedVar:Z
        echo #8 !p!!p!~$undefinedVar:Z = %%~$undefinedVar:Z
        echo #9 !p!!p!~$:Z             = %%~$:Z
    )
)

The output is:

Code: Select all

X contains dummy.tmp
Y contains E:\syntax\subdir\dummy.tmp
------ Using X -------
#1 %%~$__definedVar:X = E:\syntax\subdir\dummy.tmp
#2 %%~$undefinedVar:X =
#3 %%~$:X             =
------ Using Y -------
#4 %%~$__definedVar:Y = E:\syntax\subdir\dummy.tmp
#5 %%~$undefinedVar:Y =
#6 %%~$:Y             = E:\syntax\subdir\dummy.tmp
------ Using Z -------
#7 %%~$__definedVar:Z = %~$__definedVar:Z
#8 %%~$undefinedVar:Z = %~$undefinedVar:Z
#9 %%~$:Z             = %~$:Z

Here are some interesting things
1) Undefined variables and an empty variable names behave differently (#5 and #6). Perhaps an empty variable name accesses some random existing variable
2) If the meta variable isn't defined, the complete expression does not expand (#7, #8, #9)
3) If the meta variable contains an absolute filename, the searchVariable is no longer used, only the meta variable is used.
instead the absolute filename is used
4) If the meta variable contains a relative filename (starts with a dot or backslash), the searchVariable won't be used anymore, instead the relative filename will be appended to the current directory use

And here are a few other findings:
5) The tilde-dollar expression work also outside of FOR-Loops with normal percent expansion, but only the meta-vars 0..9 are valid

Code: Select all

  echo %~$PATH:1 -- works always
  echo %~$PATH:9 -- works always
  echo %~$PATH:A -- fails with a syntax error
  echo %~$PATH:* -- fails with a syntax error

5b) No meta-variables seem to exist in the command line context, but no syntax error is generated for this either

6) The expansion in FOR-Loops happens just BEFORE a loop block is executed, but the expansion is done for each loop!

Code: Select all

  setlocal EnableDelayedExpansion
  set "var=invalid-path"
  FOR %%# in (calc.exe CALC.EXE) DO (
    set "var=C:\windows\system32"
    echo var='!var!'  expr = %%~$var:#
  )

Only the second loop shows a result

Code: Select all

var='C:\windows\system32'  expr =
var='C:\windows\system32'  expr = C:\Windows\System32\calc.exe

7) pseudo variables are not found, works like undefined variables

Code: Select all

setlocal
set "var=is defined"
set "undef="
set "DATE=never set a pseudo variable"
FOR %%# in (C:\windows) DO (
    echo #1        var = %%~$var:#
    echo #2      undef = %%~$undef:#
    echo #3       time = %%~$time:#
    echo #4         cd = %%~$cd:#
    echo #5     __cd__ = %%~$__cd__:#
    echo #6 cmdcmdline = %%~$cmdcmdline:#
    echo #7       date = %%~$date:#
)

Output

Code: Select all

#1        var = C:\Windows
#2      undef =
#3       time =
#4         cd =
#5     __cd__ =
#6 cmdcmdline =
#7       date = C:\Windows

The syntax for the searchVariable is a bit unexpected
- You can't create a searchVariable where the name is invalid. In other words every searchVariable can be crafted

Code: Select all

FOR %%# in (C:\windows) DO (
    echo #1 %%~$varcraft:#
    echo #2 %%~$=varcraft=:#
    echo #3 %%~$, =;varcraft, =; Even this can be defined:#

The naming rules for searchVariables are:
- Remove all delimiters (<space>, <tab>, comma, semi colon, equal sign) until the first non delimiter is found
- Only the last space stops the name (this space isn't part of the name)
- The parsing ends at the first colon, obviously a colon can not be part of any searchVariable name

For expample #3 "%%~$, =;varcraft, =; Even this can be defined:#" a matching variable can be defined by

Code: Select all

set "varcraft, =; Even this can be=content"

Useful behavior
Currently I use it in my batch library to detect if meta-variables are defined somewhere in the call chain or not by a simple:

Code: Select all

set "undef="
FOR %%1 in (1) do (
  if "%%~$undef:X" == "" (
    echo  meta-var X is defined
  ) else (
        echo  meta-var X is NOT defined
  )
)

Fixing weak meta-variables
Theses characters are weak meta-variables "adfnpstxzADFNPSTXZ", because they are also valid modifiers for meta-variables.
It's only a problem, when using this meta-variables with a tilde, without they are also safe.

This example shows the problem

Code: Select all

@echo off

call :func
echo(
FOR %%s in (hello) DO (
    call :func
)
exit /b

:func
for /L %%d in (1 1 2) do (
    echo Count: %%d Duration %%~dseconds
)

Count: 1 Duration 1seconds
Count: 2 Duration 2seconds

Count: 1 Duration E:econds
Count: 2 Duration E:econds

This can be solved by adding a tilde-dollar expression

Code: Select all

@echo off
setlocal DisableDelayedExpansion

REM *** The $FOR-BREAK can be used for the expansion of WEAK for-variables with tilde
REM *** %% ~f % $FOR-BREAK%  (without spaces)
REM *** This works as long as no percent meta-variable exists
set "$FOR-BREAK=%%~$=_FOR-variable_DOLLAR_is_required_=:$"

call :func
echo(
FOR %%s in (hello) DO (
    call :func
)
exit /b

:func
for %%$ in (1) do for /L %%d in (1 1 2) do (
    echo Count: %%d Duration %%~d%$FOR-BREAK%seconds
)

Count: 1 Duration 1seconds
Count: 2 Duration 2seconds

Count: 1 Duration 1seconds
Count: 2 Duration 2seconds

----
These were some notes on tilde-dollar expressions.
I hope that some interesting aspects were included.

jeb

[jeb]

1) Undefined variables and an empty variable names behave differently (#5 and #6). Perhaps an empty variable name accesses some random existing variable

After some more testing, I found the behavior for an empty searchPath-variable name.

If you use the expression "%%~$:X" or ""%%~$====:X" the name of the searchPath-variable is empty.
But this results in an access to the first variable of your system, in my case it is the hidden "=::" variable

Code: Select all

@echo off
cls
setlocal EnableDelayedExpansion
call :cleanup
call :init_dummy_file

call :test

call :SetHiddenVariable
call :test

call :cleanup
exit /b

:test
echo(
call :showHiddenVariable
for /F %%P in ("%%%%") do FOR %%X in (dummy.tmp) do (
  echo X contains %%X
  echo -----------------
  echo #1 %%P~$:X  [empty varname]     = %%~$:X
)
exit /b

:init_dummy_file
REM *** Create special directory to be able
REM *** to set the hidden variable "=::" to "::\substroot;subdir"
mkdir "substroot" 2> NUL
mkdir "substroot;subdir" 2> NUL

REM *** Create testfile: subdir\dummy.tmp
mkdir subdir 2> NUL
echo x > subdir\dummy.tmp
exit /b

:SetHiddenVariable
REM *** Create Drive "::"
subst :: "%~dp0" 2> NUL

REM *** Modify the hidden variable ""=::" to contain ""::\;subdir"
REM *** Prepare the drive "::" and set the current directory for that drive
REM *** This is tricky: CD doesn't work as expected, it can't switch back to drive C:
REM *** This can be solved by using PUSHD, CD and POPD
PUSHD "::\substroot;subdir" 2> NUL
CD .
POPD
exit /b

:showHiddenVariable
setlocal DisableExtensions
echo The hidden variable "=::" contains "%=::%"
endlocal
exit /b

:cleanup
subst /D :: > NUL
rmdir /q /s substroot 2> NUL
rmdir /q /s subdir 2> NUL

The hidden variable "=::" contains "::\"
X contains dummy.tmp
-----------------
#1 %%~$:X [empty varname] =

The hidden variable "=::" contains "::\substroot;subdir"
X contains dummy.tmp
-----------------
#1 %%~$:X [empty varname] = C:\Users\jeb\bat\syntax\subdir\dummy.tmp

This works as the "::\substroot;subdir" is used as searchPath and it's spilt at every semi colon.
And relative path components like "subdir" are combined with the current working directory.

The simple search rule for a variable seems to be:
1) Append to the search variable name one "=" and use this as search string
2) For all stored variables do:
- Take n-characters from the from, where n is the length of the search string
- Compare both strings, when they match take the remaining characters of the variable after the n-th character, as the search-path

Then for the path search itself
3) Split the search-path at every semi colon ";", ignore any quotes or escape characters.

Therefore you never can find a directory or file with a directory name containing a semi colon.

[aGerman]

Great investigation as always, jeb!
However, you probably don't believe my first reaction when I saw the topic
Like: "Wait a second ... What the ...? I could have also used anything else but PATH in all the years that I write batch code? D'oh "

[jeb]

After reading Batch: How to Properly Use CHOICE Inside of CALL Function?
The problem was probably a previously set of the errorlevel variable.

I was curious whether it is possible to recognize this programmatically.

"if defined variableName" doesn't work here, because it's true for pseudovariables, too.

But there is a way to recognize if a pseudo variable is shadowed with the %~$ syntax.

It's using the fact that "%~$searchVarname:#" always expands to an empty string if searchVarname is not a defined (normal) variable.
But it expands to a filename if the meta-var # contains a valid absolute filename and searchVarname is defined.
The content of searchVarname doesn't matter in that case.

Code: Select all

@echo off
setlocal
call :checkPseudoVariable res errorlevel
echo #1 errorlevel is %res%

set "errorlevel=1"
call :checkPseudoVariable res errorlevel
echo #2 errorlevel is %res%

set "errorlevel="
call :checkPseudoVariable res errorlevel
echo #3 errorlevel is %res%

call :checkPseudoVariable res date
echo #4 date is %res%

exit /b

:checkPseudoVariable <resultVar> <PseudoVar>
setlocal
set "_resultVar=%~1"
set "_pseudoVar=%~2"
FOR %%# in ("%~f0") do (
  if "%%~$%_pseudoVar%:#" == "" (
      set "_result=is undefined"
  ) ELSE (
      set "_result=is defined"
  )
)
(
  endlocal
  set %_resultVar%=%_result%
)
exit /b