and add my own findings while using them for the :CreatePipe routine.
The command interpreter gives access to 10 handles, numbered 0 to 9.
The first three always exist: Handle 0 is standard input; 1 is standard output; 2 is standard error.
The next seven (number 3 to 9) can be defined by the user or a script, using the syntaxes: N>FILENAME or N>&M
The problem is that a batch routine has no way to know which handles have been used beyond the first three, and which are available for use.
Of course the enclosing script can keep track of which handles it is using internally.
But, in general, it may have been invoked in a context where other handles were defined beforehand, either by the user or by other scripts.
This is a problem in some cases, where routines depend on the exact list of free or used handles.
The goal here is to document reliable methods to find which handles are used, and which ones are free.
And, of course, all that without side effects on the existing handles. They must not be written to, or read from.
The only method discovered so far for detecting the presence of a given handle, without side effects, is to try to duplicate it.
Example where handle 2 exists (which is always true), and handle 9 does not (which is the default, but may change):
Code: Select all
C:\Temp>break >&2 && (echo succeeded) || (echo failed)
succeeded
C:\Temp>break >&9 && (echo succeeded) || (echo failed)
The handle could not be duplicated
during redirection of handle 1.
failed
C:\Temp>But doing so exposes another problem: Redirecting a used handle saves it in the first free handle.
So 2>NUL saves the old handle 2 in the first free slot, which is handle 3 by default, but may be further if handle 3, etc, are already in use.
This modifies the phenomenon that we're trying to observe! Some have compared that to quantum uncertainty
Second problem: The redirection test itself will save its first handle, if it's in use, into the first remaining free slot.
This is what happens for handle 1 above: It is saved in the first free slot. (Handle 3 also by default, but possibly much further.)
This second problem can be avoided by using a free handle (which needs not be saved) instead of handle 1 for the first handle.
But initially we don't know which handles are free... Argh! A second cause of quantum uncertainty!
Still, despite all that quantum uncertainty, it's possible to identify with certainty free and used handles in many practical cases.
The initial idea that was proposed was to use handle 9, which presumably has the least chances of being used, for trying to duplicate all others in turn.
Then once some free handles have been found for sure, start over, and use one of them to repeat the test.
The latest routine (as of this writing) using this method is presented in this post.
In its comments, it documents its own limits. In summary, in some cases, there's one, and sometimes two, free handle that are listed as used.
I reused the above routine for my implementation of the :CreatePipe routine in this post.
More precisely, I used an adapted version generating lists instead of arrays, and fixing a bug where some handles were missing in the used list, in cases where many handles were in use.
Also, I added a return variable evaluating the precision of the result. (I hoped)
Still, despite reviewing that code many times, I was not 100% sure it avoided quantum loopholes in all cases! And I was not sure of the reliability of my precision variable.
The size of the problem being small, 7x7 = 49 possible pairs of redirections, I decided to make an exhaustive test.
I wrote this simple test:
Code: Select all
@echo off
setlocal EnableExtensions EnableDelayedExpansion
for /L %%h in (3,1,9) do (
set "free=" &:# List of free file handles
set "used=" &:# List of used file handles
for /L %%i in (3,1,9) do (
if not %%h==%%i call :TryRedir %%h %%i free used
)
echo Redirecting %%h finds free:!free! and used:!used!
)
exit /b
:TryRedir %1=Handle to redirect; %2=Handle to duplicate; %3=Free var; %4=Used var
2>nul (
break %1>&%2 && ( :# The redirection succeeded
set "%4=!%4! %2" &rem The handle %2 existed. Add it to the used list
) || ( :# The redirection failed, and an error message was written to stderr
set "%3=!%3! %2" &rem The handle %2 did not exit. Add it to the free list
)
)
exit /b
Code: Select all
C:\Temp>test
Redirecting 3 finds free: 5 6 7 8 9 and used: 4
Redirecting 4 finds free: 5 6 7 8 9 and used: 3
Redirecting 5 finds free: 4 6 7 8 9 and used: 3
Redirecting 6 finds free: 4 5 7 8 9 and used: 3
Redirecting 7 finds free: 4 5 6 8 9 and used: 3
Redirecting 8 finds free: 4 5 6 7 9 and used: 3
Redirecting 9 finds free: 4 5 6 7 8 and used: 3
C:\Temp>test 3>NUL
Redirecting 3 finds free: 6 7 8 9 and used: 4 5
Redirecting 4 finds free: 6 7 8 9 and used: 3 5
Redirecting 5 finds free: 6 7 8 9 and used: 3 4
Redirecting 6 finds free: 5 7 8 9 and used: 3 4
Redirecting 7 finds free: 5 6 8 9 and used: 3 4
Redirecting 8 finds free: 5 6 7 9 and used: 3 4
Redirecting 9 finds free: 5 6 7 8 and used: 3 4
C:\Temp>test 4>NUL
Redirecting 3 finds free: 6 7 8 9 and used: 4 5
Redirecting 4 finds free: 6 7 8 9 and used: 3 5
Redirecting 5 finds free: 6 7 8 9 and used: 3 4
Redirecting 6 finds free: 5 7 8 9 and used: 3 4
Redirecting 7 finds free: 5 6 8 9 and used: 3 4
Redirecting 8 finds free: 5 6 7 9 and used: 3 4
Redirecting 9 finds free: 5 6 7 8 and used: 3 4
C:\Temp>test 9>NUL
Redirecting 3 finds free: 5 6 7 8 and used: 4 9
Redirecting 4 finds free: 5 6 7 8 and used: 3 9
Redirecting 5 finds free: 4 6 7 8 and used: 3 9
Redirecting 6 finds free: 4 5 7 8 and used: 3 9
Redirecting 7 finds free: 4 5 6 8 and used: 3 9
Redirecting 8 finds free: 4 5 6 7 and used: 3 9
Redirecting 9 finds free: 5 6 7 8 and used: 3 4
C:\Temp>test 3>NUL 4>NUL
Redirecting 3 finds free: 7 8 9 and used: 4 5 6
Redirecting 4 finds free: 7 8 9 and used: 3 5 6
Redirecting 5 finds free: 7 8 9 and used: 3 4 6
Redirecting 6 finds free: 7 8 9 and used: 3 4 5
Redirecting 7 finds free: 6 8 9 and used: 3 4 5
Redirecting 8 finds free: 6 7 9 and used: 3 4 5
Redirecting 9 finds free: 6 7 8 and used: 3 4 5
C:\Temp>test 5>NUL 6>NUL
Redirecting 3 finds free: 7 8 9 and used: 4 5 6
Redirecting 4 finds free: 7 8 9 and used: 3 5 6
Redirecting 5 finds free: 7 8 9 and used: 3 4 6
Redirecting 6 finds free: 7 8 9 and used: 3 4 5
Redirecting 7 finds free: 4 8 9 and used: 3 5 6
Redirecting 8 finds free: 4 7 9 and used: 3 5 6
Redirecting 9 finds free: 4 7 8 and used: 3 5 6
C:\Temp>test 5>NUL 8>NUL
Redirecting 3 finds free: 6 7 9 and used: 4 5 8
Redirecting 4 finds free: 6 7 9 and used: 3 5 8
Redirecting 5 finds free: 6 7 9 and used: 3 4 8
Redirecting 6 finds free: 4 7 9 and used: 3 5 8
Redirecting 7 finds free: 4 6 9 and used: 3 5 8
Redirecting 8 finds free: 6 7 9 and used: 3 4 5
Redirecting 9 finds free: 4 6 7 and used: 3 5 8
C:\Temp>test 5>NUL 6>NUL 7>NUL 8>NUL
Redirecting 3 finds free: 9 and used: 4 5 6 7 8
Redirecting 4 finds free: 9 and used: 3 5 6 7 8
Redirecting 5 finds free: 9 and used: 3 4 6 7 8
Redirecting 6 finds free: 9 and used: 3 4 5 7 8
Redirecting 7 finds free: 9 and used: 3 4 5 6 8
Redirecting 8 finds free: 9 and used: 3 4 5 6 7
Redirecting 9 finds free: 4 and used: 3 5 6 7 8
C:\Temp>test 3>NUL 4>NUL 5>NUL 6>NUL 7>NUL
Redirecting 3 finds free: and used: 4 5 6 7 8 9
Redirecting 4 finds free: and used: 3 5 6 7 8 9
Redirecting 5 finds free: and used: 3 4 6 7 8 9
Redirecting 6 finds free: and used: 3 4 5 7 8 9
Redirecting 7 finds free: and used: 3 4 5 6 8 9
Redirecting 8 finds free: and used: 3 4 5 6 7 9
Redirecting 9 finds free: and used: 3 4 5 6 7 8
C:\Temp>Whatever handle is used, the same number of free and used handles is found, but not always the same ones!
Another approach was necessary.
Let's review what we know for sure:
- Redirecting 2>NUL uses the first free handle
- Redirecting a used handle uses another free handle, whereas redirecting a free handle does not
- These two unknown handles are below the first remaining free handle
To remove the quantum uncertainty, instead of trying to avoid these unwanted used handles (which I just proved was impossible), we actually must force their creation!
The algorithm for doing that is easy:
- We know that handles 0 to 3 are always used. 0 to 2 by cmd.exe; 3 either by the user, another script, or the 2>NUL redirection within our test.
Let's use 3 as the redirected handle.
- Redirecting handle 3 creates a backup in the next free slot. So Handle 4 will also be in use, either by the user, another script, or the 3>&M redirection within our test.
- It's sufficient to start testing duplications at handle 5, since 3 & 4 will be found used in any case.
Furthermore, we know that below that first free handle, there are exactly TWO unidentified free handles.
Identifying the second one is easy:
We know that 4 <= H < first free handle. (As handle 3 is still used at least by the first unidentified handle.)
So doing a redirection test of the first free handle, duplicating handles 4 to first_free_handle-1, gives it to us.
Only one unidentified handle remains, below that last one we found. The quantum uncertainty has shrunk some more
Here, we apply another fact that was discovered in the previous thread:
- If only one used handle remains in the list, it's necessarily the unknown one used by the 2>NUL redirection.
- If only one used handle remains below the last identified handle, it's necessarily the unknown one used by the 2>NUL redirection.
This worked very well for the first pipe... But the second call to :CreatePipe failed to find the four free handles it needed.
Why? Because the two handles created by the first call caused four handles to remain before the last identified handle, not just one. (See the 'test 3>NUL 4>NUL' results above, which reproduces the effect.)
And only three free handles were identified for sure above that, where four were needed.
But the script knew about these two handles it created in the first call! This knowledge could be tracked and used.
This forced me to improve the last stage as follows:
- If only one used handle remains below the last identified handle, after eliminating all known handles, it's necessarily the unknown one used by the 2>NUL redirection.
Here's a test with the final handle enumeration routine:
Code: Select all
@echo off
:# TestEnumHandles.bat - Test the :EnumHandles routine
:# 2019-01-24 JFL Published at https://www.dostips.com/forum/viewtopic.php?p=58721#p58721
:# 2019-01-25 JFL Fixed a bug in :EnumHandles, that broke the elimination of known used handles.
:# Added lots of comments.
setlocal EnableExtensions EnableDelayedExpansion
call :debug.init
goto :main
:debug.init
set "IFDEBUG=if "%DEBUG%"=="1""
set "ECHO.D=%IFDEBUG% echo"
set "ECHOVARS.D=%IFDEBUG% call :echovars"
exit /b
:echovars %*=variables names
setlocal EnableExtensions EnableDelayedExpansion
for %%s in (%*) do echo set "%%s=!%%s!"
endlocal & exit /b
:# Note: In the routines below, we call "lists" batch variables with elements separated by a space.
:# The advantage of such lists is that they can be parsed, or looped on, using just the "for" command.
:# We build lists in a way that inserts an extra space ahead of the first element.
:# For performance reasons, we do not bother removing that space all along.
:# We remove it only in the final list returned in the end.
:# Also, we rely on the fact that elements in a list of handles here are all 1-digit numbers.
:#----------------------------------------------------------------------------#
:# Handle enumeration routine - Return lists of used and free file I/O handles
:EnumHandles %1=freeListVar %2=usedListVar %3=quantumLevelVar %4=knownList (Not including 0 1 2)
:# Returns %freeHandles%, %usedHandles%, %nUnknownHandles%
setlocal EnableExtensions EnableDelayedExpansion
%ECHO.D% call %0 %*
:# Search the top free handles
:# Our :TryRedir routine uses 2>NUL, so itself always uses the first free handle.
:# => Handle 3 will always be found in use: Either it was already, or 2>NUL will use it.
:# Using handle 3 for redirection tests forces cmd.exe to use the second free handle to save it.
:# => Handle 4 will always be found in use: Either it was already, or 3>&%%h will use it.
:# Try duplicating handles 5 to 9, to see if they exist.
set "freeHandles=" &:# List of free file handles
set "usedHandles= 3 4" &:# List of used file handles. 3 and 4 will be in use.
for /L %%h in (5,1,9) do call :TryRedir 3 %%h freeHandles usedHandles
%ECHOVARS.D% freeHandles usedHandles
:# Search the second missed free handle, used for saving handle 3 above
if defined freeHandles ( :# This can only work by using another free handle
set "firstFreeHandle=!freeHandles:~1,1!" &:# The first free one we've found so far
set /a "tryLast=firstFreeHandle-1" &:# The last used handle before the first free one
set "freeHandle=" &:# 1-element list with the free handle we missed in the above loop
:# Again, no need to test handle 3, it's bound to be found in use.
for /L %%h in (4,1,!tryLast!) do if not defined freeHandle call :TryRedir !firstFreeHandle! %%h freeHandle usedHandles2
:# Move that free handle from the used list to the free list
for %%h in ("!freeHandle!") do set "usedHandles=!usedHandles:%%~h=!"
set "freeHandles=!freeHandle!!freeHandles!"
)
%ECHOVARS.D% freeHandles usedHandles
:# Search the first missed free handle, used for saving handle 2 above
:# If the first used handle is followed by the first free handle, then we know
:# it's the one that was used by the 2>NUL redirection. So it's actually free.
:# More generally, if all used handles before the first free one are known used
:# handles passed in %4, except for one, then that unknown one is actually free.
if defined freeHandles ( :# [Else there may actually be two unknown handles]
set "firstFreeHandle=!freeHandles:~1,1!" &:# The first free one we've found so far
set "knownUsedHandles=%~4" &:# This list may be empty, or partial
set /a "nUnknownHandles=firstFreeHandle-3"
%ECHOVARS.D% firstFreeHandle nUnknownHandles knownUsedHandles
set "unknownHandles="
set /a "tryLast=firstFreeHandle-1"
for /l %%h in (3,1,!tryLast!) do set "unknownHandles=!unknownHandles! %%h"
if defined knownUsedHandles for %%h in (!knownUsedHandles!) do (
if %%h lss !firstFreeHandle! ( :# Then remove it from the unknown handle list
set /a "nUnknownHandles-=1"
set "unknownHandles=!unknownHandles: %%h=!"
)
)
%ECHOVARS.D% nUnknownHandles unknownHandles
if !nUnknownHandles!==1 ( :# OK, this single used handle is actually free.
:# Move it from the used list to the free list
for %%h in ("!unknownHandles!") do set "usedHandles=!usedHandles:%%~h=!"
set "freeHandles=!unknownHandles!%freeHandles%"
set "quantumLevel=0"
) else ( :# One unidentified handle in the used list is actually free.
set "quantumLevel=1"
)
) else ( :# No free handle found. Up to 2 used handles may actually be free.
set "quantumLevel=2"
)
:# Cleanup and return
for %%v in (freeHandles usedHandles) do if defined %%v set "%%v=!%%v:~1!" &:# Remove the head space
endlocal & (
set "%1=%freeHandles%"
set "%2=%usedHandles%"
set "%3=%quantumLevel%"
%ECHOVARS.D% freeHandles usedHandles quantumLevel
) & (%ECHO.D% return %ERRORLEVEL%) & exit /b
:TryRedir %1=Handle to redirect; %2=Handle to duplicate; %3=Free var; %4=Used var
2>NUL ( :# Prevent error messages written to stderr from being visible.
break %1>&%2 && ( :# The redirection succeeded.
set "%4=!%4! %2" &rem The handle %2 existed. Add it to the used list.
(call,) &rem Clear ERRORLEVEL, which might be non-0 despite success here.
) || ( :# The redirection failed, and an error message was written to stderr.
set "%3=!%3! %2" &rem The handle %2 did not exit. Add it to the free list.
)
)
exit /b &:# Returns 0=Used, 1=Free
:#----------------------------------------------------------------------------#
:usage
echo %~nx0 - Test the :EnumHandles routine
echo Usage: %~nx0 [OPTIONS] [HANDLE ...]
echo Options:
echo -? Display this help
echo -d Debug mode
echo handles: List of known handles (possibly void or incomplete), to pass
echo to the enumeration routine.
exit /b 0
:main
set "knownHandles="
goto :get_arg
:next_arg
shift
:get_arg
if "%~1"=="" goto :start
if "%~1"=="-?" goto :usage
if "%~1"=="-a" set "ACTION=TryAll" & goto :next_arg
if "%~1"=="-d" set "DEBUG=1" & call :debug.init & goto :next_arg
set "knownHandles=%knownHandles% %1"
goto :next_arg
:start
call :EnumHandles freeHandles usedHandles quantumLevel "!knownHandles!"
echo Free handles: !freeHandles!
echo Used handles: !usedHandles!
if !quantumLevel! equ 1 (
echo Warning: One unknown handle in the used list is actually free.
) else if !quantumLevel! gtr 1 (
echo Warning: Up to two unknown handles in the used list might be free.
)
exit /b
- freeHandles=A list of the known free handles
- usedHandles=A list of the possibly used handles
- quantumLevel: This last variable is important: It allows knowing the amount of quantum uncertainty that remains:
- 0=The two lists are exact
- 1=One used handle is actually free. Note that the caller can eliminate that uncertainty by doing a 2>NUL:
This fills up the unknown hole, and all that remains is certain!
- 2=All handles seem to be used, even though up to two may actually be free. Sorry, you're out of luck, so far, we can't do anything about that.
Future directions:
Another thing is bothering me: Redirections succeed even when all handles are in use.
This means that cmd.exe has more than 10 handles internally, as the backup handles are necessarily allocated beyond 9.
Actually it probably has at least 20, which is the default for Microsoft standard C libraries.
If we found a way to use them, we'd be able to further reduce the quantum uncertainty. (All those pesky cases I flag with Q-level 2)
Also I've still not lost all hope of identifying the first unknown handle (cases flagged as Q-level 1):
It's not feasible with handle duplications, but I suspect that pipes are another possible road: They reshuffle handles in predictable ways.
I'll keep searching.
