More fun with redirection and file handles

[dbenham]

Here is a script that demonstrates various redirection concepts, most of which I have seen posted before.

- The same file can be opened for input and output independently.

- A given file can only be opened once for output due to an exclusive lock

- The same file can be opened multiple times for input. Each input file stream maintains its own file pointer independent of the others.

- Redirection to an existing file handle via something like &1 shares the stream with only 1 file pointer between them.

- FIND resets the file pointer to the beginning of file before scanning.

- FINDSTR does not reset the file pointer (I first saw this in an Aacini post)

- Writing to an input handle fails with a puzzling error message

- Reading from an output handle fails simply returns nothing (fails?), but there is no error message.

Code: Select all

@echo off
0<test.txt 7>test.txt 8<&0 9<test.txt (
  call :read 0
  call :write 1
  call :write 2
  call :write 3
  call :read 0
  call :read 8
  call :read 9
  call :write 4
  call :read 8
  call :read 8
  call :read 0
  echo(
  echo FINDSTR from 9 does not reset file pointer to beginning, and ends at EOF:
  0<&9 findstr "^"
  echo(
  call :read 9
  0<test.txt (
    echo(
    echo Temporarily reopen input file via 0
    call :read 0
    call :read 0
    echo Close and resume old 0
    echo(
  )
  call :write 5
  call :read 0
  call :read 9
  call :read 0
  call :read 8
  call :read 9
  echo(
  echo FINDSTR from 0 does not start at beginning:
  findstr "^"
  echo(
  echo FIND from 0 resets file pointer to beginning, and ends at EOF:
  find /v ""
  echo(
  call :read 0
  echo(
  echo Cannot reopen output file because of exclusive lock
  1>test.txt (
    1>&2 echo This is never executed
  )
  echo(
  echo Cannot read from non-existent file
  0<notExists (
    set /p "ln=This is never executed"
  )
  echo(
  1>&0 (
    1>&2 echo Cannot write to input handle, but redirection worked and error message puzzling
    echo This fails when try to write to input handle
  )
  echo(
  0<&1 set /p "ln=Why does read from output handle not generate an error message?"
  echo(
)
exit /b


:read  handle
setlocal
if "%1" neq "0" set "src=0<&%1"
set "ln="
set /p "ln=" %src%
echo Reading from %1 ==^> %ln%
exit /b

:write  txt
1>&7 echo %*
echo Wrote %*
exit /b

--OUTPUT--

Code: Select all

Reading from 0 ==>
Wrote 1
Wrote 2
Wrote 3
Reading from 0 ==> 1
Reading from 8 ==> 2
Reading from 9 ==> 1
Wrote 4
Reading from 8 ==> 3
Reading from 8 ==> 4
Reading from 0 ==>

FINDSTR from 9 does not reset file pointer to beginning, and ends at EOF:
2
3
4

Reading from 9 ==>

Temporarily reopen input file via 0
Reading from 0 ==> 1
Reading from 0 ==> 2
Close and resume old 0

Wrote 5
Reading from 0 ==> 5
Reading from 9 ==> 5
Reading from 0 ==>
Reading from 8 ==>
Reading from 9 ==>

FINDSTR from 0 does not start at beginning:

FIND from 0 resets file pointer to beginning, and ends at EOF:
1
2
3
4
5

Reading from 0 ==>

Cannot reopen output file because of exclusive lock
The process cannot access the file because it is being used by another process.

Cannot read from non-existent file
The system cannot find the file specified.

Cannot write to input handle, but redirection worked and error message puzzling
The system cannot find the file specified.

Why does read from output handle not generate an error message?

Note that I did not redirect handle 3 after redirecting 0 because of this issue: http://stackoverflow.com/a/9880156/1012053


Dave Benham

[Liviu]

- Reading from an output handle fails simply returns nothing (fails?), but there is no error message.

Code: Select all

  0<&1 set /p "ln=Why does read from output handle not generate an error message?"

Focusing on just this tiniest bit of your whole exercise, but I think line-oriented input assumes an implicit end-of-line at the end-of-stream (whether it's been closed, or never opened at all). Probably the same reason why "<nul pause" works without error, and without waiting for a key - which happens to be a very useful behavior, whether intentional or accidental.

Liviu

[penpen]

According to http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/redirection.mspx?mfr=true this happens:

if you start a program with <&2, all attempts to read handle 0 fail because handle 2 is initially opened with write-only access.

I assume the same happens when using STDIN.
So the output of 'set /P "input=output"' is generated, but reading the input fails and the errorlevel is set to != 0 (1 on my WinXP).

Using ">&0" works similar:

if you start a program with redirection >&0, all attempts to write handle 1 fail because handle 0 is initially opened with read-only access.

penpen

[Ed Dyreen]

Trying to find a nice elegant way to get this working properly.

Code: Select all

@echo off &setlocal enableDelayedExpansion

echo.
echo.test0
(set/A=)3>&2>&1>nul&&echo.yes||echo.no

echo.
echo.test1
(set/A=)3>nul 2>nul >nul&&echo.yes||echo.no

echo.
echo.test2
(((set/A=)3>&2)2>&1)>nul&&echo.yes||echo.no

echo.
echo.test3
(set/A=)1>&2 2>&3 3>nul&&echo.yes||echo.no

pause
exit

Code: Select all

test0
Er ontbreekt een operand.
no

test1
no

test2
no

test3
Er ontbreekt een operand.

The 1st most logical solution fails !? Both the 2nd & 3th solution work but they don't make me happy, I have a feeling this can be done in a more elegant way... Test3 successfully hangs the script, won't be needing that feature for now

[penpen]

Test3 successfully hangs the script, won't be needing that feature for now

This should not hang your script, but it should reassign the used streams to the given handles.
Assumed that handles 3++ are not in use this should happen:

Code: Select all

handle   before   after
     0   STDIN    STDIN
     1   STDOUT   nul
     2   STDERR   STDERR
     3   unused   STDOUT
    4+   unused   unused

The default handle to STDOUT is reassigned to nul, so you don't see the shells output.

To proof this, you could delete the "pause" "exit" from your batch file (assumed its name is "test.bat") and then you could start a new command shell instance using ">&3 cmd":

Code: Select all

Z:\>test.bat

test0
Fehlender Operand
no

test1
no

test2
no

test3
Fehlender Operand
>&3 cmd
Microsoft Windows [Version 6.3.9600]
(c) 2013 Microsoft Corporation. Alle Rechte vorbehalten.

Z:\>

To avoid such unwanted reassignments of streams to handles, you should redirect one handle only per instruction, or block.
So you should use the syntax of your third example ("test2"):

Code: Select all

(((set/A=)3>&2)2>&1)>nul&&echo.yes||echo.no
(((set/A=)3>nul)2>nul)>nul&&echo.yes||echo.no
:: ...

penpen

[OperatorGK]

Whoa.
Thanks to you, I've just discovered another bug ! Try:

Code: Select all

0>&1 echo TEST 3>con

Strangely, it works ONLY in command line, when not in braces, for loop, cmd /c and so on. Can anybody explain me why this is happening?

[penpen]

Initially assignemnts (using c++ terminology for STDIN, STDOUT, STDERR; CON analogous):
- CON: (in your case) an OutputStream (write only access).
- STDIN: InputStream (read only access).
- STDOUT: OutputStream (write only access).
- STDERR: OutputStream (write only access).

Your command "0>&1 echo TEST 3>con" probably produces the following mapping (step by step):

Code: Select all

handle mapping   handle    initial   assign             restore
                                     0>&1     3>con     0        3
                      0    STDIN     STDOUT   STDOUT    CON      CON
                      1    STDOUT    STDOUT   STDOUT    STDOUT   STDOUT
                      2    STDERR    STDERR   STDERR    STDERR   STDERR
                      3    unused    STDIN    CON       unused   STDIN
                      4    unused    unused   STDIN     STDIN    unused
                     5+    unused    unused   unused    unused   unused
restore handle             -         0        0,3       3        -
        with               -         3        3,4       4        -

You could check that handle 3 is STDIN using '@(0>&1 echo TEST 3>con & set /P "=test" <&3)' (no endless loop while processing 'set/P').

The command shell uses an endless loop (simplified pseudo code) like this:

Code: Select all

String input;
boolean echoEnabled = true;
while (true) do {
   if (echoEnabled) streamFromHandle(1).write(getPrompt());
   input = streamFromHandle(0).readLine();
   execute(input);
}

After your command streamFromHandle(0) equals CON (in this case an OutputStream).
Reading an OutputStreamit fails, so "input" is set to an empty String, or null (i don't know it).
Then this "input" is executed (obviously nothing is displayed).
Then the next loop iteration is performed.
So you will see the prompt string (probably something like "C:\Foo\>") printed to screen again and again.


penpen

[dbenham]

I've neglected this thread far too long after I started it.

According to http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/redirection.mspx?mfr=true this happens:

if you start a program with <&2, all attempts to read handle 0 fail because handle 2 is initially opened with write-only access.

I assume the same happens when using STDIN.
So the output of 'set /P "input=output"' is generated, but reading the input fails and the errorlevel is set to != 0 (1 on my WinXP).

- Reading from an output handle fails simply returns nothing (fails?), but there is no error message.

Code: Select all

  0<&1 set /p "ln=Why does read from output handle not generate an error message?"

Focusing on just this tiniest bit of your whole exercise, but I think line-oriented input assumes an implicit end-of-line at the end-of-stream (whether it's been closed, or never opened at all). Probably the same reason why "<nul pause" works without error, and without waiting for a key - which happens to be a very useful behavior, whether intentional or accidental.

Yes, I noticed the similarity, but it still surprises me a bit. I can see how reading from NUL could be designed to instantly fail without error, but I wouldn't have thought reading from a write only handle would be treated the same way.

Even more surprising, there is no error message if an input file becomes unavailable after it has been opened.

I have my H: drive pointing to a removable thumb drive.

Code: Select all

C:\test>3<&1 <h:test.txt (<&3 pause&set /p val=First time I leave the thumb drive plugged in)
Press any key to continue . . .
First time I leave the thumb drive plugged in
C:\test>echo %errorlevel%
0

C:\test>set val
val=ok

C:\test>set "val="

C:\test>3<&1 <h:test.txt (<&3 pause&set /p val=Now I remove the thumb drive before pressing a key)
Press any key to continue . . .
Now I remove the thumb drive before pressing a key
C:\test>echo %errorlevel%
1

C:\test>set val
Environment variable val not defined

C:\test>

No error message
But at least the error can be detected via %ERRORLEVEL% (or the || operator, tested but not shown)

Using ">&0" works similar:

if you start a program with redirection >&0, all attempts to write handle 1 fail because handle 0 is initially opened with read-only access.

I fully expect it to fail, but the error message is not at all what I expect: "The system cannot find the file specified."

The redirection has already taken place when I attempt to write to stdin, so it should not be looking for a file. The redirection outside the parentheses was successful, it is only the attempt to write via ECHO that is failing.

And now for a really horrendous discovery - a failed ECHO does not set any error code
Again, I've got H: pointing to a removable thumb drive.

Code: Select all

C:\test>>h:test.txt (echo I remove the thumb drive while pausing&pause >&2&echo This failed ECHO does not set a detectable error code||>&2echo FAILURE!)
Press any key to continue . . .
The system cannot find the file specified.

C:\test>ECHO %errorlevel%
0

After reconnecting the thumb drive, I see that test.txt is empty, even though the first ECHO succeeded. This is probably due to a buffering issue.

But the important thing is that || does not detect the failure to write to the output file, and %ERRORLEVEL% is still 0
The weird thing is, CMD.EXE obviously detected the error because it printed out an error message.

This is a horrible bug in my book. The only way to detect a failure when writing output is to redirect stderr to a file and then check if the file is not empty after attempting to write.


Dave Benham

[dbenham]

@Ed Dyreen - I'm not sure I understand what your goal was with your convoluted experiments. It seems to me your desired result can be achieved simply using:

Code: Select all

2>nul set/a=&&echo.yes||echo.no

The issues with Ed's test 3 and OperatorGK's code are both related to http://stackoverflow.com/q/9878007/1012053

penpen has the proper analysis, and the behavior is full explained within two posts at viewtopic.php?p=14612#p14612


Dave Benham

[Ed Dyreen]

@Ed Dyreen - I'm not sure I understand what your goal was with your convoluted experiments. It seems to me your desired result can be achieved simply using:

Code: Select all

2>nul set/a=&&echo.yes||echo.no

I generally use stream 0 for default output, stream 2 for error's, and stream 3 for headers and footers. When a function throws an error it's often suppressed by the calling function. The caller often uses it's own header and footer thus will often suppress stream 3 aswell. Stream 1 is often suppressed, but not always. I have not encountered any problems with '3>nul 2>nul >nul' yet. Was hoping for '3>&2>&1>nul' to work, but it doesn't work as expected.

[dbenham]

OK, I understand now. I think you mean you use stream 1 for default output, not 0.

You could also use 3>nul 2>&3 >&3 or 3>nul 2>&3 >&2. The assignments must be done in descending order (or staged as penpen suggested), otherwise the original definitions are not restored properly when the command finishes.

You could never do anything like 3>&2>&1>nul. Each redirection must be specified in full, and the redirections are processed from left to right.



Dave Benham