set /p problems with pipes

[jeb]

Relating to this question at stackoverflow Piping multiple values into a program in a batch script ...

The main question is the question, why multiple set/p fail to read a block of multiple echo'd lines?

Code: Select all

(
echo Line1
echo Line2
echo Line3
echo Line4
) | (
  set /p var1=
  set /p var2=
  set /p var3=
  set /p var4=
  set var
)

On the most system you will see only one variable

var1=Line1

We can test it with more lines

Code: Select all

@echo off
(
   for /L %%n in (1 1 10) do echo Line%%n
) | (
  FOR /L %%n in (1 1 10) do @(
    set "var=empty"
    set /p var=
    set var
  )
)

On my system I only get empty lines.

var=empty
var=empty
var=empty
var=empty
var=empty
var=empty
var=empty
var=empty
var=empty
var=empty

But when I change the first 10 to 100 I get this

var=empty
var=
var=empty
var=Line48
var=empty
var=Line68
var=Line78
var=Line88
var=Line100
var=empty

Or this, or something different ...

var=empty
var=
var=Line55
var=Line64
var=Line74
var=Line85
var=Line95
var=empty
var=empty
var=empty

So the result is random.

to be continued ... (in some hours)

[Squashman]

Now I am confused by this output. I put an additional SET VAR after the code block. I really never understand the cmd processor.

Code: Select all

var1=Line1
Environment variable var not defined

[Yury]

Code: Select all

@echo off
(
 for /l %%n in (1 1 10) do @echo Line%%n&>nul timeout 1
)|(
 for /l %%n in (1 1 10) do @(
  set var=empty
  set /p var=
  set var
 )
)

var=Line1
var=Line2
var=Line3
var=Line4
var=Line5
var=Line6
var=Line7
var=Line8
var=Line9
var=Line10

[penpen]

Before i saw the code from Yury i thought that the echo produces this behaviour by processing \r\n in the same way as it does within the console: It sets the cursor to the start of the line and then puts \n to the stream, so the receiver only reads an empty line.

Without the \r it works just fine on my win xp home:

Code: Select all

@echo off
cls
setlocal disableDelayedExpansion
set \n=^


setlocal enableDelayedExpansion
@<nul (
   cmd /V:ON /E:ON /C "@for /L %%n in (1 1 100) do @set /P "=Line%%n^!\n^!""
) | (
  FOR /L %%n in (1 1 100) do @(
    set "var=empty"
    set /p "var="
    set var
  )
)
endlocal
goto :eof

But the code above confuses me.... .

penpen

[jeb]

After I finished my christmas preparations, some more thoughts.

Set /p works with pipes different than with redirects.
With redirects it works like a charm.

But with pipes I assume the behaviour is a bit strange.
Each _set/p_ waits for input, but it sets the variable only to the first \r, the rest will be dropped.

But when the pipe is closed, then _set/p_ doesn't wait any longer for input it reads simply nothing.

The trick of penpen to use only \n for the line end solves the problem of dropped content, but a _set/p_ still reads multiple lines when available (tested in Win7).

But findstr or more are both are able to split the input buffer always at the line ends.

So this works

Code: Select all

(
   for /L %%n in (1 1 10) do @(
      echo Line%%n
   )
) | (
  FOR /F "delims=" %%R in ('more') do @(
   echo %%R
  )
)

But now there is another problem, as a FOR /F loop will wait until all data is fetched before entering the body, it waits until _Line10_ is produced and the pipe is closed, before the first echo is executed.

But for this exists another solution, using set /p with redirection

The question is how to use _set/p_ with redirection when we are in a pipe situation?

It's possible by using a temporary file

Code: Select all

@echo off
setlocal DisableDelayedExpansion
set "outfile=%~2"

if "%~1" == "-internal" goto :readPipe
start "" /b cmd /c %~dpf0 -internal "%~1" "
find /N /V "" > "%temp%\tee.tmp"
echo ----------------------------------- > con
echo EOF> "%temp%\tee.tmp"
exit /b

:readPipe
cd.>"%outfile%"
< "%temp%\tee.tmp" (
  for /L %%n in (0 0 99) DO (
   set "line="
   set /p line=
   if defined line (
      setlocal EnableDelayedExpansion
      set "line=!line:*]=!"
      echo(!line!
      echo(!line!>> "%outfile%"
      if "!line!"=="EOF" exit
      endlocal
   )
  )
)

Here are two tasks working, the first uses FIND /N to get the data from the pipe and write it to a file.
The second read the data instantly from the file with set/p only one line when data is available or nothing when no data is currently written.
As all lines written from the FIND/N are prefixed by a number they can be identifyed as data.
If the pipe is closed a line without number is written only with the text _EOF_ to mark this as the end.

[Aacini]

This problem is related to the synchronization of the two asynchronous processes at each side of the pipe. When I was in the developement of my Tee.bat program I did several tests on this point and never found an exact cause of the error nor a pattern of it. The only solution for me was to use a semaphore file as inter-process flag that allows the right side of pipe to read the data as soon as the left side indicate that data was available...

Antonio

[dbenham]

Yes, and the SET /P seems to read multiple lines even if \r is encountered. But only the content up until the \r is preserved. The rest is discarded.

In case it is not obvious to others, penpen's no \r "solution" doesn't really solve anything. As jeb says, no content is dropped, but multiple lines may be assigned to a single variable.

Here is a sample output on my Win 7 machine:

Code: Select all

var=Line1
var=Line2
Line3
Line4
var=Line5
Line6
Line7
Line8
Line9
var=Line10
var=empty
var=empty
var=empty
var=empty
var=empty
var=empty

Not good

There is a weird timing issue involved. Yury's solution of a 1 second delay between each ECHO is painful. But you don't need nearly that much time.

On my machine, simply introducing a CALL before ECHO on the left solves the problem, but sometimes I get a The process tried to write to a nonexistent pipe. error.

I've also tried putting a FOR /L loop delay before the left ECHO, with varied number of iterations. As the number of iterations increases, the result gets better and better. Eventually I get the correct end result.

It really mystifies me why Microsoft implemented SET /P in such a way that reading from a pipe is radically different than reading from a redirected file.


Dave Benham

[foxidrive]

but multiple lines may be assigned to a single variable.

That would be the holy grail of newbie questions, if it worked reliably.

[dbenham]

but multiple lines may be assigned to a single variable.

That would be the holy grail of newbie questions, if it worked reliably.

It can be made reliable, but it is not exactly fit for newbies.

It relies on a file based semaphore, just like Aacini talked about in his post. Not only does the right side wait for the appearance of the file before it continues, the left side waits for the file to be deleted before it continues. In this way the two processes are fully synchronized.

left.bat

Code: Select all

@echo off
setlocal enableDelayedExpansion

set ^"LF=^

^" Empty blank line above is critical - DO NO REMOVE

for /l %%A in (1 1 10) do (
  for /l %%B in (1 1 %%A) do <nul set /p "=Line%%B!LF!"
  echo go>signal.txt
  call :wait
)
exit /b

:wait
if exist signal.txt goto :wait
exit /b

right.bat

Code: Select all

@echo off
setlocal enableDelayedExpansion

for /l %%N in (1 1 10) do (
  call :wait
  set "var="
  set /p "var="
  del signal.txt
  echo [!var!]
  echo(
)
exit /b

:wait
if not exist signal.txt goto :wait

Demonstration:

Code: Select all

C:\test>left|right

Reading 1 lines with 1 SET /P:
[Line1]

Reading 2 lines with 1 SET /P:
[Line1
Line2]

Reading 3 lines with 1 SET /P:
[Line1
Line2
Line3]

Reading 4 lines with 1 SET /P:
[Line1
Line2
Line3
Line4]

Reading 5 lines with 1 SET /P:
[Line1
Line2
Line3
Line4
Line5]

Reading 6 lines with 1 SET /P:
[Line1
Line2
Line3
Line4
Line5
Line6]

Reading 7 lines with 1 SET /P:
[Line1
Line2
Line3
Line4
Line5
Line6
Line7]

Reading 8 lines with 1 SET /P:
[Line1
Line2
Line3
Line4
Line5
Line6
Line7
Line8]

Reading 9 lines with 1 SET /P:
[Line1
Line2
Line3
Line4
Line5
Line6
Line7
Line8
Line9]

Reading 10 lines with 1 SET /P:
[Line1
Line2
Line3
Line4
Line5
Line6
Line7
Line8
Line9
Line10]

Note how there is no linefeed before the end bracket. That is because SET /P discards trailing control characters. The number of lines that can be read is limited because SET /P cannot read more than 1021 bytes.


Dave Benham

[foxidrive]

It can be made reliable, but it is not exactly fit for newbies.

[dbenham]

Eureka
I think I got it, thanks to OJBaker's excellent thread: How Set/p works

Here are OJBaker's rules (with one small correction)

A) Reading characters:
Characters are read from the input stream and put in a character buffer until one of three conditions is true:

[1]: The 2 byte end-of-line sequence is read, either CRLF or LFCR.
[2]: There are 1023 characters in the char buffer. (buffer full)
[3]: A timeout occurs

B) Processing the char buffer:

• All control-characters from the end of the char buffer are discarded (possible data loss)
• If there is a NUL character in the char buffer, then all characters following the first NUL in the char buffer
  will be discarded (data loss)

C) Moving from char buffer to Var:

• If char buffer is empty report error condition: Set errorlevel to 1 (meaning No value entered)
• If char buffer is not empty: Move the string from char buffer to the variable named in the set/p command.

D) Set/p is done and returns control to the batch file.

OJBakker states he mostly tested with redirection. But then jeb tested with pipes and got slightly different results, though not fully explained.

I think OJBakker's rules are almost perfect except:

During Step A)

• When SET /P is reading piped data it does NOT stop reading from the input stream when CRLF or LFCR is read. Condition [1] only applies if reading from a redirected file or directly from the console.
  
• There is no timeout condition when reading directly from the console. Condition [3] only applies when reading from redirected or piped input.

During Step B)

• All characters after the first occurrence of CRLF or LFCR or NUL are discarded.
  
• Any remaining trailing control characters are also discarded.

This accounts for nearly all observed behavior:

The fact that condition 1 does not apply to piped data explains the original problem cited in this thread.

The timeout condition explains why introducing a delay between each ECHO on the left causes the piped data to be read properly. The trick is in getting the minimum delay that always produces correct results.

And here is some interesting code that shows how a piped version can give the correct result without introducing a delay. Each value is printed normally with trailing CRLF. The trick is to follow each value with fill data such that the length of value + CRLF + filler is exactly 1023 bytes.

left.bat

Code: Select all

@echo off
setlocal enableDelayedExpansion

set ^"LF=^

^" Empty blank line above is critical - DO NO REMOVE

for /f %%A in ('copy /Z "%~dpf0" nul') do set "CR=%%A"

set "buffer="
for /l %%N in (1 1 1200) do set "buffer=!buffer!."

for /l %%A in (1 1 20) do (
  set "val=line%%A!CR!!LF!!buffer!"
  echo !val:~0,%1!
)
exit /b

right.bat

Code: Select all

@echo off
setlocal enableDelayedExpansion

for /l %%N in (1 1 20) do (
  set "var="
  set /p "var="
  echo [!var!]
)
exit /b

When left.bat writes in chunks of 1023 (1021 + CRLF from ECHO), then everything works great!

Code: Select all

C:\test> left 1021 | right
[line1]
[line2]
[line3]
[line4]
[line5]
[line6]
[line7]
[line8]
[line9]
[line10]
[line11]
[line12]
[line13]
[line14]
[line15]
[line16]
[line17]
[line18]
[line19]
[line20]

But when left.bat writes too many or too few bytes, then things quickly get out of synch:

Code: Select all

C:\test> left 1020 | right
[line1]
[line2]
[ine3]
[ne4]
[e5]
[6]
[]
[
......................................................... data removed to shrink line ....................................................................]
[........................................................ data removed to shrink line .....................................................................]
[........................................................ data removed to shrink line ....................................................................]
[........................................................ data removed to shrink line ...................................................................]
[........................................................ data removed to shrink line ..................................................................]
[........................................................ data removed to shrink line .................................................................]
[........................................................ data removed to shrink line ................................................................]
[........................................................ data removed to shrink line ...............................................................]
[........................................................ data removed to shrink line ..............................................................]
[........................................................ data removed to shrink line .............................................................]
[........................................................ data removed to shrink line ............................................................]
[........................................................ data removed to shrink line ...........................................................]
[........................................................ data removed to shrink line ..........................................................]

C:\test> left 1022 | right
[line1]
[
line2]
[]
[.]
[..]
[...]
[....]
[.....]
[......]
[.......]
[........]
[.........]
[..........]
[...........]
[............]
[.............]
[..............]
[...............]
[................]
[.................]

Note that the CRLF is not needed at the end of the 1023 byte chunk. A slight change to left.bat demonstrates:

left2.bat

Code: Select all

@echo off
setlocal enableDelayedExpansion

set ^"LF=^

^" Empty blank line above is critical - DO NO REMOVE

for /f %%A in ('copy /Z "%~dpf0" nul') do set "CR=%%A"

set "buffer="
for /l %%N in (1 1 1200) do set "buffer=!buffer!."

for /l %%A in (1 1 20) do (
  set "val=line%%A!CR!!LF!!buffer!"
  <nul set /p "=!val:~0,%1!"
)
exit /b

When writing too few bytes, it first seems that it works, but that is only because SET /P is slow, introducing enough delay for the timeout to kick in:

Code: Select all

C:\test>left2 1022|right
[line1]
[line2]
[line3]
[line4]
[line5]
[line6]
[line7]
[line8]
[line9]
[line10]
[line11]
[line12]
[line13]
[line14]
[line15]
[line16]
[line17]
[line18]
[line19]
[line20]

If output of left2 is captured in a file, and then the file is piped into right, then the problems are manifested:

Code: Select all

C:\test> left2 1022 >test1022.txt

C:\test> type test1022.txt | right
[line1]
[ine2]
[ne3]
[e4]
[5]
[]
[
................................................ data removed to shrink line ..............................................................line8]
[............................................... data removed to shrink line ...............................................................line9]
[............................................... data removed to shrink line ..............................................................line10]
[............................................... data removed to shrink line .............................................................line11]
[............................................... data removed to shrink line ............................................................line12]
[............................................... data removed to shrink line ...........................................................line13]
[............................................... data removed to shrink line ..........................................................line14]
[............................................... data removed to shrink line .........................................................line15]
[............................................... data removed to shrink line ........................................................line16]
[............................................... data removed to shrink line .......................................................line17]
[............................................... data removed to shrink line ......................................................line18]
[............................................... data removed to shrink line .....................................................line19]
[............................................... data removed to shrink line ....................................................line20]
[............................................... data removed to shrink line ...................................................]

C:\test> left2 1024 >test1024.txt

C:\test> type test1024.txt | right
[line1]
[.line2]
[..line3]
[...line4]
[....line5]
[.....line6]
[......line7]
[.......line8]
[........line9]
[.........line10]
[..........line11]
[...........line12]
[............line13]
[.............line14]
[..............line15]
[...............line16]
[................line17]
[.................line18]
[..................line19]
[...................line20]

But if the correct length of 1023 is output, then everything works:

Code: Select all

C:\test> left2 1023 > test1023.txt

C:\test> type test1023.txt | right
[line1]
[line2]
[line3]
[line4]
[line5]
[line6]
[line7]
[line8]
[line9]
[line10]
[line11]
[line12]
[line13]
[line14]
[line15]
[line16]
[line17]
[line18]
[line19]
[line20]

I'm happy now because at least the mystery is solved
Well, actually I'm still pissed because MicroSoft made SET /P pretty much worthless with pipes.


Dave Benham

[penpen]

I should have been more exact:
The code from Yury confuses me, because it works even on my winxp (with no timeout command).
This seems to work always reliably (i made a small change; tested on 3 different machines):

Code: Select all

@echo off
(
 for /l %%n in (1 1 1000) do @echo Line%%n&>nul 2>nul command_does_not_exist
)|(
 for /l %%n in (1 1 1000) do @(
  set var=empty
  set /p var=
  set var
 )
)

penpen

[dbenham]

I modified my OJBakker rule modifications in my prior post just a bit to account or the fact that SET /P does not timeout when reading directly from the console.

I should have been more exact:
The code from Yury confuses me, because it works even on my winxp (with no timeout command).

I think the most likely explanation is that the XP machines are slower - slow enough that the timeout condition fires between each ECHO. I think this is especially likely if the XP is a virtual machine running over top of Win 7.

This could also explain how such a nasty behavior worked its way into production. Perhaps machines were never fast enough during development of CMD.EXE to expose the problem when SET /P reads piped data on a fast machine.

I suppose an alternate explanation is that XP SET /P uses different rules (it could honor condition [1] when reading piped data). But I think this is unlikely.


Dave Benham

[Aacini]

I did some tests with this matter and found a case that does not follow the last stated rules. Rule A) [3] states that a timeout occurs, and Dave added that "Condition [3] only applies when reading from redirected or piped input". In other words, SET /P don't waits and don't change previous variable value when reading from redirected file or piped input and there is no data available at that moment, right?

Well, this rule not works on piped input:

Code: Select all

@echo off

(
   ping -n 5 localhost > nul
   echo Data from pipe
) | (
   set "var=Timeout"
   set /p var=
   set var
  )
)

Output:

Code: Select all

var=Data from pipe

I think the During Step A) rule should said: "There is no timeout condition when reading directly from the console or piped input. Condition [3] only applies when reading from redirected file.

Antonio

[dbenham]

Damn you Antonio - my nice ordered world has crumbled around me

So there seems to be evidence that there is a timeout for piped data, and counter evidence that there is not. It reminds me of the particle/wave dual nature of light.

I took your test, and added a bit more:

test2.bat

Code: Select all

@echo off

(
   echo start
   ping -n 2 localhost > nul
   for %%c in (H e l l "o " w o r l d) do @set /p "=[%%~c]" <nul 
   echo(
   echo Data from pipe
) | (
   set "var=Timeout"
   set /p var=
   set var
   set /p var=
   set var
   set /p var=
   set var
   set /p var=
   set var
   set /p var=
   set var
   set /p var=
   set var
   set /p var=
   set var
   set /p var=
   set var
   set /p var=
   set var
   set /p var=
   set var
   set /p var=
   set var
)

The results of 4 successive runs have me ready to check into the nearest loony bin. It appears to defy all logic.

Code: Select all

D:\test>test2
var=start
var=[H]
var=
var=Data from pipe
var=[l]
var=[o ]
var=
var=[o]
var=[r]
var=
var=Data from pipe

D:\test>test2
var=start
var=[H]
var=
var=[e]
var=[l]
var=
var=[l]
var=Data from pipe
var=
var=[w]
var=Data from pipe

D:\test>test2
var=start
var=[H]
var=
var=Data from pipe
var=[e]
var=
var=Data from pipe
var=[l]
var=[l]
var=[o ]
var=[w]

D:\test>test2
var=start
var=[H]
var=
var=Data from pipe
var=[e]
var=
var=[l]
var=
var=
var=
var=[o]

Is someone warping the space time continuum


Dave Benham