weird error from gnuwin32 head, seems pipe related

[taripo]

So the for statement generates a file..

Grep is like the windows find command. The "." is not current directory, it's a regex pattern (like findstr uses). It means any character

I think i've narrowed it down to head rather than grep or some combination of head and grep..

Though interesting that one of the error messages made it look a bit like grep was giving the error. But I see I can get an error with head alone.

Code: Select all

C:\ff\ff>for /L %f in (1,1,500) do @echo zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz >>testfile

C:\ff\ff>type testfile | grep "." | head -n 1
The process tried to write to a nonexistent pipe.
zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz



C:\ff\ff>type testfile | grep "." | head -n 1
zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz
grep: writing output: Invalid argument
grep: writing output: Invalid argument
..


Interesting how I got different errors from the same command there. Sometimes i've only had the the "grep: writing output: Invalid argument" error.


As for what version of commands i'm using

C:\ff\ff>grep --v
GNU grep 2.5.4

Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://g
This is free software: you are free to change and re
There is NO WARRANTY, to the extent permitted by law

C:\ff\ff>head --version
head (GNU coreutils) 5.3.0
Written by David MacKenzie and Jim Meyering.

Copyright (C) 2005 Free Software Foundation, Inc.
This is free software; see the source for copying co
warranty; not even for MERCHANTABILITY or FITNESS FO

C:\ff\ff>


C:\ff\ff>type testfile | findstr "." | head -n 1
zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz
FINDSTR: Write error
The process tried to write to a nonexistent pipe.

C:\ff\ff>


C:\ff\ff>head -n 1 testfile
zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz

C:\ff\ff>type testfile | head -n 1
zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz
The process tried to write to a nonexistent pipe.

C:\ff\ff>

though this works

C:\ff\ff>type testfile | findstr "." | find /n /v "asdf" | head -n 1
[1]zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz

C:\ff\ff>
 

And this problem exists whether the file has windows line separators, (\r\n), or *nix new lines (\n).

UPDATE

interestingly I get the same error with sed (sed has an option which makes it behave like head, so I thought maybe sed wouldn't give the error and i'd use sed instead of head)

Code: Select all

C:\ff\ff>sed "2q" testfile
zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz
zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz

C:\ff\ff>type testfile| sed "2q"
zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz
zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz
The process tried to write to a nonexistent pipe.

C:\ff\ff>

No issue with grep but definitely an issue with gnuwin32 sed and head.

C:\ff\ff>type testfile| sed "1q" >a
The process tried to write to a nonexistent pipe.

C:\ff\ff>type testfile| grep "." >a

C:\ff\ff>

I recall there was this weird issue mentioned by dave benham with pipe and sides of it running in separate processes viewtopic.php?f=3&t=5905&start=15 where cmd /c resolved it.. But it doesn't apply / isn't relevant here, as even with cmd /c the error still appears.

Code: Select all

C:\ff\ff>cmd /c "type testfile | sed ^"1q^""
zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz
The process tried to write to a nonexistent pipe.

C:\ff\ff>cmd /c "type testfile | head -n 1"
zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz
The process tried to write to a nonexistent pipe.

C:\ff\ff>

It wouldn't surprise me if it was purely a bug in gnuwin32 implementations of sed and head.

[penpen]

Well i think my statement will surprise you:
There is no bug, and all is working as intended.

The instructions 'head -n 1' and 'sed "1q"' both do the same:
Print one line from the given input (here stdin) to stdout and then stop.

Stop := The executable exits closing all associated streams, including the output stream of the anonymous pipe, which is in a half closed state now.

You told grep to write all stdin to stdout (analogous for the type/findstr/... command), so it doesn't stop after the first line is printed, but continues writing to an anonymous pipe which is half closed.
Writing to a half closed pipe fails, raising an exception, which is handled by grep (or type) by printing the error message to stderr.

There are two possible solutions.

1. Redirect stderr from the pipe source to nul:

Code: Select all

Z:\>2>nul type testfile | head -n 1
zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz

Z:\>type testfile | 2>nul  grep "." | head -n 1
zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz

Z:\>type testfile | 2>nul findstr "." | head -n 1
zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz

This solution has the drawback, that all error messages are redirected to nul (for example: unknown parameter).

2. Add another command to consume the 'unwanted' data:

Code: Select all

Z:\>type testfile | (head -n 1 & findstr /v "^")
zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz

Z:\>type testfile | grep "." | (head -n 1 & findstr /v "^")
zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz

Z:\>type testfile | findstr "." | (head -n 1 & findstr /v "^")
zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz

The drawback of this method is the higher consumption of cpu time.
Nevertheless this solution is my preferred.


penpen

[taripo]

thanks..

(I think I figured out exactly what's going on but mid post.. though i'm still not 100% if my conclusions are totally correct)


i'm trying to figure out what's going on with this command you have that works..

Code: Select all

Z:\>type testfile | grep "." | (head -n 1 & findstr /v "^")

I did a test to try to figure it out

Code: Select all

C:\blah\aeea2>echo abc|(xxd -p && more && dir)
6162630d0a

 Volume in drive C has no label.
....
C:\blah\aeea2>

So I understand that.. The abc gets piped to xxd -p, it returns an errorlevel 0, no error,
so it then does more, but sends a ^Z char to it to indicate no more text char to it so we get no prompt from more.
And then it runs dir, probably piped a ^Z to it .but dir doesn't read stdin 'or' anything piped to it anyway.

I'm still a bit baffled by the & findstr line

I get this..

Code: Select all

type testfile | findstr /v "^"

in that the findstr /v "^" would return nothing.

So I thought maybe..

Code: Select all

Z:\>type testfile | grep "." | (head -n 1 & findstr /v "^")

is working by directing all the stdin to head -n 1, which doesn't read all of stdin, but type doesn't give an error, because it then sends the rest of stdin to findstr /v "^"

That was my theory, and I see that's true but kind of oversimplified, I see there's some interesting things going on.. in that it's not just one line is sent to head which reads one line.. Head is sent a bunch of lines / load of chars, (some buffer size), and reads just one line of it, and the rest - what head was not given as its beyond that buffer size and head had no need for it, is sent onto the next command.

Code: Select all

C:\blah\aeea2>(echo abc & echo def)| (head -n 1 & sed "s/./r/g")
abc

C:\blah\aeea2>

So it's not that the first line is displayed as is, and the rest is transformed by sed.

What happens is the first line is displayed by head, then a bunch of lines are skipped, those would be part of what is buffered.

And the rest is transformed by sed.


I think the following proves that, and that the buffer is 512 characters/bytes

(for those unfamiliar with *nix commands)

wc -l (counts the number of lines)
wc -c (counts the number of characters
sed "s/././" (reads the first character of each line and replaces it with itself and outputs the line, so basically does nothing)
bc does basic math(s) calculations
'less' is the same kind of command as 'more'.

Code: Select all

C:\blah\aeea2>type testfile | wc -l
500

C:\blah\aeea2>type testfile | wc -c
19500

C:\blah\aeea2>more testfile | head -n 1
zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz

C:\blah\aeea2>more testfile | head -n 1 | wc -c
39

C:\blah\aeea2>type testfile | head -n 1 | wc -c
39
The process tried to write to a nonexistent pipe.

C:\blah\aeea2>type testfile | (head -n 1 & sed "s/././")| wc -c
19027

C:\blah\aeea2>echo so first line + difference would be the buffer
so first line + difference would be the buffer

C:\blah\aeea2>bc 5+5
File 5+5 is unavailable.

C:\blah\aeea2>echo 5+5 |bc
10

C:\blah\aeea2>echo 19500-19027|bc
473

C:\blah\aeea2>echo 473+39|bc
512

C:\blah\aeea2>

So when you pipe data to (programA & program B)

and of course we're talking about a case where both programA and programB execute so a case where programA doesn't return an error.

e.g. piping to (head -n 1 | findstr /v "^" )

then when head does a ReadLine then 512 bytes/char of the file are sent to it and it reads the first line. head -n 1 displays that first line and exits. Head has displayed the first line, and happily eaten the rest of the 512 byte buffer of characters which were beyond that line, they disappear into the ether so to speak. The rest of the bytes of the file, All characters after the first 512 bytes, now remain.. findstr /v "^" will be fed that - the rest rest of the stdin file and that operates on every line of the file so it won't leave anything. (and of course for each line of the file it returns nothing).

Is that right?

And if so , I notice it is rather complex.. And i'm curious what (if anything) made you aware of that?

That is a pretty clever solution you came up with.

And also, I notice that for some reason, 'less' doesn't have the issue. So if you do less instead of type, you don't get that error. So no workaround required in the case of 'less'.

[penpen]

I'm unsure if your explaination just sounds as if it were slightly wrong, or if it is wrong.

So i do my best to explain what happens in detail and short.

The complex instruction "instruction1 | (instruction2 & instruction3)" is processed by the command line interpreter within the given environment (offering stdin, stdout, and stderr) as follows:
1. First the pipe (P) is created:
A pipe is just an object which offers an input stream pipeIn (to read data from the pipe) and an output stream pipeOut (to write data to the pipe).
A pipean (internal) error stream pipeErr, which is connected to stderr to be able to report errors:
P.err := stderr.

2. A sub-thread (B) is created to process the instructions inside the block, and its internal input (B.in), output (B.out)and errorstream (B.err) are connected to (directly or as a subsequent stream as denoted):
- B.in := P.in,
- B.out := Stream(stdout),
- B.err := Stream(stderr).
(Note: The subsequent streams don't close the referenced streams when it is closed.)

3. A sub-process (P1) is created and its internal streams are connected to the environment as follows:
- P1.in := Stream(stdin),
- P1.out := P.out,
- P1.err := Stream(stderr).

4. Now B and P1 are started:
The following may not happen in exactly that order, because the involved process and thread are running concurrently.

5. P1 consumes its input and produces its output.
In case P1 is "type testfile" it reads the whole content of file named "testfile" to the internal output P1.out which is connected to the pipe).
If the pipe buffer (in this case 1024 wchar) is full, then writing to the pipe is blocked, blocking process P1, until some data is read from the pipe.
Finally P1 closes its streams (closing P.out) and terminates.

6. The block consists of only 1 complex conjuncted instruction "instruction2 & instruction3", so B executes this instruction:
(i) A process P2 is created with subsequent Stream objects connected to the environment as follows:
- P2.in := Stream(B.in)
- P2.out := Stream(B.out)
- P2.err := Stream(B.err)
(ii) P2 is started consumes its input and produces its output;
in case P2 is "head -n 1", P2 reads some data from P2.in until its internal buffer of 512 wchar (as you found out) is filled, writes the first line to B.out (repeating refilling the buffer as needed).
After that P2 closes its internal streams (setting them to the null pointer "null"):
- P2.in := null
- P2.out := null
- P2.err := null
(Note that B.in, B.out, and stderr are NOT closed: That are different objects.)
(iii) P2 is terminated.
(iv) A process P3 is created with subsequent Stream objects connected to the environment as follows:
- P3.in := Stream(B.in)
- P3.out := Stream(B.out)
- P3.err := Stream(B.err)
(v) P3 is started consumes its input and produces its output;
in case P3 is "findstr /v "^"", P3 reads just all data from P3.in.
After that P3 closes its internal streams (setting them to the null pointer "null"):
- P3.in := null
- P3.out := null
- P3.err := null
(Again without closing B.in, B.out, and B.err.)
(vi) P3 is terminated.
(vii) No more instructions, so B closes its internal streams (setting them to the null pointer "null"):
- B.in := null
- B.out := null
- B.err := null
(Closing P.in.)

7. After B and P1 are terminated, the Block is deleted.

(I really hope this is helpfull for you.)


penpen

[taripo]

Thanks, that was very informative.

In what programming language(s) have you had the experience of creating pipe objects?

Was the environment *nix or windows, and were there big differences?

[penpen]

I used ((un)named) pipes in Turbo Pascal, C++ and Java under MS-DOS 6.0 and Windows xyz only (although the Java version should also work under *nix),
so i don't know what the big differences are - i assume there is no difference since Win95;
except when using MS-DOS, but i assume this is not of much interest for you.

I think, all needed to know about pipes could be found here:
- https://msdn.microsoft.com/en-us/library/windows/desktop/aa365780(v=vs.85).aspx
- https://docs.oracle.com/javase/7/docs/api/java/nio/channels/Pipe.html


penpen

[foxidrive]

Thanks, that was very informative.

In what programming language(s) have you had the experience of creating pipe objects?

Was the environment *nix or windows, and were there big differences?

I don't think he understood you penpen. His reply would have been more descriptive about his problem.

My own head began to meltdown as I read through your description but that's coz it's half melted anyway from chronic sleep deprivation.

You go to great lengths to answer highly technical issues

[penpen]

I again thought about how to describe the details, without writing 'wall of texts' (like above):
The only other method are images, ... but i'm not sure if they are easy to understand (without further explaination).

Nevertheless, here is my try (i've summarized multiple steps) using the notation of my above post:

Code: Select all

1.-5.: CLI creates P, B, P1, connect streams, start B, P1 (CLI blocked)
   ┌───────────────────────────────────────────────────────────────────────────────────┐
   │ CLI    ┌─────────────┐  ┏━━━━━━━━━━━━━┓  ┌─────────────────────────────┐          │
   │        │ P1          │  ┃ P           ┃  │ B                           │          │
   │        │             │  ┃             ┃  │                             │          │
   │ stdin >┼─ in    out ─┼──╂> out    in >╂──┼─ in                    out ─┼─> stdout │
   │        │             │  ┃             ┃  │                             │          │
   │        │             │  ┃             ┃  │                             │          │
   │        │       err ─┐│  ┃      errP ─┐┃  │                       err ─┐│          │
   │        └────────────┼┘  ┗━━━━━━━━━━━━┿┛  └────────────────────────────┼┘          │
   │                     └────────────────┴────────────────────────────────┴─> stderr  │
   └───────────────────────────────────────────────────────────────────────────────────┘

6.(i)-(ii): B creates and starts P2 (B blocked)
   ┌───────────────────────────────────────────────────────────────────────────────────┐
   │ CLI    ┌─────────────┐  ┏━━━━━━━━━━━━━┓  ┌─────────────────────────────┐          │
   │        │ P1          │  ┃ P           ┃  │ B    ┌─────────────┐        │          │
   │        │             │  ┃             ┃  │      │             │        │          │
   │ stdin >┼─ in    out ─┼──╂> out    in >╂──┼─ in >┼─ in    out ─┼─> out ─┼─> stdout │
   │        │             │  ┃             ┃  │      │      err2 ─┐│        │          │
   │        │             │  ┃             ┃  │      └────────────┼┘        │          │
   │        │       err ─┐│  ┃      errP ─┐┃  │                   └─> err ─┐│          │
   │        └────────────┼┘  ┗━━━━━━━━━━━━┿┛  └────────────────────────────┼┘          │
   │                     └────────────────┴────────────────────────────────┴─> stderr  │
   └───────────────────────────────────────────────────────────────────────────────────┘

6.(iii)-(v): P2 terminates, B (not blocked anymore) creates and starts P3 (B blocked again)
   ┌───────────────────────────────────────────────────────────────────────────────────┐
   │ CLI    ┌─────────────┐  ┏━━━━━━━━━━━━━┓  ┌─────────────────────────────┐          │
   │        │ P1          │  ┃ P           ┃  │ B    ┌─────────────┐        │          │
   │        │             │  ┃             ┃  │      │ P3          │        │          │
   │ stdin >┼─ in    out ─┼──╂> out    in >╂──┼─ in >┼─ in    out ─┼─> out ─┼─> stdout │
   │        │             │  ┃             ┃  │      │       err ─┐│        │          │
   │        │             │  ┃             ┃  │      └────────────┼┘        │          │
   │        │       err ─┐│  ┃      errP ─┐┃  │                   └─> err ─┐│          │
   │        └────────────┼┘  ┗━━━━━━━━━━━━┿┛  └────────────────────────────┼┘          │
   │                     └────────────────┴────────────────────────────────┴─> stderr  │
   └───────────────────────────────────────────────────────────────────────────────────┘

(Created usingt notepad.exe with 'Consolas' font - in browser displaying these characters seems to be suboptimal.)
The important thing is context is that P2 and P3 are not running at the same time, and use in, out and err of sub thread B (block),
so the output of P1 is partitioned; P1 reads the first, P2 a second part.


penpen

[taripo]

I don't think he understood you penpen. His reply would have been more descriptive about his problem.

Hmm not sure why I didn't get notified of the reply to this thread at the time, but was just checking back on this topic now to refresh my memory on penpen's findstr workaround, and saw that there had been two replies since.

Don't jump to conclusions foxidrive. My reply asking penpen about programming languages was because I wanted to program something e.g. a program called myhead, to test my understanding of what he was saying, where i'd write one that consumed all of stdin and one that didn't, and see if it gives an error when it left lots of data in stdin unconsumed, and if it gave no error when consuming all from stdin. So if he'd have said a language that I know i'd have picked that one to maximise my chances of reproducing what he was saying. Nevertheless, I used C#, I recall that I was able to confirm what he had said since from what I recall, when the myhead program consumed all of stdin then there was no error, and crucially, when there was quite a bit of data in stdin that I left unconsumed, then I got the same error that gnuwin32 head gave. which meant I had a good understanding of what was happening with the gnuwin32 head. As I said, penpen's answer was very informative, and in case that wasn't clear, I could add, it was very helpful!

[penpen]

Foxidrive passed away only a few days after his above post:
viewtopic.php?f=3&t=7529.

penpen

[taripo]

I am very sorry to hear that.

His technical posts were of great benefit.

He will be missed.