Memory leak when reading large text files

[vin97]

Ok, I will test that.

How does CMD work out piping internally or more specifically, what counts as a "critical pipe" capable of causing memory issues? Is it just piping into a command (with "|") or does it apply to "redirecting" in general?
For example, could I avoid the problems by writing to tempory files and letting FIND read them directly?
Of course, wherever possible, I will use the IF syntax you proposed since it's much faster.

[penpen]

If i remember right, then a pipe should creates two cmd.exe instances, which you should be able to test somehow like the following:

Code: Select all

set "test="
(set "test=a" & set test) | findstr "^"
set test
(>nul pause) | (set "test=b" & set test)
set test

But it's not the pipe itself causing that issue, but any process if you start enough in short enough time.
Every executable file will also create it's own process.
So the following line starts 3 processes (cmd.exe for left pipe, cmd.exe for right pipe, findstr.exe) each time:

Code: Select all

echo "!line!" | find /i ":\" > nul

Redirecting typically doesn't start any process, except if you redirect to sth that needs a driver that isn't loaded at that time (for example when redirecting to LPT or COM device) - but i don't think that driver is loaded more than once (except if the producer was mad).

You will reduce the amount of processes created by not using pipes, but if you start one find.exe-process every loop-iteration, then that also might be sufficient.


penpen

[vin97]

Ok, going to start rewriting my program then.
Learning a lot here, thanks again!

[miskox]

Any news on this. I am really curious what is causing this memory leak. I would assume if this is a system process (that allocates nonpaged pool) it would free it when this allocation is not needed anymore.

Did you test your program with some SLEEP in in it (so the number of processes created is during the longer period and the system has time to deallocate the memory)? While your batch is executing you should monitor your system state (nonpaged pool...).

Did you try to find the file (.dll...) that is causing this with the appropriate tool?

Saso

[vin97]

I was busy with other stuff but I will probably not bother finding the exact cause since I was planning on reinstalling my OS anyhow.
Also, I am pretty certain that with the tipps provided here and some other simple optimizations to the code, it will be possible to avoid these memory-hungry loops/commands entirely.