FINDSTR memory leak? (XP)

[miskox]

Hi all,

I have a strange situation on one computer (32 bit, XP PRO):

I have a .cmd procedure which (in short) reads an input .txt file (with 20000+ records (delimited), based on a field in this file makes a search (findstr) in another file (or set of file, for example: folder\*.txt). And based on that result it writes some new data to a new file.

Looks like this .cmd causes my computer to have a memory leak - after few times I run this .cmd no new processes can be created, no network connections cannot be established (I cannot access internet, I cannot access maped drives etc.)

I suspect findstr might be the reason.

Any ideas how to find the cause/repair this etc.?

This test triggered the problems in 30 minutes (2GB RAM, closed all other programs). Folder _TXT contains ~60 TXT files. (all of them contain ~25000 records)

Code: Select all

@echo off
:0
>nul (findstr /c:ASDFASDF _TXT\*.*)
goto 0

Anyone can confirm this?

Thanks.
Saso

P.S. I will take the test on another computer.

[foxidrive]

no new processes can be created, no network connections cannot be established (I cannot access internet, I cannot access maped drives etc.)

IMHO It's not likely to be findstr.

Without more information about the process and what else is running, it's impossible to guess what is causing the problems that you see.

Do the files have 20,000 lines and more? What length are the lines?

[miskox]

Memory leak occurs when I run many findstr commands that is why I am suspecting it (I will take another test on another computer as soon as possible - but it takes time as we know).

Lines don't exceed 60 characters (characters being only 7bit characters (numbers, letters A-Z, a-z and a comma (decimal separator))) (+ of course TAB+ CR+LF).

Files contain diffferent number of records, in total ~20000.

.txt files contain something like this (tab delimited):
(if first character of 2nd column starts with 3 then there are 9 columns, if it starts with 1 or 2 then there are 11 columns)

Code: Select all

20.3.2014   36084361   363   FbFcd   0   0   0   11   72      
20.3.2014   36073452   181   hEL18   0   0   0   34   2      
20.3.2014   212000452   1,8   c08o1   0,0   c0L00   113466   3   0   20.0.   0
20.3.2014   212000595   0,6   P31hd   0,0   P3b7E   32886   0   0   20.0.   0
20.3.2014   212000612   4,9   An0tP   0,0   Andti   133927   1   0   20.0.   0

Thanks.
Saso

[foxidrive]

How much RAM does the Windows machine have? Is it 32 bit?

[miskox]

How much RAM does the Windows machine have? Is it 32 bit?

All the info is there, let me repeat it:

32bit WIN XP PRO, 2GB RAM

I tried checking this with Process Explorer (Sysinternals). No luck there. Couldn't find the reason.

Saso

[miskox]

I was going thru Aacini's posts and there was a link to this: viewtopic.php?f=3&t=2128&p=9720#p9720

Can this be a reason?

Quote:

"None of the commands close the input stream, even if they are at the end the stream is still open."

Saso

[Ed Dyreen]

I was going thru Aacini's posts and there was a link to this: viewtopic.php?f=3&t=2128&p=9720#p9720

That link references a reply by jeb and the topic was started by jeb also. There is not a single post there by Aacini. So I have no clue...

Anyways I managed to install windows XP professional on an i486 90Mhz 32MB SD-RAM.
The system had access to virtual memory but only after TUI setup had completed.

Low memory causes low performance, and lots of pageFile activity.
It's usual for the system/application to notify the user when more memory is needed to accomplish a task.

It seems likely you are dealing with a software conflict or corruption.

I would remove any unnecessary hardware, revert to generic drivers and if the problem persist analyze the installation in a virtual environment to exclude hardware issues.

[miskox]

I did some checking today.

Looks like I ran out of non-paged pool (symptoms are the same as Mark Russinovich mentions in his blog - http://blogs.technet.com/b/markrussinov ... 11216.aspx)

Looks like svchost.exe (the one with the following services) is just increasing its virtual size:

svchost.exe 1432 AudioSrv, BITS, Browser, CryptSvc, Dhcp,
dmserver, ERSvc, EventSystem, helpsvc,
HidServ, LanmanServer, lanmanworkstation,
Netman, Nla, RasMan, Schedule, seclogon,
SENS, SharedAccess, ShellHWDetection,
srservice, TapiSrv, Themes, TrkWks, W32Time,
winmgmt, wscsvc, wuauserv, WZCSVC

I ran the test on another XP computer (x86, 2GB) and there is the same problem.
Test on x86 Windows 7 Starter: no problems (nonpaged pool stays the same).

More test will follow.

Saso

[miskox]

Update:

looks like ESET Endpoint antivirus is causing memory leaks.

If you have XP (PRO or HOME), 32 bit you can test it. The following code will create a file, search thru it and do this many times. Every one minute the program will check the current NONPAGEDPOOL size. When it exceeds 200000kB program will stop.

Code: Select all

@echo off
cls
REM max nonpaged pool value (200000 kB if RAM GTR 1.2 GB). If this value is too high system can have problems running.
REM That is why we have to stop early enough.
set maxnonp=200000

set tmp_vbs="tmp.vbs"
set tmp_tmp="tmp.tmp"
>%tmp_vbs% (
echo Set objWMIService = GetObject^("winmgmts:" _
echo    ^& "{impersonationLevel=impersonate}!\\.\root\cimv2"^)
echo set objRefresher = CreateObject^("WbemScripting.SWbemRefresher"^)
echo Set colItems = objRefresher.AddEnum _
echo    ^(objWMIService, "Win32_PerfFormattedData_PerfOS_Memory"^).objectSet
echo objRefresher.Refresh
echo.
echo For Each objItem in colItems
echo   objRefresher.Refresh
echo        wscript.Echo "Pool Nonpaged Bytes: #" ^& objItem.PoolNonpagedBytes
echo next
)

call :GETNONPAGEDPOOL
set nonp_start=%nonp%
set old_min=-1

REM ESET drivers are probably the cause of the memory leak:
REM poolmon.exe reveals 'Mloc' tag which probably belongs to:
REM epfwtdir.sys
REM ehdrv.sys
REM edevmon.sys
REM eamon.sys

:0
>x.x (echo THE QUICK BROWN FOX JUMPS OVER A LAZY DOG)
>>x.x (echo THE QUICK BROWN FOX JUMPS OVER A LAZY DOG)
findstr /C:"THE" x.x>y.y
del x.x
del y.y
if %nonp% GTR %maxnonp% goto :finish

for /f "tokens=2 delims=:" %%f in ("%time%") do set min=%%f

if not "%old_min%"=="%min%" set old_min=%min%&&call :GETNONPAGEDPOOL


goto :0

:GETNONPAGEDPOOL

>%tmp_tmp% (cscript //E:VBS //NOLOGO %tmp_vbs%)
for /f "tokens=1,2 delims=#" %%f in ('type %tmp_tmp%') do set nonp=%%g
set /a nonp=nonp / 1024
echo %date% %time% %nonp% kBytes
goto :eof


:finish
echo NONPAGEDPOOL FULL. Finish.
echo.
echo NONPAGED POOL at the beginning: %nonp_start%
echo NONPAGED POOL now: %nonp%
echo.
echo REBOOT required.
echo.
if exist tmp.tmp del tmp.tmp
if exist tmp.vbs del tmp.vbs
if exist x.x del x.x
if exist y.y del y.y

goto :EOF

Here is the output:

Code: Select all

wed 30.04.2014 13:43:15,28 94844 kBytes
wed 30.04.2014 13:43:15,43 94816 kBytes
wed 30.04.2014 13:44:00,14 97844 kBytes
wed 30.04.2014 13:45:00,12 101864 kBytes
wed 30.04.2014 13:46:00,20 105924 kBytes
wed 30.04.2014 13:47:00,12 109872 kBytes

ESET support contacted.

Saso

[miskox]

Update:

confirmed - ESET AV was causing the problems. Developer corrected the error.

Saso