Counting records

[miskox]

Hi all,

I have .txt files with thousand of records. Records contain numbers only (upto 9 characters long+CR+LF). There might be some empty lines - and these cause problems.

I have many .txt files:

a.txt
b.txt
c.txt
etc...

For example a.txt:

Code: Select all

34095142

34095143
34095144
34095212

34095145
34095146
34095578

34095581
34095202

34094740
34103146
34095167

and now the code:

Code: Select all

@echo off

set /a cnt=0

for /f "tokens=2 delims=:" %%f in ('find /V /C "" *.txt') do set /a cnt=cnt + %%f

echo find_count: %cnt%

set /a cnt=0

for /f "tokens=1" %%f in (a.txt) do set /a cnt=cnt+1

echo for/f count: %cnt%

I use first command (find /v /c...) to get the numbers of records.

Then I process each file and count the records (and do other things). As you can see (and we all know) that FOR /F skips empty lines so the total number of records is not the same.

Code: Select all

c:\a.cmd
find_count: 16
for/f count: 12

c:\

Any other solution to get the correct number of records with the FIND command (without empty lines)? At the end I can replace the code with the FOR /F counting.

Thanks.
Saso

[foxidrive]

This works here with my log files:

Code: Select all

@echo off
for %%z in (*.log) do (
for /f "delims=:" %%a in (' findstr "[0-9]" "%%z" ^|find /c /v "" ') do echo %%z - %%a
)
pause

[Squashman]

In your first FOR loop you are using the FIND command to count the lines of ALL text files in the working directory.

But in your 2nd For loop you are just using the A.txt file and then incrementing a counter for each line it processes.

Not quite understanding what you are trying to do. The find command works just fine for counting the correct number of lines in the file. Why are you trying to do it with a counter.

Why not make nested FOR LOOPS if you need to process all the files in a directory. The outer FOR loop processes each file. The Inner For loop processes the amount of lines in the file with the FIND command and if you need a running total of all the lines in all the files then set a total count variable with that inner loop as well.

[miskox]

In your first FOR loop you are using the FIND command to count the lines of ALL text files in the working directory.

But in your 2nd For loop you are just using the A.txt file and then incrementing a counter for each line it processes.

Not quite understanding what you are trying to do. The find command works just fine for counting the correct number of lines in the file. Why are you trying to do it with a counter.

This is just to show that FIND correctly counts all the lines and that we have 'incorrect' number of lines with FOR /F.
As I mentioned I have *many* .txt files - a.txt is just an example of one file. First I used a FIND command to get the total number of records in all the .txt files (quicker than the FOR /F and set /a cnt=cnt+1).

I need the total number of records because I later in the program count the records I succesfully processed so I can display a progress bar (I must say it looks very good - and keeps the user busy and not nervous).

Code: Select all

+-------------------------+
!XXXXXXXX                 !
+-------------------------+

(but with characters 219, 218 etc.)

And I encountered a difference in the total number of records and the number of processed records.

Why not make nested FOR LOOPS if you need to process all the files in a directory. The outer FOR loop processes each file. The Inner For loop processes the amount of lines in the file with the FIND command and if you need a running total of all the lines in all the files then set a total count variable with that inner loop as well.

Yes, this is the way I do it now. I used the FIND command just becuase it was quicker.

I use Jeb's solution (short).

Thank you all.
Saso

[miskox]

Here is a photo of this:



EDIT: 12-apr-2014: Uploaded english version of the screen so translation is not needed anymore.

When errors are found they are displayed below the bottom progress bar. If there are more than 10 errors, lines are moved in FIFO order (upper line goes off the screen, next line goes up one line etc. and the new error is displayed at the bottom) - progress bars stay where they are.

Saso

Edit 19-feb-2015: Image host does not work anymore. Moved to another server.

[Matt Williamson]

Is that progress bar from another program or did you do it in batch? If it's in batch, I'd love to see the code for it.

[miskox]

Is that progress bar from another program or did you do it in batch? If it's in batch, I'd love to see the code for it.

Pure batch.

I will prepare the code (in English) and post it as soon as possible.

Saso

[miskox]

Here it is:

cnt_all is the number of records (increase this number if your computer is too fast).

Code: Select all

@echo off
cls
call :createbinvalue 218
call :createbinvalue 196
call :createbinvalue 191
call :createbinvalue 179
call :createbinvalue 192
call :createbinvalue 217
call :createbinvalue 219

set spaces=                                                                                &rem
set fillr13=%chr219%%chr219%%chr219%%chr219%%chr219%%chr219%%chr219%%chr219%%chr219%%chr219%%chr219%%chr219%%chr219%
set fillr=%fillr13%%fillr13%%fillr13%%fillr13%%fillr13%%fillr13%
cls
set /a cnt=0
set /a cnt_all=1111
set /a old_percent=-1

:00
set /a cnt=cnt+1
set /a percent=100 * cnt / cnt_all


if not %percent%==%old_percent% set old_percent=%percent%&&call :create_progress_bar %percent%
if %cnt%==%cnt_all% goto :EOF
goto :00

:create_progress_bar

set /a proc=(77 * %1 + 22 ) / 100
title %1 %%

call set strng1=%%fillr:~0,%proc%%%
set strng2=%strng1%%spaces%
set strng3=%strng2:~0,77%

set lines=%chr196%%chr196%%chr196%%chr196%%chr196%%chr196%%chr196%%chr196%%chr196%%chr196%%chr196%%chr196%%chr196%
set line0=0 %%                                 50 %%                                 100 %%&rem
set line1=%chr218%%lines%%lines%%lines%%lines%%lines%%chr196%%chr196%%chr196%%chr196%%chr196%%chr196%%chr196%%chr196%%chr196%%chr196%%chr196%%chr196%%chr191%
set line2=%chr179%%strng3%%chr179%
set line3=%chr192%%lines%%lines%%lines%%lines%%lines%%chr196%%chr196%%chr196%%chr196%%chr196%%chr196%%chr196%%chr196%%chr196%%chr196%%chr196%%chr196%%chr217%

cls
echo %line0%
echo %line1%
echo %line2%
echo %line3%

goto :eof


:createbinvalue
set par1=%1
type nul > t.tmp
makecab /d compress=off /d reserveperdatablocksize=26 /d reserveperfoldersize=%par1% t.tmp %par1%.chr > nul
type %par1%.chr | ( (for /l %%N in (1,1,38) do pause)>nul & findstr "^" > temp.tmp )
>nul copy /y temp.tmp /a %par1%.chr /b
del t.tmp temp.tmp
for /F "delims=" %%a in (%par1%.chr) do set "chr%par1%=%%a"
del %par1%.chr
goto :EOF

This is just a start. Of course all static variables should be placed outside of the subprogram for more speed.

Only if we could move cursor to 0,0 without clearing the screen with pure batch...

Saso

[foxidrive]

Only if we could move cursor to 0,0 without clearing the screen with pure batch...

Saso

It looks good in Windows 8.1 here.

A small binary that resets the cursor position to 0,0 could be used and certutil can be used to create the base64 encoded text for the batch file,
and then to create the binary for use in the batch file.

[Aacini]

The Batch file below create CursorHome.exe, a small auxiliary program that just move the cursor to 0,0 coordinates in the screen:

Code: Select all

@echo off

echo Extracting CursorHome.exe
call :ExtractBinaryFile CursorHome.exe
echo CursorHome.exe file created
goto :EOF


rem Extract Binary File from hexadecimal digits placed in a "resource" in this .bat file

:ExtractBinaryFile filename.ext[.cab]
setlocal EnableDelayedExpansion
set "start="
set "end="
for /F "tokens=1,3 delims=:=>" %%a in ('findstr /N /B "</*resource" "%~F0"') do (
   if not defined start (
      if "%%~b" equ "%~1" set start=%%a
   ) else if not defined end set end=%%a
)
(for /F "skip=%start% tokens=1* delims=:" %%a in ('findstr /N "^" "%~F0"') do (
   if "%%a" == "%end%" goto decodeHexFile
   echo %%b
)) > "%~1.hex"
:decodeHexFile

rem Modified code based on :genchr subroutine
type nul > t.tmp
(for /F "usebackq" %%a in ("%~1.hex") do (
   set input=%%a
   set i=0
   for /L %%I in (0,2,122) do for %%i in (!i!) do if "!input:~%%i,1!" neq "" (
      set hex=!input:~%%i,2!
      set /A i+=2
      if "!hex:~0,1!" neq "[" (
         set /A chr=0x!hex!
         if not exist !chr!.chr call :genchr !chr!
         type !chr!.chr
      ) else (
         for /L %%J in (1,1,5) do for %%i in (!i!) do if "!input:~%%i,1!" neq "]" (
            set "hex=!hex!!input:~%%i,1!"
            set /A i+=1
         )
         if not exist 0.chr call :genchr 0
         for /L %%J in (1,1,!hex:~1!) do type 0.chr
         set /A i+=1
      )
   )
)) > "%~1"
del *.chr
del t.tmp temp.tmp
del "%~1.hex"

rem Expand created file if extension is .cab
set "filename=%~1"
if /I "%filename:~-4%" equ ".cab" (
   expand "%filename%" "%filename:~0,-4%" > NUL
   del "%filename%"
)

exit /B


:genchr
REM This code creates one single byte. Parameter: int
REM Teamwork of carlos, penpen, aGerman, dbenham
REM Tested under Win2000, XP, Win7, Win8
set "options=/d compress=off /d reserveperdatablocksize=26"
if %~1 neq 26 (
   makecab %options% /d reserveperfoldersize=%~1 t.tmp %~1.chr > nul
   type %~1.chr | ( (for /l %%N in (1,1,38) do pause)>nul & findstr "^" > temp.tmp )
   >nul copy /y temp.tmp /a %~1.chr /b
) else (
   copy /y nul + nul /a 26.chr /a >nul
)
exit /B

<resource id="CursorHome.exe">
4d5a900003[3]04[3]ffff[2]b8[7]40[35]b0[3]0e1fba0e00b409cd21b8014ccd21546869732070726f6772616d2063616e6e6f74206265207275
6e20696e20444f53206d6f64652e0d0d0a24[7]dd3855de99593b8d99593b8d99593b8d1746288d9f593b8d6579298d98593b8d5269636899593b8d
[8]5045[2]4c01020020150a53[8]e0000f010b01050c0002[3]02[7]10[3]10[3]20[4]40[2]10[3]02[2]04[7]04[8]30[3]02[6]03[5]10[2]10
[4]10[2]10[6]10[11]1020[2]28[84]20[2]10[27]2e74657874[3]28[4]10[3]02[3]02[14]20[2]602e7264617461[2]90[4]20[3]02[3]04[14]
40[2]40[8]6af5e815[3]6a0050e813[3]6a00e8[4]ff2508204000ff2500204000ff25042040[473]5620[2]6620[2]4820[6]3820[10]8220[3]20
[22]5620[2]6620[2]4820[6]9b004578697450726f63657373006a0147657453746448616e646c65[2]6d02536574436f6e736f6c65437572736f72
506f736974696f6e[2]6b65726e656c33322e646c6c[370]
</resource>

The source code of CursorHome is below:

Code: Select all

;   Stripped down version of CursorPos that just move cursor to 0,0

        include         \masm32\include\masm32rt.inc

        Main            PROTO

    .code

Main    PROC

        invoke  GetStdHandle, STD_OUTPUT_HANDLE     ;EAX = console output handle
        invoke  SetConsoleCursorPosition, eax, 0    ;move cursor to Home position
        invoke  ExitProcess, 0
        
Main    ENDP

        end     Main

Antonio

[foxidrive]

This has a nice little batch written by Aacini which also has cursor changing capability.
Search for pacman.bat on the page.

http://stackoverflow.com/questions/1513 ... r-position

[miskox]

Thanks. I tried with CursorPos.exe and/or CursorHome.exe - it works great. BTW, is this possible with .vbs script?

If anyone has problems with the frame: I have a CP852 in my command prompt window. So check the CP852 table and your CP table.

Here is a screen shot of the frame with the decimal codes:



I did such a progress bar in DCL on OpenVMS more than 20 years ago. And it looked great, too.

Saso

[einstein1969]

Thanks. I tried with CursorPos.exe and/or CursorHome.exe - it works great. BTW, is this possible with .vbs script?

If anyone has problems with the frame: I have a CP852 in my command prompt window. So check the CP852 table and your CP table.

Here is a screen shot of the frame with the decimal codes:



I did such a progress bar in DCL on OpenVMS more than 20 years ago. And it looked great, too.

Saso

I have searched for moving cursor in wsh but i have found nothing...
for wsh i thinks that there is a wrapper and/or it is possible if installed excel... I never tried because i haven't office.

But there is in powershell... or ansi.

einstein1969