Conditional Actions Using Data From Text File

[Samir]

I have several text files that on line 8 either have text that I can use in a conditional to execute something, or are blank, which indicates another action needs to be taken.

What's a good methodology for this?

Here's some example files to work with:
FILE TYPE 1

Code: Select all

-------------------------------------
ADDR
ADDR
ADDR
MISC


      Credit Batch 014 Summary
#  Account Number        Amount Status
- MORE MISC BELOW THIS LINE -

FILE TYPE 2

Code: Select all

-------------------------------------
ADDR
ADDR
ADDR
MISC


      Debit Batch 014 Summary
#  Account Number        Amount Status
- MORE MISC BELOW THIS LINE -

FILE TYPE 3

Code: Select all

-------------------------------------
ADDR
ADDR
ADDR
MISC



   EMS Batch Summary
- IRRELEVANT INFO ON THIS LINE -
     NO INFORMATION TO REPORT     

- MORE MISC BELOW THIS LINE -

FILE TYPE 4

Code: Select all

-------------------------------------
ADDR
ADDR
ADDR
MISC



   Special Handling Summary
- IRRELEVANT INFO ON THIS LINE -
     NO INFORMATION TO REPORT     

- MORE MISC BELOW THIS LINE -

(The conditional that I plan to code is to take file type 1 and 2 and sum the total number found in each file and output the sum to a file. There can be multiple instances of files type 1 and 2, but the batch numbers tie them together, but I'm not exactly sure how I plan to implement that. The other conditional is when line 8 is blank indicating file type 3 or 4, I will search for 'NO INFORMATION TO REPORT' and if this statement is not found in those files, I will output the filename to a problems.txt file.)

[aGerman]

Hmm. Maybe that way

Code: Select all

@echo off 
setlocal EnableDelayedExpansion

set "filepattern=report_*.txt"
set "sumfile=sum.txt"
set "errfile=problems.txt"

set "sum=0"
for %%i in ("!filepattern!") do (
  set "num=" &set "noinfo="
  <"%%~i" (
    for /l %%j in (1 1 7) do set /p "="
    set "ln=" &set /p "ln="
    if defined ln (
      for /f "tokens=3" %%j in ("!ln!") do set /a "num=1000%%j %% 1000"
    ) else (
      for /l %%j in (1 1 2) do set /p "="
      set "ln=" &set /p "ln="
      if "!ln:~5,24!"=="NO INFORMATION TO REPORT" set "noinfo=1"
    )
  )

  if defined num (
    set /a "sum+=num"
  ) else if not defined noinfo (
    >>"!errfile!" echo "%%~i"
  )
)

>"!sumfile!" echo %sum%

[Samir]

It took me a bit to understand the algorithm in your code, but I think I got it now.

I plan to use this code to get the filenames that are needed to be parsed:

Code: Select all

FOR /F "tokens=1-5" %a in ('dir bat*.rpt /a-d^|findstr /c:"%currentDate%"') do

How would I incorporate this into the code you wrote?

I know this line would be changed

Code: Select all

for %%i in ("!filepattern!") do (

but I'm not sure if this would work:

Code: Select all

FOR /F "tokens=1-5" %%i in ('dir bat*.rpt /a-d^|findstr /c:"%currentDate%"') do (

[aGerman]

I plan to use this code to get the filenames that are needed to be parsed:

Code: Select all

FOR /F "tokens=1-5" %a in ('dir bat*.rpt /a-d^|findstr /c:"%currentDate%"') do

You may try this line ...

Code: Select all

FOR /F "tokens=3*" %%h in ('dir bat*.rpt /a-d^|findstr /bc:"%currentDate%"') do (

... and leave the %%i as is in the rest of the code (because it should contain the file name).

If the output of the DIR command has 12 hours time style (with AM or PM appended) you may have to change it a little to "tokens=4*" because there should be one more token in a line in that case.

[Samir]

Something is not right.

Even with the original code, I'm not getting the right output for problems.txt. I'm not sure about sum since it truncates floating point numbers.

I'll see if I can't post some more details when I get some time.

[aGerman]

I'll see if I can't post some more details when I get some time.

Would help indeed.

I'm not sure about sum since it truncates floating point numbers.

That's the kind of Input that I tried to process:

Code: Select all

      Credit Batch 014 Summary

Third token is 014 that I thought should be interpreted as 14. Since a prepended 0 marks it as octal number i tried to get rid of it by catenating 1000 and modulus 1000 afterwards. There is no floating point number in your example and as you probably know batch cannot calculate floating point numbers directly.

I'm not getting the right output for problems.txt.

Hmm. If line 8 is blank it checks whether or not the 6th till 29th character in line 11 is "NO INFORMATION TO REPORT". That means if neither the number in line 8 nor the string in line 11 was found the file name will be appended to problems.txt.

[Samir]

That's the kind of Input that I tried to process:

Code: Select all

      Credit Batch 014 Summary

Third token is 014 that I thought should be interpreted as 14. Since a prepended 0 marks it as octal number i tried to get rid of it by catenating 1000 and modulus 1000 afterwards. There is no floating point number in your example and as you probably know batch cannot calculate floating point numbers directly.

Hmm. If line 8 is blank it checks whether or not the 6th till 29th character in line 11 is "NO INFORMATION TO REPORT". That means if neither the number in line 8 nor the string in line 11 was found the file name will be appended to problems.txt.

Ahh, I see what you were trying to do now. For the 'NO INFORMATION TO REPORT' check, I was thinking to just use FIND with an errorlevel check as opposed to a line check.

I had to get something working very quickly as deadlines were approaching, so I basically hacked up your code with what limited understanding I had of how it worked. I came up with the following working code that I'm using now:

Code: Select all

setlocal EnableDelayedExpansion
SETLOCAL ENABLEEXTENSIONS

FOR /F "tokens=4*" %%h in ('dir bat*.rpt /a-d^|findstr /bc:"%currentDate%"') do (
  set "num=" &set "noinfo=" 
  <"%%~i" ( 
    for /l %%j in (1 1 7) do set /p "="
    set "ln=" &set /p "ln="
    if defined ln (
      for /f "tokens=1-3" %%j in ("!ln!") do (
        IF "%%j"=="Credit" (
          for /f "tokens=1,2,3,4 delims= " %%m in ('FINDSTR /C:"Total Approved 0" %%i') DO SET CREDIT=%%p
        ) 
        IF "%%j"=="Debit" (
          for /f "tokens=1,2,3,4 delims= " %%m in ('FINDSTR /C:"Total Approved 0" %%i') DO SET DEBIT=%%p
        )
      )
IF NOT "!CREDIT!"=="" IF NOT "!DEBIT!"=="" for /f %%Q in ('S:\SORT\ITWORK\DOSCALC\CALC.EXE !CREDIT!+!DEBIT!') DO (
SET CREDIT=
SET DEBIT=
ECHO D%MM%/%DD%'%YYYY:~2,2%>> IMPORT.QIF
ECHO T-%%Q >> IMPORT.QIF
ECHO L[CC Receivable]>> IMPORT.QIF
for /f "tokens=3" %%j in ("!ln!") do echo PBatch %%j >> IMPORT.QIF
ECHO ^^>> IMPORT.QIF
)
    ) else (
FIND "NO INFORMATION TO REPORT" %%i > NUL
IF ERRORLEVEL 1 ECHO NOTEPAD %%i >> PROBLEMS.TXT
    )
  )
)

%currentDate%, %MM%, %DD%, and %YYYY% are defined earlier as this segment of code was integrated into another batch.

Let me know what you think of my methodology. I'm sure there were much better ways to code various aspects in my 'solution'.

[aGerman]

FINDSTR is an external executable (%SystemRoot%\system32\findstr.exe). Since you're keen on having a good performance I tried to avoid external tools because loading another process for each file will waste your precious time (even more if you call it several times for each file)
As I understood your examples there was no need to call findstr because the informations of interest were always found at the same line numbers. Now I'm unsure. Is there any rule how to find the data without using FINDSTR?

[Samir]

I can replace the findstr with this find equivalent for the same output:

Code: Select all

TYPE %%i|FIND "Total Approved 0"

but is the pipe going to make this less efficient than the findstr?

[aGerman]

FIND is an external tool (find.exe) too. Again my question is if you can define a rule for getting the data in your files. E.g. certain data is always in line #X or is always in the line that begins with string Y.

[Samir]

FIND is an external tool (find.exe) too. Again my question is if you can define a rule for getting the data in your files. E.g. certain data is always in line #X or is always in the line that begins with string Y.

So the line method is faster?

[aGerman]

Most likely.

Since you're keen on having a good performance I tried to avoid external tools because loading another process for each file will waste your precious time (even more if you call it several times for each file)

Otherwise I wouldn't have asked you several times.

FOR /L runs fast and you will adjust it that SET /P only reads until the lines of interest are found. FIND or FINDSTR are external programs. Every time you call one of them a process has to be loaded to the stack (which I remember took ~250-300 ms on my old XP Netbook). You can't adjust FIND and FINDSTR to stop at the first occurence. Thus, always the whole file will be read and analyzed.

[Squashman]

You can't adjust FIND and FINDSTR to stop at the first occurence. Thus, always the whole file will be read and analyzed.

If that was the case wouldn't my fixed text attribute program that Dave helped me with take forever to run? It takes all of a couple seconds to output the information I need using FINDSTR on files that are gigabytes in size.
viewtopic.php?p=12603#p12603

[Samir]

Most likely.

Since you're keen on having a good performance I tried to avoid external tools because loading another process for each file will waste your precious time (even more if you call it several times for each file)

Otherwise I wouldn't have asked you several times.

FOR /L runs fast and you will adjust it that SET /P only reads until the lines of interest are found. FIND or FINDSTR are external programs. Every time you call one of them a process has to be loaded to the stack (which I remember took ~250-300 ms on my old XP Netbook). You can't adjust FIND and FINDSTR to stop at the first occurence. Thus, always the whole file will be read and analyzed.

Sorry if I missed that.

And you bring up a good point about parsing the entire file, especially since the information I want is easily at the top thanks to the %skip% variable. That is a big waste, especially if you add in the launch of the external programs. Something I'll need to look at when I get a chance to optimize the code.

You can't adjust FIND and FINDSTR to stop at the first occurence. Thus, always the whole file will be read and analyzed.

If that was the case wouldn't my fixed text attribute program that Dave helped me with take forever to run? It takes all of a couple seconds to output the information I need using FINDSTR on files that are gigabytes in size.
viewtopic.php?p=12603#p12603

But one thing about 'faster' is that aGerman points out that it's in miliseconds. If you have just one call, it might not be noticeable, but in my code, I use it several times which should have a more significant impact.

[aGerman]

I agree that findstr is optimized for big files. What I'm talking about is that calling external tools multiple times will slow down the processing of the code tremendously.

~~~~~~~~~~~~~~~~~~~~~~~~~~~
Example:

"test.txt" with 20 lines and the data I'm after is in line 10.

Code: Select all

bla
bla
bla
bla
bla
bla
bla
bla
bla
data of interest
bla
bla
bla
bla
bla
bla
bla
bla
bla
bla

~~~~~~~~~~~~~~~~~~~~~~~~~~~
1) Running FINDSTR 1000 times

Code: Select all

@echo off &setlocal
echo start %time%
for /l %%i in (1 1 1000) do (
  for /f "delims=" %%j in ('findstr "interest" "test.txt"') do echo %%j
)
echo end %time%
pause

Code: Select all

start 18:52:27,80
data of interest
... etc.
data of interest
end 18:53:34,78
Drücken Sie eine beliebige Taste . . .

67 seconds

~~~~~~~~~~~~~~~~~~~~~~~~~~~
2) Using FOR /L and SET /P 1000 times

Code: Select all

@echo off &setlocal EnableDelayedExpansion
echo start %time%
for /l %%i in (1 1 1000) do (
  <"test.txt" (
    for /l %%j in (1 1 9) do set /p "="
    set /p "ln="
    echo !ln!
  )
)
echo end %time%
pause

Code: Select all

start 18:54:29,87
data of interest
... etc.
data of interest
end 18:54:42,85
Drücken Sie eine beliebige Taste . . .

13 seconds