how to process ALL lines of text in a file, including BLANKS

[alan_b]

I need code to process each line of text held by INPUT.TXT and write it to OUTPUT.TXT

My first stage is to create OUTPUT.TXT the same as INPUT.TXT,
after which I can code appropriate tweaks in the output depending on the contents of each input line.

This is what I have so far

Code: Select all

@echo off & setlocal EnableDelayedExpansion

IF EXIST OUTPUT.TXT DEL OUTPUT.TXT
SET /A N=1000
FOR /F "eol= tokens=1* delims=" %%d IN ('type INPUT.TXT') DO (
  SET /A N+=1
  ECHO %%d>> OUTPUT.TXT
)

Right now I am stuck because the input includes blank lines, and the FOR loop excludes them.
I believe SET /P is able to deliver blank lines,
but I cannot see how to make SET /P iterate through each line of INPUT.TXT.

N.B. A plausible alternative is to accept that blank lines are not delivered.
and to deduce that there was one immediately before any line that starts "[" and ends "*]".

The final result will process an INI configuration file with many blocks of code such as

Code: Select all

[NVIDIA Updates*]
LangSecRef=3021
DetectFile=%CommonAppData%\Nvidia\Updates
Default=False
FileKey1=%CommonAppData%\Nvidia\Updates|*.bak

[O&O Defrag*]
LangSecRef=3024
Detect=HKCU\Software\O&O\O&O Defrag
Default=False
FileKey1=%Documents%\O&O\O&O Defrag\data\reports|*.*|RECURSE

I will have greater confidence that a block has started with "[" and ends "*]" if I also detect the immediately preceeding blank line

Regards
Alan

[aGerman]

Hi Alan,
Have a look at this technique.
Quote:

Code: Select all

setlocal enableDelayedExpansion
<%file% (
  for /f %%n in ('type %file%^|find /c /v ""') do for /l %%l in (1 1 %%n) do (
    set /p "ln="
    echo(!ln!
    set "ln="
  )
)>%out%
endlocal

Regards
aGerman

[alan_b]

Thanks

I have already seen the topic viewtopic.php?f=3&t=2128
but could not make the jump from processing a fixed quantity of variables to repeatedly setting the same variable to each line in turn.

Having seen your code I still do not understand it but I have confidence in you.
I am now tired and more than ready for bed, but will get stuck into this tomorrow.

Regards
Alan

[alan_b]

Fantastic.

Even a binary file comparison cannot see any difference between the input and out.

I now see how the code works, but doubt I could reproduce from scratch by myself.

Many thanks
Regards

Alan

[alan_b]

Please explain WHEN the quotes might be needed in

Code: Select all

    set /p "ln="
    echo(!ln!
    set "ln="

e.g. does CMD.EXE do silly things when reading a file that contains "special" character permutations ?

This is more concise, does it have any disadvantages please

Code: Select all

    set ln=& set /p ln=
    echo(!ln!

QUESTION - HOW CAN I SEND DEBUG TO THE SCREEN ?
There is no urgency because I fixed the problems without the debug,
but if there is an easy fix I would like to see it for future needs.

This is my failed attempt at using :ECHO_2 to deliver to Stdout the debug message
"DEBUG %%Y %X%!ln!

Code: Select all

<%file% (
  for /f %%n in ('type %file%^|find /c /v ""') do (
    for /l %%l in (1 1 %%n) do (
      set ln=& set /p ln=
      if "!ln:~0,1!"=="[" if "!ln:~-2!"=="*]" (
        REM Start of new definition block
        SET PRESENT=0
        FOR /L %%Y IN (101,1,%X%) DO IF "!A_%%Y!"=="!ln!" (
          SET PRESENT=1
          CALL :ECHO_2 "DEBUG %%Y %X%!ln!"
        )
      )
      IF "!PRESENT!"=="1" echo(!ln!
    )
  )> %out%
)
GOTO :EOF

:ECHO_2
ECHO %1
GOTO :EOF

I successfully send the correct message, but NOT to Stdout but to %out%
This is a portion of the %out% file with DEBUG wrongly appearing at the start of each configuration block.

"DEBUG 102 123[ATI/AMD Graphics Driver Install Files*]"
[ATI/AMD Graphics Driver Install Files*]
LangSecRef=3024
Detect=HKLM\Software\Ati Technologies\Install
Default=False
FileKey1=%SystemDrive%\ATI\Support|*.*|RECURSE
FileKey2=%SystemDrive%\AMD|*.*|RECURSE

"DEBUG 101 123[ATI/AMD Graphics Card Driver Install Logs*]"
[ATI/AMD Graphics Card Driver Install Logs*]
LangSecRef=3024
Detect=HKLM\Software\Ati Technologies\Install
Default=False
FileKey1=%ProgramFiles%\ATI\CIM\Reports|*.xml

"DEBUG 103 123[Config.msi Folder*]"
[Config.msi Folder*]
LangSecRef=3025
DetectFile=%WinDir%\System32\msiexec.exe
Default=False
FileKey1=%SystemDrive%\Config.msi|*.*|RECURSE

Regards
Alan

[aGerman]

Hi Alan.

Please explain WHEN the quotes might be needed

Well, it's only to protect you from trailing spaces. Your code should work as well, but that wouldn't ...

Code: Select all

set ln= & set /p ln=

... due to the space between equal sign and ampersand.

I'm not sure about your second problem. You could try to work with user-defined streams. Example:

Code: Select all

@echo off &setlocal

5>"test1.txt" 6>"test2.txt" (
  echo 1st line for stdOut
  >&6 echo 1st line for "test2.txt"
  >&5 echo 1st line for "test1.txt"
  >&5 echo 2nd line for "test1.txt"
  echo 2nd line for stdOut
  >&6 echo 2nd line for "test2.txt"
)

start "" notepad "test2.txt"
start "" notepad "test1.txt"

pause

Regards
aGerman

[alan_b]

Many thanks. Never used user-defined streams before.
I only knew that >&1 >&2 and >&3 were related to StdOut, StdErr and StdIn

I first adapted your suggestion to this, which took 2520 mSec to execute :-

Code: Select all

SET file=Winapp2.ini
SET OUT=WINMIN.TXT

<%file% 4> @%OUT% (
  for /f %%n in ('type %file%^|find /c /v ""') do (
    for /l %%l in (1 1 %%n) do (
      set ln= & set /p ln=
      if "!ln:~0,1!"=="[" if "!ln:~-2!"=="*]" (
        REM Start of new definition block
        SET PRESENT=0
        FOR /L %%Y IN (101,1,%X%) DO IF "!A_%%Y!"=="!ln!" (
          SET "PRESENT=1"
          ECHO %%Y %X% -!A_%%Y!-
        )
      )
      IF "!PRESENT!"=="1" >&4 echo(!ln!
      IF "!PRESENT!"=="1" >&9 echo(!ln!
    )
  )
)9> %OUT% 

FC /B %OUT% #%OUT%
FC /B @%OUT% #%OUT%

N.B.
The file #%OUT% is a "Gold Standard" which has been previously validated as correct
and the latest outputs %OUT% and @%OUT% perfectly match #%OUT%
It makes almost no difference to execution time whether I send output to only >&4 or >&9 or to both >&4 and >&9

I experimented with sending the Debug message via predefined stream >&2 ECHO %%Y %X% -!A_%%Y!-
and struck lucky with only 1220 mSec execution time :-

Code: Select all

SET file=Winapp2.ini
SET out=WINMIN.TXT

<%file% (
  for /f %%n in ('type %file%^|find /c /v ""') do (
    for /l %%l in (1 1 %%n) do (
      set ln= & set /p ln=
      if "!ln:~0,1!"=="[" if "!ln:~-2!"=="*]" (
        REM Start of new definition block
        SET PRESENT=0
        FOR /L %%Y IN (101,1,%X%) DO IF "!A_%%Y!"=="!ln!" (
          SET "PRESENT=1"
          >&2 ECHO %%Y %X% -!A_%%Y!-
        )
      )
      IF "!PRESENT!"=="1" echo(!ln!
    )
  )
)> %out%

FC %OUT% #%OUT%

N.B.
The original code without any debug at all took 1110 mSec to execute,
so only a 10% increase in execution duration when adding this pre-defined stream >&2 debug such as
"DEBUG 101 123[ATI/AMD Graphics Card Driver Install Logs*]"

I assume that user-defined stream redirection within the main processing loop is much more disruptive to code flow than predefined stream redirection.

N.B. The Input file is 110 KB long and the output is only 6 KB consisting of 23 blocks of relevant Header and configuration code such as

[ATI/AMD Graphics Card Driver Install Logs*]
LangSecRef=3024
Detect=HKLM\Software\Ati Technologies\Install
Default=False
FileKey1=%ProgramFiles%\ATI\CIM\Reports|*.xml

Again, many thanks. Perfect solution.

Regards
Alan