New technic: set /p can read multiple lines from a file

[dbenham]

I have updated timings for my Vista machine.

Remember I said I had a mysterious .5 second delay that randomly appears when I have to instantiate CMD.EXE? Well I've also been having problems for months with my anti-virus scans - they crashed every time. I was using BitDefender. When the license expired this past week I uninstalled BitDefender and installed Avast 6 (free version) instead. Well it crashed with every scan as well.

I had no other apparent symptoms - though I'm sure there were plenty that I was just missing.

Well any way, I ran CHKDSK /R /F (scheduled for next restart), and it found and repaired 3 files with corrupt clusters and marked as unusable 5 additional clusters.

The result - faster reboots, no more mysterious delays, and no more anti-virus scan crashes

Here are my revised timings for my Vista machine (same code and test cases as before):

Code: Select all

                        A V E R A G E   T I M E  ( s e c o n d s )
    Size  Lines  runs  Read1  Read2  Read3   Read4   Read5   Read6
     ~1k     24    10   0.04   0.05   0.05    0.05    0.07    0.08
     ~2k     48    10   0.05   0.07   0.07    0.07    0.12    0.12
     ~4k     96    10   0.07   0.11   0.10    0.11    0.20    0.20
     ~8k    192    10   0.12   0.17   0.17    0.18    0.37    0.37
    ~16k    384    10   0.20   0.31   0.32    0.32    0.71    0.72
    ~32k    768    10   0.36   0.58   0.60    0.61    1.41    1.42
    ~50k   1685     3   0.75   1.34   1.43    1.43    3.09    3.11
   ~100k   3370     3   1.45   2.67   2.98    2.98    6.31    6.35
   ~200k   6740     3   2.89   5.31   6.59    6.65   13.28   13.32
   ~400k  13480     3   5.80  10.56  15.83   15.95   29.29   29.79
   ~800k  26960     3  11.50  21.07  42.43   43.02   69.43   70.11
  ~1600k  53920     3  22.94  42.23 128.49  130.93  182.75  184.90

The timings are all very consistent for a change. For example, whereas the 1k Read1 test used to take either ~.10 or ~.6 seconds, it now consistently takes .05 seconds. The results are also much more linear for small file sizes. I don't know if this improvement is because of CHKDSK or because AVAST monitoring is less resource intensive than BitDefender.

The methods still sort in the same order as far as performance. But these tests show a more pronounced impact of the addition of the pipe to Read2 vs. Read1.

But this is puzzling because Read5 (FINDSTR) and Read6 (FIND), have nearly identical timings. Yet Read3 (FINDSTR DO SET /P) and Read4 (TYPE | FIND DO SET /P) also have nearly identical timings even though Read4 has a pipe that Read3 does not.

These results still demonstrate non-linear behavior with the FOR /F ('command') construct when the outputs are large.

Dave Benham

[OJBakker]

Nice work on the timings.

Here are two more variants to play with.
Both are based on the set/p goto loop limited by filesize.

The first counts characters per line, the counting slows it down.

Code: Select all

@echo off
setlocal EnableDelayedExpansion
set File=%~f1
for /F %%f in ("%File%") do set FileSize=%%~zf
echo FileSize=%FileSize%
set/a size=0
<"%File%" call :DynamicRead "%FileSize%"
endlocal
exit/b

:DynamicRead
   set /p line=
   if errorlevel 1 set "line=" & verify>nul
   echo(!line!
   set "localstr=!line!#"
   set "lineLen=0"
   for %%P in (4096 2048 1024 512 256 128 64 32 16 8 4 2 1) do (
       if "!localstr:~%%P,1!" NEQ "" (
           set /a "lineLen+=%%P"
           set "localstr=!localstr:~%%P!"
       )
   )
   set /a lineLen+=2
   set /a size+=lineLen
   if !size! LSS !FileSize! goto :DynamicRead
exit/b

The second is faster because no charcounting is involved but uses a temporary file.
This second version is dynamic, while reading new content can be added to the inputfile and will still be processed. This can not be done with the for-loop variant.

Code: Select all

@echo off
setlocal EnableDelayedExpansion
set File=%~f1
set tmpFile=%File%.tmp
echo File=%File%
echo tmpFile=%tmpFile%
>nul copy nul "%tmpFile%"
for /F %%f in ("%File%") do set/a FileSize=%%~zf
echo FileSize=%FileSize%
set/a size=0
<"%File%" >"%tmpFile%" call :DynamicRead "%FileSize%"
del %tmpFile%
endlocal
exit/b

:DynamicRead
   set /p line=
   if errorlevel 1 set "line=" & verify>nul
   echo(!line!
   for /F %%f in ("%File%") do set/a FileSize=%%~zf
   for /F %%f in ("%tmpFile%") do set/a tmpFileSize=%%~zf
   if !tmpFileSize! LSS !FileSize! goto :DynamicRead
exit/b

OJB

[jeb]

I thought also about the character counting way, but I can't see a solution for the line end counting.

You simply add 2 for each line (CR and LF), but as Dave shows set/p can read new content even without a newline,
if it is appended later.

jeb

[dbenham]

Another problem with BOTH file size methods is they assume lines are terminated by <CR><LF>. But I frequently find files with lines that are simply terminated by <LF>, even though it is not standard for Windows. Such files will throw off the size computation.

I'm pretty sure the other methods that are based on the number of lines will work regardless.

Another problem with the STOP teminator / GOTO loop method is the STOP terminator could be missed if the last line of the original file is not terminated with <CR><LF> or <LF> etc. This could be avoided by appending a blank line before the STOP string, but that will introduce an extra line to the input when the final line DOES have the line terminator.

Dave Benham

[OJBakker]

The size-calculation is intended for internal use of the batchfile.
It is used to determine when the end-of-input is reached.

Both routines will process files OK if the following rules are respected.
All Lines must end with CRLF (exception is the last line, there CRLF is optional)
The file contains no control characters.
Processed with the above restrictions will result in exact size-calculations.

If file does not use windows-line-endings (CRLF) these routines will fail. See later in this post.
Control characters can be stripped by set/p under certain condition.
This will cause the charactercount to be less than expected.
Nevertheless the entire file will be read and all lines will be processed but some control chars will be lost.
When end-of-input-stream is reached the read-filesize will be less than expected so the loop continues
but set/p returns only no-input errors.
Every cycle adds 2 to the read-filesize so the loop will still finish but it will appear as if we got some trailing empty lines.

Files with UNIX-line-endings are a different problem.
I expected read/p to just read until the LF separator, but this assumption was wrong.
If set/p would read until LF the first routine would read the file without problem, processing
would work with just some trailing empty lines at the end.
The second routine would fail because the tmpcopy would grow faster than the original filesize so the end-condition would be reached before the end-of-inputstream.
But the above is no longer relevant because set/p behaviour is entirely different.

set/p accepts only CRLF as line-separator.
So unix-files can not be read similar to windows-files.
My test also shows a new behaviour of set/p: blockread
If lines are not separated by CRLF set/p just keeps reading characters until its charbuffer is full
and than returns the full buffer as result.
This blockread is not limited to UNIX-files.
It is just how set/p handles extremely long lines.

So UNIX-files can be read using set/p, but the read block must be split in lines in the readloop in the batchfile.

@Dave
The last-line-line-ending problem is a general problem for all read/copy methods posted in this thread.
Your tests just don't show this behaviour.
If the last line of the inputfile has no CRLF at the end all copy routines will add a CRLF after the last line so all copies will be larger than the original.
This does not show in your tests because filecompare in textmode just simply ignores these differences!
Change your test to fc/b and you will get a warning of size-difference.

I will post my analysis of the set/p behaviour in a new thread.
See http://www.dostips.com/forum/viewtopic.php?f=3&t=2160

[Ed Dyreen]

... and two more interesting effects.

1. Dave discover the same, the errorlevel is set to 1 for an empty line, but you can't decide if it is an empty line or if the end of the file is reached, but the errorlevel isn't reset to 0, if the next set /p read a none empty line.

2. (again demonstrated by walid2me Alternative creation of LF)
This demonstrates the usage of PAUSE, pause reads excactly one character, it can even split a CR/LF at a line end.

Code: Select all

@echo off

:)
setlocal enabledelayedexpansion
>nul,(pause&set /p LF=&pause&set /p LF=)<%0
set LF=!LF:~0,1!

echo 1!LF!2!LF!3

jeb

This alternative way of $lf only takes a single line of code, which is good.

Something is wrong, it doesn't seems to work here

Code: Select all

1e2e3
Druk op een toets om door te gaan. . .

Code: Select all

@prompt _$

setlocal enabledelayedexpansion
>nul,(pause&set /p LF=&pause&set /p LF=)<%0
set LF=!LF:~0,1!

echo 1!LF!2!LF!3

pause
exit

Code: Select all

_setlocal enabledelayedexpansion

_(pause & set /p LF=  & pause & set /p LF= ) 1>nul 0<"F:\ADMIN\REPAIR\OS\ED's_Wi
nXPPro32x86\EDInstall_WinXPPro32x86 beta v1\TST.CMD"

_set LF=!LF:~0,1!

_echo 1!LF!2!LF!3
1@2@3

_pause
Druk op een toets om door te gaan. . .

Wait a minute, what's that smiley doing there ?
You did that on purpose didn't you, let's not make it too easy for tooComplex to figure it out
ok jeb you lost me again

Code: Select all

@echo off

:)
setlocal enabledelayedexpansion
>nul,(pause&set /p LF=&pause&set /p LF=)<%0
set LF=!LF:~0,1!

echo 1!LF!2!LF!3

pause
exit

Code: Select all

1
2
3
Druk op een toets om door te gaan. . .

I did my homework, now I demand a little bit of explaining from the master please
First of all, why do u need to tell DOS you are really very happy ?
Next, why the comma behind nul, it also seems to work with a space ?
Next, <"%~0" how does this self-redirection actually work, I don't really understand that.
But now additional coding is needed to push it over endlocal.
That doesn't look very a good

[jeb]

Hi Ed,

First of all, why do u need to tell DOS you are really very happy ?
Next, why the comma behind nul, it also seems to work with a space ?
Next, <"%~0" how does this self-redirection actually work, I don't really understand that.
But now additional coding is needed to push it over endlocal.

I didn't wrote it, the idea comes from walid2me, even the comma and the smiley are from him.

The comma is equivalent to a space, equal sign, or semi colon, it's only a delimiter.

The self redirect with <"%~0" is simply the input file for the code, else you would need an extra file.
But for a stable behaviour you should use "%~f0" instead, else the code fails if you call it via
myBatch instead of myBatch.bat

The smiley isn't neccessary, any content should work there.

The first set/p reads the first line, in this case the "@echo off".
The pause reads the next character, in this case the next line is empty, so it reads a <CR>.
The next set/p reads now the <LF> and the next line "<"%~f0" >nul (set /p LF=&pause&set /p LF=)"
But as we only want the LF, so we have to remove the rest with set LF=!LF:~0,1!

Why do you wan't to push it over the endlocal, and delayed expansion isn't required, but it's easier to use the LF

Code: Select all

@echo off

<"%~f0" >nul (set /p LF=&pause&set /p LF=)
set LF=^%LF:~0,1%%LF:~0,1%

setlocal EnableDelayedExpansion
echo 1!LF!2!LF!3

hope it helps
jeb

[Ed Dyreen]

'
Definetly:

Code: Select all

@echo off

setlocal EnableDelayedExpansion
::(
   <"%~f0" >nul (set /p $lf=&pause&set /p $lf=)
::)
for %%r in ( "!$lf:~0,1!" ) do endlocal &set "$lf=%%~r" &echo.This%%~rworks

[alan_b]

I found this entire thread very interesting, but I have difficulty understanding exactly what is happening.

Since I was out of my depth I decided to cast aside any assumptions of how this sort of code worked.
I created a tool and experimented and observed.

As the very first stage I copied and composed 25 different ways of combining
'|' pipe and "<" redirection and "Type", and "Find" and "Findstr".

I found that only 3 of the 25 combinations would give the same number of output lines as were held by the input file.

I still do not fully comprehend but at least I now know what works.

You may find benefit from my tool at
viewtopic.php?f=3&t=2568

This was much more complicated than I thought it would be. Results are NOT as expected - in fact, I find some of them shocking!

1) This does NOT work

Code: Select all

<%file% (
  for /f "skip=2" %%i in ('find /n /v "" %file%') do (
    set "ln="
    set /p "ln="
    echo(!ln!
  )
)>%out%

It reads and writes the correct number of lines - but the lines are empty
I don't understand the mechanism of failure.

Dave Benham

My tool replaced your "<%file%" with "<%1", and your ('find /n /v "" %file%') with ('find /n /v "" %2')

I ASSUMED that the "echo(!ln!" would deliver text held by %2 but instead it comes from %1

That's pretty cool jeb

But I am still mystified how the 'FIND /N "" %FILE%' in my FOR IN() clause is sharing the same input stream (or file handle?) with the <%file% redirection
Dave Benham

Could my observation throw some light on both these peculiarities of reality ?

Regards
Alan

[ideleon007]

So, after reading this entire thread,
I'm more confused.
I am trying to create a batch file to run under Windows 7 x64 to clean up a drive by
deleting contents from the following folders.

C:\users\%USERNAME%\APPDATA\Local\Temp\*.*
C:\Windows\Temp\*.*[/list][/list]

I am hung up on reading the C:\Temp\Users.txt file and adding each line into a consistent variable
so that I can change the directory under C:\Users.

I hope this makes sense and here is what I have.

Code: Select all

@Echo Time to clean up your drive...
@ECHO OFF
REM DIR C:\Users\*. /B >c:\Temp\users.txt
REM @ECHO STOP>>c:\Temp\users.txt
Set Count=1
Goto Loop

:Loop
< c:\Temp\users.txt (
  set /p 1=
  set /p 2=
  set /p 3=
  set /p 4=
  set /p 5=
  set /p 6=
  set /p 7=
  set /p 8=
  set /p 9=
  set /p 10=
  set /p 11=
  set /p 12=
  set /p 13=
  set /p 14=
  set /p 15=
  set /p 16=
  set /p 17=
  set /p 18=
  set /p 19=
  set /p 20=
) 
set EndUser 

@Echo %EndUser%
If %COUNT%=='STOP' Goto Next_Process
@ECHO Just DELETED %COUNT%'s TEMP FILES
REM del /F /S /Q C:\users\%Count%\APPDATA\Local\Temp\*.*
set /A COUNT=%COUNT%+1
ECHO The COUNT is now %COUNT%
pause
Goto Loop


:Next_Process
@ECHO THIS IS THE NEXT PROCESS
pause

REM Snipits take from http://www.robvanderwoude.com/clevertricks.php
REM del /F /S /Q C:\users\%USERNAME%\APPDATA\Local\Temp\*.*
REM del /F /S /Q C:\Windows\Temp\*.*

Any help appreciated...

[Squashman]

I am not sure why you trying to do what you are doing. It does not make any sense to me to redirect the output of the DIR command to a file and then read that file to assign them to variables and then use those variables. You are jumping through a lot of hoops.

But to shed some light on one of your problems, using numeric variable names is not a good idea.
http://stackoverflow.com/a/30109942

[Aacini]

Perhaps this array management is what you refer as "consistent variable":

Code: Select all

@echo off
setlocal EnableDelayedExpansion

set Count=0
for /F "delims=" %%a in ('DIR C:\Users\*. /B') do (
   set /A Count+=1
   set "user[!Count!]=%%a"
)

echo All users:
set user[

echo Processing each user:
for /L %%i in (1,1,%Count%) do (
   echo User #%%i- "!user[%%i]!"
)

Antonio

[ideleon007]

Understood Sqashman & thanks for the input Antonio.
The goal is to delete the content each of the users' temporary files found in :

C:\users\%USERNAME%\APPDATA\Local\Temp

Then I would also delete the contents of C:\Windows\Temp.
I am open to which ever way I can accomplish that, but was confusing myself (and you) in the process.
Although,
by doing a
DIR C:\Users\*. /B >c:\Temp\users.txt
This would simply give me a txt file containing all the folders within C:\Users excluding and further unwanted info like "." and ".."

By using your example Antonio, which worked well,
how then would I be able to go through each of the directories in the C:\Users folder and
delete the temporary files?

I would like to delete the content of the following folders:
C:\users\%USER1%\APPDATA\Local\Temp
C:\users\%USER2%\APPDATA\Local\Temp
C:\users\%USER3%\APPDATA\Local\Temp\
etc..

[aGerman]

You may try this code:

Code: Select all

@echo off &setlocal
REM Successively assign the profile folders to %%i in a loop
for /d %%i in ("C:\Users\*") do (
  REM Successively assign the subfolders in the temp folder of a certain user profile to %%j and remove them
  for /d %%j in ("%%~i\AppData\Local\Temp\*") do rd /s /q "%%~j"
  REM Delete all files in the temp folder of a certain user profile
  del /f /q "%%~i\AppData\Local\Temp\*.*"
)
pause

The question is have you even permissions to access other user profiles than your own?

Regards
aGerman

[thefeduke]

Thank you for reviving this five year old topic. Great reading in the early going.

I have always liked the idea of having a script with imbedded test data and this technique lends itself to the cleanest method that I have encountered because it uses WYSIWYG test data.

There is no editing SET and ECHO commands to alter the content. I managed to reduce a data locating FOR and an extraction FOR to one pass with the multiple line SET /P commands. I was shocked at the lack of traditional FINDSTR commands to find things. This script adjusts to the number of input records. Here is a .bat file with a name of your choice [Edited to handle blank lines.] :

Code: Select all

   @ECHO Off
:: jeb - aGerman - Post subject: Re: New technic: set /p can read multiple lines from a file
:: http://www.dostips.com/forum/viewtopic.php?p=9598#p9598
GoTo :EndOfDataFile
Help=0
Args=0
Set=0
Defaults=0
Options=0
LastRun=0
SaveRun=0
:EndOfDataFile
    SETLOCAL EnableDelayedExpansion
    Set /A "Stop=Stopped=0"
    For /f %%a in ('type %~f0^|find /c /v ""') do set lines=%%a
   (For /l %%i in (1,1,%lines%) do (
        If "!Stop!" EQU "0" (
            Set /p "Test="
            Set "Test=!Test:~0,11!"
            If /I "!Test!" EQU "GoTo :EndOf" Set "Stop=1"
        ) Else (
            If "!Stopped!" EQU "0" (
                Set "Keep="
                Set /p "Keep="
                Set "Test=!Keep:~0,6!"
                If /I "!Test!" EQU ":EndOf" Set "Stopped=1"
            )
            If "!Stopped!" EQU "0" Echo.!Keep! 
        )
    ))< "%~f0" >"%Temp%\%~n0_Input.txt" 
    Type "%Temp%\%~n0_Input.txt" 
    Exit /B

I hope that you are drawn to the simple data format of the imbedded freeform file.

John A.