Optimizing These Loops

[Samir]

I don't know if the following is much faster, but you could try to avoid using environment variables (instead this uses a tempfile; not tested):

Code: Select all

@echo off
setlocal
:: ... set current date ... 

>list.temp (
   for %%f in ("DAY*.RPT" "PLU*.RPT" "SKU*.RPT" "*.txt") do echo(%%~tf %%~nx
)
for /F "token=2" %%f in ('findstr "^%currentdate%" "list.tmp"') do (
   for /F "token=1 delims=YU" %%t in ("%%~f") do (
      if        "DA" == "%%~t" ( SET DAY=%%f
      ) else if "PL" == "%%~t" ( SET PLU=%%f
      ) else if "SK" == "%%~t" ( SET SKU=%%f
      )
   )
)
del "list.temp"
endlocal
goto :eof

The first loop may be replaced by a dir command (may be faster); in that case you probably have to adjust the tokens in the other loops.


penpen

This is kind of what I was thinking too as a way to speed it up. I like the delimiters usage--very clever! I could replace the for loop with simply

Code: Select all

DIR *.RPT > list.temp

but it would probably be more efficient to make a temp file for only the three types of files like so:

Code: Select all

DIR DAY*.RPT > list.temp
DIR SKU*.RPT >> list.temp
DIR PLU*.RPT >> list.temp

I did some quick testing and looping the way I was is much, much slower than writing the file list to a file and searching that file.

But then I started re-thinking the whole task of getting these filenames into the variables.

My data set consists of just one "DAY" file, one "SKU" file, and one "PLU" file per date. A

Code: Select all

DIR DAY*.RPT|FIND "%currentdate%"

would find that one file much faster, but how would I put the filename that into the variable?

[Samir]

If your system has short names enabled, then you could try this

Code: Select all

@echo off
    setlocal enableextensions enabledelayedexpansion
    
    set "currentdate=11/03/2015"
    
    for /f "tokens=1,2 eol=|" %%a in ('
        dir *.rpt /a-d /-n ^| findstr /c:"%currentdate%"
    ') do for %%c in ("%%~a.%%~b?") do (
        set "filetype=%%~nc"
        set "_!filetype:~0,3!=%%~nxc"
    )

    echo DAY: %_day%
    echo PLU: %_plu%
    echo SKU: %_sku%

I believe I do. Let me try this and see what happens. Seems to be quite a bit faster. I'm going to execute the original code and compare.

Original code took almost a minute, so definitely an improvement!

[Samir]

I've been able to tweak this even a bit more to make it execute almost instantly now.

Code: Select all

:REPORTSTART
    setlocal enableextensions enabledelayedexpansion
FOR %%F IN (DAY PLU SKU) DO (
    for /f "tokens=1,2 eol=|" %%a in ('
        dir %%F*.rpt /a-d /-n /o-d ^| findstr /c:"%currentDate%"
    ') do for %%c in ("%%~a.%%~b?") do (
        set "filetype=%%~nc"
        set "!filetype:~0,3!=%%~nxc"
    )
)
REM    echo DAY: %day%
REM    echo PLU: %plu%
REM    echo SKU: %sku%

ECHO !DAY!
ECHO !PLU!
ECHO !SKU!

Executing the complete batch file used to take a minute. Now it's under 15 seconds. Thank you very for the great responses everyone!

[penpen]

If you differ between the filetypes, then you probably could simplify to:

Code: Select all

FOR %%F IN (DAY PLU SKU
) DO for /f "tokens=1,2 eol=|" %%a in ('
    dir %%F*.rpt /a-d /-n /o-d ^| findstr /c:"%currentDate%"
') do set "%%~F=%%~a.%%~b"

I'm a bit unsure, because of the question mark in "%%~a.%%~b?":
Actually i can't see a benefit using this character, but i may be wrong.


penpen

[Samir]

If you differ between the filetypes, then you probably could simplify to:

Code: Select all

FOR %%F IN (DAY PLU SKU
) DO for /f "tokens=1,2 eol=|" %%a in ('
    dir %%F*.rpt /a-d /-n /o-d ^| findstr /c:"%currentDate%"
') do set "%%~F=%%~a.%%~b"

I'm a bit unsure, because of the question mark in "%%~a.%%~b?":
Actually i can't see a benefit using this character, but i may be wrong.


penpen

I'm actually not exactly sure how this section works either. I know that %%~a expands that without removing quotes, and that it's the same with %%~b. I think the question mark part of the trick of using the 8.3 filenames back to normal.

I'm amazed at the speed improvements with just small changes. I knew there had to be a better and faster way to do what I needed.

[Squashman]

You can do multiple file masks with the DIR command.

[Samir]

You can do multiple file masks with the DIR command.

I checked the help on DIR and didn't see it, but tried and you're absolutely right! Learned something big today! Thank you! I'll see how I can best incorporate this. Probably won't change speed, but definitely will make the code easier to read.

[mcnd]

I'm a bit unsure, because of the question mark in "%%~a.%%~b?":

When writing the code I was trying to use a dir command and filter the full list using the %currentdate% variable with findstr so the list is only retrieved once (a dir command with several masks execute several searchs, so I decided not to include it) and the for command will only process the files that match the date.

The problem is that now the code has to process the lines generated by the dir command. To know where to locate inside the lines the name and extension of the file, I used the `/-n` to place the short names at the start of the line.

But now there is another problem. We only need the DAY, PLU and SKU files, but it is posible that the short name does not include any of this characters, so, we need to convert the short name into a long name and to do it a for command with a wildcard is used to force a filesystem query, locate the file and retrieve the full name.

[penpen]

I was too imprecise and posted too ambiguous (/poor):
Sorry for that.

I didn't want to say, that i don't know the meaning of a "?" in a for command.

I was aware of the benefit in your (mcnd) solution.

But if one differs between the filetypes then it is of not much use for the pure algorithm.
But Samir might need the long name for other reasons; for example to create an infofile containing the path, so other users may access this file via a network, while the short name is not accessible (possible network limitation).


penpen

[mcnd]

I didn't want to say, that i don't know the meaning of a "?" in a for command.

I was sure of it. Excuse me if I was not clear enough, I was just trying to indicate why I used it.

When writting the code my first try was to execute the inner for command without the wildcard, but I was unable to retrieve the full name when using the short one as reference, so at the end I included the wildcard. I see no simpler way to retrieve the full name.

[Squashman]

I checked the help on DIR and didn't see it,

Well there is no concrete example showing it but the help file does use the plural of the word file.

Code: Select all

[drive:][path][filename]
            Specifies drive, directory, and/or files to list.

[Samir]

I'm a bit unsure, because of the question mark in "%%~a.%%~b?":

When writing the code I was trying to use a dir command and filter the full list using the %currentdate% variable with findstr so the list is only retrieved once (a dir command with several masks execute several searchs, so I decided not to include it) and the for command will only process the files that match the date.

The problem is that now the code has to process the lines generated by the dir command. To know where to locate inside the lines the name and extension of the file, I used the `/-n` to place the short names at the start of the line.

But now there is another problem. We only need the DAY, PLU and SKU files, but it is posible that the short name does not include any of this characters, so, we need to convert the short name into a long name and to do it a for command with a wildcard is used to force a filesystem query, locate the file and retrieve the full name.

Thank you for the detailed explanation! This helps to understand the code quite a bit.

[Samir]

I checked the help on DIR and didn't see it,

Well there is no concrete example showing it but the help file does use the plural of the word file.

Code: Select all

[drive:][path][filename]
            Specifies drive, directory, and/or files to list.

I always thought of 'files' as just the normal list of files.

When did this feature in dir come about? I know it wasn't there in the dos 6.0 days. Did it start in win95? Or was it with win2k or xp?

[Squashman]

When did this feature in dir come about? I know it wasn't there in the dos 6.0 days. Did it start in win95? Or was it with win2k or xp?

I would assume Windows NT.

[Samir]

When did this feature in dir come about? I know it wasn't there in the dos 6.0 days. Did it start in win95? Or was it with win2k or xp?

I would assume Windows NT.

I'll have to try it on my NT system. Thank you!