How to force processing of certain files first in for loop?

[SIMMS7400]

HI All -

I have a data process I'm building for a client. All files are staged in a single directory so I'm using a for loop to consume all files and process accordingly. However, there are certain files I need to process first just to requirement from within the target application that I"m importing the files into.

Working with the source IT team has been painful so rather than reach out to them to have them name the files differently or drop them in a specific order, is there another way to do this? My thoughts are before I process I can loop through and rename the files I need to process first with "_" at the front and then sort them? Perhaps their a better option? Thank you!

[Aacini]

You have not stated a minimum of specifications. Have the file names spaces or special characters? Have the files to process the same or different extensions? How do you define the first files: via a list? via a file with names? Must the "first files" be processed in a certain order (or it is enough to process they before the "rest of files")? How many files exists? All these points define and modify the method used to implement a solution...

In the solution below I assumed that: 1-File names have no spaces nor special characters, 2-Files have the same extension, 3-First files are defined via a list, 4-First files will be processed in the natural FOR order, 5-There are less than 1000 total files.

The method consist of: Process all files via FOR command and create an ordered array with their names. When a "first file" name appears, put it in the first places of the array. At end, process the array.

Code: Select all

@echo off
setlocal EnableDelayedExpansion

rem Define the "First Files" list
set "firstFiles=/First/Second/Third/"

rem Process all files and create "file[i]" ordered array
set "i=1000"
for %%a in (*.ext) do (
   set /A i+=1
   if "!firstFiles:/%%~Na/=!" equ "%firstFiles%" (
      rem Is not a "First File": insert it in its natural order
      set "file[!i!]=%%~Na"
   ) else (
      rem Insert it at beginning of the array
      set "file[0!i:~1!]=%%~Na"
   )
)

rem Process the "file[i]" array
echo Files will be processed in this order:
for /F "tokens=2 delims==" %%a in ('set file[') do echo %%a.ext

The method could also use a rename of the first files so a posterior FOR command process all files, but I am pretty sure that such a method would be slower than this one, specially if the number of "first files" is large...

Antonio

[SIMMS7400]

HI Antonio -

Yes you're right, allow me to provide some more context!

1. Can be any number of files (1-1000)
2. Can have diferent extensions
3. The "first files" can be processed in any order so long as they are processed BEFORE the others
4. I have a list (or an array) setup which defines the files in a specific order. The variable "OLU_TYPE" dictates the order of processing. Anything with "AH" must be processed before anything tagged as "D".

Below is my list/array I initialize :

Code: Select all

FOR /F "skip=1 tokens=2-5 delims=|" %%A IN (
	"#|||||"^

	"|CITY|CITY|CITY_Errors.zip|AD|"^

	"|COUNTY|COUNTY|COUNTY_Errors.zip|AD|"^

	"|REGION|REGION|REGION_Errors.zip|AD|"^

	"|SITE_TYPE|SITE_TYPE|SITE_TYPE_Errors.zip|AD|"^

	"|STANDING|STANDING|STANDING_Errors.zip|AD|"^

	"|STATE|STATE|STATE_Errors.zip|AD|"^

	"|SITE|SITE|SITE_Errors.zip|D|" 
) DO SET "STR[%%A].OLU_JOB=%%B" & SET "STR[%%A].OLU_ERR=%%C & SET "STR[%%A].OLU_TYPE=%%D"

And then at another part of the script, I then read from a specific directory and match up the first part of my list above to the file name and check if there's a match:

Code: Select all

::-- Set Data Management file name as variable for processing --::
SET "FILENAME="
FOR %%A IN ("%FILE_BIN%*") DO SET "FILENAME=%%~nxA"

::-- Determine Array Definition --::
FOR %%a IN ("DLR" "OLU_JOB") DO (
    FOR /L %%A IN (100,-1,1) DO (

        CALL SET "MASK=%%FILENAME:~0,%%A%%"
        CALL SET "_CHECK=%%STR[!MASK!].%%~a%%"
        
        REM Breakout of loop once match is found
        IF DEFINED _CHECK EXIT /B 0
        
    )
)
>>"%LOGFILE%" (
ECHO.
ECHO Warning    : The file %FILENAME% or a portion of it's name could not be matched to an array definition
ECHO              Please confirm the array defintion is setup correctly
)

CALL :PROC_ERR "Import"
SET "ERR=T" & EXIT /B 1

The file names, while in my example above are straight forward, they could have spaces, special chracters etc. I like your idea, the challenge is processing within another for loop.

[SIMMS7400]

Well, I figured out a way to do it and it's working as expected although not sure if this is an acceptable approach. Essentially, I"m running over the files first before I process them to check an array definition I set. If a specific OLU_TYPE is set, then modfy the data modify. Then, when I begin to process in the main body, I'm pulling in OLDEST first in my for loop which will get me the newest each time in my FILENAME variable.

Code: Select all

IF /I ["%OLU_CNT%"]==["0"] IF DEFINED OLU_M_F_POV (

	PUSHD "%FILE_BIN%"
		FOR %%A IN ("%FILE_BIN%*") DO (
			SET "FILENAME=%%~nxA"
			
			REM Ensure %FILENAME% can be matched to an array definition
			CALL :MASK || EXIT /B 1
			
			REM Get OLU_TYPE
			CALL SET "_OLU_TYPE=%%STR[!MASK!].OLU_TYPE%%"

			REM Modify file properties to ensure files to be processed first
			IF /I ["!_OLU_TYPE!"]==["AD"] COPY "!FILENAME!",,+
			SET /A "OLU_CNT+=1"
		)
	POPD

)

::-- Set Data Management file name as variable for processing --::
SET "FILENAME="
FOR /F %%A IN (
	'DIR /a-d /B /O:-D "%FILE_BIN%*"2^>nul'
) DO SET "FILENAME=%%~nxA" & IF DEFINED FILENAME GOTO SETPOV




:MASK

::-- Determine Array Definition --::
FOR %%a IN ("DLR" "OLU_JOB") DO (
    FOR /L %%A IN (100,-1,1) DO (

        CALL SET "MASK=%%FILENAME:~0,%%A%%"
        CALL SET "_CHECK=%%STR[!MASK!].%%~a%%"
        
        REM Breakout of loop once match is found
        IF DEFINED _CHECK EXIT /B 0
        
    )
)
>>"%LOGFILE%" (
ECHO.
ECHO Warning    : The file %FILENAME% or a portion of it's name could not be matched to an array definition
ECHO              Please confirm the array defintion is setup correctly
)

CALL :PROC_ERR "Import"
SET "ERR=T" & EXIT /B 1