List XML files after a certain file

[MattW76]

This is kind of convoluted but I'll explain as best as I can. I need to extract all of the file names between 2 files that have the same date in the file name and after the latest one. Here is my very sparse code so far to extract the first file:

Code: Select all

@echo off
setlocal

pushd \\server1.com\developer\inetpub\Ftproot\XMLs
echo.
for /f "tokens=2 delims=_" %%a in ('dir /b /o-d ABC*.xml') do (
    for /f %%b in ('dir /b /o-d ABC*%%a*.xml') do (echo %%~b - %%~tb &&goto:out)
)
:out
popd

This just finds the latest ABC xml file in the directory. Now I need to find the 2nd latest and list all of the files between the 2nd latest ABC xmls and all of the files after the latest ABC xml file.

I can't really do a dir listing for this as even a small subset would be hundreds of files. Here is a very small subset of what I expect to return.

Code: Select all

08/20/2015  05:50 PM             8,829 BC6009490001_ACCTV21_20150820_CHG_367_107431247A.xml
08/20/2015  05:50 PM            11,552 BC6009490001_BALMIS_20150820_CHG_367_107432647B.xml
08/20/2015  05:49 PM               679 ABC_20150820_107481745D.xml
08/20/2015  05:48 PM             7,647 ID1779750001_BALMIS_20150820_NAC_5_1074336956.xml
08/20/2015  05:48 PM             7,650 ID1779740001_BALMIS_20150820_NAC_5_107447244U.xml
08/20/2015  05:48 PM             7,604 ID1778730001_BALMIS_20150820_NAC_5_107434944I.xml
08/20/2015  05:48 PM             7,786 ID1808430001_BALMIS_20150820_NAC_2_1074206446.xml
08/20/2015  05:43 PM             8,911 SB1522960001_ACCTV21_20150820_EXT_3_107249691J.xml
08/20/2015  05:43 PM             6,141 SB1522960001_BALMIS_20150820_EXT_3_107400191K.xml
08/20/2015  05:43 PM               623 ABC_20150820_1074331234.xml

So, I can find the latest ABC*.xml file easily but I want to get the 5:43 one and everything above it. The search date has to be extracted from the file name because sometimes it differs from the date dir returns.

[foxidrive]

This is kind of convoluted but I'll explain as best as I can. I need to extract all of the file names between 2 files that have the same date in the file name and after the latest one.

It's not clear what you are trying to filter, without using the DIR timestamp.

Code: Select all

08/20/2015  05:50 PM             8,829 BC6009490001_ACCTV21_20150820_CHG_367_107431247A.xml
08/20/2015  05:50 PM            11,552 BC6009490001_BALMIS_20150820_CHG_367_107432647B.xml
08/20/2015  05:49 PM               679 ABC_20150820_107481745D.xml
08/20/2015  05:48 PM             7,647 ID1779750001_BALMIS_20150820_NAC_5_1074336956.xml
08/20/2015  05:48 PM             7,650 ID1779740001_BALMIS_20150820_NAC_5_107447244U.xml
08/20/2015  05:48 PM             7,604 ID1778730001_BALMIS_20150820_NAC_5_107434944I.xml
08/20/2015  05:48 PM             7,786 ID1808430001_BALMIS_20150820_NAC_2_1074206446.xml
08/20/2015  05:43 PM             8,911 SB1522960001_ACCTV21_20150820_EXT_3_107249691J.xml
08/20/2015  05:43 PM             6,141 SB1522960001_BALMIS_20150820_EXT_3_107400191K.xml
08/20/2015  05:43 PM               623 ABC_20150820_1074331234.xml

So, I can find the latest ABC*.xml file easily but I want to get the 5:43 one and everything above it. The search date has to be extracted from the file name because sometimes it differs from the date dir returns.

Where is the search date in the filenames? They all look to have 20150820 and which is the same.

[MattW76]

Now that I think about it, I'll have to use the DIR date and time. I can't see a way not to since the filename doesn't have the time in it.

In most cases when I'm doing this, I just need to select all of the files that are newer than the 2nd ABC*.xml file. This is just to prevent having to scroll through hundreds if not thousands of files to make sure they all are there and later I have to check a sql database to make sure the same number of files are written to a table. I really want to have the first line of output to show the count of files found and then output all of the list starting at the 2nd ABC*.xml and all *.xml files that are newer to a file that I can open in notepad++ and eyeball really quick. I hope that explains it a bit better. And now that I've written this, I have some code coming through in my head right now but I do appreciate anything you have to offer.

[foxidrive]

It's almost clear Matt.

My only question left is explained here - you are doing a dir /b to get a list of filenames with the newest at the top, and you wish to filter all the files from top, down to and including the 2nd ABC*.xml file - correct?

Plus the count of those files?

[MattW76]

Yes, That is exactly what I'm trying to do.

Thanks for your help Foxi. I appreciate it.

[MattW76]

Here is what I came up with but I think I'm falling prey to the Dir read for large directories. There are 32k files and this takes about 30 seconds to run but the dir is just going to keep growing until the end of the month. I can't write the file out because I only have read access on the server. I'll figure some way around that.

Code: Select all

@echo off 
setlocal enabledelayedexpansion

set "found=0"
pushd \\server1\developer\inetpub\Ftproot\XMLs
echo.
for /f "tokens=1,2 delims=[]" %%a in (
   'dir /b /o:-d *.xml^|find /i /n /v "^"'
   ) do ( 
      set "f3=%%b"
      IF "!f3:~0,3!" EQU "ABC" set /a "found+=1"
      IF "!found!" EQU "2" (
            Echo %%b
            Echo %%a Files Total
            Goto :out
         ) ELSE ( 
            echo %%b
      )
   )
:out
popd

Any tweaks of recommendations are welcome.

[foxidrive]

I'm not sure if this will be faster but it eliminates a problem when using a FOR loop with a lot of filenames/long filenames and an execution delay starts to become very long.

There's no reasonable way to stop the DIR during the listing and the FOR loop doesn't exit cleanly using this method either, as it processes every line but just doesn't show anything once the goto out is executed. This adds to the time taken also.

There are methods to exit a FOR cleanly - I haven't remembered them but the others here may chime in, or you can search the forum for them.

Code: Select all

@echo off
dir /b /o:-d "\\server1\developer\inetpub\Ftproot\XMLs\*.xml" >"%temp%\filterABC.tmp"
set "found="
set "files="
setlocal enabledelayedexpansion
(
for /f "usebackq delims=" %%a in ("%temp%\filterABC.tmp") do (
      set "f3=%%a"
      echo %%a
      set /a files+=1
      IF /i "!f3:~0,3!" EQU "ABC" IF defined found goto :out
      IF /i "!f3:~0,3!" EQU "ABC" set "found=1"
   )
)>"%temp%\filterABC2.tmp"
:out
echo located !files! files in total
pause
type "%temp%\filterABC2.tmp"
pause
del "%temp%\filterABC.tmp"

[MattW76]

I can't do it that way. I only have read access on the server. I also can't write back from the server to my workstation I tried that already. I can only run it from my workstation against the server and get the output in the cmd window. I have my buffer size set to 999 and num of buffers set to 999 as well and it still isn't enough. I do have RDP access to the server though, so I'm going to play around with that same more.

[Squashman]

Explain to us how you saved the batch file to execute it if you have no write access?

[MattW76]

I created and run it from my workstation against the server. Since I have read access, I'm able to pushd and popd to get access and have it output to my local workstation. I just can't write a file to the server or my workstation from the server.

[Squashman]

You are executing the batch file from your workstation. Why would you be saying you can't write a file from the server to your workstation. You are not executing the batch file from the server. If you are executing the batch file from your workstation, you most definitely can write the output of the batch file to your workstation.

[foxidrive]

I can't do it that way. I only have read access on the server. I also can't write back from the server to my workstation I tried that already.

As Squashman mentions - the batch script reads from the server and writes to the temp folder on your workstation.

If there is an error on the screen can you tell us what it says?

[MattW76]

Thanks Foxi. I have this working perfectly now. You forgot the /a on location and I just used notepad to show the file list as well as sending the number of files there too. Your code is way faster than mine too. It takes less than a second for it to pop up the populated notepad window.

[foxidrive]

Thanks Foxi. I have this working perfectly now. You forgot the /a on location and I just used notepad to show the file list as well as sending the number of files there too. Your code is way faster than mine too. It takes less than a second for it to pop up the populated notepad window.

Brilliant result there Matt!

The execution delay in the FOR command (with stacks of files/long names) must have been the primary problem in your case.

I added the /a in set /a files+=1 that I'd missed. Ta.

[MattW76]

Given this code that Foxidrive helped me with:

Code: Select all

@echo off
set "xmlpath=\\Server1\developer\inetpub\Ftproot\XMLs\*.xml"
dir /b /o:-d %xmlpath% >"%temp%\filterEOD.tmp"
set "found="
set "files="
setlocal enabledelayedexpansion
(
for /f "usebackq delims=" %%a in ("%temp%\filterEOD.tmp") do (
      set "f3=%%a"
      echo %%a
      set /a files+=1
      IF /i "!f3:~0,3!" EQU "EOD" IF defined found goto :out
      IF /i "!f3:~0,3!" EQU "EOD" set "found=1"
   )
)>"%temp%\filterEOD2.tmp"
:out
echo.
echo located !files! files in total>>"%temp%\filterEOD2.tmp"
echo.
start "Filter EOD Files" "%temp%\filterEOD2.tmp"
del "%temp%\filterEOD.tmp"

I need to add the Date and Time after the EOD message on the same line. This is much harder than I thought it would be. I tried just inserting it in the

Code: Select all

IF defined found goto :out

but that doesn't include the first EOD file reference. Am I missing something obvious?

Thanks

-Matt