Organizing XML files into year month format

Discussion forum for all Windows batch related topics.

Moderator: DosItHelp

Post Reply
Message
Author
Matt Williamson
Posts: 82
Joined: 30 Dec 2013 10:16
Location: United States by the big waterfall

Organizing XML files into year month format

#1 Post by Matt Williamson » 18 May 2015 11:31

I need to re-organize close to a million xml files that are all in the same directory so they are in year/month folders. I have written the following code to do that. The problem I'm having is how long it is taking. I ran the script and it took over an hour just to output how many files are in the directory. I'm thinking about breaking it up by copying about 1,000 files at a time to another directory, running the script there until they're all accounted for and then moving them back. What would be the best way to do that? Also, any constructive criticism on the code is welcome. I'll only have to run it against this many files 1 time. After the initial seed, I'll run it once a month on only a few thousand files. Here is my code so far:

Code: Select all

@ECHO OFF
SETLOCAL
Title Reorganizing XMLs - DO NOT CLOSE THIS WINDOW!
color 0F
mode con: cols=100 lines=6
prompt $t $d$_$p$g

::Get start time
for /F "tokens=1-4 delims=:.," %%a in ("%time%") do (
   set /A "start=(((%%a*60)+1%%b %% 100)*60+1%%c %% 100)*100+1%%d %% 100"
)

Echo Start time: %start%

set "sourcedir=C:\Temp\xmltemp"
set "tempdir=C:\temp\xmlreorgtemp"

pushd %sourcedir%
SET "spinChars=\|/-"
for /f %%a in ('"prompt $H&for %%b in (1) do rem"') do set "BS=%%a"
set "spaces=          "
SET /a filesMoved = 0, spinPos = 0, prev = 0

echo Moving XML Files...

setlocal enabledelayedexpansion
for /L %%I in (1,1,7) do set "BS=!BS!!BS!"
for /L %%I in (1,1,3) do set "spaces=!spaces!!spaces!"

For %%A in (%sourcedir%\*.xml) do set /a cnt+=1
echo.
Echo Total XML files: %cnt%
echo.

FOR /f "tokens=1*delims=" %%a IN ('dir /b /a-d "%sourcedir%\*.xml" ' ) DO (
      set /a filesmoved += 1
      call :spinner !filesmoved! "%%~nxa"
)
call :spinner %filesMoved% Done.

for /F "tokens=1-4 delims=:.," %%a in ("%time%") do (
   set /A "end=(((%%a*60)+1%%b %% 100)*60+1%%c %% 100)*100+1%%d %% 100"
)

echo End time: %end%
set /A elapsed=end-start

rem Show elapsed time:
set /A hh=elapsed/(60*60*100), rest=elapsed%%(60*60*100), mm=rest/(60*100), rest%%=60*100, ss=rest/100, cc=rest%%100
if %mm% lss 10 set mm=0%mm%
if %ss% lss 10 set ss=0%ss%
if %cc% lss 10 set cc=0%cc%
echo Elapsed Time: %hh%:%mm%:%ss%
endlocal & echo;
exit /b 0

:spinner <filecount> <filename>
set /a spinPos += 1, spinPos %%= 4, ten = %~1 / 10 * 10
if "%~2"=="Done." set ten=%~1
set "str=[!spinChars:~%spinPos%,1!] %ten% files moved... [%~2]"
set "str=%str:~0,79%"
call :length len "%str%"
set /a diff = 79 - len
if %diff% gtr 0 set "str=%str%!spaces:~-%diff%!"
set /P "=!BS:~-79!%str%"<NUL
if "%~2" NEQ "Done." call :process %~2
exit /b 0

:length <return_var> <string>
setlocal enabledelayedexpansion
if "%~2"=="" (set ret=0) else set ret=1
set "tmpstr=%~2"
for %%I in (4096 2048 1024 512 256 128 64 32 16 8 4 2 1) do (
      if not "!tmpstr:~%%I,1!"=="" (
            set /a ret += %%I
            set "tmpstr=!tmpstr:~%%I!"
      )
)
endlocal & set "%~1=%ret%"
exit /b 0

:process
FOR /f "tokens=2,3,6delims=_" %%m IN ("%~1") DO SET "date1=%%m"&SET "date2=%%n"&SET "whichdate=%%o"
IF DEFINED whichdate SET "date1=%date2%"
IF NOT DEFINED date2 exit /b 1
If not exist .\%date1:~0,4%\%date1:~4,2% MD .\%date1:~0,4%\%date1:~4,2%
MOVE %~1 .\%date1:~0,4%\%date1:~4,2%\ > nul

foxidrive
Expert
Posts: 6031
Joined: 10 Feb 2012 02:20

Re: Organizing XML files into year month format

#2 Post by foxidrive » 19 May 2015 06:50

Code: Select all

FOR /f "tokens=1*delims=" %%a IN ('dir /b /a-d "%sourcedir%\*.xml" ' ) DO (


This style of the FOR command has a bug when processing huge loads of files, and when filenames are also long even fewer will trigger the bug. It takes exponential amounts of time to merely start, as the file name length and count rises.

One way around it to dir /b /a-d "%sourcedir%\*.xml" into a temporary file and then use that file in the for loop.

Matt Williamson
Posts: 82
Joined: 30 Dec 2013 10:16
Location: United States by the big waterfall

Re: Organizing XML files into year month format

#3 Post by Matt Williamson » 19 May 2015 12:32

Thanks Foxidrive, I wasn't aware of this bug.

-Matt

Post Reply