Batch to archive files

[foxidrive]

EDIT: I tested the /L switch and it treats strings as literal and *.txt doesn't work.

However if you use this it works to exclude *.txt

.*\.txt



I think if the first character in a line is a regular expression syntax then it takes that line as a regular expression, and * is one of the characters that will trigger that.

[tiagocampos]

That did the trick

Can I ask you one more thing at the risk of abusing of your huge good will?

It would be great to have a log file of the archiving process, to register what has been done for future troubleshooting.
Can you assist me?

[foxidrive]

Which details do you need in a log file?

[tiagocampos]

Which details do you need in a log file?

I was thinking about recording all the files moved and zipped, like a logbook with the timestamp in the beginning of each line.

[foxidrive]

replace this line:

forfiles /m "%%b" /d -%day% /c "cmd /c MD 0x22%%e_%%d0x22 2>nul & move @file 0x22%%e_%%d0x22 >nul"

With this one:

forfiles /m "%%b" /d -%day% /c "cmd /c MD 0x22%%e_%%d0x22 2>nul & move @file 0x22%%e_%%d0x22 >nul & >>0x22logfile.txt0x22 echo %dt% moving @file to 0x22%dt%-%%e_%%d0x22.zip"


I think it should give you a log like this:

date moving "c:\folder\filename.ext" to "date-2013_03.zip"

[tiagocampos]

Worked perfectly Thank you

[tiagocampos]

I changed the file type to .log and put the timestamp as the log name:

Code: Select all

forfiles /m "%%b" /d -%day% /c "cmd /c MD 0x22%%e_%%d0x22 2>nul & move @file 0x22%%e_%%d0x22 >nul & >>0x22%dt%.log0x22 echo %dt% moving @file to 0x22%dt%-%%e_%%d0x22.zip"

One question:

How can I make these logs a single file? Right now it creates 1 log in each folder where the script is executed. It would be easier to me if I had only 1 log file, saved in the directory where the script is.

[foxidrive]

Add this red line below:

@echo off
call :getdate today -30
set "folder=%cd%"

and change this in red.


forfiles /m "%%b" /d -%day% /c "cmd /c MD 0x22%%e_%%d0x22 2>nul & move @file 0x22%%e_%%d0x22 >nul & >>0x22%folder%\%dt%.log0x22 echo %dt% moving @file to 0x22%dt%-%%e_%%d0x22.zip"

[tiagocampos]

Perfect

[tiagocampos]

Hi again foxidrive,

I've been experiencing some issues with the script lately. After finishing executing, the script deletes the zips from previous executions, leaving only the new ones in the folder Archive.

I can't seem to find what's causing this behavior, can you please take a look?

Here is the current state:

Code: Select all

@echo off
call :getdate today -30
set "folder=%cd%"

:: get timestamp in YYYYMMDD_HHMMSS format
for /f "delims=" %%a in ('wmic OS Get localdatetime  ^| find "."') do set dt=%%a
set dt=%dt:~0,8%_%dt:~8,6%


:: To specify the location of the file from where we import the list of directories to take action
set file=C:\MTSOMS\Scripts\FileArchiver\dg1_higeco_http_folders.txt
:: To specify the location of the file from where we filters the files we don't want to archive
set exclusion=C:\MTSOMS\Scripts\FileArchiver\dg1_higeco_http_exceptions.txt
:: To specify the location of the program that will handle the compression
set programzip=C:\Program Files\7-Zip\7z.exe

::for /f "delims=" %%a in ('dir /a:d /b') do (
for /f "delims=" %%a in (%file%) do (

pushd "%%a"
 for /f "delims=" %%b in ('dir /b ^|findstr /v /g:%exclusion%') do (
  for /f "tokens=1,2,3 delims=/-" %%c in ('forfiles /M "%%b" /c "cmd /c echo @fdate"') do (
   echo checking "%%b"
   forfiles /m "%%b" /d -%day% /c "cmd /c MD 0x22%%e_%%d0x22 2>nul & move @file 0x22%%e_%%d0x22 >nul & >>0x22%folder%\%dt%.log0x22 echo %dt%;@file;moving @file to 0x22%dt%-%%e_%%d0x22.zip"
  )
 )
popd
)
SET LOG_FOLDER=C:\MTSOMS\Scripts\FileArchiver\LOGS
SET datetime="%date:~6,4%%date:~3,2%%date:~0,2%_%time:~0,2%%time:~3,2%%time:~6,2%"
SET LOG_FILE=%LOG_FOLDER%\LOG_%datetime%.CSV

::for /f "delims=" %%a in ('dir /a:d /b') do (
for /f "delims=" %%a in (%file%) do (
pushd "%%a"
md Archive
for /f "delims=" %%b in ('dir "????_??" /ad /b') do (
   pushd "%%b"
   echo %%a;%%b>>%LOG_FILE%
   echo Making ZIP file from "%%b" folder
   "%programzip%" a -r -tzip "%%b_%dt%.zip" *
   move "%%b_%dt%.zip" .. >nul
   popd
   rd /s /q "%%b"
   move /y "%%b_%dt%.zip" Archive
   )
popd
)

pause
goto :EOF


:getdate
:: Date foward & backward
@echo off
:: from code by Phil Robyn
setlocal
if [%1]==[] (
  echo to get todays date use
  echo call "%~n0" today 0
  echo.
  echo to get yesterdays date use
  echo call "%~n0" today -1
  echo.
  echo to get the date 25 days ago:
  echo call "%~n0" today -25
  echo.
  echo to get the date 1250 days in the future
  echo call "%~n0" today +1250
  goto :EOF)

set date1=%1
set qty=%2
if /i "%date1%" EQU "TODAY" (
 set date1=now
) else (
 set date1="%date1%"
)
echo >"%temp%\%~n0.vbs" s=DateAdd("d",%qty%,%date1%)
echo>>"%temp%\%~n0.vbs" WScript.Echo year(s)^&_
echo>>"%temp%\%~n0.vbs"         right(100+month(s),2)^&_
echo>>"%temp%\%~n0.vbs"         right(100+day(s),2)
for /f %%a in (
  'cscript //nologo "%temp%\%~n0.vbs"') do set result=%%a
del "%temp%\%~n0.vbs"
endlocal& set "day=%result:~6,2%-%result:~4,2%-%result:~0,4%"
:: echo %%day%% is set to "%day%" (without the quotes)

[foxidrive]

The only places it moves files or removes a folder is here:

In the forfiles command which is logged: move @file 0x22%%e_%%d0x22
Does the file logged for that move command get deleted?

And in this segment, which moves the same file twice, and removes directory.

move "%%b_%dt%.zip" .. >nul
popd
rd /s /q "%%b"
move /y "%%b_%dt%.zip" Archive

Apart from that, there's one del command and that just removes the temp VBS script.

[tiagocampos]

What's really buzzing me is that this only happens when I do big runs.

I was just testing with a single folder trying to recreate the error but it was ok.
But when I run for all folders with sometimes 20.000 files each, this issue happens.

Regarding your question, I don't think so. The script is not supposed to "read" files inside the "Archive" folder.
You think it might help if add an option to write the output of that command to a log?

[foxidrive]

I don't know what is being parsed in %file% and the %exclusions% file could have an issue sometimes, if the archive folder is not being treated correctly. The exclusion file is regular expression based IIRC.

[tiagocampos]

Well I saved the list of folders and exclusions used in 2 past runs where this issue occurred:

In the 1st the exclusions were:

Code: Select all

infoLog.txt

And the folders: http://pastebin.com/GhM6iP5M

In the 2nd the exclusions were:

Code: Select all

.csv

And the folders: http://pastebin.com/mp2HhVP5

[foxidrive]

In the section of code at the bottom:

change

Code: Select all

for /f "delims=" %%b in ('dir "????_??" /ad /b') do (

to

Code: Select all

for /f "delims=" %%b in ('dir "????_??" /ad /b') if /i not "%%b"=="Archive" do (

as the archive folder is in the same folder. This will ensure that it isn't processed.
It shouldn't be targeted but short filename and wildcard matches aren't always straight forward as Dave Benham has shown in his post about how wildcards are parsed.

Code: Select all

::for /f "delims=" %%a in ('dir /a:d /b') do (
for /f "delims=" %%a in (%file%) do (
pushd "%%a"
md Archive
for /f "delims=" %%b in ('dir "????_??" /ad /b') do (
   pushd "%%b"
   echo %%a;%%b>>%LOG_FILE%
   echo Making ZIP file from "%%b" folder
   "%programzip%" a -r -tzip "%%b_%dt%.zip" *
   move "%%b_%dt%.zip" .. >nul
   popd
   rd /s /q "%%b"
   move /y "%%b_%dt%.zip" Archive
   )
popd
)