Recursively copy text files and remove duplicate as well

[abc0502]

Hi, This is the batch pawan26sas wanted, he tested it and it worked with him fine, so here it is in case some one else needed it.

Code: Select all

@echo off
cls
:: 1st it set the current dir to a variable to use it later with the second for loop, then
:: go throw all directories taking all csv files and merging them to a file named based
:: on the folder it is in with a temporary extention ".297".
:: 2nd for loop will start with the same directory we saved it to a variable then rename all
:: ".297" files to a ".csv" file types.
For /R %%a in (.) Do (
set "directory=%~dp0"
   pushd %%a
      For /f "tokens=*" %%b in ('dir /S /B /A:-D *.csv') do (
         For /f "tokens=*" %%c in ('type "%%b"') do echo %%c>>"%%~na.297"
      )
CD ..
)
Pushd %directory%
For /R %%d in (.) Do (
   pushd %%d
      Ren "*.297" "*.csv"
   CD ..
)

[pawan26sas]

Hi abc0502,


Thanks a Lot. Yes, this script is working fine for me with all my aspect. Thanks a lot again for your valuable time.


Thanks and Regards,
Pawan Sarswat

[pawan26sas]

Hi abc0502,

I need little bit change in this above working script ----

1.) File should be save at one level up in the folder hierarchy but the file name will remain same.
2.) File_name like -- <Folder_name>_AutoMerge_<Timestamp in DDMMYYY>.csv -- (Little bit modification in file-name also needed)

ex.-

Inside d:\dos\ -- Before running the script.

folder_1
folder_2
file1.csv
file2.csv

Inside d:\dos\ -- After running the script.

folder_1
folder_2
file1.csv
file2.csv
folder_1_AutoMerge_24072012.csv (Result file having merged data from folder_1 only)
folder_2_AutoMerge_24072012.csv (Result file having merged data from folder_2 only)

but "folder_1_AutoMerge_24072012.csv" file will contain merged data only from folder_1 inside,
but this file should not have any data from outside csv file, like file1.csv and file2.csv -- in this case.


My last requirement is --

Before running the same batch script -
it should also delete the files of type -- %_AutoMerge_%.csv recursively -- to prevent the duplicacy.

means, before running the script it must delete the old created files having "AutoMerge" in the file-name string.

[pawan26sas]

Can anyone please help me in this below code ........

@echo off
cls

:: Delete All Old Merged csv files, if there.
For /R %%a in (.) Do (
set "directory=%~dp0"
pushd %%a
For /f "tokens=*" %%b in ('dir /S /B /A:-D *_AutoMerge.csv') do (
For /f "tokens=*" %%c in ('type "%%b"') do (
del "*_AutoMerge.csv"
)
)
CD ..
popd
)
del ../"*_AutoMerge.csv"

:: 1st it set the current dir to a variable to use it later with the second for loop, then
:: go throw all directories taking all csv files and merging them to a file named based
:: on the folder it is in with a temporary extention ".297".
:: 2nd for loop will start with the same directory we saved it to a variable then rename all
:: ".297" files to a ".csv" file types.

For /R %%a in (.) Do (
set "directory=%~dp0"
pushd %%a
For /f "tokens=*" %%b in ('dir /S /B /A:-D *.csv') do (
For /f "tokens=*" %%c in ('type "%%b"') do (
echo %%c>>"%%~na""_AutoMerge.297"
)
)
CD ..
popd
)

Pushd %directory%
For /R %%d in (.) Do (
pushd %%d
Ren "*.297" "*.csv"
CD ..
popd
)

:: Moving Merged csv files at one level up.
For /R %%a in (.) Do (
pushd %%a
For /f "tokens=*" %%b in ('dir /S /B /A:-D *_AutoMerge.csv') do (
For /f "tokens=*" %%c in ('type "%%b"') do (
move "*_AutoMerge.csv" ../
)
)
CD ..
popd
)

1.) in this section -- " :: Delete All Old Merged csv files, if there. "
last statement -- del ../"*_AutoMerge.csv" is not working .. it is working only when i do
cd ..
del "*_AutoMerge.csv" -- but then further code is not working as the directory got changed.
.
.
2.) in this Section -- " :: Moving Merged csv files at one level up. "
code is working fine but the problem is in deletion of topmost folder file has now moved to one level up.
Thats why the problem-1 is also occurring.

Note -- Inspite the code is working but giving so many error message as well.

[pawan26sas]

Now the modified is like --

It is fulfilling my 98% requirement....



@echo off
cls

For %%a in (.) do set "folder=%%~na"

:: Delete All Old Merged csv files, if there.
For /R %%a in (.) Do (
set "directory=%~dp0"
pushd %%a
For /f "tokens=*" %%b in ('dir /S /B /A:-D *_AutoMerge.csv') do (
For /f "tokens=*" %%c in ('type "%%b"') do (
del "*_AutoMerge.csv"
)
)
CD ..
popd
)
cd ..
del "*_AutoMerge.csv"
cd %folder%

:: 1st it set the current dir to a variable to use it later with the second for loop, then
:: go throw all directories taking all csv files and merging them to a file named based
:: on the folder it is in with a temporary extention ".297".
For /R %%a in (.) Do (
set "directory=%~dp0"
pushd %%a
For /f "tokens=*" %%b in ('dir /S /B /A:-D *.csv') do (
For /f "tokens=*" %%c in ('type "%%b"') do (
echo %%c>>"%%~na""_AutoMerge.297"
)
)
CD ..
popd
)

:: 2nd for loop will start with the same directory we saved it to a variable then rename all
:: ".297" files to a ".csv" file types.
Pushd %directory%
For /R %%d in (.) Do (
pushd %%d
Ren "*.297" "*.csv"
CD ..
popd
)

:: Moving Merged csv files at one level up.
For /R %%a in (.) Do (
pushd %%a
For /f "tokens=*" %%b in ('dir /S /B /A:-D *_AutoMerge.csv') do (
For /f "tokens=*" %%c in ('type "%%b"') do (
move "*_AutoMerge.csv" ../
)
)
CD ..
popd
)

[foxidrive]

Now the modified is like --

It is fulfilling my 98% requirement....



@echo off
cls

For %%a in (.) do set "folder=%%~na"

:: Delete All Old Merged csv files, if there.
For /R %%a in (.) Do (
set "directory=%~dp0"
pushd %%a
For /f "tokens=*" %%b in ('dir /S /B /A:-D *_AutoMerge.csv') do (
For /f "tokens=*" %%c in ('type "%%b"') do (
del "*_AutoMerge.csv"
)
)
CD ..
popd
)
cd ..
del "*_AutoMerge.csv"
cd %folder%

You can probably replace that bit with this

del "d:\folder\abc\*_AutoMerge.csv" /s /f /q

[Squashman]

Can anyone please help me in this below code ........

@echo off
cls

:: Delete All Old Merged csv files, if there.
For /R %%a in (.) Do (
set "directory=%~dp0"
pushd %%a
For /f "tokens=*" %%b in ('dir /S /B /A:-D *_AutoMerge.csv') do (
For /f "tokens=*" %%c in ('type "%%b"') do (
del "*_AutoMerge.csv"
)
)
CD ..
popd
)
del ../"*_AutoMerge.csv"

:: 1st it set the current dir to a variable to use it later with the second for loop, then
:: go throw all directories taking all csv files and merging them to a file named based
:: on the folder it is in with a temporary extention ".297".
:: 2nd for loop will start with the same directory we saved it to a variable then rename all
:: ".297" files to a ".csv" file types.

For /R %%a in (.) Do (
set "directory=%~dp0"
pushd %%a
For /f "tokens=*" %%b in ('dir /S /B /A:-D *.csv') do (
For /f "tokens=*" %%c in ('type "%%b"') do (
echo %%c>>"%%~na""_AutoMerge.297"
)
)
CD ..
popd
)

Pushd %directory%
For /R %%d in (.) Do (
pushd %%d
Ren "*.297" "*.csv"
CD ..
popd
)

:: Moving Merged csv files at one level up.
For /R %%a in (.) Do (
pushd %%a
For /f "tokens=*" %%b in ('dir /S /B /A:-D *_AutoMerge.csv') do (
For /f "tokens=*" %%c in ('type "%%b"') do (
move "*_AutoMerge.csv" ../
)
)
CD ..
popd
)

Could you stop using SMALL SIZED text! Makes it hard to read.

[pawan26sas]

I have done with the script, but now there is only one problem -- please help me.

@echo off
cls

For %%f in (.) do set "folder=%%~nf"

:: Delete All Old Merged csv files, if there.
cd ..
del "*_AutoMerge_*.csv" /S /F /Q
cd %folder%

:: 1st it set the current dir to a variable to use it later with the second for loop, then
:: go throw all directories taking all csv files and merging them to a file named based
:: on the folder it is in with a temporary extention ".297".
For /R %%a in (.) Do (
set "directory=%~dp0"
pushd %%a
For /f "tokens=*" %%b in ('dir /S /B /A:-D *.csv') do (
For /f "tokens=*" %%c in ('type "%%b"') do (
set "Timestamp=%date:~7,2%-%date:~4,2%-%date:~-4%_%time:~0,2%:%time:~3,2%:%time:~6,2%_%Date:~0,3%"
echo %%c>>"%%~na""_AutoMerge_"%Timestamp%".297"
)
)
CD ..
popd
)

:: 2nd for loop will start with the same directory we saved it to a variable then rename all
:: ".297" files to a ".csv" file types.
Pushd %directory%
For /R %%d in (.) Do (
pushd %%d
Ren "*.297" "*.csv"
CD ..
popd
)

:: Moving Merged csv files at one level up.
For /R %%k in (.) Do (
pushd %%k
move "*_AutoMerge_*.csv" ../
CD ..
popd
)


In these 2 below lines --

set "Timestamp=%date:~7,2%-%date:~4,2%-%date:~-4%_%time:~0,2%:%time:~3,2%:%time:~6,2%_%Date:~0,3%"
echo %%c>>"%%~na""_AutoMerge_"%Timestamp%".297"

What I want is - Timestamp value should be appended to the filename.
but it is not coming..

Please help..

Thanks in advance..

[Squashman]

Code tags would be nice as well, like everyone else uses.

A double redirection symbol is used to append to the end of a FILE. Not to the end of the file name. If you want the file Renamed then use the RENAME command.

[pawan26sas]

Yes it is redirecting -

echo %%c >> "%%~na""_AutoMerge_"%Timestamp%".297"


but this filename - "%%~na""_AutoMerge_"%Timestamp%".297"

is not containing Timestamp ..



output :- somefilename_AutoMerge_.297

Desired O/P :- somefilename_AutoMerge_<value_of_Timestamp>.297


I am new to Batch script. please help me.

[Squashman]

Your time stamp is inside a for loop. You need to use delayed expansion which means you need to use Exclamation points with your variables instead of percent symbols.

Do you really want to calculate your time stamp at that specific point in your batch file. It is basically going to change when you are streaming the file and redirecting it as it processes each line of input.

[abc0502]

Hi, the time stamp has illegal sign that can not be in a file or folder name which is ":" so it won't create the file

so the time stamp should be like that:

%date:~7,2%-%date:~4,2%-%date:~-4%_%time:~0,2%-%time:~3,2%-%time:~6,2%_%Date:~0,3%

notice the red signs replaces : with -

[Squashman]

Code: Select all

:: 1st it set the current dir to a variable to use it later with the second for loop, then
:: go throw all directories taking all csv files and merging them to a file named based
:: on the folder it is in with a temporary extention ".297".
For /R %%a in (.) Do (
set "directory=%~dp0"
pushd %%a
For /f "tokens=*" %%b in ('dir /S /B /A:-D *.csv') do (
For /f "tokens=*" %%c in ('type "%%b"') do ( 
set "Timestamp=%date:~7,2%-%date:~4,2%-%date:~-4%_%time:~0,2%:%time:~3,2%:%time:~6,2%_%Date:~0,3%"
echo %%c>>"%%~na""_AutoMerge_"%Timestamp%".297"
)
)

How many CSV files are we talking about here. This may run faster if you just built a COPY command to merge them together. I actually have code that does this in a batch file at work. But depending on how many file names you are merging together you may exceed the 8192 byte limit. I usually don't have much more than 15 files that I copy together at a time so it is not a big deal for me to build a copy command.

[foxidrive]

Now that you mention it, one way around the 8192 byte limit in this case is to copy every file in each sublevel job to a temp folder, using numerical names in ascending order, and then use

Code: Select all

COPY *.TXT outputfile.tmp
copy outputfile.tmp "%targetfolder%\%directoryname%.txt"
del outputfile.tmp

[pawan26sas]

Hi Thanks to all of you ... .

The timestamp issue has got resolved now..

but what is this 8192 byte limit ..??

As this script is using echo and then redirection to merge these all csv to one single csv in one shot, recursively.

And I also got the concept of copy but which one would be faster from performance point of view..

Is there any limit in echo also...??