Page 1 of 3

Recursively copy text files and remove duplicate as well

Posted: 17 Jul 2012 02:59
by pawan26sas
I want to copy each text file recursively at each level with the parent_directory named file, but there should be no duplication in the copied files.

for example --

Parent Directory -- Parent_dir contain 2 directory called
parent_1 & parent_2,
and 2 text files called
parent_dir_1.txt & parent_dir_2.txt

now again parent_1 and parent_2 is also having two files inside that - - called ..

parent_1 --------- parent_1_1.txt parent_1_2.txt
parent_2 --------- parent_2_1.txt parent_2_2.txt

Now requirement is -- I need to copy files (like merging) at each level (with some new name) starting from extreme child.

In this way ---

parent_1 will contain one merged text file called -- (<parent_directory_name>.txt) in this case ... parent_1.txt
i.e -- parent_1.txt will contain the data of both (parent_1_1.txt & parent_1_2.txt) the files ---- .

same in the case of Parent_2 as well.


Like this I need to merge the data upto parent_directory level.

So, finally at extreme parent level -- i.e ... Parent_dir ----- final file named should be -- Parent_dir.txt

but the condition is -- this file should not contain the duplication or redundancy. that is --

for ex. we have 7 text files (before final copy) ...

parent_1.txt = parent_1_1.txt + parent_1_2.txt
same as ---- parent_2.txt = parent_2_1.txt + parent_2_2.txt

when we recursively merge (copy the contents of files) all these 6 files in -- parent level directory (i.e - parent_dir) in a single file as well as the contents of parent directroy level files as well.

i.e -- parent_dir.txt = (
( (parent_1.txt + parent_1_1.txt + parent_1_2.txt)
+
(parent_2.txt + parent_2_1.txt + parent_2_2.txt)
)
+
parent_dir_1.txt + parent_dir_2.txt
)

(without duplication of the contents)




Note -- This is an example of upto level-2 .. it could be upto any level of recursion.

Please help me out, I am new to DOS Batch File.

Re: Recursively copy text files and remove duplicate as well

Posted: 17 Jul 2012 03:54
by foxidrive
pawan26sas wrote:Please help me out, I am new to DOS Batch File.


I got lost reading your request - your description is not very clear.

Can you use some real names in the description rather than parent this and 1_2 that?

It seems like you have two text files in the root and also in each subdirectory. You want to copy the ones in the root, down the tree or something...

Re: Recursively copy text files and remove duplicate as well

Posted: 17 Jul 2012 04:35
by pawan26sas
Vechicle (Parent Directory) --

Directory -- Bike, Car , etc..

Files -- Vec1.txt , Vec2.txt, etc.

Now inside Bike --

Directory -- Wheel, BikeEngine , etc. (We can have files inside Wheel and BikeEngine as well)

Files -- Bik1.txt, Bik2.txt, etc..

same for Car as well ... with some name...


Now Starting from extreme child level, (ex. - Inside Bike),

merge the contents of both files Bik1.txt and Bik2.txt into a new file (Inside same Bike directory)

which name must be -- <parent_dir_name>.txt i.e Bike.txt in this case.



This should keep on recursively go on from extreme child to extreme parent level recursively.


and any file should not contain duplicate data.


lets suppose -- Bike.txt = Bike1.txt and Bike2.txt


While again merge these file - into Vehicle.txt

= that will contain only data from Bike.txt (not from Bike1.txt and Bike2.txt) --- (This will remove data duplication in parent_file.)


Final File at extreme level will be -- Vehicle.txt.

Re: Recursively copy text files and remove duplicate as well

Posted: 17 Jul 2012 05:23
by foxidrive
So you start with this:

Vechicle\Vec1.txt
Vechicle\Vec2.txt
Vechicle\Bike\Bik1.txt
Vechicle\Bike\Bik2.txt
Vechicle\Bike\Wheel\Bik1.txt
Vechicle\Bike\Wheel\Bik2.txt
Vechicle\Bike\BikeEngine\Bik1.txt
Vechicle\Bike\BikeEngine\Bik2.txt
Vechicle\Car\Bik1.txt
Vechicle\Car\Bik2.txt

and you want to end with this:

Vechicle\Vehicle.txt
Vechicle\Bike\Bike.txt
Vechicle\Bike\Wheel\Wheel.txt
Vechicle\Bike\BikeEngine\BikeEngine.txt
Vechicle\Car\Car.txt

but each txt file is the sum of two .txt files from the same folder.
IE for the last one the code would be
copy bik1.txt + bik2.txt car.txt
del bik?.txt

is that right?

I'm not sure where the duplication comes into this.

Re: Recursively copy text files and remove duplicate as well

Posted: 17 Jul 2012 05:41
by pawan26sas
Vechicle\Vec1.txt
Vechicle\Vec2.txt
Vechicle\Bike\Bik1.txt
Vechicle\Bike\Bik2.txt
Vechicle\Bike\Wheel\Whel1.txt
Vechicle\Bike\Wheel\Whel2.txt
Vechicle\Bike\BikeEngine\BkEn1.txt
Vechicle\Bike\BikeEngine\BkEn2.txt
Vechicle\Car\Car1.txt
Vechicle\Car\Car2.txt

and I want to end like with this :---

Vechicle\Vehicle.txt
Vechicle\Bike\Bike.txt
Vechicle\Bike\Wheel\Wheel.txt
Vechicle\Bike\BikeEngine\BikeEngine.txt
Vechicle\Car\Car.txt


Wheel.txt = Whel1.txt + Whel2.txt

BikeEngine.txt = BkEn1.txt + BkEn2.txt


Now ---

Bike.txt = Wheel.txt + BikeEngine.txt + Bik1.txt + Bik2.txt

Car.txt = Car1.txt + Car2.txt

And Finally File Now --

Vehicle.txt = Bike.txt + Car.txt


2 main Condition --

1.) each new copied file name should be -- its <parent_directory_name>.txt

2.) No Duplication in the contents... like --


If I Do.. XCOPY or ..

somewhat Recursive copy like --

Vehicle.txt ==

Bike.txt + Car.txt + Car1.txt + Car2.txt + Wheel.txt + BikeEngine.txt + Bik1.txt + Bik2.txt + Whel1.txt + Whel2.txt + BkEn1.txt + BkEn2.txt


This will definately have a duplicate contents..

Re: Recursively copy text files and remove duplicate as well

Posted: 17 Jul 2012 06:10
by foxidrive
Ok, lets break this into smaller tasks - it is still a little confusing.

In each folder there are two text files. The name changes.

In each folder you want the two text files copied together and named the same as the directory, and the text files removed.

This is untested: It will do what I outlined here hopefully - test it on a copy of your files.

It requires an NTFS formatted drive or else the text files will probably not be copied together in alphabetical order.

Code: Select all

@echo off
for /f "delims=" %%a in ('dir /ad /b /s') do (
pushd "%%a"
for /f "delims=" %%b in ("%cd%") do (
copy *.txt "%%~nxb.tmpabc"
del *.txt
ren "%%~nxb.tmpabc" "%%~nxb.txt"
)
popd
)

Re: Recursively copy text files and remove duplicate as well

Posted: 17 Jul 2012 06:22
by pawan26sas
@echo off
For /F %%a in ('dir /S /B /A:-D *.txt') do (
for /f "tokens=*" %%b in (%%a) do echo %%b >> final_file.txt
)

For single final file at extreme parent level -- this above code is working fine.

but I want the functionality kind off --- the below one --
_________________________________________________________________________________________________________

@echo off

SET CDIR=%~p0
SET CDIR=%CDIR:~1,-1%
SET CDIR=%CDIR:\=,%
SET CDIR=%CDIR: =_%

For /F %%i in ('dir /S /B /A:-RD') do ( :: 'dir /S /B /A:-RD' -- shows all the sub-directories recursively.
::CD (%%i) :: trying to move inside recursively
FOR %%k IN (%CDIR%) DO SET "CNAME=%%k"
SET CNAME=%CNAME:_= % :: CNAME is holding parent directory name.

For /F %%a in ('dir /S /B /A:-D *.txt') do (
for /f "tokens=*" %%b in (%%a) do echo %%b >> %CNAME%.txt :: Actual Code
)
)

___________________________________________________________________________________________________________


If you can please help me out in this above code.

Thanks a Lot ..

Re: Recursively copy text files and remove duplicate as well

Posted: 17 Jul 2012 07:05
by foxidrive
I posted a batch file - does it do something wrong?

Re: Recursively copy text files and remove duplicate as well

Posted: 17 Jul 2012 07:23
by pawan26sas
It is achieving my 1/4th of purpose ..

it is creating a file named <extreme_parent_directory>.txt but it should create <parent_folder>.txt ...

and the new file contains the data ONLY of that directory level.. not with the subdirectory level.

Re: Recursively copy text files and remove duplicate as well

Posted: 17 Jul 2012 07:42
by foxidrive
It should be giving you this:

Vechicle\Vehicle.txt
Vechicle\Bike\Bike.txt
Vechicle\Bike\Wheel\Wheel.txt
Vechicle\Bike\BikeEngine\BikeEngine.txt
Vechicle\Car\Car.txt

pawan26sas wrote:and the new file contains the data ONLY of that directory level.. not with the subdirectory level.


Yes, that is what it is designed to do. Break this up into steps. Now you can explain what more you want.

Re: Recursively copy text files and remove duplicate as well

Posted: 18 Jul 2012 03:24
by pawan26sas
Thank you very much -- @ Foxidrive
___________________________________________________________

@echo off
:: This will copy into single final file at extreme parent level.
For /F %%a in ('dir /S /B /A:-D *.txt') do (
for /f "tokens=*" %%b in (%%a) do echo %%b >> pawan.txt
)
___________________________________________________________

@echo off

SET CDIR=%~p0
SET CDIR=%CDIR:~1,-1%
SET CDIR=%CDIR:\=,%
SET CDIR=%CDIR: =_%

FOR /F "delims=" %%a in ('dir /ad /b /s') do (
pushd "%%a"

FOR %%k IN (%CDIR%) DO SET "CNAME=%%k"
SET CNAME=%CNAME:_= %

FOR /F "delims=" %%b in ("%cd%") do (

FOR /F %%a in ('dir /S /B /A:-D *.txt') do (
FOR /F "tokens=*" %%b in (%%a) do echo %%b >> pawan.txt
)
)
popd
)
___________________________________________________________

These above 2 script are working fine ...


But now i am facing only 2 issues -- -
___________________________________________________________
one is -- this below script is giving me immediate parent folder name.

@echo off
SET CDIR=%~p0
SET CDIR=%CDIR:~1,-1%
SET CDIR=%CDIR:\=,%
SET CDIR=%CDIR: =_%
FOR %%k IN (%CDIR%) DO SET "CNAME=%%k"
SET CNAME=%CNAME:_= %
echo %CNAME%


My both script (mentioned above) are creating file(s) with the name -- pawan.txt
but I want file name with -- <immediate parent folder name>.txt
example -- %CNAME%.txt but i am unable to so.
___________________________________________________________


___________________________________________________________
and the other problem is --

these above 2 script are working fine --
only when the file name and directory/sub-directories are not having space
in their name.

but is any space is there in between the names of file or subdirectories,
both the scripts are failing -- saying --

The system can not find the file.

___________________________________________________________


Please help me out in this... I will be thankful to you..

Thanks in Advance.

Re: Recursively copy text files and remove duplicate as well

Posted: 23 Jul 2012 03:13
by pawan26sas
Is there anyone -- who can please help me in this ....

Re: Recursively copy text files and remove duplicate as well

Posted: 23 Jul 2012 03:26
by abc0502
what i understood is that every folder has 3 things 2 txt files and a folder
you want each folder have extra txt file that has all the data from the curent txt files in that folder and all sub folders
and that extra txt file should be named in the same name as the folder taht extra txt file inside
is that right ??

Re: Recursively copy text files and remove duplicate as well

Posted: 23 Jul 2012 03:36
by pawan26sas
That I have done with batch script ...

but now my first and formost concern is -

I need to remove spaces from the directory name.

this script is working for this -- but only for current directory but not for recursively sub-directories.

@ECHO OFF
SETLOCAL EnableDelayedExpansion

FOR /f "tokens=*" %%a IN ('DIR /s /b /ad') DO (
SET Var=%%~na
SET Var=!Var: =_!
REN "%%a" "!Var!"
)


And this one also I got just now --

@ECHO OFF
SETLOCAL ENABLEDELAYEDEXPANSION

for /f "delims=" %%A in ('dir /b /s /ad') do (
set f=%%~A
set f=!f: =_! &rem replace space with underscore
if "%%~A" NEQ "!f!" echo ren "%%~A" !f!
)

This one working fine - but printing the commands ...
not executing them...
when i removes echo from ren .. and then it is giving error -- The Syntx of the command is incorrect.

Re: Recursively copy text files and remove duplicate as well

Posted: 23 Jul 2012 04:17
by abc0502
leave converting spaces into _ at the end, thaimportant now is getting a for loop to recrusively search and make the extra files.

I'm almost complete a code and send it to you to test