Recursively copy text files and remove duplicate as well

[foxidrive]

8192 is the command line limit for one command under XP.

Your code echos the text in one file at a time so it will not encounter that limit - but you will lose any blank lines if your text has any.
I would use TYPE to append the files, which doesn't have a character or blank line problem and is more efficient than parsing a file in a for-in-do loop.

replace this

Code: Select all

For /f "tokens=*" %%b in ('dir /S /B /A:-D *.csv') do (
For /f "tokens=*" %%c in ('type "%%b"') do (
echo %%c>>"%%~na""_AutoMerge.297"
)
)

with this

Code: Select all

For /f "tokens=*" %%b in ('dir /S /B /A:-D *.csv') do (
type "%%b">>"%%~na_AutoMerge.297"
)

[Squashman]

Hi Thanks to all of you ... .

The timestamp issue has got resolved now..

but what is this 8192 byte limit ..??

As this script is using echo and then redirection to merge these all csv to one single csv in one shot, recursively.

And I also got the concept of copy but which one would be faster from performance point of view..

Is there any limit in echo also...??

I could probably test it when I get some time today. I work with very large text files on a daily basis.

[Squashman]

I took 9 files that totaled about 1.7GB.
I have a batch file that builds a copy command to combine all the files it finds into one file.
My batch file took 4 minutes 6 seconds to combine all 9 files into one.
Using the TYPE command the way Foxidrive showed you took 9 minutes 29 seconds.

[foxidrive]

TYPE would have to open, seek and append with each file used. Not so bad with small files though.

[Squashman]

TYPE would have to open, seek and append with each file used. Not so bad with small files though.

I agree. Most people don't deal with large text files like I do. Probably wouldn't be that significant if you are only dealing with files that only have a couple hundred lines. I ended up combining 863,251 lines. Line Length of each line was 2060 bytes. 2,062 if you count the CR\LF.

[foxidrive]

I think even FIND would barf at such long lines...

and FINDSTR is even worse.