Search large folders, copy found files, paste to new folder?

Discussion forum for all Windows batch related topics.

Moderator: DosItHelp

Message
Author
Bobbo Jones
Posts: 48
Joined: 18 Oct 2012 21:27

Re: Search large folders, copy found files, paste to new fol

#31 Post by Bobbo Jones » 28 Oct 2012 20:24

SO to remove the renaming section of the script, it looks as if i delete

Code: Select all

set "name="

for /f "delims=" %%a in ('type "list2.tmp"') do (
if /i "%%~nxa"=="!name!" >>"samefile.txt" echo Same filename - "!name!"
set /a c=c+1
set num=000!c!
set num=!num:~-3!
echo copying "!num! %%~nxa"
copy /b "%%a" "%userprofile%\desktop\copied files\!num! %%~nxa" >nul
set "name=%%~nxa"
)


I am trying this now


Bobbo Jones
Posts: 48
Joined: 18 Oct 2012 21:27

Re: Search large folders, copy found files, paste to new fol

#33 Post by Bobbo Jones » 28 Oct 2012 21:35

is this correct to simply remove the renaming function? it seems to be working.

Code: Select all

@echo off
set "list=list.txt"

setlocal EnableExtensions EnableDelayedExpansion

dir /a:-d /o:n /b /s >filelist.tmp

del list2.tmp 2>nul
del notfound.txt 2>nul
del samefile.txt 2>nul


for /f "delims=" %%a in ('type "%list%"') do echo finding "%%a"& findstr /r /i /c:"\\\%%a$" filelist.tmp >>list2.tmp || >>notfound.txt echo %%a

md "%userprofile%\desktop\copied files" 2>nul

set "name="

for /f "delims=" %%a in ('type "list2.tmp"') do (
if /i "%%~nxa"=="!name!" >>"samefile.txt" echo Same filename - "!name!"
echo copying "!num! %%~nxa"
copy /b "%%a" "%userprofile%\desktop\copied files\!num! %%~nxa" >nul
set "name=%%~nxa"
)

del filelist.tmp 2>nul
if not exist "samefile.txt" del list2.tmp 2>nul
echo.
if exist notfound.txt echo Check NOTFOUND.TXT for filenames that were missing
if exist "samefile.txt" (
ren "list2.tmp" "samefile.list.txt"
echo Check "samefile.txt" and "samefile.list.txt" for the list of repeated files
)
echo.
echo done
echo.
pause

Bobbo Jones
Posts: 48
Joined: 18 Oct 2012 21:27

Re: Search large folders, copy found files, paste to new fol

#34 Post by Bobbo Jones » 28 Oct 2012 22:05

If I am searching the same root directory, would the filelist.tmp be the same each time? as in, time could be saved each search if it didnt have to recreate it each search?

I also still need this "So I need a version that does not rename the files, and deletes duplicates" although I have something which so far seems to be working

Aacini
Expert
Posts: 1914
Joined: 06 Dec 2011 22:15
Location: México City, México
Contact:

Re: Search large folders, copy found files, paste to new fol

#35 Post by Aacini » 28 Oct 2012 23:01

The Batch file below use a method that should be the fastest one to achieve this process with a large number of files: it first get a list of the folders to search in, and then search each file name in that list using the fastest way to do that. This way, the total number of files have not any effect in the processing time, just the number of folders (that is much lower).

Code: Select all

@echo off
setlocal EnableDelayedExpansion

rem source = Path to the file that contain the names list "names without the extension"
rem location = The top folder to search in
rem destination = Path to the destination folder (even if it not exist)

set "source=C:\Users\Evan\Desktop\breaks filenames.txt"
set "location=C:\Users\Evan\Desktop\location folder"
set "destination=C:\Users\Evan\Desktop\Collating folder"

if not exist "%destination%" (
   md "%destination%"
) else (
   del "%destination%\NotFound.txt" 2>NUL
)

rem Get a list of existent folders in location folder
set folders=
for /R "%location%" /D %%d in (*) do set folders=!folders!%%d;

rem Process each file name in the list
set n=1000
for /F "usebackq delims=" %%n in ("%source%") do (
   set /A n+=1
   set num=!n:~-3!
   rem Search this file name in the folders
   for %%f in ("%%n.wav") do set "filePath=%%~dp$folders:f"
   if defined filePath (
      echo Copying:   !num!- %%n.wav
      copy /b "!filePath!%%n.wav" "%destination%\!num! %%n.wav" >NUL
   ) else (
      echo Not found: !num!- %%n.wav
      echo %%n.wav>> "%destination%\NotFound.txt"
      rem Create an empty file in the destination folder
      cd . > "%destination%\!num! %%n.wav  ---  NOT FOUND"
   )
)

if exist "%destination%\NotFound.txt" echo Check NotFound.txt for filenames that were missing
echo/
pause

This Batch file:

- Does NOT search the .wav files directly in the location folder, but only in sub-folders beneath it (to speed up the process). If this is required, a small modification is needed.

- Copy the first file found in the folders, taking they in alphabetical order.

- Will not find a .wav file if its name include an exclamation mark! This problem may be fixed, if required.

- Create empty files in the destination folder when a file name is not found, to preserve the order of the original list.

Please, try it and report the results! 8)

Antonio

Bobbo Jones
Posts: 48
Joined: 18 Oct 2012 21:27

Re: Search large folders, copy found files, paste to new fol

#36 Post by Bobbo Jones » 28 Oct 2012 23:32

Hey Antonio, thanks for your help!
can you please clarify what this sentence meant?
- Copy the first file found in the folders, taking they in alphabetical order.


thanks!

Bobbo Jones
Posts: 48
Joined: 18 Oct 2012 21:27

Re: Search large folders, copy found files, paste to new fol

#37 Post by Bobbo Jones » 29 Oct 2012 00:08

Hey Antonio, it does seem to be a bit faster, but it did not find the files. It seems to have renamed them all with an additional '.wav' before searching for them?? which means of course they won't be found.

What I got was a collating folder full of files like this
001 03_Crazy_White_Boy___Love_You_Better__Villanova_Remix_.wav.wav --- NOT FOUND
it should have been searching for just
03_Crazy_White_Boy___Love_You_Better__Villanova_Remix_.wav

The notfound.txt had a list of files like this
03_Crazy_White_Boy___Love_You_Better__Villanova_Remix_.wav.wav
06_Crazy_White_Boy___Love_You_Better__Eric_Faria_Remix_.wav.wav
03_Like_That__feat__Ono_.wav.wav
07_Like_A_Bird.wav.wav
02_Justin_Martin___Hood_Rich.wav.wav
DB071_072_V_04_Justin_Martin___Hood_Rich__DJ_Version_.wav.wav
02_Abigail_Bailey__and__Nick_Tcherniak___Sundance___Leisuregroove_Remix_.wav.wav

any advice?> cheers!

foxidrive
Expert
Posts: 6031
Joined: 10 Feb 2012 02:20

Re: Search large folders, copy found files, paste to new fol

#38 Post by foxidrive » 29 Oct 2012 02:24

Do you want to *delete* the duplicates? This ignores the duplicates, it just copies the 1st file in a set of the same nameed files. It doesn't add a prefix either (untested).

Code: Select all

@echo off
set "list=list.txt"

setlocal EnableExtensions EnableDelayedExpansion

dir /a:-d /o:n /b /s >filelist.tmp

del list2.tmp 2>nul
del notfound.txt 2>nul
set c=0

for /f "delims=" %%a in ('type "%list%"') do echo finding "%%a"& findstr /r /i /c:"\\\%%a$" filelist.tmp >>list2.tmp || >>notfound.txt echo %%a

md "%userprofile%\desktop\copied files" 2>nul

set "name="

for /f "delims=" %%a in ('type "list2.tmp"') do (
   if /i not "%%~nxa"=="!name!" (
      echo copying "%%~nxa"
      copy /b "%%a" "%userprofile%\desktop\copied files\" >nul
   )
set "name=%%~nxa"
)

del filelist.tmp 2>nul
del list2.tmp 2>nul
echo.
if exist notfound.txt echo Check NOTFOUND.TXT for filenames that were missing
echo.
echo done
echo.
pause

Bobbo Jones
Posts: 48
Joined: 18 Oct 2012 21:27

Re: Search large folders, copy found files, paste to new fol

#39 Post by Bobbo Jones » 29 Oct 2012 08:04

thats right, yeah I dont want to delete it, just ignore it. Thanks! ill try this out tomorrow

Aacini
Expert
Posts: 1914
Joined: 06 Dec 2011 22:15
Location: México City, México
Contact:

Re: Search large folders, copy found files, paste to new fol

#40 Post by Aacini » 29 Oct 2012 10:20

@Bobbo Jones: I am pretty sure that file names in the list include .wav extensions although the program said: "names without the extension". You have two choices to fix this problem:

- Remove all .wav extensions in your list of names, or
- Replace all %%n.wav parts in the program by just %%n. Choose whichever be better for you.

Bobbo Jones wrote:can you please clarify what this sentence meant?
- Copy the first file found in the folders, taking they in alphabetical order.



If the location folder have two subfolders A and C (or more) and both (or several) have the same.wav file, just the file in A subfolder is copied. My program can NOT detect duplicated files in more than one subfolder.

Antonio

Bobbo Jones
Posts: 48
Joined: 18 Oct 2012 21:27

Re: Search large folders, copy found files, paste to new fol

#41 Post by Bobbo Jones » 29 Oct 2012 17:58

Thanks again for your assistance guys, i will go through these scripts now.

Last night I left 8 instances of the last script i got from Foxidrive running, in a attempt to speed up my process. I need to search a 6 terrabyte drive 30 times to gather all the sound files I need into smaller genre related folders. It is taking about 50 minutes per search at the moment. Which is a lot faster than I could do it manually!!

I will need to do this in the future every 3 months as well.

I attempted to do this by changing the script to look to different list.txt files, and different copy folders as well.
It hasn't worked, for any of them, although 2 seemed to copy a handful of files across.

I do not know the mechanics behind all this, but from other tests a week ago it seems like scripts can run alongside eachother without slowing down much, although I had only tested 2 at a time previously.

I assume the issue was the 'filelist.tmp' part of the script, as all 8 scripts would have been fighting for the same document?

thanks again for any thoughts.
Evan

Bobbo Jones
Posts: 48
Joined: 18 Oct 2012 21:27

Re: Search large folders, copy found files, paste to new fol

#42 Post by Bobbo Jones » 29 Oct 2012 18:34

ah. woops. My test last night with the 8 scripts was useless, my list.txt files were empty. goose!

Bobbo Jones
Posts: 48
Joined: 18 Oct 2012 21:27

Re: Search large folders, copy found files, paste to new fol

#43 Post by Bobbo Jones » 29 Oct 2012 19:44

ok so I just tested this again, and it works if you write different list and copy folder names. The only thing that isn't working, is the notfound.txt
I tried to add a different digit after notfound, for each script running (ie notfound1.txt)

it did not see this, and only made 1 notfound file which did not have a number associated

any ideas whats going on here?
thanks again for all this help. Its massive

foxidrive
Expert
Posts: 6031
Joined: 10 Feb 2012 02:20

Re: Search large folders, copy found files, paste to new fol

#44 Post by foxidrive » 29 Oct 2012 22:29

EDIT2: I altered the way it uses the list?.txt files, and the code again - which might reduce the time taken markedly if the folder tree has hundreds of thousands of files.
Keep the list?.txt filename simple with no spaces and it should be fine.


This should work: it includes the list filename for each invocation, and uses that name in all the filenames and the copy folder.

You will need to change the list file for each run, I presume, so launch the batch file each time with a different list.txt
You can run another batch file to call the searching batch file, with a different list file on each invocation. EG:

@echo off
call batchfilename.bat "list1.txt"
call batchfilename.bat "list2.txt"
call batchfilename.bat "list3.txt"
call batchfilename.bat "list4.txt"
call batchfilename.bat "list5.txt"
call batchfilename.bat "list6.txt"
call batchfilename.bat "list7.txt"
call batchfilename.bat "list8.txt"



Code: Select all

@echo off
set "list=%~1"

setlocal EnableExtensions EnableDelayedExpansion

:rnum
set randnum=%list%
REM if exist filelist.tmp.%randnum% goto :rnum

dir /a:-d /o:n /b /s >filelist.tmp.%randnum%

del list2.tmp.%randnum% 2>nul
del notfound.txt.%randnum% 2>nul
set c=0

for /f "delims=" %%a in (%list%) do echo finding "%%a"& findstr /r /i /c:"\\\%%a$" filelist.tmp.%randnum% >>list2.tmp.%randnum% || >>notfound.txt.%randnum% echo %%a

md "%userprofile%\desktop\copied files.%randnum%" 2>nul

set "name="

for /f "delims=" %%a in ('type "list2.tmp.%randnum%"') do (
   if /i not "%%~nxa"=="!name!" (
      echo copying "%%~nxa"
      copy /b "%%a" "%userprofile%\desktop\copied files.%randnum%\" >nul
   )
set "name=%%~nxa"
)

del filelist.tmp.%randnum% 2>nul
del list2.tmp.%randnum% 2>nul
echo.
if exist notfound.txt.%randnum% echo Check NOTFOUND.TXT.%randnum% for filenames that were missing
echo.
echo done
echo.
pause

foxidrive
Expert
Posts: 6031
Joined: 10 Feb 2012 02:20

Re: Search large folders, copy found files, paste to new fol

#45 Post by foxidrive » 29 Oct 2012 22:49

Note that the code in the message above has changed.

Post Reply