Search large folders, copy found files, paste to new folder?

[Aacini]

@Bobbo Jones: Did you tested my program again? Remember that you must delete .wav extensions from your name list or from the program, but not both!

I am pretty sure that my solution will fulfill your timing requirements...

Antonio

[foxidrive]

You will need to change the list file for each run, I presume, so launch the batch file each time with a different list.txt
You can run another batch file to call the searching batch file, with a different list file on each invocation. EG:

@echo off
call batchfilename.bat "list1.txt"
call batchfilename.bat "list2.txt"
call batchfilename.bat "list3.txt"
call batchfilename.bat "list4.txt"
call batchfilename.bat "list5.txt"
call batchfilename.bat "list6.txt"
call batchfilename.bat "list7.txt"
call batchfilename.bat "list8.txt"

viewtopic.php?p=21378#p21378

Did that work??

[Bobbo Jones]

Sorry i wont be able to get back to my windows machine until monday.

Foxidrive, will your update give different not founds per list? and do they use the same tmp files for the search?

aacini I think I have, I will need to check again on monday, sorry i haven't reported in on it yet.

Thanks!!

[foxidrive]

Foxidrive, will your update give different not founds per list? and do they use the same tmp files for the search?

Yes, different notfounds per list.

All files and folder use the searchfilename1.txt as part of the names, so they are as unique as the search list filename you are feeding it.

[Bobbo Jones]

Hi guys, thanks again,

I have tried to test these but I think I am getting confused, I having success with the tools, but am struggling to apply the recent changes you added foxidrive.

ie this

You will need to change the list file for each run, I presume, so launch the batch file each time with a different list.txt
You can run another batch file to call the searching batch file, with a different list file on each invocation. EG:

@echo off
call batchfilename.bat "list1.txt"
call batchfilename.bat "list2.txt"
call batchfilename.bat "list3.txt"
call batchfilename.bat "list4.txt"
call batchfilename.bat "list5.txt"
call batchfilename.bat "list6.txt"
call batchfilename.bat "list7.txt"
call batchfilename.bat "list8.txt"

I will try again when I am less tired asap

[foxidrive]

It occurs to me that the call keyword would make it sequential and they would run one after the other.

If you want to run them at the same time then try this.

Code: Select all

@echo off
start "" batchfilename.bat "list1.txt"
start "" batchfilename.bat "list2.txt"
start "" batchfilename.bat "list3.txt"
start "" batchfilename.bat "list4.txt"
start "" batchfilename.bat "list5.txt"
start "" batchfilename.bat "list6.txt"
start "" batchfilename.bat "list7.txt"
start "" batchfilename.bat "list8.txt"

[Bobbo Jones]

Hey Foxi, going back to single searches at one time, I have 1 tool which does not rename the files, and ignores dupes. Its working great.
I also have one which DOES rename the files, but does NOT ignore the dupes. I need this to also ignore the dupes. originally we kept the dupes in, so I can listen to them and ensure they are the correct song. But after some use I see that there are too many dupes for that to be practical.

How would I change its code to do this? here is what I have.

Code: Select all

@echo off
set "list=list.txt"

setlocal EnableExtensions EnableDelayedExpansion

dir /a:-d /o:n /b /s >filelist.tmp

del list2.tmp 2>nul
del notfound.txt 2>nul
del samefile.txt 2>nul
set c=0

for /f "delims=" %%a in ('type "%list%"') do echo finding "%%a"& findstr /r /i /c:"\\\%%a$" filelist.tmp >>list2.tmp || >>notfound.txt echo %%a

md "%userprofile%\desktop\copied files" 2>nul

set "name="

for /f "delims=" %%a in ('type "list2.tmp"') do (
if /i "%%~nxa"=="!name!" >>"samefile.txt" echo Same filename - "!name!"
set /a c=c+1
set num=000!c!
set num=!num:~-3!
echo copying "!num! %%~nxa"
copy /b "%%a" "%userprofile%\desktop\copied files\!num! %%~nxa" >nul
set "name=%%~nxa"
)

del filelist.tmp 2>nul
if not exist "samefile.txt" del list2.tmp 2>nul
echo.
if exist notfound.txt echo Check NOTFOUND.TXT for filenames that were missing
if exist "samefile.txt" (
ren "list2.tmp" "samefile.list.txt"
echo Check "samefile.txt" and "samefile.list.txt" for the list of repeated files
)
echo.
echo done
echo.
pause

[Bobbo Jones]

secondly with the multiple bat files running at once idea, you indicated that the code was now updated, are these updates necessary for my 2 tools I mentioned above?

The search tool with rename will NOT need to run multiple instances. The search tool without rename will regularly be used with multiple instance.

thanks for your time mate!

[foxidrive]

secondly with the multiple bat files running at once idea, you indicated that the code was now updated, are these updates necessary for my 2 tools I mentioned above?

No, the only change was to allow more than one instance of the batch file at the same time.

You're lucky I can recall how it works - it's been a long while!

This should ignore the dupes.

Code: Select all

@echo off
set "list=list.txt"

setlocal EnableExtensions EnableDelayedExpansion

dir /a:-d /o:n /b /s >filelist.tmp

del list2.tmp 2>nul
del notfound.txt 2>nul
del samefile.txt 2>nul
set c=0

for /f "delims=" %%a in ('type "%list%"') do echo finding "%%a"& findstr /r /i /c:"\\\%%a$" filelist.tmp >>list2.tmp || >>notfound.txt echo %%a

md "%userprofile%\desktop\copied files" 2>nul

set "name="

for /f "delims=" %%a in ('type "list2.tmp"') do (
if /i not "%%~nxa"=="!name!" (
set /a c=c+1
set num=000!c!
set num=!num:~-3!
echo copying "!num! %%~nxa"
copy /b "%%a" "%userprofile%\desktop\copied files\!num! %%~nxa" >nul
)
set "name=%%~nxa"
)

del filelist.tmp 2>nul
if not exist "samefile.txt" del list2.tmp 2>nul
echo.
if exist notfound.txt echo Check NOTFOUND.TXT for filenames that were missing
if exist "samefile.txt" (
ren "list2.tmp" "samefile.list.txt"
echo Check "samefile.txt" and "samefile.list.txt" for the list of repeated files
)
echo.
echo done
echo.
pause

[Bobbo Jones]

Thanks foxi, that multiple search is brilliant.
One odd thing has occurred, the copied files folders don't appear on my desktop. But when go follow the path from my start bar to the desktop folder, they are there. I try and drag them onto the desktop, and I get a message saying this is already where they are.. there is no visual indication of their existence, yet they exist! wierd!

[Bobbo Jones]

After restarting it appears...

[foxidrive]

Glad it helps. Pressing F5 should refresh the desktop when it is in focus, too.

[Bobbo Jones]

Well i just discovered an issue with the multiple search, I attempted to have 10 searches going at once in a single folder with 4500 tracks in it, but of course only one file can be copied at a time from any given folder, so it essentially stopped the batch files in their tracks. I realise I can only expect the multiple search to work when I am searching a root directory, with many subfolders under it.

Another strange thing, my computer is now unable to perform single searches in the background, it is only able to do the one thing... odd behaviour!

[foxidrive]

Well i just discovered an issue with the multiple search, I attempted to have 10 searches going at once in a single folder with 4500 tracks in it, but of course only one file can be copied at a time from any given folder, so it essentially stopped the batch files in their tracks. I realise I can only expect the multiple search to work when I am searching a root directory, with many subfolders under it.

A hard drive and computer can have multiple concurrent threads open but doing a DIR of the same folders at the same time could slow it appreciably.
Similarly, copying 10 files at the same time will slow all the transfers markedly.

You can copy more than one file from the same folder at the same time though - just a lot slower.

Another strange thing, my computer is now unable to perform single searches in the background, it is only able to do the one thing... odd behaviour!

"single"? "one thing"?

I'm not sure I follow your comment.

[Aacini]

Hi Bobbo Jones, I wonder if you would like to test the last version of my solution. Here it is:

Code: Select all

@echo off
setlocal EnableDelayedExpansion

set list=list.txt
set blockSize=20


rem If foldersList.txt file not exist: create it
if not exist foldersList.txt (
   echo FoldersList.txt file not found, creating it...
   (for /R /D %%a in (*) do (
      set folder=%%a
      echo !folder:%cd%\=!
   )) > foldersList.txt
)

rem Load folders List
echo Loading folders list...
set folders=
for /F "delims=" %%a in (foldersList.txt) do set folders=!folders!%%a;
echo/

md "%userprofile%\desktop\copied files" 2>NUL
del notFound.txt 2>NUL

rem Process the list of file names in blocks of blockSize
set n=0
set num=
set block=
set blockN=0
for /F "usebackq delims=" %%a in ("%list%") do (
   set block=!block! "%%a"
   set /A blockN+=1
   if !blockN! equ %blockSize% call :ProcessBlock %1
)
if defined block call :ProcessBlock %1
echo/
if exist notFound.txt echo Check notFound.txt for filenames that were missing
echo/
pause
goto :EOF


:ProcessBlock [NOT]
for %%b in (!block!) do (
   set /A n+=1
   if /I "%~1" neq "NOT" (
      set num=000!n!
      set "num=!num:~-3! "
   )
   set "filePath=%%~$folders:b"
   if defined filePath (
      echo Copying:   !n! - %%~b
      copy "!filePath!" "%userprofile%\desktop\copied files\!num!%%~b" >NUL
   ) else (
      echo Not found: !n! - %%~b
      echo %%~b >> notFound.txt
      rem Create an empty file in the destination folder
      cd . > "%userprofile%\desktop\copied files\!num!%%~b  ---  NOT FOUND"
   )
)
set block=
set blockN=0
exit /B

Previous program:

- Always ignore duplicated files.
- Allows to cancel the rename of files if word NOT is included as parameter. For example:

Code: Select all

rem Execute the Batch file renaming files:
batchfile

rem Execute the Batch file preserving file names:
batchfile NOT

The first time the Batch file run it creates a file called foldersList.txt with the names of all existent sub-folders beneath current folder; this mean that from the second time on the program will run faster. If anyone of the sub-folders change (NOT the .wav files!), just delete this file so the program create it again.

Next paragraphs below refers to details that may speed up the program a little. You may ignore they the first time if you wish.

When .wav files are searched, sub-folders in foldersList.txt file are processed in that order. You may speed up the program if you place at beginning the sub-folders that contain the most frequently copied files, and the folders with less copied files at end. Be careful to not modify this file in any other way; in particular, don't insert any space!

Files in the list are processed in blocks, not one by one. The number of files in each block affect the speed of the program, but there is no way to predict in which way. You may do several timing tests changing the value of blockSize variable and choose the best one for you; the current value is 20.

Antonio

PS - I'd appreciate it if you could let me know what happened with this reply. If you just want not use it for any reason, just said it; there is not any problem here!