Move from FTP and count in Target folder automated

[aaksar]

hi,

Now its working fine... now i am thinking.. i dont have any control on files.

so rather than checking the file no. , if we make the list of file name.

like :
1st ftp location : we need to get 16 files. so we make another ftp.lst file which contains only the file names.
i think this work you are doing in command2.ftp . in this you are making list of tgt folder files which are already copied.

so by implementing above thing, it doesn't matter whether we have some useless folder or files in FTP location folder. it will pick only the files which are there in .lst.

and counting things will remain same for poling.

so in that case if near future we make another entry in ftp.set file then we need to add 1 more column name.lst file which contains the file names.
and it will be reusable..

thanks
ankur

[abc0502]

So you already know the names of the files that will be downloaded and the folder names that contain this folders ?

[aaksar]

Yup that can be requeated while other team will make entry in ftp.ser file.

[abc0502]

Edited,
Fixed multi FILE.lst
Fixed pause and ftp server problem when connecting
added "create local folder if not exist

Ok, This is the Batch Files:
Start.bat : same as the last one (no changes)
FTP.set : same as the last one (no changes)
Code.bat : changed the For loop in line 62 that put the get command to command_2.ftp
FILE.lstX : this is the file where you put a list of all the files that will be downloaded (new)

Start.bat

Code: Select all

@Echo OFF
Title FTP Main Download Script

:: To prevent errors of existing directory when a previous run of this batch
:: wasn't completed successfully, so create a new one instead
set C1=0
set C2=%RANDOM%
setlocal enabledelayedexpansion
For /F "skip=1 tokens=1-7 delims=;" %%A in ('Type "FTP.set"') Do (

   set /a C1 += 1
   set "file_list=%%A
   set "remote_server=%%B"
   set "remote_server_username=%%C"
   set "remote_server_password=%%D"
   set "total_number=%%E"
   set "local_folder=%%F"
   set "remote_folder=%%G"
   
   IF exist "%temp%\!C2!" (
      Set /a C2 += 51
      MD "%temp%\!C2!" >nul
      ) Else ( MD "%temp%\!C2!" >nul )
   
   start "%%B" cmd.exe /c Code.bat "!file_list!" "!remote_server!" "!remote_server_username!" "!remote_server_password!" "!total_number!" "!local_folder!" "!remote_folder!" "!C2!"
   )

Code.bat

Code: Select all

@Echo OFF
:: Don't Forget to Specify the Zipped file type in line 18

::"file_list" --------------------> %~1
::"remote_server" ----------------> %~2
::"remote_server_username" -------> %~3
::"remote_server_password" -------> %~4
::"total_number" -----------------> %~5
::"local_folder" -----------------> %~6
::"remote_folder"  ---------------> %~7
::"FTP servers temp folder" ------> %~8

::Create Local Folder if not exist
IF not Exist "%~6" MD "%~6" >nul

:loop
set "zip_program=%programfiles%\7-Zip\7z.exe"
set "zipped=7z"

:: FTP commands to get list of files on the server
IF "%~7"=="" (
   (
   Echo open %~2
   Echo %~3
   Echo %~4
   Echo ls
   Echo disconnect
   Echo Bye
   )>"%temp%\%~8\commands_1.ftp"
) Else (
   (
   Echo open %~2
   Echo %~3
   Echo %~4
   Echo cd "%~7"
   Echo ls
   Echo disconnect
   Echo Bye
   )>"%temp%\%~8\commands_1.ftp"
   )

:: start FTP session and log output
Ftp -v -i -s:"%temp%\%~8\commands_1.ftp" >"%temp%\%~8\dump.log"

:: extract file name from the ftp dump file
For /F "delims=" %%a in ('Type "%temp%\%~8\dump.log"') Do Echo "%%a" |findstr /I /V /R ^ftp^> |findstr /I /V /R ^bye >>"%temp%\%~8\server_list.lst"

:: 2nd FTP commands to download only new files that dosn't exist locally [Part1]
IF "%~7"=="" (
   (
   Echo open %~2
   Echo %~3
   Echo %~4
   Echo lcd "%~6"
   )>"%temp%\%~8\commands_2.ftp"
) Else (
   (
   Echo open %~2
   Echo %~3
   Echo %~4
   Echo lcd "%~6"
   Echo cd "%~7"
   )>"%temp%\%~8\commands_2.ftp"
   )

:: compare files that will be downloaded with the file.list and if there is a match
:: it see if this match isn't in the local folder so it can download that file. [Part2]
For /F "delims=" %%a in ('Type "%~1"') Do (
   For /F "delims=" %%b in (%temp%\%~8\server_list.lst) Do (
   Set "sf=%%b"
   setlocal enabledelayedexpansion
      IF "%%a" EQU "!sf:~1,-2!" (
         IF NOT EXIST "%~6\%%b" echo get %%b >>"%temp%\%~8\commands_2.ftp"
      )
   endlocal
   )
)

:: Add disconnect & bye commands to 2nd FTP command file [Part3]
(
Echo disconnect
Echo Bye
)>>"%temp%\%~8\commands_2.ftp"

:: start FTP session and download new files
Ftp -v -i -s:"%temp%\%~8\commands_2.ftp"

:: Check files count
Setlocal EnableDelayedExpansion
set Fcount=0
For /F "delims=" %%a in ('Dir /B "%~6"') Do set /a Fcount += 1

:: Check Condition to Exit or Continue Looping
:: GEQ means equal to total file number or greater in case there was new files added
:: and user didn't update the FTP.set file.
:: but if the files was become less and user didn't update the file it will keep looping
:: forever and will not extract files from zip archives.
Setlocal EnableDelayedExpansion
IF "%Fcount%" EQU "%~5" (
   "%zip_program%" x -y "%~6\*.%zipped%" -o"%~6" >nul
   Del /F /Q "%~6\*.%zipped%" >nul
   RMDIR /S /Q "%temp%\%~8" >nul
   Goto :EOF
) Else (
   :: Wait 5 min
   Ping Localhost -l 1 -n 300 >nul
   Goto :loop
   )

FTP.set

FILE_LIST;FTP_SERVER;USER_NAME;PASSWORD;MAX_NUMBER_TO_DOWNLOAD;LOCAL_FOLDER;REMOTE_FOLDER
FILE.lst1;ftp.domain1.com;username;password;3;C:\documents and settings\admin\desktop\1234;
FILE.lst2;ftp.domain2.com;username;password;4;C:\documents and settings\admin\desktop\FTP Project;

FILE.lst

File1.txt
File2.scr
File3

>In FTP.set, If the files in the parent directory on the ftp server leave the REMOTE_FOLDER section empty don't leave space in it.
>All 4 Files MUST be in SAME folder.
>The first token "FILE.lst1" is for the first ftp server, and for every extra ftp server you change the number from 1 to 2 to 3 ..etc & each one of these file will contain the names of files belong to it's ftp server

Description:
When you start the Start.bat file it will take inputs from FTP.set file and pass it to the Code.bat file,
Which will then create a command_1.ftp file that will be used to connect to the FTP server and get a list of all
files and folders in the REMOTE_FOLDER "if specified" and put this list in Server_list.lst.
Then, compare list of files in the FILE.lst1 with the file & folder names in the Server_list.lst,
If there is a match it check to see if this match already downloaded or not, if downloaded it doesn't download it again.
If not it will put it's name in the command_2.ftp file to be downloaded.
After Compare finish, will connect again to FTP server and download the files to the LOCAL_FOLDER
Then, count the files in that folder,
If they reached the total number of the files that should be downloaded it will remove temporary files, unzip the zipped files and remove the zipped file after, then exit,
If not reached the count it will wait 5 min (300 sec) then do the same again till it complete the amount of the files.

This description will be applied on multi-FTP servers too.

Don't Forget to change The zipped File Tye in line 18 to match your's

[aaksar]

Thanks Buddy, will test tmrw and update you.

[aaksar]

tell me 1 more thing,

IF in ftp.set file we have 4 ftp location, so it will take every location sequentially or it will hit all the 4 ftp location parallely.

it should go parallel otherwise it would be problem like if 1 ftp is poling for files so it wont come to 2nd ftp loc until 1st ftp completed.
what do u say

[abc0502]

This version is exactly like the previous one, it will process all 4 locations at once ( each ftp server in it's separate cmd window).
The only difference here is that it compare list of files with what already in the ftp server and if it match it will get that files, so in this case it won't try to download any unnessacery files or a folders

[aaksar]

file.lst is a comman file?

it should be individual for every FTP location, otherwise if all the file name in 1 file and using 4 ftp location, it wont problem?

i think each ftp location should have 1 file.lst file which contain all the file names which needs to be fetched from that FTP location.

and file.lst file should mention in ftp.set file with another column, so that it pick the file from list and than compare.

[aaksar]

Its not moving files. and when i run script it ask" press any key to continue"

[abc0502]

file.lst is a comman file?

it should be individual for every FTP location, otherwise if all the file name in 1 file and using 4 ftp location, it wont problem?

i think each ftp location should have 1 file.lst file which contain all the file names which needs to be fetched from that FTP location.

and file.lst file should mention in ftp.set file with another column, so that it pick the file from list and than compare.

you are write, i only was testing on 1 ftp so didn't notice.
Edited the Codes above, note all changes in RED and changed start.bat, Code.bat and FTP.set

[aaksar]

you are not giving File.lst in ftp.set file.
if its there then it will pick the file from that row.
seems like you are doing dynamically file1--fileX.

[aaksar]

ignore my previous post.

let me test.

[abc0502]

don't forget change 7z to zip in the code.bat line 14
and i made other changes take a nother copy of it again

[aaksar]

seems like problem.
command1.ftp is use to login FTP

Code: Select all

::"file_list" --------------------> %~1
::"remote_server" ----------------> %~2
::"remote_server_username" -------> %~3
::"remote_server_password" -------> %~4
::"total_number" -----------------> %~5
::"local_folder" -----------------> %~6
::"remote_folder"  ---------------> %~7
::"FTP servers temp folder" ------> %~8

:loop
set "zip_program=%programfiles%\7-Zip\7z.exe"
set "zipped=7z"

:: FTP commands to get list of files on the server
IF "%~7"=="" (
   (
   Echo open %~1
   Echo %~3
   Echo %~4
   Echo ls
   Echo disconnect
   Echo Bye
   )>"%temp%\%~8\commands_1.ftp"

but you are opening %~1 which is file list.

and when i am running the script its opening 2 command window. and says"Press any key to continue."

[abc0502]

yes i just noticed that and working on it