Move from FTP and count in Target folder automated

[aaksar]

i need to pole the FTP location , and move the file to my Informatica location, unzip the file (8 files coming as zip rest of files are unzipped).
every 5 mins it will wakeup and check the file, everytime check in my informatica dir whether it reached the count of files like (assume i need to receive 15 files) so if its 15 then create a 0 byte file and stop polling.

and after putting remove special char from the files.

Thanks in advance

[aaksar]

and can it be make dynamic , like take FTP loc, username, pswd, filename, and count of file, from some other table or file, and put the file from FTP loc to my folder and unzip the file. so in future if i need to add another FTP location for other files i make an entry in file/table and it will read it from there.

Thanks In advance

[abc0502]

I don't understand much in ftp but i wanted to do something similar to that so i guess i can help.

Q:
from what you want to pull the ftp location?
what is the Informatica location ?
When it check the files , where is these files, in the ftp or in the "Informatica location" ?

can you describe more

EDITED:
This is The Final Batch That is Working [ For downloading from multi-FTP servers at same time ]
For downloading specific files types from the FTP server have a look at this : link
Start.bat

Code: Select all

@Echo OFF
Title FTP Main Download Script

:: IF The batch dosen't downloading your files make sure to provide full location
:: of these files in "FTP.set" and "Code.bat" at this Batchfile "Start.bat"
:: also provide the full location of the "FILE.lstX" inside "FTP.set file

set C1=0
set C2=%RANDOM%
setlocal enabledelayedexpansion
For /F "skip=1 tokens=1-7 delims=;" %%A in ('Type "FTP.set"') Do (

   set /a C1 += 1
   set "file_list=%%A
   set "remote_server=%%B"
   set "remote_server_username=%%C"
   set "remote_server_password=%%D"
   set "total_number=%%E"
   set "local_folder=%%F"
   set "remote_folder=%%G"
   
   IF exist "%temp%\!C2!" (
      Set /a C2 += 51
      MD "%temp%\!C2!" >nul
      ) Else ( MD "%temp%\!C2!" >nul )
   
   start "%%B" cmd.exe /c Code.bat "!file_list!" "!remote_server!" "!remote_server_username!" "!remote_server_password!" "!total_number!" "!local_folder!" "!remote_folder!" "!C2!"
   )

> If you have any errors when downloading files or can't download files make sure you provide the full location for FTP.set and Code.bat in the Start.bat batch file, also the full location of the FILE.lst inside the FTP.set file.

Code.bat

Code: Select all

@ECHO OFF

:: Don't Forget to Specify the Zipped file type in line 20

::"file_list" --------------------> %~1
::"remote_server" ----------------> %~2
::"remote_server_username" -------> %~3
::"remote_server_password" -------> %~4
::"total_number" -----------------> %~5
::"local_folder" -----------------> %~6
::"remote_folder"  ---------------> %~7
::"FTP servers temp folder" ------> %~8

::Create Local Folder if not exist
::Echo Creating LOCAL FOLDER
IF not Exist "%~6" MD "%~6" >nul

:loop
set "zip_program=%programfiles%\7-Zip\7z.exe"
set "zipped=zip"

:: FTP commands to get list of files from the server
::Echo Creating Commands_1.ftp file
IF "%~7"=="" (
   (
   Echo open %~2
   Echo %~3
   Echo %~4
   Echo ls
   Echo disconnect
   Echo Bye
   )>"%temp%\%~8\commands_1.ftp"
) Else (
   (
   Echo open %~2
   Echo %~3
   Echo %~4
   Echo cd "%~7"
   Echo ls
   Echo disconnect
   Echo Bye
   )>"%temp%\%~8\commands_1.ftp"
   )

:: start FTP session and log output
::Echo Connicting to FTP server to get files list
Ftp -v -i -s:"%temp%\%~8\commands_1.ftp" >"%temp%\%~8\dump.log"

:: extract file name from the ftp dump file
::Echo Processing dump.log File and creating server_list.lst
For /F "delims=" %%a in ('Type "%temp%\%~8\dump.log"') Do (
   Echo "%%a"|findstr /I /V /R ^ftp^>|findstr /I /V /R ^bye>>"%temp%\%~8\server_list.lst"
   )

:: 2nd FTP commands to download only new files that dosn't exist locally [Part1]
::Echo Creating commands_2.ftp File (part1)
IF "%~7"=="" (
   (
   Echo open %~2
   Echo %~3
   Echo %~4
   Echo lcd "%~6"
   )>>"%temp%\%~8\commands_2.ftp"
) Else (
   (
   Echo open %~2
   Echo %~3
   Echo %~4
   Echo lcd "%~6"
   Echo cd "%~7"
   )>>"%temp%\%~8\commands_2.ftp"
   )

:: compare files that will be downloaded with the file.list and if there is a match
:: it see if this match isn't in the local folder so it can download that file. [Part2]
::Echo Creating commands_2.ftp File (part2)
For /F "delims=" %%a in ('Type "%~1"') Do (
   For /F "delims=" %%b in ('Type "%temp%\%~8\server_list.lst"') Do (
   Set "sf=%%b"
   setlocal enabledelayedexpansion
      IF "%%a"=="!sf:~1,-1!" (
         IF NOT EXIST "%~6\%%b" echo get %%b>>"%temp%\%~8\commands_2.ftp"
      )
   endlocal
   )
)

:: Add disconnect & bye commands to 2nd FTP command file [Part3]
::Echo Creating commands_2.ftp File (part3)
(
Echo disconnect
Echo Bye
)>>"%temp%\%~8\commands_2.ftp"

:: start FTP session and download new files
::Echo Connecting to FTP server to download files
Ftp -v -i -s:"%temp%\%~8\commands_2.ftp"

:: Check files count
::Echo Counting files in local folder
set Fcount=0
For /F "delims=" %%a in ('Dir /B "%~6\*.*"') Do set /a Fcount += 1

:: Check Condition to Exit or Continue Looping
::Echo Check count number
Setlocal EnableDelayedExpansion
IF "%Fcount%" EQU "%~5" (
   "%zip_program%" x -y "%~6\*.%zipped%" -o"%~6" >nul
   Del /F /Q "%~6\*.%zipped%" >nul
   RMDIR /S /Q "%temp%\%~8" >nul
   Goto :EOF
) Else (
   Del /F /Q "%temp%\%~8\server_list.lst" >nul
   :: Wait 5 min
   Ping Localhost -l 1 -n 300 >nul
   Goto :loop )

FTP.set

FILE_LIST;FTP_SERVER;USER_NAME;PASSWORD;MAX_NUMBER_TO_DOWNLOAD;LOCAL_FOLDER;REMOTE_FOLDER
file.lst1;ftp.domain1.com;USERNAME;PASSWORD;3;%userprofile%\desktop\1234;test_folder
file.lst2;ftp.domain2.com;USERNAME;PASSWORD;3;%userprofile%\desktop\new2;test_folder_2

> if you don't have a remote folder leave it empty and don't leave any spaces instead.
> if you can't download files from FTP server, make sure to provide the full location of the FILE_LIST file inside the FTP.set file.

FILE.lst

file_name_1.txt
file 2.zip
file3
test.png

> This File should contain all the files names that will be downloaded from FTP server.
Any other information, read the topic

[aaksar]

I don't understand much in ftp but i wanted to do something similar to that so i guess i can help.

Q:
from what you want to pull the ftp location?
what is the Informatica location ?
When it check the files , where is these files, in the ftp or in the "Informatica location" ?

can you describe more

1: I need to copy files from FTP location, we r pulling file thro BAtch script
2: informatica location dont matter as we r giving path,
2: it checks the file in informatica loc evrytime, whether reach the desired count or not, if not then again pole the ftp location after 5 mins.

[abc0502]

So, you want to :
1- input ftp location , username and password
2- the batch log in and download all file in that ftp location locally and unzip them then delete the zip files.
3- then count the files to see if they reached the total allowed files, if not keep looping and downloading files every 5 min till get them all, else exit.

Is that right ?

[aaksar]

Currently we have only 1ftp location.
But in future it will be 7-8, so i dont want to hard code.
I want to make it reusable for everybody, like it pick the ftp loc, username, pswd, tgt loc, and file coming frequency.

So it check in the table/ascii file, pick ftp loc, user name, pswd and other field,
And copy the file to desire location.
So any time if i need to add another ftp loc, so i just need to make an entry in file, it will pick all the things from it.

Thanks buddy, i m in deep trouble.

[abc0502]

will the content of the zipped files has the same name of the zip file it self ?
I need to know that so i can compare the already downloaded files and the files on the ftp location.

and if it is OK to unzip the files at the end, after the whole files reached the specified number that will be easier.

[abc0502]

For Now, This work for one FTP location and the settings is set inside the batch.
It can be easily modified to take input from user or a file that contain the inputs.
But for now till i finish and make it easy for me to test i'm using the setting inside the Batch.

Code: Select all

@Echo Off
Title FTP Downloader
Mode 60,15

:loop
:: variables
set "local_folder=%userprofile%\desktop\1234"
set "zip_program=%programfiles%\7-Zip\7z.exe"
set "remote_folder="
set "remote_server=ftp.domain.com"
set "remote_server_username=user"
set "remote_server_password=pass"
set "remote_file_types=.7z"
set "total_number=3"

:: FTP commands to get list of files on the server
(
Echo open %remote_server%
Echo %remote_server_username%
Echo %remote_server_password%
Echo ls
Echo disconnect
Echo Bye
)>commands_1.ftp

:: start FTP session and log output
Ftp -v -i -s:"commands_1.ftp">dump.log

:: extract file name from the ftp dump file based on files type
For /F "delims=" %%a in ('Type "dump.log"') Do Echo "%%a" |find "%remote_file_types%" >>server_list.lst

:: 2nd FTP commands to download only new files that dosn't exist locally
(
Echo open %remote_server%
Echo %remote_server_username%
Echo %remote_server_password%
Echo lcd "%local_folder%"
)>commands_2.ftp

:: get the all file names that dosn't exist in the local folder and put then in ftp command file 2 to be downloaded
For /F "tokens=* delims=" %%a in ('Type "server_list.lst"') Do (
   IF not exist "%local_folder%\%%a" echo get %%a >>commands_2.ftp
   )

:: Add disconnect & bye commands to 2nd FTP command file
(
Echo disconnect
Echo Bye
)>>commands_2.ftp

:: start FTP session and download new files
Ftp -v -i -s:"commands_2.ftp"

:: Delete lst, log, ftp files
Del /F /Q "*.ftp" "*.lst" "*.log" >nul

:: Check files count
Setlocal EnableDelayedExpansion
set Fcount=0
For /F "delims=" %%a in ('Dir /B "%local_folder%"') Do set /a Fcount += 1

:: Check Condition to Exit or Continue Looping
IF "%Fcount%"=="%total_number%" (
   "%zip_program%" x -y "%local_folder%\*%remote_file_types%" -o"%local_folder%" >nul
   Del /F /Q "%local_folder%\*%remote_file_types%" >nul
   Goto :EOF
) Else (
   :: Wait 5 min
   Ping Localhost -l 1 -n 300 >nul
   Goto :loop )

However, when taking more than one FTP server the problem will happen when there is still missing files, it will keep looping on that server and the rest of the server will not be processed till that on complete it's files.
If it is ok to run a separate cmd window for all FTP servers that is missing files that will make things a bit easier but will make your desktop full of cmd windows

[Squashman]

Thanks buddy, i m in deep trouble.

Why and How?

[abc0502]

This is for downloading specific file types
Edited:
Added double qoutes arround the %~7 and %~6 in Code.bat

OK, here is the batch file(S):
It consist of two batch files, one has the download code like the above & the other is the one that will extract the ftp server and username and password ..etc
and one txt file that hold all info about the ftp servers

Start.bat

Code: Select all

@Echo OFF
Title FTP Main Download Script

setlocal enabledelayedexpansion
set C=0
For /F "skip=1 tokens=1-7 delims=;" %%A in ('Type "FTP_servers.txt"') Do (
   set /a c +=1
   MD !c!
   set "remote_server=%%A"
   set "remote_server_username=%%B"
   set "remote_server_password=%%C"
   set "remote_file_types=%%D"
   set "total_number=%%E"
   set "local_folder=%%F"
   set "remote_folder=%%G"
   start /MIN "%%A" cmd.exe /c Code.bat "!remote_server!" "!remote_server_username!" "!remote_server_password!" "!remote_file_types!" "!total_number!" "!local_folder!" "!remote_folder!" "!c!"
   )

Code.bat

Code: Select all

@Echo OFF

::"remote_server" ----------------> %~1
::"remote_server_username" -------> %~2
::"remote_server_password" -------> %~3
::"remote_file_types" ------------> %~4
::"total_number" -----------------> %~5
::"local_folder" -----------------> %~6
::"remote_folder" ----------------> %~7
::"Forlder for each FTP server" --> %~8

:loop
set "zip_program=%programfiles%\7-Zip\7z.exe"

:: FTP commands to get list of files on the server
IF "%~7"=="" (
   (
   Echo open %~1
   Echo %~2
   Echo %~3
   Echo ls
   Echo disconnect
   Echo Bye
   )>"%~8\commands_1.ftp"
) Else (
   (
   Echo open %~1
   Echo %~2
   Echo %~3
   Echo cd "%~7"
   Echo ls
   Echo disconnect
   Echo Bye
   )>"%~8\commands_1.ftp"
   )
   
:: start FTP session and log output
Ftp -v -i -s:"%~8\commands_1.ftp" >"%~8\dump.log"

:: extract file name from the ftp dump file
For /F "delims=" %%a in ('Type "%~8\dump.log"') Do Echo "%%a" |find "%~4" >>"%~8\server_list.lst"

:: 2nd FTP commands to download only new files that dosn't exist locally
IF "%~7"=="" (
   (
   Echo open %~1
   Echo %~2
   Echo %~3
   Echo lcd "%~6"
   )>"%~8\commands_2.ftp"
) Else (
   (
   Echo open %~1
   Echo %~2
   Echo %~3
   Echo lcd "%~6"
   Echo cd "%~7"
   )>"%~8\commands_2.ftp"
   )

:: get the all file names that dosn't exist in the local folder and put then in ftp command file 2 to be downloaded
For /F "tokens=* delims=" %%a in ('Type "%~8\server_list.lst"') Do (
   IF not exist "%~6\%%a" echo get %%a >>"%~8\commands_2.ftp"
   )

:: Add disconnect & bye commands to 2nd FTP command file
(
Echo disconnect
Echo Bye
)>>"%~8\commands_2.ftp"

:: start FTP session and download new files
Ftp -v -i -s:"%~8\commands_2.ftp"

:: Check files count
Setlocal EnableDelayedExpansion
set Fcount=0
For /F "delims=" %%a in ('Dir /B "%~6"') Do set /a Fcount += 1

:: Check Condition to Exit or Continue Looping
IF "%Fcount%"=="%~5" (
   "%zip_program%" x -y "%~6\*%~4" -o"%~6" >nul
   Del /F /Q "%~6\*%~4" >nul
   Del /F /Q "%~8\*.ftp" "%~8\*.lst" "%~8\*.log" >nul
   RMDIR /S /Q "%~8" >nul
   Goto :EOF
) Else (
   :: Wait 5 min
   Ping Localhost -l 1 -n 300 >nul
   Goto :loop )
   )

FTP_servers.txt

FTP_SERVER;USER_NAME;PASSWORD;FILE_TYPE;TOTAL_NUMBER_OF_FILES;LOCAL_FOLDER;REMOTE_FOLDER
ftp.domain1.com;username;password;.7z;3;C:\documents and settings\admin\desktop\1234;test-folder
ftp.domain2.com;username;password;.7z;5;C:\documents and settings\admin\desktop\FTP Project

The FTP_server.txt is like csv files, and the first line explain the order of the inputs seprated by ";"

Notes
>> To start the download you will run the "Start.bat" and every ftp server will run in a separate minimized cmd window.
>> To make it run normally remove "/MIN" from the "Start.bat".
>> All 3 files "start.bat", "code.bat" & "FTP_servers.txt" must be in the same folder.
>> You can't connect to one FTP server using multi-connections in the same time.
>> This code

Code: Select all

   Ping Localhost -l 1 -n 300 >nul

is to make the batch file wait 300 sec "5 min" between every try till it reach the total number of files that should be in the local folder.

Edit:
Forgot to say, i'm using 7z to unzip the files and the file type i used is .7z

[aaksar]

when i am running the script, i am getting error" specified file can not found"

i have checked all the files in same folder, seems like its not going to code.bat, coz if it will go to code.bat then, i should get error, as i dont have permission to create any file in FTP location, and in the script its creating couple of files.

seems like its not finding the .txt file.

thanks in advance

[aaksar]

in the .txt file you are mentioning the file type, but dont need to mention the type, as some files are coming without extension and some are .zip.

so the count of files represent the count of all the files( without ext + zip)
and on that count it will check whether all the files are there or not.

Command1 and command 2.ftp should be create in the dir where scripts are, as i dont hv permission to create TP loc

thanks

[abc0502]

When you said unzip i supposedit will be all zipped files,
any way these files "zip + other files" are in one folder and there is no other folder with them?
because you can't download folders.

and about the missing file, i was having this problem but i fixed it, it was a missing ")".
I will check again

i dont hv permission to create TP loc

Do you mean you cant's create temporary folder in the same location of the batch?
if so, then can you create folders in the temp directory "%temp%"

[aaksar]

File not found error gone, i have made some changes in start.bat

Login failed.
ftp> open ftp.us.abcdfg.com
User (ftp.us.abcdfg.com:(none)):

User FtpTemp0013 cannot log in.
Login failed.
ftp> lcd "C:\folder\"
\folder\: File not found
ftp> disconnect
ftp> Bye

its not taking password, might be my password starting with special char
pswd is like "^abcde23".

and giving another error

A subdirectory or file 1 already exists.

@Echo OFF
Title FTP Main Download Script

setlocal enabledelayedexpansion
set C=0
For /F "skip=1 tokens=1-7 delims=;" %%A in ('Type "C:\Folder\abc.txt"') Do (
set /a c +=1
MD !c!
set "remote_server=%%A"
set "remote_server_username=%%B"
set "remote_server_password=%%C"
set "remote_file_types=%%D"
set "total_number=%%E"
set "local_folder=%%F"
set "remote_folder=%%G"
start "%%A" C:\Folder\Code.bat "!remote_server!" "!remote_server_username!" "!remote_server_password!" "!remote_file_types!" "!total_number!" "!local_folder!" "!remote_folder!" "!c!"
)

i have removed the minimize window coz i wanted to see what is going on, after that i received the above error

[abc0502]

Forget the above code cause it work for specific file types, i'm now changing the code to work generally for any file type.

but answer this quesion:
> Is There folders in the same location on the ftp server with the files that will be downloaded ?
> Do you have permission to create folders in the temp directory "%temp%" or "C:\DOCUME~1\admin\LOCALS~1\Temp\" ?
> Is the zipped files always have the .zip extension ?

Edited:
and about your errors:
> the special character in the password:
Only these special characters will need to be escaped so the batch can read them:
% sign : should add another % before it ---> %abc0502 become %%abc0502
^ sign : should add another ^ before it -----> ^abc0502 become ^^abc0502
& sign : should add ^ before it -------------> &abc0502 become ^&abc0502
whether these characters was at the beginning, at the middle or at the end, do the same as instructed above.

> A subdirectory or file 1 already exists.
it happen becasuse the batch should remove the temp folder after it complete, and it semes that the batch didn't the last time so it didn't remove it, i will do a fix for that in the new batch.

I'm almost done here.