Page 1 of 1
FTP pull data files without overwrite
Posted: 02 Dec 2008 19:49
by CheeseFox
Hi ppl,
its nice to see such a forum still around. Batch files are sooo very useful at automating some tasks.
I'm writing a script to pull datalogs from an offsite FTP server.
I'm having trouble finding way to write it so that the Script when pulling data does not auto-overwrite old files on local.
the server generates data files at say 10files a day and keeps a backlog of up to a month. I'm trying not to have to pull all 300+ files everyday just to have the few most updated files in my system. extremely bad for server since i'm not the only one doing such monitoring
in dos, theres the "Echo n | copy *.* c:\store" where it will auto N to overwrite.
can we do the same for Dos FTP?
Thanks people
Code: Select all
REM FTP System311Monitoring Daily Script
REM Going to storage Directory
cd System311Monitoring
REM Creating temporary script file
> script.ftp ECHO <username>
>>script.ftp ECHO <password>
>>script.ftp ECHO hash
>>script.ftp ECHO cd DataStore
>>script.ftp ECHO ascii
>>script.ftp ECHO prompt n
>>script.ftp ECHO mget *SysMon311*.*
>>script.ftp ECHO bye
:: Use the temporary script for unattended FTP
FTP -v -d -s:script.ftp <ftp address>
:: Delete script file after use.
DEL script.ftp
REM Back to Root directory
cd ..
REM FTP Log Pull Script Complete
Posted: 05 Dec 2008 01:10
by DosItHelp
CheeseFox,
Check this out:
http://www.dostips.com/DtTipsFtpBatchScript.php#Batch_FtpBatchGetNewFilesOnly
This batch connects twice to the FTP server. First time it retrieves a list of files on the FTP server. This list is being trimmed to contain only files that don`t already exist locally. The files in the trimmed list are then downloaded during a second connection.
DosItHelp?
Posted: 06 Dec 2008 02:29
by CheeseFox
Hi DosItHelp,
been going thru the code since i saw the post...
i've changed << %temp%\%~n0.ftp >> to something akin of << c:\temp.ftp>> to get it to create the temporary script file
figured out why the extractFileSection won't work with Echo On. (i like to use echo on to learn how things work... heeh... ended up Echo Off before extractFileSection and Echo On after it. )
but have hit a stumbling point..
on unmasking (removing REM) of the Notepad lines to check the script file, i noted that the script file was only changed from "ls" to "mget" with nothing else.
but... i cant seem to figure out how this section works :
Code: Select all
REM -- Execute Ftp Script, collect File Names
Set "FileList="
For /F "Delims=" %%A In ('"Ftp -v -i -s:"%temp%\%~n0.ftp"|Findstr %FindStrArgs%"') Do (
Call Set "FileList=%%FileList%% "%%A""
)
i didnt see the script connecting to the FTP server the first time to get the list. I only saw the ftp connecting the 2nd time to run mget with no file list.
Any pointers on how that for loop works?
Thanks a million!
Posted: 06 Dec 2008 23:16
by DosItHelp
CheeseFox,
Code: Select all
REM -- Execute Ftp Script, collect File Names
Set "FileList="
For /F "Delims=" %%A In ('"Ftp -v -i -s:"%temp%\%~n0.ftp"|Findstr %FindStrArgs%"') Do (
Call Set "FileList=%%FileList%% "%%A""
)
is functionally identical with:
Code: Select all
REM -- Execute Ftp Script, collect File Names
Set "FileList="
For /F "Delims=" %%A In ('"Ftp -v -i -s:"%temp%\%~n0.ftp"|Findstr %FindStrArgs%"') Do (
Set "FileList=!FileList! "%%A""
)
... except it's not using Delayedexpansion.
It executes the first ftp script (ls) and sends the output through FINDSTR in order to filter the file names that match the pattern defined in the FindStrArgs variable, i.e. all files ending with ".txt".
The FOR loop iterates all resulting file names and adds them to FileList. The resulting string in the FileList variable should be a space separated list of all file names in the remote directory.
What you could try is:
After the first FTP script is created in c:\temp.ftp run it and verify that it returns the list of files in the remote directory, i.e. by running
Ftp -v -i -s:"c:\temp.ftp"
Let me know how it goes.
DosItHelp
Posted: 09 Dec 2008 20:58
by CheeseFox
Hi DosItHelp,
i'm getting the error "The input line is too long" at the line
Call Set "FileList=%%FileList%% "%%A""
................................
a side question, is it possible to pass the %%A variable value into a %temp1% variable for each loop?
i also noticed that the FTP server replies have a certain hidden character at the end of each line. only visible when opened in edit.com ... i think its screwing some things here.
Posted: 09 Dec 2008 21:11
by CheeseFox
I also noticed some junk characters (#) when the 'ls' list is long.
So i guess i'll need to put the %%A into a variable to trim off the hidden character at the end and to remove all the junk #s in the filenames.
But i cant seem to do a "set temp1=%%A" or similar at all..
been a decade since i last did such stuff. Still kinda fun.. lolx...
Thanks man!
Posted: 10 Dec 2008 04:40
by CheeseFox
here i am talking to myself... LMAO
ok.. i got it to work... in a fairly different way than the original code. but the basis is still there...
took me a day to figure out that i needed the "setlocal enabledelayedexpansion" environment variable for some stuff...
i'll clean up the code before posing it here...
hurrah!
thanks dude
Posted: 22 Dec 2008 23:04
by CheeseFox
Not the shortest nor the most elegant of codes. but it works well.
theres a simple logging function encoded too that shows Start Time, New Files downloaded and End time.
the extension thing is a workaround as some FTP servers add a LFCR or such character at the end of every reply.
the "#" removal is to remove junk '#' characters that sometimes appear when listing many many files...
works for long file names...
yea.. i think thats all..
To use... replace all <<????>> with the appropriate things
Thanks DosItHelp for the inspiration
Cheers,
CheeseFox
Code: Select all
@echo off
REM -- Define Local Folder to store files
REM -- Full directory location pls. Eg: C:\temp
Set HomeLoc=<<local directory>>
REM -- Define logfile
Set LogFile=<<logfile name, eg:C:\temp\FTPAUTO.txt>>
REM -- Define FTP information
Set FTPAddress=<<ftp server>>
Set FTPUser=<<ftp username>>
Set FTPPassword=<<ftp password>>
REM -- Define Remote Folder to pull files
REM -- tip: the directory name after logging into ftp server
Set RemoteLoc=<<remote directory>>
Set ls_string=<<search string: either "ls" for all files or "ls *item1*" to download specific files>>
REM -- Define File Filter,
REM -- :: example: files with extension .txt
REM -- :: Set FileType=ASCII or Binary?
REM -- :: Set Extension=.txt
Set FileType=ASCII
Set Extension=.txt
REM ################################################################################################
REM ################################################################################################
REM ################################################################################################
>>%LogFile% Echo.===========================
>>%LogFile% Echo Start Time:
Echo exit|cmd /q /k prompt $D $T>>%LogFile%
>>%LogFile% Echo.
>>%LogFile% Echo.Local Dir: %HomeLoc%
>>%LogFile% Echo.Remote ftp: %FTPAddress%
>>%LogFile% Echo.Remote Dir: %RemoteLoc%
>>%LogFile% Echo.
>>%LogFile% Echo.ver0.74 : 16 Dec 2008 9.30am
>>%LogFile% Echo.===========================
>>%LogFile% Echo.
Set FindStrArgs=/E /C:"%Extension%"
Set ScpFile1=%HomeLoc%\ftpscript1.ftp
Set ScpFile2=%HomeLoc%\ftpscript2.ftp
cd %HomeLoc%
>>%LogFile% Echo.**Listing
REM -- Creating FTP script file for listing out FTP files
REM -------------------------------------------
>>%ScpFile1% Echo !Title Connecting to %FTPAddress%...
>>%ScpFile1% Echo open %FTPAddress%
>>%ScpFile1% Echo %FTPUser%
>>%ScpFile1% Echo %FTPPassword%
>>%ScpFile1% Echo
>>%ScpFile1% Echo !Title Setting variables... %FTPAddress%...
>>%ScpFile1% Echo cd %RemoteLoc%
>>%ScpFile1% Echo lcd %HomeLoc%
>>%ScpFile1% Echo %FileType%
>>%ScpFile1% Echo hash
>>%ScpFile1% Echo
>>%ScpFile1% Echo !Title Retrieving File List... %FTPAddress%...
>>%ScpFile1% Echo %ls_string%
>>%ScpFile1% Echo !Title Disconnecting... %FTPAddress%...
>>%ScpFile1% Echo disconnect
>>%ScpFile1% Echo bye
REM -------------------------------------------
REM -- Execute Ftp Script, collect File Names
Set "Listing="
Set "FtpCommand=get"
REM -------------------------------------------
>>%ScpFile2% Echo !Title Connecting to %FTPAddress%...
>>%ScpFile2% Echo open %FTPAddress%
>>%ScpFile2% Echo %FTPUser%
>>%ScpFile2% Echo %FTPPassword%
>>%ScpFile2% Echo
>>%ScpFile2% Echo !Title Setting variables... %FTPAddress%...
>>%ScpFile2% Echo cd %RemoteLoc%
>>%ScpFile2% Echo lcd %HomeLoc%
>>%ScpFile2% Echo %FileType%
>>%ScpFile2% Echo hash
>>%ScpFile2% Echo
>>%ScpFile2% Echo !Title Downloading Files... %FTPAddress%...
>>%LogFile% Echo.**Sorting
>>%LogFile% Echo.=== New Files Start ===
REM -------------------------------------------
setlocal enabledelayedexpansion
set Junksearch=#
set Junkreplace=
set NewFiles=0
For /F "Delims=" %%A In ('"Ftp -v -s:%ScpFile1%|Findstr %FindStrArgs%"') Do (
set newline=%%~nA
set newline=!newline:%Junksearch%=%Junkreplace%!
set emptyline=
if Not !newline!==!emptyline! (
If Not Exist "!newline!%extension%" (
>> %ScpFile2% Echo.%FtpCommand% !newline!%extension%
>>%LogFile% Echo.%FtpCommand% !newline!%extension%
set NewFiles=1
)
)
)
>>%LogFile% Echo.=== New Files End ===
setlocal disabledelayedexpansion
REM -------------------------------------------
>>%ScpFile2% Echo !Title Disconnecting... %FTPAddress%...
>>%ScpFile2% Echo disconnect
>>%ScpFile2% Echo bye
if %NewFiles%==1 (
REM -- Execute Ftp Script, download files
>>%LogFile% Echo.**Retrieving
ftp -v -d -s:%ScpFile2%
>>%LogFile% Echo.ERROR LEVEL:: %ERRORLEVEL%
) ELSE (
>>%LogFile% Echo.**No New Files
)
Del %ScpFile1%
Del %ScpFile2%
>>%LogFile% Echo.**Ending Script
>>%LogFile% Echo.===========================
>>%LogFile% Echo End time:
Echo exit|cmd /q /k prompt $D $T>>%LogFile%
>>%LogFile% Echo.
>>%LogFile% Echo.===========================
>>%LogFile% Echo.
>>%LogFile% Echo.
>>%LogFile% Echo.