Page 1 of 1
needs to create incremental synch local folder with ftp
Posted: 28 Jan 2011 01:29
by rizutov
Hi guys,
I suppose it's really simple but I could not go through this task...
There is an ftp folder where every day every 15 mins new files are put automatically. I have server with local folder which should contain the same files. I wrote a script which copies all this files from FTP to this local folder:
bat file:
Code: Select all
ftp -s:start_download.txt ftp.mysite.com
start_download.txt
Code: Select all
ftplogin
ftppassword
lcd D:\ftp
cd pub
prompt
mget *
bye
But it replaces all files in local folder. Since this ftp folder's size is 15Gb it's inconvenient and slow. It needs somehow to add missing files to the local folder and not to copy all files from FTP. I have some idea about copying list of files to the txt file(ls comand) but I don't know how to parse it and check new filenames in it. I wonder if there is a standard decision for this...
thanks in advance!
Re: needs to create incremental synch local folder with ftp
Posted: 28 Jan 2011 12:48
by ChickenSoup
This would be ugly with the windows built in FTP client (and probably not completely reliable). It would be better to use a 3rd party with command line like
WinSCP. This has command line parameters here:
http://winscp.net/eng/docs/scripting#commands. The synchronize command or keepuptodate command may be what you need. Let me know if you want a Windows FTP solution. There may be a ugly way to do it.
Re: needs to create incremental synch local folder with ftp
Posted: 31 Jan 2011 01:40
by rizutov
Ok then. I downloaded it and tried on my computer first. Everything looks fine. But the question is:
1. Can I adjust this WinSCP portable version to start when the system starts?
2. Very interesting option keepuptodate but I failed to manage it's timetable. How can I check wich timeframe it uses to check if there are some changes in remote ftp directory?
Re: needs to create incremental synch local folder with ftp
Posted: 31 Jan 2011 04:33
by rizutov
one more update:
- keepuptodate doesn't suit as it allows to update only in one direction: from local dir to ftp. I need vise versa direction.(I have read rights on FTP folder and have to copy all new files from ftp server to local directory)
- I got through with synchronization but I wonde if I can make this procedure scheduled as this synch should be held on a constant basis.
Re: needs to create incremental synch local folder with ftp
Posted: 31 Jan 2011 08:10
by ChickenSoup
Once you have your script written you would have to use windows task scheduler to run it in the background or some other task scheduler. So you would create a task that launches winscp like so:
Code: Select all
winscp.exe /console /script=yourscript.txt
yourscript.txt (untested)
Code: Select all
option batch on
option confirm off
option transfer automatic
open user:password@example.com
synchronize both c:\local\path /remote/path
close
exit
Re: needs to create incremental synch local folder with ftp
Posted: 31 Jan 2011 09:43
by rizutov
This is may batch file:
Code: Select all
C:\...\winscp.exe /console /script=ftp_connect.txt
And this is my ftp_connect.txt file:
Code: Select all
option batch on
option confirm off
option transfer automatic
open login:pass@194.154.80.6:21
synchronize local D:\ftp /export
close
exit
When I connect using GUI with the same login details it's OK. But from bathfile it says waiting for data timed out...
Code: Select all
. 2011-01-31 18:37:07.677 Looking up host "194.154.80.6"
. 2011-01-31 18:37:07.677 Connecting to 194.154.80.6 port 21
. 2011-01-31 18:37:22.677 Waiting for data timed out, asking user what to do.
. 2011-01-31 18:37:22.677 Asking user:
. 2011-01-31 18:37:22.677 Host is not communicating for 15 seconds.
Where is the bug?
Re: needs to create incremental synch local folder with ftp
Posted: 31 Jan 2011 09:55
by rizutov
. 2011-01-31 18:53:29.502 Asking user:
. 2011-01-31 18:53:29.502 Host is not communicating for 15 seconds.
. 2011-01-31 18:53:29.502
. 2011-01-31 18:53:29.502 Wait for another 15 seconds? ()
. 2011-01-31 18:53:29.502 Attempt to close connection due to fatal exception:
* 2011-01-31 18:53:29.502 Terminated by user.
. 2011-01-31 18:53:29.502 Closing connection.
. 2011-01-31 18:53:29.502 Sending special code: 12
Re: needs to create incremental synch local folder with ftp
Posted: 31 Jan 2011 10:09
by ChickenSoup
You probably need it in passive mode. Try changing 'open' line to:
Code: Select all
open login:pass@194.154.80.6:21 -passive
Re: needs to create incremental synch local folder with ftp
Posted: 01 Feb 2011 01:14
by rizutov
I've added this option (-passive) to open string but it seems the same problem (please see log's content below)
This is logged when I connect using GUI:Code: Select all
. 2011-02-01 09:13:20.580 WinSCP Version 4.2.9 (Build 938) (OS 5.2.3790 Service Pack 2)
. 2011-02-01 09:13:20.580 Login time: 1 Февраль 2011 г. 9:13:20
. 2011-02-01 09:13:20.580 --------------------------------------------------------------------------
. 2011-02-01 09:13:20.580 Session name: user@194.154.80.6
. 2011-02-01 09:13:20.580 Host name: 194.154.80.6 (Port: 21)
. 2011-02-01 09:13:20.580 User name: ALGLExport (Password: Yes, Key file: No)
. 2011-02-01 09:13:20.580 Tunnel: No
. 2011-02-01 09:13:20.580 Transfer Protocol: SFTP (SCP)
. 2011-02-01 09:13:20.580 Ping type: -, Ping interval: 30 sec; Timeout: 15 sec
. 2011-02-01 09:13:20.580 Proxy: none
. 2011-02-01 09:13:20.580 SSH protocol version: 2; Compression: No
. 2011-02-01 09:13:20.580 Bypass authentication: No
. 2011-02-01 09:13:20.580 Try agent: Yes; Agent forwarding: No; TIS/CryptoCard: No; KI: Yes; GSSAPI: No
. 2011-02-01 09:13:20.580 Ciphers: aes,blowfish,3des,WARN,arcfour,des; Ssh2DES: No
. 2011-02-01 09:13:20.580 SSH Bugs: -,-,-,-,-,-,-,-,-
. 2011-02-01 09:13:20.580 SFTP Bugs: -,-
. 2011-02-01 09:13:20.580 Return code variable: Autodetect; Lookup user groups: Yes
. 2011-02-01 09:13:20.580 Shell: default
. 2011-02-01 09:13:20.580 EOL: 0, UTF: 2
. 2011-02-01 09:13:20.580 Clear aliases: Yes, Unset nat.vars: Yes, Resolve symlinks: Yes
. 2011-02-01 09:13:20.580 LS: ls -la, Ign LS warn: Yes, Scp1 Comp: No
. 2011-02-01 09:13:20.580 Local directory: default, Remote directory: home, Update: No, Cache: Yes
. 2011-02-01 09:13:20.580 Cache directory changes: Yes, Permanent: Yes
. 2011-02-01 09:13:20.580 DST mode: 1
. 2011-02-01 09:13:20.580 --------------------------------------------------------------------------
Code: Select all
. 2011-02-01 09:13:20.580 Looking up host "194.154.80.6"
. 2011-02-01 09:13:20.580 Connecting to 194.154.80.6 port 21
. 2011-02-01 09:13:35.595 Waiting for data timed out, asking user what to do.
. 2011-02-01 09:13:35.595 Asking user:
. 2011-02-01 09:13:35.595 Host is not communicating for 15 seconds.
. 2011-02-01 09:13:35.595
. 2011-02-01 09:13:35.595 Wait for another 15 seconds? ()
. 2011-02-01 09:13:35.595 Attempt to close connection due to fatal exception:
* 2011-02-01 09:13:35.595 Terminated by user.
. 2011-02-01 09:13:35.595 Closing connection.
. 2011-02-01 09:13:35.595 Sending special code: 12
By the way, I tried to change password and username entries. And this didn't change anything. This means the problem is in connection i.e. in "open" string. But still I can not figure what it says by
Code: Select all
"Waiting for data timed out, asking user what to do.
. 2011-02-01 09:13:35.595 Asking user:
"
Maybe there is another way to make this sync...
Re: needs to create incremental synch local folder with ftp
Posted: 01 Feb 2011 06:07
by rizutov
sorry!
It works, the problem was in the session I adjusted through GUI.
I put off port number and saved password in GUI version and now it works perfect!
Re: needs to create incremental synch local folder with ftp
Posted: 01 Feb 2011 07:03
by rizutov
ChickeSoup, thank you very much! Great thanx!