needs to create incremental synch local folder with ftp

[rizutov]

Hi guys,
I suppose it's really simple but I could not go through this task...
There is an ftp folder where every day every 15 mins new files are put automatically. I have server with local folder which should contain the same files. I wrote a script which copies all this files from FTP to this local folder:

bat file:

Code: Select all

ftp -s:start_download.txt ftp.mysite.com

start_download.txt

Code: Select all

ftplogin
ftppassword
lcd D:\ftp
cd pub
prompt
mget *
bye

But it replaces all files in local folder. Since this ftp folder's size is 15Gb it's inconvenient and slow. It needs somehow to add missing files to the local folder and not to copy all files from FTP. I have some idea about copying list of files to the txt file(ls comand) but I don't know how to parse it and check new filenames in it. I wonder if there is a standard decision for this...
thanks in advance!

[ChickenSoup]

This would be ugly with the windows built in FTP client (and probably not completely reliable). It would be better to use a 3rd party with command line like WinSCP. This has command line parameters here: http://winscp.net/eng/docs/scripting#commands. The synchronize command or keepuptodate command may be what you need. Let me know if you want a Windows FTP solution. There may be a ugly way to do it.

[rizutov]

Ok then. I downloaded it and tried on my computer first. Everything looks fine. But the question is:
1. Can I adjust this WinSCP portable version to start when the system starts?
2. Very interesting option keepuptodate but I failed to manage it's timetable. How can I check wich timeframe it uses to check if there are some changes in remote ftp directory?

[rizutov]

one more update:
- keepuptodate doesn't suit as it allows to update only in one direction: from local dir to ftp. I need vise versa direction.(I have read rights on FTP folder and have to copy all new files from ftp server to local directory)
- I got through with synchronization but I wonde if I can make this procedure scheduled as this synch should be held on a constant basis.

[ChickenSoup]

Once you have your script written you would have to use windows task scheduler to run it in the background or some other task scheduler. So you would create a task that launches winscp like so:

Code: Select all

winscp.exe /console /script=yourscript.txt

yourscript.txt (untested)

Code: Select all

option batch on
option confirm off
option transfer automatic
open user:password@example.com
synchronize both c:\local\path /remote/path
close
exit

[rizutov]

This is may batch file:

Code: Select all

C:\...\winscp.exe /console /script=ftp_connect.txt

And this is my ftp_connect.txt file:

Code: Select all

option batch on
option confirm off
option transfer automatic
open login:pass@194.154.80.6:21
synchronize local D:\ftp /export
close
exit

When I connect using GUI with the same login details it's OK. But from bathfile it says waiting for data timed out...

Code: Select all

. 2011-01-31 18:37:07.677 Looking up host "194.154.80.6"
. 2011-01-31 18:37:07.677 Connecting to 194.154.80.6 port 21
. 2011-01-31 18:37:22.677 Waiting for data timed out, asking user what to do.
. 2011-01-31 18:37:22.677 Asking user:
. 2011-01-31 18:37:22.677 Host is not communicating for 15 seconds.

Where is the bug?

[rizutov]

. 2011-01-31 18:53:29.502 Asking user:
. 2011-01-31 18:53:29.502 Host is not communicating for 15 seconds.
. 2011-01-31 18:53:29.502
. 2011-01-31 18:53:29.502 Wait for another 15 seconds? ()
. 2011-01-31 18:53:29.502 Attempt to close connection due to fatal exception:
* 2011-01-31 18:53:29.502 Terminated by user.
. 2011-01-31 18:53:29.502 Closing connection.
. 2011-01-31 18:53:29.502 Sending special code: 12

[ChickenSoup]

You probably need it in passive mode. Try changing 'open' line to:

Code: Select all

open login:pass@194.154.80.6:21 -passive

[rizutov]

I've added this option (-passive) to open string but it seems the same problem (please see log's content below)

This is logged when I connect using GUI:

Code: Select all

. 2011-02-01 09:13:20.580 WinSCP Version 4.2.9 (Build 938) (OS 5.2.3790 Service Pack 2)
. 2011-02-01 09:13:20.580 Login time: 1 Февраль 2011 г. 9:13:20
. 2011-02-01 09:13:20.580 --------------------------------------------------------------------------
. 2011-02-01 09:13:20.580 Session name: user@194.154.80.6
. 2011-02-01 09:13:20.580 Host name: 194.154.80.6 (Port: 21)
. 2011-02-01 09:13:20.580 User name: ALGLExport (Password: Yes, Key file: No)
. 2011-02-01 09:13:20.580 Tunnel: No
. 2011-02-01 09:13:20.580 Transfer Protocol: SFTP (SCP)
. 2011-02-01 09:13:20.580 Ping type: -, Ping interval: 30 sec; Timeout: 15 sec
. 2011-02-01 09:13:20.580 Proxy: none
. 2011-02-01 09:13:20.580 SSH protocol version: 2; Compression: No
. 2011-02-01 09:13:20.580 Bypass authentication: No
. 2011-02-01 09:13:20.580 Try agent: Yes; Agent forwarding: No; TIS/CryptoCard: No; KI: Yes; GSSAPI: No
. 2011-02-01 09:13:20.580 Ciphers: aes,blowfish,3des,WARN,arcfour,des; Ssh2DES: No
. 2011-02-01 09:13:20.580 SSH Bugs: -,-,-,-,-,-,-,-,-
. 2011-02-01 09:13:20.580 SFTP Bugs: -,-
. 2011-02-01 09:13:20.580 Return code variable: Autodetect; Lookup user groups: Yes
. 2011-02-01 09:13:20.580 Shell: default
. 2011-02-01 09:13:20.580 EOL: 0, UTF: 2
. 2011-02-01 09:13:20.580 Clear aliases: Yes, Unset nat.vars: Yes, Resolve symlinks: Yes
. 2011-02-01 09:13:20.580 LS: ls -la, Ign LS warn: Yes, Scp1 Comp: No
. 2011-02-01 09:13:20.580 Local directory: default, Remote directory: home, Update: No, Cache: Yes
. 2011-02-01 09:13:20.580 Cache directory changes: Yes, Permanent: Yes
. 2011-02-01 09:13:20.580 DST mode: 1
. 2011-02-01 09:13:20.580 --------------------------------------------------------------------------

And this was logged while trying batch file:

Code: Select all

. 2011-02-01 09:13:20.580 Looking up host "194.154.80.6"
. 2011-02-01 09:13:20.580 Connecting to 194.154.80.6 port 21
. 2011-02-01 09:13:35.595 Waiting for data timed out, asking user what to do.
. 2011-02-01 09:13:35.595 Asking user:
. 2011-02-01 09:13:35.595 Host is not communicating for 15 seconds.
. 2011-02-01 09:13:35.595 
. 2011-02-01 09:13:35.595 Wait for another 15 seconds? ()
. 2011-02-01 09:13:35.595 Attempt to close connection due to fatal exception:
* 2011-02-01 09:13:35.595 Terminated by user.
. 2011-02-01 09:13:35.595 Closing connection.
. 2011-02-01 09:13:35.595 Sending special code: 12

By the way, I tried to change password and username entries. And this didn't change anything. This means the problem is in connection i.e. in "open" string. But still I can not figure what it says by

Code: Select all

 "Waiting for data timed out, asking user what to do.
. 2011-02-01 09:13:35.595 Asking user:
"

Maybe there is another way to make this sync...

[rizutov]

sorry!
It works, the problem was in the session I adjusted through GUI.
I put off port number and saved password in GUI version and now it works perfect!

[rizutov]

ChickeSoup, thank you very much! Great thanx!