renaming files

[jon123]

Hi Guys,

I'm pretty much a noob to all this batch file business.

Anyways atm I have a bat file that effectively uses ftp to pull off some log files.. however these log files are in a bad format to read clearly.

:1
cd U:\hc\server
ftp -i -s:serverftp.txt
goto end

the serverftp.txt file has the commands to get the ftp files.

the rest of the script simply copies them across

cd U:\server
copy file.112009.csv U:\server\output\112009\file.112009.csv


Basically I want have a batch file that will select the files from the last week and rename them to the appropriate server they have been sourced from.

Cheers,
Jon

[aGerman]

Hmm, no easy task.
Batch can't calculate with DateTime values, but it would be possible to find the date one week ago using the date2jdate function.
First question is: How can we find out how old a file is? By file name? Or have we to use the FTP DIR command to get informations about the file date?
Second question: If we have to use the DIR command, could you post an example output?

Code: Select all

@echo off &setlocal
(
  echo open yourHostName.com
  echo yourUserName
  echo yourPassword
  echo dir "yourRemoteDirectory" "dir.txt"
  echo disconnect
  echo bye
)>"test.ftp"
ftp.exe -i -s:"test.ftp"

After that we could look forward ...

Regards
aGerman

[jon123]

Hi aGerman thanks for your response.

In reply to your first question the log files are formatted e.g. ainslie.igs.27112010.log. I suppose you could count back 7 days from the current date and select those files somehow?

I'm not sure I understand your second question regarding the DIR command? Do you mean I should try putting

echo dir "yourRemoteDirectory" "dir.txt" in the script?


Apart from that the script calls a .txt file with the ftp commands at the moment MGET igs.*2010.log I want to have it so that it just selects the log files for the past month. Would I have to manually change the script every month to MGET igs.*102010?

Regards,

Jon

*edit* I think maybe I could have been clearer with what I'm trying to do.

Basically I want to wget with ftp files from the last week with the extension .log and .csv and rename them to LocationNamedate.log & LocationNamedate.csv . At the moment the format is simply date.log & date.csv.

[ghostmachine4]

Use a programming language with FTP support to do this kind of stuff, not batch. Why?
1) Date arithmetic in batch (cmd) is a pain. Programming languages with good DateTime modules make it easier to program date maths and a lot more , eg Date formatting, converting etc.
2) String/File manipulation/ is a pain. there is only primitive string/text manipulation functions such that you have to do a lot more ( in terms of lines of code) to get simple job done (using programming structures called arrays/ or more advanced ones like hashes/dictionaries)
3) the FTP client that comes with windows is not the meant for automation. (its only primitive) and you have to create an extra file (to put in your commands). More maintenance. Plus, there is no good mechanism for error catching. Often times, if the task goes beyond merely getting and uploading files, you have to log in the FTP server several times...which is ridiculous.

Here's one example done with Python (similarly if you know other languages) ... that gets files for the last 1 week and download from server.

Code: Select all

import ftplib,time,datetime
server="localhost"
user="anonymous"
password="test@email.com"
today = datetime.date.today()
now = datetime.date(today.year, today.month,today.day)
minuswks = datetime.timedelta(weeks=-1)  #set number of weeks here
wksago = now + minuswks 
wksago = str(wksago).replace("-","")
filelist=[]
def download(ftp, filename):
    ftp.retrbinary("RETR " + filename, open(filename,"wb").write)
try:
   ftp = ftplib.FTP()
   #ftp.debug(3)
   ftp.connect(server,21)
   ftp.login(user,password)
except Exception,e:
    print e
else:
    #ftp.cwd("directory to change if any")
    ftp.retrlines('LIST',filelist.append)
    for latest in filelist:
        latest=latest.split()
        whatiwant=latest[5:]
        if latest[0].startswith("-"):
            timestamp,filename = whatiwant[:3], ''.join(whatiwant[3:])
            if ":" in timestamp[-1] :
                timestamp[1]=timestamp[1].zfill(2)
                timestr=str(today.year)+''.join(timestamp[:2])
                p=time.strptime(timestr,"%Y%b%d")
                filedate=time.strftime("%Y%m%d",p)
                if str(filedate) >= wksago:
                    try:
                        download(ftp,filename)
                    except Exception,e:
                        print e
                    else:
                        print "Download ok"

    ftp.quit()

[jon123]

Thanks for your response ghostmachine.. I did suspect such a complex task may have been beyond the scope of batch files.

If I decided to use python for example, what would I have to have installed on my local machine & server for it to run?.. I ask because the server is on AIX and I don't know if I will be allowed to install python on there.

With python would I also be able to extract the files into an excel spreadsheet? That would sorta be the ultimate goal, although some of the files are already in CSV format, others are in .log format.

Cheers,
Jon

[ghostmachine4]

If I decided to use python for example, what would I have to have installed on my local machine & server for it to run?..

You need to get the Python distribution. I useActiveState Python.
You also need to get familiarize with the language (trust me, its easy to learn, and you can do your admin stuff with ease)

[ghostmachine4]

I ask because the server is on AIX and I don't know if I will be allowed to install python on there.

I am not sure what's your setup, but generally, to get and put files to and from an ftp client, you don't have to install the client everywhere.
You can install Python on a another server/PC that is not AIX for example, then get and put files to AIX server, all the while using another machine as client. You just need to run the FTP daemon at AIX for FTP to work. That's all. However, if you want to install Python on AIX and use the script there, its your choice. there is Python for AIX here.

There is also a Python to exe converter (not the only one). You can code your script in one machine, convert it to exe and bring anywhere if needed.

[ghostmachine4]

With python would I also be able to extract the files into an excel spreadsheet?

of course. You can use dedicated excel modules for that. See here for examples.
there is also the CSV module that comes with Python for reading/writing CSV file for import to
Excel.

[jon123]

Your feedback is much appreciated ghostmachine. I have come across python before but only played with some very basic functions (i remember some kind of game where you had to solve riddles using python to progress to the next web page?).

I will certainly try installing Activestate and the excel module and see what happens.

Regards,
Jon

[aGerman]

I'm not sure I understand your second question regarding the DIR command? Do you mean I should try putting

echo dir "yourRemoteDirectory" "dir.txt" in the script?

No, I thought you would customize this script (yourHostName.com, yourUserName, yourPassword, yourRemoteDirectory) and you would post (only the first 3 or 4 lines of) "dir.txt" enclosed in code tags. This was just to see the format (order) of the data in the line.

But I agree with ghostmachine4. Batch is not the best way for this kind of stuff. But if you're interrested in, we also could try to find a (probably slow and complicated) solution for batch.

Regards
aGerman