INTELLIGENT WORK FORUMS
FOR COMPUTER PROFESSIONALS

Log In

Come Join Us!

Are you a
Computer / IT professional?
Join Tek-Tips Forums!
  • Talk With Other Members
  • Be Notified Of Responses
    To Your Posts
  • Keyword Search
  • One-Click Access To Your
    Favorite Forums
  • Automated Signatures
    On Your Posts
  • Best Of All, It's Free!

*Tek-Tips's functionality depends on members receiving e-mail. By joining you are opting in to receive e-mail.

Posting Guidelines

Promoting, selling, recruiting, coursework and thesis posting is forbidden.

Jobs

Create a .SH script to download from external ftp to internal ftp and transfer to windows share...

Create a .SH script to download from external ftp to internal ftp and transfer to windows share...

Create a .SH script to download from external ftp to internal ftp and transfer to windows share...

(OP)
Hi there, I hope you're good...

I am new to linux commands and shell scripting and would like to create a .sh script on a linux server to download data from an external FTP site to our internal ftp site and then transfer the data to a share on a windows fileserver. i would also like to create a notification email when the process is complete. Any help would be greatly appreciated?

Here is a script we use to upload data to the external FTP but I now need to create one for downloading the data:

#!/bin/bash
# FTP settings
HOST='example.co.uk'
USER='user'
PASSWD='example'
TMPUPABC='/ftp/FTPSITE/tmpupload'
MNTDIRABC='/mnt/server_mastershare/FTPSITE/To_example_ABC'
MNTARCABC='/mnt/server_mastershare/FTPSITE/Archive_Upload_ABC'
# Email settings
SUBJECT="FTP upload for ABC"
EMAIL="Joe.Bloggs@example.co.uk"
EMAILMESSAGEABC="/ftp/FTPSITE/FTPSITEemailmessageABC.txt"
FTPRESULT="/ftp/FTPSITE/ftpresult.txt"
# Other variables
DATE=`date +%Y-%m-%d_%H%M`
# Start of main script
# look for empty dir and generate email if files found
if [ "$(ls -A $MNTDIRABC)" ]; then
mkdir /mnt/server_mastershare/FTPSITE/Archive_upload_ABC/$DATE
cp -r $MNTDIRABC/* /mnt/server_mastershare/FTPSITE/Archive_upload_ABC/$DATE
mv -f $MNTDIRABC/* $TMPUPABC
cd $TMPUPABC
echo "FTP to FTPSITE for ABC : found the following files and will now attempt to upload them"> $EMAILMESSAGEABC
echo "**********************************************************************">> $EMAILMESSAGEABC
tree $TMPUPABC >>$EMAILMESSAGEABC
echo "**********************************************************************">> $EMAILMESSAGEABC
echo "Listing of remote server">> $EMAILMESSAGEABC
echo "**********************************************************************">> $EMAILMESSAGEABC
# Connect to remote server using lftp and sync 'up'
lftp -u user,example sftp://example.co.uk <<EOD
cd from_gbo
cd ABC
mirror -R --Remove-source-files
ls >> $EMAILMESSAGEABC
quit 0
EOD
/usr/bin/mail -s "$SUBJECT" "$EMAIL" < $EMAILMESSAGEABC
# Tidy files & folders
rm -rf $TMPUPABC/*
rm -rf $EMAILMESSAGEABC
else
echo "$TMPDUPABC is Empty"
fi

RE: Create a .SH script to download from external ftp to internal ftp and transfer to windows share...

(OP)
Help!

Please can anybody offer any direction, or if you need more information just ask?

Thanks in advance

RE: Create a .SH script to download from external ftp to internal ftp and transfer to windows share...

Well, it appears having that upload script already gives you a lot of clues; why don't you just try studying the code a little, rearranging it and changing the appropriate FTP commands to do what you require? Besides, it seems to use a special FTP client and commands ("lftp" and "mirror") that I personally haven't encountered, so you may have better luck anyway.

If you get stuck with the scripting, post here with specific questions.

Annihilannic
tgmlify - code syntax highlighting for your tek-tips posts

RE: Create a .SH script to download from external ftp to internal ftp and transfer to windows share...

(OP)
Thanks for your post above, I appreciate that the script already gives alot of clues but I'm a complete novice when it comes to creating a script from the begining and manipulating the above upload script to a downloads script is a daunting task and I have no idea where to start? I have been studying the code but I'm having problems with structuring the script.

I assume I need to add the following variables to my downloads script in the same way as on the uploads script?

#!/bin/bash
# FTP settings
HOST='example.co.uk'
USER='user'
PASSWD='example'
TMPDOABC='/ftp/FTPSITE/tmpdownload'
MNTDIRABC='/mnt/server_mastershare/FTPSITE/From_example_ABC'
MNTARCABC='/mnt/server_mastershare/FTPSITE/Archive_Download_ABC'
# Email settings
SUBJECT="FTP upload for ABC"
EMAIL="Joe.Bloggs@example.co.uk"
EMAILMESSAGEABC="/ftp/FTPSITE/FTPSITEemailmessageABC.txt"
FTPRESULT="/ftp/FTPSITE/ftpresult.txt"
# Other variables
DATE=`date +%Y-%m-%d_%H%M`


If yes, I assume I then need to add the code directly below it for the following tasks:

1. connect to external ftp sites directory to be downloaded
2. download data to internal ftp
3. move data to internal windows share
4. create an archive copy on windows share
5. Add code to email out if data has been recieved.

If anybody can help me structure this script then it would be a real help, much appreciated in advance.

RE: Create a .SH script to download from external ftp to internal ftp and transfer to windows share...

(OP)
smilePlease don't laugh. Here is what I have so far, it needs a lot more work. I'm just finding out from one of the senior helpdesk guys how the email EMAILMESSAGEEVF="/ftp/ABCEVF/ABCupemailmessageEVF.txt" file works as i haven't been able to find it on the server at that path! I'm hoping this text file can be copied and amended to be used as a template to be added to the script to send out an email when the transfer is complete.

Can a member on here have a look and see if the code below is even close to being correct for this process? Also how would I go about debugging the script on a windows machine?

#!/bin/bash
# FTP settings
HOST='example.co.uk'
USER='user'
PASSWD='password'
MNTDIREVF='/mnt/sc-la-fs-01_mastershare/ABC/From_ABC_EVF'
MNTARCEVF='/mnt/sc-la-fs-01_mastershare/ABC/Archive_Download_EVF'
# Email settings
SUBJECT="ABC FTP Download for EVF"
EMAIL="joe.bloggs@example.co.uk"
EMAILMESSAGEEVF="/ftp/ABCEVF/ABCdownemailmessageEVF.txt"
FTPRESULT="/ftp/ABCEVF/ftpdownloadresult.txt"
# Other variables
DATE=`date +%Y-%m-%d_%H%M`


# Start of main script
# Connect to remote server using lftp and sync 'down'
lftp -u gbo,password sftp://example.co.uk <<EOD

mirror /to_gbo/DMEVF /ftp://sc-la-app-01/ABC_SFTP/For_system_EVF/
quit 0
EOD
# look for empty dir and generate email if files found
if [ "$(ls -A $MNTDIREVF)" ]; then
mkdir /mnt/sc-la-fs-01_mastershare/ABC/Archive_Download_EVF/$DATE
cp -r $MNTDIREVF/* /mnt/sc-la-fs-01_mastershare/ABC/Archive_Download_EVF/$DATE


cd ?
echo "FTP from ABC for EVF : found the following files and will now attempt to download them"> $EMAILMESSAGEEVF
echo "**********************************************************************">> $EMAILMESSAGEEVF
tree $TMPUPEVF >>$EMAILMESSAGEEVF
echo "**********************************************************************">> $EMAILMESSAGEEVF
echo "Listing of remote server">> $EMAILMESSAGEEVF
echo "**********************************************************************">> $EMAILMESSAGEEVF
fi

Thanks in advance, I'm finding unix scripting a slow learning process shadessad

RE: Create a .SH script to download from external ftp to internal ftp and transfer to windows share...

The email message is created by the script. The first echo "blahblah" >$EMAILMESSAGEEVF creates the file, and the subsequent echo "****" >>$EMAILMESSAGEEVF lines append to that file. So there is no separate template.

You could debug the script on a Windows system by using Cygwin which allows you to run a bash shell and a bunch of GNU utilities under Windows; however I would test it on the target server just using test files and directory locations until you're ready to switch them to the real locations. That way you know that the commands and utilities you are depending on are definitely available (such as, for example, that lftp client you are using, which may not be available for Cygwin).

Annihilannic
tgmlify - code syntax highlighting for your tek-tips posts

RE: Create a .SH script to download from external ftp to internal ftp and transfer to windows share...

(OP)
Thanks Annihilannic, your posts are much appreciated. Here is the script as it stands, can you see any glaring mistakes before I start testing it on the unix server? Thanks

#!/bin/bash
# FTP settings
HOST='example.co.uk'
USER='user'
PASSWD='password'
TMPDOEVF='/ftp/ABCEVF/tmpdownload'
MNTDIREVF='/mnt/sc-la-fs-01_mastershare/ABC/From_ABC_EVF'
MNTARCEVF='/mnt/sc-la-fs-01_mastershare/ABC/Archive_Download_EVF'
# Email settings
SUBJECT="ABC FTP Download for EVF"
EMAIL="joe.bloggs@example.co.uk"
EMAILMESSAGEEVF="/ftp/ABCEVF/ABCdownemailmessageEVF.txt"
FTPRESULT="/ftp/ABCEVF/ftpdownloadresult.txt"
# Other variables
DATE=`date +%Y-%m-%d_%H%M`


# Start of main script
# Connect to remote server using lftp and sync 'down'
lftp -u gbo,password sftp://example.co.uk <<EOD

mirror /to_gbo/DMEVF /ftp://sc-la-app-01/ABC_SFTP/For_system_EVF/
quit 0
EOD
# look for empty dir and generate email if files found
if [ "$(ls -A $MNTDIREVF)" ]; then
mkdir /mnt/sc-la-fs-01_mastershare/ABC/Archive_Download_EVF/$DATE
cp -r $MNTDIREVF/* /mnt/sc-la-fs-01_mastershare/ABC/Archive_Download_EVF/$DATE
cd $TMPDOEVF
echo "FTP from ABC for EVF : found the following files and will now attempt to download them"> $EMAILMESSAGEEVF
echo "**********************************************************************">> $EMAILMESSAGEEVF
tree $TMPDOEVF >>$EMAILMESSAGEEVF
echo "**********************************************************************">> $EMAILMESSAGEEVF
echo "Listing of remote server">> $EMAILMESSAGEEVF
echo "**********************************************************************">> $EMAILMESSAGEEVF

/usr/bin/mail -s "$SUBJECT" "$EMAIL" < $EMAILMESSAGEEVF
# Tidy files & folders
rm -rf $TMPDOEVF/*
rm -rf $EMAILMESSAGEEVF
else
echo "$TMPDOPEVF is Empty"
fi

RE: Create a .SH script to download from external ftp to internal ftp and transfer to windows share...

Since you've bothered to define the MNTARCEVF variable, you may as well use it here:

CODE

mkdir $MNTARCEVF/$DATE
cp -r $MNTDIREVF/* $MNTARCEVF/$DATE 

Also the code seems to be slighly confused between MNTDIREVF and TMPDOPEVF and TMPDOEVF... e.g. your conditional statement checks for the presence of MNTDIREVF, but then the error message reports that TMPDOPEVF (which is not defined anywhere... extra "P" perhaps) is empty.

When the email message says "will now attempt"... is it referring to a future action (to be implemented) or the lftp mirror step that was just completed?



Annihilannic
tgmlify - code syntax highlighting for your tek-tips posts

RE: Create a .SH script to download from external ftp to internal ftp and transfer to windows share...

(OP)
Thanks for the last post, I have made the changes as you suggested and think I'm ready for testing. neutral

I need help to debug the shell script in an effective way but have no idea how, I've heard that using the SH command is the most effective way but am finding this difficult to read up on. I have been looking for a dummys guide online but nothing is simple enough to follow.

Do I just copy the script onto the linux server and set the scripts permissions to read and execute first and then use SH (inside script or when running script?) and then add flags to show the relevant errors?

Any instructions will be much appreciated on how to debug this effectively, the simpler the better! Thanks

RE: Create a .SH script to download from external ftp to internal ftp and transfer to windows share...

(OP)
PHV, thanks for the post.

I'm using Putty SSH to connect to the server and the above command isn't working, I assume this needs to be run on the local server? Is there a way of doing this through putty?

Thanks

RE: Create a .SH script to download from external ftp to internal ftp and transfer to windows share...

The exact error message you are getting would be helpful.

Try bash -x /path/to/your/script perhaps?

Annihilannic
tgmlify - code syntax highlighting for your tek-tips posts

Red Flag This Post

Please let us know here why this post is inappropriate. Reasons such as off-topic, duplicates, flames, illegal, vulgar, or students posting their homework.

Red Flag Submitted

Thank you for helping keep Tek-Tips Forums free from inappropriate posts.
The Tek-Tips staff will check this out and take appropriate action.

Reply To This Thread

Posting in the Tek-Tips forums is a member-only feature.

Click Here to join Tek-Tips and talk with other members!

Resources

Close Box

Join Tek-Tips® Today!

Join your peers on the Internet's largest technical computer professional community.
It's easy to join and it's free.

Here's Why Members Love Tek-Tips Forums:

Register now while it's still free!

Already a member? Close this window and log in.

Join Us             Close