Tek-Tips is the largest IT community on the Internet today!
Members share and learn making Tek-Tips Forums the best source of peer-reviewed technical information on the Internet!
-
Congratulations Chriss Miller on being selected by the Tek-Tips community for having the most helpful posts in the forums last week. Way to Go!
ftp upload completion 1
- Thread starter arcnon
- Start date
There's an easy way to do this if you have control over the process that is sending the file.
The process sending the file using ftp should do this:
Create a lock file (file.lck maybe)
Send the real file.
Delete the lock file.
The process waiting for the real file should only take it if there is no lock file.
Mike
Want to get great answers to your Tek-Tips questions? Have a look at faq219-2884
It's like this; even samurai have teddy bears, and even teddy bears get drunk.
The process sending the file using ftp should do this:
Create a lock file (file.lck maybe)
Send the real file.
Delete the lock file.
The process waiting for the real file should only take it if there is no lock file.
Mike
Want to get great answers to your Tek-Tips questions? Have a look at faq219-2884
It's like this; even samurai have teddy bears, and even teddy bears get drunk.
- Thread starter
- #3
OK maybe I was not real clear about my question. I am polling my FTP directory every second to see if there is a file/files there.
The files that are uploaded there come from a scanner cluster with a proprietary language that I cannot make changes to even if I wanted to. The company that owns them has setup my ftp directory as the place to dump them. This is a good thing because I can use my log files as a support to error checking, etc etc.
Each scan will be about 8mg so I want to be able to make sure they are finished uploaded before I move them to other directories or ftp them to a differant server for processing.
The reason that I dont try and do it at say 12 midnight is the forwarded scans are then ocr'ed and a reply is sent back to original location.
The files that are uploaded there come from a scanner cluster with a proprietary language that I cannot make changes to even if I wanted to. The company that owns them has setup my ftp directory as the place to dump them. This is a good thing because I can use my log files as a support to error checking, etc etc.
Each scan will be about 8mg so I want to be able to make sure they are finished uploaded before I move them to other directories or ftp them to a differant server for processing.
The reason that I dont try and do it at say 12 midnight is the forwarded scans are then ocr'ed and a reply is sent back to original location.
SirCharles
Programmer
How about a system call?
$test = `ps -ef |grep ftp`;
$test = `ps -ef |grep ftp`;
- Thread starter
- #5
in this test will/should it return
the filename,0,1,true,false
I think I might understand the call
ps postscript?
ef end of file?
lgrep to communicate the 2 afore mentioned attributes to my ftp server?
or is $test = $the_name_of_the_file
and it returns a $_[0]; or $ARGV[0];
thanks for the help cw
the filename,0,1,true,false
I think I might understand the call
ps postscript?
ef end of file?
lgrep to communicate the 2 afore mentioned attributes to my ftp server?
or is $test = $the_name_of_the_file
and it returns a $_[0]; or $ARGV[0];
thanks for the help cw
ps is how to return a list of all running processes on *nix box
-ef are switches that are passed to it
The results of this are piped to the command grep ftp which will return any related results with "ftp" in it. Word of warning though if you have an ftp daemon running you may need to check that ftpd is not going to affect your decision to copy the file.
If you're asking these questions is it safe to assume you're on windows? Chances are if you try to copy a file before completion the OS will chuck an error, so check the results of a 'move' while a file is being copied
HTH
==Paul
-ef are switches that are passed to it
The results of this are piped to the command grep ftp which will return any related results with "ftp" in it. Word of warning though if you have an ftp daemon running you may need to check that ftpd is not going to affect your decision to copy the file.
If you're asking these questions is it safe to assume you're on windows? Chances are if you try to copy a file before completion the OS will chuck an error, so check the results of a 'move' while a file is being copied
HTH
==Paul
- Thread starter
- #7
arcnon,
In a similar situation, and with timescales suck it and see works best for me, have a go, if it doesn't pass your testing (8meg - concat about 100 together) to amke sure your move/rename and ftp occur in the same time slice
HTH
--Paul
In a similar situation, and with timescales suck it and see works best for me, have a go, if it doesn't pass your testing (8meg - concat about 100 together) to amke sure your move/rename and ftp occur in the same time slice
HTH
--Paul
Arcnon,
I understood your question, I just didn't answer it real well.
The problem is that there's no reliable way (AFAIK) to tell if FTP has finished delivering a file.
In the past I've used two methods.
1 - The method I described above - a good and reliable method but it only works if you have control of the program delivering the file with FTP
2 - "Watch the size of the file". If you can say - if the size of this file doesn't change for 30 seconds, then FTP has finished - then you can do that. The only problem is that FTP might *not* have finished...
Mike
Want to get great answers to your Tek-Tips questions? Have a look at faq219-2884
It's like this; even samurai have teddy bears, and even teddy bears get drunk.
I understood your question, I just didn't answer it real well.
The problem is that there's no reliable way (AFAIK) to tell if FTP has finished delivering a file.
In the past I've used two methods.
1 - The method I described above - a good and reliable method but it only works if you have control of the program delivering the file with FTP
2 - "Watch the size of the file". If you can say - if the size of this file doesn't change for 30 seconds, then FTP has finished - then you can do that. The only problem is that FTP might *not* have finished...
Mike
Want to get great answers to your Tek-Tips questions? Have a look at faq219-2884
It's like this; even samurai have teddy bears, and even teddy bears get drunk.
-
1
- #10
Hi,
Look at the output of the following command :
ps -ef --cols 300 | grep myfilename
returns
username 23115 22308 6 15:24 ? 00:00:14 ftpd: myclient.machine: ftpusername: STOR myfilename
username 23153 22839 0 15:30 pts/1 00:00:00 grep myfilename
This was while downloading the file "myfilename".
So this command returns at least two lines. If the file is currently downloading, you may find one of the lines matching the pattern "STOR".
So what you have to do in your perl script is to set your grep command :
$filename = "myfilename";
$command = "ps -ef --cols 300 | grep $filename" ;
You have to catch the results of this command in an array :
@results = `$command`;
You have to see if one of these lines matches the "STOR" string :
foreach $line (@results) {
if ($line~=/STOR/) {print "Still downloading $filename \n";}
}
All what remains is to write a script with nice loops & checks
Hope this help
Look at the output of the following command :
ps -ef --cols 300 | grep myfilename
returns
username 23115 22308 6 15:24 ? 00:00:14 ftpd: myclient.machine: ftpusername: STOR myfilename
username 23153 22839 0 15:30 pts/1 00:00:00 grep myfilename
This was while downloading the file "myfilename".
So this command returns at least two lines. If the file is currently downloading, you may find one of the lines matching the pattern "STOR".
So what you have to do in your perl script is to set your grep command :
$filename = "myfilename";
$command = "ps -ef --cols 300 | grep $filename" ;
You have to catch the results of this command in an array :
@results = `$command`;
You have to see if one of these lines matches the "STOR" string :
foreach $line (@results) {
if ($line~=/STOR/) {print "Still downloading $filename \n";}
}
All what remains is to write a script with nice loops & checks
Hope this help
While it is hard for a watcher to determine if an ftp xfer goes to successful completion, the ftp protocol knows.
if you can (re)install the ftp server (ftpd), take a look at wu-ftp( It appears to have extensive logging facilities which might answer the "is it EOT yet?" query.
if you can (re)install the ftp server (ftpd), take a look at wu-ftp( It appears to have extensive logging facilities which might answer the "is it EOT yet?" query.
gordonisnz
Programmer
How is the 'company' FTP'ing the original file ?
if its manual - They COULD FTP a 2nd smaller (1 KB) file - eg. finish.txt as well - AFTER the original file is copied
- You then do a check on if BOTH files exist - then do your transfer (& delete the finish.txt)
- Just a suggestion - Not sure if the company is manual or automatic
G.
if its manual - They COULD FTP a 2nd smaller (1 KB) file - eg. finish.txt as well - AFTER the original file is copied
- You then do a check on if BOTH files exist - then do your transfer (& delete the finish.txt)
- Just a suggestion - Not sure if the company is manual or automatic
G.
- Thread starter
- #13
The system that is sending them is a scanner these scanners are at different locations physically these scanners scan the image and transfers them to a central location....a linux box.
from this point it is all me until I send them to their resting place
1. check if it is downloaded complete
2. check validation (make sure that the files are not overwritting one another)
3. archieve
4. check approm. make sure I am not sending a file that is half there
5. output to the screen the stats to the screen and to a log file
6. move a copy to a differant server
7. send a notification to a ocr program that a file is ready to process
none of it is that difficult except the checking of the file to see if it is all there
After reading some stuff & reading some posts. I am starting to think that my best solution is to write my own ftp server. If I am processing the whole thing when close(socket) is sent I will know that the file has finished.
Which maybe the deal a way to ask ftpd->fork->child->was deleted
The bad part I have looked through my perl & cgi books and there are server samples but none that seem to mesh well. I know that there is telnet protocal that must be observered. I am sure there has to be some type of error checking. Etc blah blah.
Maybe I should be asking what the best approach is instead how to follow my own apprearent faulty thinking.
Once again thanks very much for your attention to my post. And offering help.
from this point it is all me until I send them to their resting place
1. check if it is downloaded complete
2. check validation (make sure that the files are not overwritting one another)
3. archieve
4. check approm. make sure I am not sending a file that is half there
5. output to the screen the stats to the screen and to a log file
6. move a copy to a differant server
7. send a notification to a ocr program that a file is ready to process
none of it is that difficult except the checking of the file to see if it is all there
After reading some stuff & reading some posts. I am starting to think that my best solution is to write my own ftp server. If I am processing the whole thing when close(socket) is sent I will know that the file has finished.
Which maybe the deal a way to ask ftpd->fork->child->was deleted
The bad part I have looked through my perl & cgi books and there are server samples but none that seem to mesh well. I know that there is telnet protocal that must be observered. I am sure there has to be some type of error checking. Etc blah blah.
Maybe I should be asking what the best approach is instead how to follow my own apprearent faulty thinking.
Once again thanks very much for your attention to my post. And offering help.
- Thread starter
- #19
Hi All,
I have an identical problem as the originator of the post, except that I am on an NT system. I need to wait until the ftp transfer has finished downloading the pdf to the folder and then print it. I figurered that the size check thing is the way to go as well. When I was working with Unix I could have managed it with a c-shell script, but now we are on NT I have started to look at Perl, but simple things like file/directory sizes are not so simple.
Does anyone know how to write this code in Perl on NT?
Cheers
Jimbo
I have an identical problem as the originator of the post, except that I am on an NT system. I need to wait until the ftp transfer has finished downloading the pdf to the folder and then print it. I figurered that the size check thing is the way to go as well. When I was working with Unix I could have managed it with a c-shell script, but now we are on NT I have started to look at Perl, but simple things like file/directory sizes are not so simple.
Does anyone know how to write this code in Perl on NT?
Cheers
Jimbo
- Status