When Is A File Done Transferring?

[menkes]

I have a simple script that scans FTP directories every 15 minutes for new files. When a file is found the script:
1. puts the file name in an array
2. moves/renames the file to another server
3. queries a DB to see who the file belonged to
4. sends an email listing the owner/file pairs

My problem comes when I receive large files. Several times a month I receive files in excess of 500MB. What happens is the notice email is sent that the file has arrived, but the file has not fully arrived yet. So, I may get 8 notices (if it takes 2 hours to receive the file) before the file is fully received.

How do I validate that the entire file is present before adding its name to the array (Step 1 above)??

 

[Wonboodoo]

Not sure if Perl has an equivalent function, but are you running on a Unix-like system. A number of Unixes (not all) have an "fuser" command that will show you the process IDs that currently have the file open. You can call "fuser" from your Perl script.

 

[menkes]

Great tip. Unfortunately, I am shackled to WinDoze.

 

[icrf]

I don't think there's anything in the ftp spec that allows a query like that. The best I can think of is to get the filesize of the remote file, sleep for five seconds or something, then check again for any difference. I don't know if that's a big enough time window for the remote user to update the file, or if the ftp server itself updates that quickly, but I'd assume yes to both. ----------------------------------------------------------------------------------
...but I'm just a C man trying to see the light

 

[arn0ld]

the 2 classic solutions are
sender puts a "done" file at end
sender renames file at end

 

[menkes]

Thanks for the suggestions. Unfortunately, they would not fit in this situation. I have thought it through and what I think I will try is moving the file first, and if the move is successful, then I will update the notification array.

 

[arn0ld]

if you are on *nix, the move will always work but this is no indication of EOT. FTP will know the file by its inode. A mv is a "rename" (at least in the same file system).
The inode will be unchanged. In fact, you can remove the file and it will still keep growing until FTP ends - then it will be gone.

PS what is your OS? If *nix, what FTP server? - there may be logs