File System Concurrency 1

[divinecomedy]

Hi, I am a Perl newbie so please bear with me. I have two scripts both of which are fired up from unix cron. One of them will write some files to a directory while the other one will FTP them to somewhere else. Assuming script 1 ( reading ) and script 2 ( writing ) are doing their stuff at the same time, can it be that script 1 will FTP an incomplete file or is the file system locked when script 2 is doing the writing ?

Due to requirements I can't combine the two. Please offer any possible solutions. Thanks.

 

[icrf]

Depends on the OS, but you could try flock()ing:

http://www.perldoc.com/perl5.8.0/pod/func/flock.html

----------------------------------------------------------------------------------
...but I'm just a C man trying to see the light

 

[divinecomedy]

Actually I should be more clear. Script 1 will actually do a copy from one directory to another directory ( the file has not existed yet ). Since I have not gotten the new file handle yet, I can't lock it, can I ?

Let me also give more details on what Script 2 is doing. It will read the directory information and see what files are under that then it will try to copy the files one by one.

Obviously the answer to this question is when will the copied file name be visible to Script 2 ? Is it after all the copying is done or sometime before ?

I am running Unix.

 

[icrf]

I would say almost certainly as soon as it starts copying. But I also assume that the first script's transfer rate is significantly higher than the second's (local file copy versus remote ftp copy). If that's the case, script2 should be able to start uploading as soon as script1 starts copying, as it's just reading the file, not writing it. So long as script1 can keep ahead, I think script2 will never know the difference.

I don't know, only real way it to test, make them run concurrently and see what happens. I'm going to guess that you won't have any problems. As the file gets bigger, EOF keeps moving, too.

----------------------------------------------------------------------------------
...but I'm just a C man trying to see the light

 

[MikeLacey]

This is always a problem, in the past I've gotten around it in three ways.

1 - Only get the file when you know it will be completed, an hour after it was put there for instance. This is the easiest solution.

2 - Create the file with a temporary name, when the put is complete rename it to its permananent name. The program that picks it up will only see it when it's complete.

3 - Create a lockfile before put'ting the file, don't let the collector program get the file while the lockfile is there. Only delete the lockfile when the put is complete.

I usually go for 2, 3 is useful when transferring a lot of files - it just saves you having to rename a load of files.

Mike

Want to get great answers to your Tek-Tips questions? Have a look at faq219-2884

It's like this; even samurai have teddy bears, and even teddy bears get drunk.