Ideas for splitting up a script

[nfaber]

Hello all,

I have a program I wrote that performs multiple database queries for a list of about 30 companies I get from opening a file. My problem is that the program takes over an hour to run for each company due to a slow database. In addition, the database is only accesible between 1:00 PM and 12:00 AM every day. What I want to do is run the program for say 10 companies at a time and then the next night another 10 the next night another 10 etc. This would not be a problem with a counter, however I need some way to keep track of what companies it has already ran for and start where it left off the next time it runs.

The program creates output files for each company, but then transfers the file to another server and deletes them. Below is the file I am opening to get the company names. If anyone has done something like this before and has any suggestions, it is appriciated.

Thanks,

Nick

Company Key	Company Name			Assignment Group Exceptions

coc,		CHICAGO,
ccc,		IRC COMCAST,
com,		CITY OF MINNEAPOLIS,
cpd,		CPD,
cta,		CTA,
gld,		GAYLORD PALMS,
hgi,		HARLEYSVILLE,
fya,		HEALTH AND HUMAN SERVICES,
ipa,		INTERNATIONAL PAPER,
ipi,		INTERNATIONAL PAPER,
lal,		LEADER,
lmr,		LE MERIDIEN,
mmc,		MAIMONIDES,
mdc,		MIAMI DADE COUNTY,
mwd,		MORGAN STANLEY,			MWD-T3-SRV MWD-T3-NET			
npa,		PORT AUTHORITY,
nwa,		NWA,
pjc,		PIPER JAFFRAY,
plx,		PLEXUS,				PLX-SMSL1 PLX_SMSNee
shr,		STARWOOD,
sp,		SUBURBAN PROPANE,
thc,		SWAN DOLPHIN,
trc,		ROUSE,
net,		NETWORKS,
gfy,		TSA,
rwb,		DHS,
ihd,		IHDHS,
glb,		GLOBAL PETROLEUM,

 

[TonyGroves]

You could maintain an XML file containing that information, along with the date each company was last processed.

Your program could then use XML::Simple to read the file into a tree of arrays and hashes, determine which companies to process, update their dates, and write the file back out again.

 

[tchatzi]

open DONE, ">>comp_done.txt";
while (<DONE>){
push (@companyssssssss,$_);
}
£last_company= £companysssssss[$#companyssssss];


....connect to the database ......and find the record which
matches the £last_company........
....start reading database from that line an after....

while (reading from database){
.....whatever else you want to do........

print DONE the_key_and_whatever,"\n";
.....whatever else you want to do........
}

close DONE;

Is something like this, that you are looking for?
Else you have to tell us what kind of db are you using, what kind of modules to connect to it, and of course some of your code to adjust it into.