Help with HTTP requests...

[Numbski]

I've got a doozie on my hand. I need to be able to do the following:

1. Request a text document via HTTP, open it and loop through it searching for strings. I've been reading the documentation for ActivePERL and I haven't been able to get it working.

2. Request a binary file via HTTP and save it to a pre-specified location on the hard drive.

3. Be able to set up a list of requests, and be able to queue them so only 2 go at a time. If I set up an array of URL's I want to download and just use a foreach() command to do them all, it'll try to do them all at once. I want them to fire two at a time, when one completes begin the next.

I have absolutely NO clue of how to do these things. I've looked around for code samples and have run into very limited success. Anyone?

 

[hmerrill]

I'm guessing that more info from you migh help.

1. What platform are you on? I assume Windows or NT
since you need ActivePerl, but am I right?

2. package LWP::Agent or LWP::Simple might be useful for
retrieving web pages - then there are other packages
that can help you parse the HTML.

On Perl 5.6.0 on Linux, I can do "perldoc LWP::Simple" to get a description of the LWP::Simple module. Once you have ActivePerl installed, I believe you can see the perldocs for LWP::Simple by bringing up a DOS window and doing "perldoc LWP::Simple" at the command line - assuming LWP::Simple is included in the version of ActivePerl that you're installing. If not, you can get LWP::Simple from CPAN(

http://www.cpan.org).

3. not sure why you need to queue requests - you could set up an array of request URL's, and request each of them in turn, but I don't know how you'd queue them. Can you give more info on this?

HTH.
Hardy Merrill
Mission Critical Linux, Inc.

http://www.missioncriticallinux.com

 

[Numbski]

I need to queue the requests because the server on the other end can only send 2 at a time. These are rather large binary files that will take some time to download. If I try to get them all going at once from array cycling in turn, most of them will time out.

I think I've figured out how to save the binary files. If I recieve the binary file in an array, I would open it's filename in the location I need it. Then do a foreach{print} on it. Close the file.

You don't happen to have a code snippet on how to request a document and store it in an array, each line an element?

 

[Numbski]

FYI...it's being coded in active perl on NT, and will be compiling a binary for Win32 and Linux.

 

[Numbski]

We'll make this a little more simple. The only thing I need to do now is save a binary file to the hard drive. I got everything else figured out. Using LWP::Simple is great BTW.

Here's what's happening:

$binary_file=get("

http://www.mydomain.com/happy_little_binary.zip");

I try to do this:

open (ROMFILE,">$rom_filename&quot || die "Can't Open $rom_filename: $!\n";

print ROMFILE $binary_file;
close(ROMFILE);

This does in fact save the data to a specified filename, HOWEVER...it's corrupted. I've tried doing binmode(print ROMFILE $binary_file);, but that doesn't help.

 

[hmerrill]

Sorry I can't really be much more help - I haven't used LWP::Simple to retrieve binary files.
Hardy Merrill
Mission Critical Linux, Inc.

http://www.missioncriticallinux.com

 

[tsdragon]

Have you tried just doing binmode on the filehandle?

open (ROMFILE,">$rom_filename") || die "Can't Open $rom_filename: $!\n";
binmode ROMFILE;
print ROMFILE $binary_file;
close(ROMFILE);

Tracy Dryden
tracy@bydisn.com

http://www.bydisn.com

Meddle not in the affairs of dragons,
For you are crunchy, and good with mustard.