HTTP Request

[vne147]

Hello all. I am trying to go to a webpage and save the html source. My code is below:

my $cookie_jar = HTTP::Cookies->new(file => "lwp_cookies.dat",autosave => 1);
my $agent = new LWP::UserAgent();
$agent->cookie_jar($cookie_jar);
$agent->credentials($URL,'','username','password');
my $request = new HTTP::Request( 'GET' => $URL );
$cookie_jar->add_cookie_header($request);
$request = new HTTP::Request( 'GET' => $URL );
my $response = $agent->request( $request );
my $html = $response->content();

So, the contents of source html of $URL should be stored at $html, right? The problem I'm running into is that $URL is a webpage on a secure site. So, when I try to go directly there the PERL script gets redirected to an authentication page. However the script isn't going to the redirected location and the only thing that gets stored at $html is a webpage with the word "Redirect" in the body. I would be greatful to anyone who can shed some light on this for me. I have alot of experience with PERL but next to none with HTML and dealing with HTTP requests. Thanks in advance.

 

[PaulTEG]

$URL=?

more code might be an idea ...

 

[KevinADC]

$URL is a webpage on a secure site

 

[vne147]

Thanks PaulTEG, there is more code. I didn't think I needed to include it all. $URL is the url of the site I am trying to download files from. Do you need to know the value of $URL to help troubleshoot my problem? Not sure what KevinADC is trying to tell me but thanks anyway for the reply. Please let me know if more information is needed. Thanks.

 

[PaulTEG]

The more code we have, the more we can help
Kevin is just pointing out a statement you made in your post, but if we don't know how $URL is made up, as in how the string is constructed we're gonna have diffs helping


DO obfucsate machine names, networks, usernames and passwords ...

Paul
------------------------------------
Spend an hour a week on CPAN, helps cure all known programming ailments ;-)

 

[vne147]

PaulTEG, thanks again for your reply. I'm sorry if I wasn't giving you enough information before and I did not mean to obfuscate (had to look it up) my question. I'm asking this question for a work related project and I wasn't sure how much info I was allowed to give out concerning the project. Below is all of my code. With the exception of 'username' & 'password' it is exactly the same in my script. Any help you could provide would be much appreciated. Thanks, and hope to hear from you soon.

use strict;
use warnings;

use LWP;
use LWP::Simple;
use LWP::UserAgent;
use HTTP::Request::Common;
use HTTP::Cookies;

our $URL = '

http://p51.ksc.nasa.gov/aps/Daily_Report/ContentListing.cfm?Orbiter=105';

&connect_url;

sub connect_url {

my $cookie_jar = HTTP::Cookies->new(file => "lwp_cookies.dat",autosave => 1);

my $agent = new LWP::UserAgent();
$agent->cookie_jar($cookie_jar);
$agent->credentials($URL,'','username','password');

my $request = new HTTP::Request( 'GET' => $URL );
$cookie_jar->add_cookie_header($request);
$request = new HTTP::Request( 'GET' => $URL );

my $response = $agent->request( $request );

if ( !$response->is_success() ) {
print("HTTP Error!\n");
}

else {
my $html = $response->content();
my $output = new IO::File "data.html", "w";
print $output "$html";
}

}#end connect_url