Log In

Come Join Us!

Are you a
Computer / IT professional?
Join Tek-Tips Forums!
  • Talk With Other Members
  • Be Notified Of Responses
    To Your Posts
  • Keyword Search
  • One-Click Access To Your
    Favorite Forums
  • Automated Signatures
    On Your Posts
  • Best Of All, It's Free!

*Tek-Tips's functionality depends on members receiving e-mail. By joining you are opting in to receive e-mail.

Posting Guidelines

Promoting, selling, recruiting, coursework and thesis posting is forbidden.

Students Click Here

LWP::UserAgent with https question

LWP::UserAgent with https question

LWP::UserAgent with https question

greetings everyone,

I have to connect to an https site, log in, browse to a particular page, input field selections, and extract certain information from it.  I wish to automate this process.  I am, however, running into a problem.  Here is what I have so far



use strict;

use URI::URL;
use LWP::UserAgent;
use HTTP::Request;
use HTTP::Request::Common;
use HTTP::Request::Form;
use HTML::TreeBuilder 3.0;
use HTTP::Cookies;

my $cookie_file = 'cookies.txt';
my $ua = LWP::UserAgent->new;

my $url = url 'https://some.page.com/UILogon.aspx?ReturnUrl=https%3a%2f%2fsome.page.com%2blah%2fFolder.aspx';

$ua->cookie_jar( HTTP::Cookies->new( file => $cookie_file ));

my $res = $ua->request(GET $url);
my $tree = HTML::TreeBuilder->new;


my @forms = $tree->find_by_tag_name('FORM');
die "No forms found" unless @forms;
my $f = HTTP::Request::Form->new($forms[0], $url);

$f->field("user", 'my_username');
$f->field("pass", 'my_password');

my $response = $ua->request($f->press("submitButton"));

print $response->status_line;
print $response->content;

When I use a browser to log in, I see a Folder page.  When I look at the output of the last line ($response->content), I  see that it's the html of the initial login page.  It's as if it simply doesn't see me passing my credentials.  If it did, it would give me an "Invalid credentials" page html.   Aldo, the status code is 302 FOUND, and the headers start with a redirect message of "Object moved to...".  It's not generating the cookies.txt file either, and I am assuming it needs it, considering I see a bunch of authentication info in my Firefox cookies after I go to the website.  What am I doing wrong here?  Thank you for any help or pointers

RE: LWP::UserAgent with https question

Ok, I figured out why it wasn't automatically handling the redirect.  Internally, the site uses POST, and POST is not redirected by default.  So have to do


push @{ $ua->requests_redirectable }, 'POST';

Red Flag This Post

Please let us know here why this post is inappropriate. Reasons such as off-topic, duplicates, flames, illegal, vulgar, or students posting their homework.

Red Flag Submitted

Thank you for helping keep Tek-Tips Forums free from inappropriate posts.
The Tek-Tips staff will check this out and take appropriate action.

Reply To This Thread

Posting in the Tek-Tips forums is a member-only feature.

Click Here to join Tek-Tips and talk with other members! Already a Member? Login

Close Box

Join Tek-Tips® Today!

Join your peers on the Internet's largest technical computer professional community.
It's easy to join and it's free.

Here's Why Members Love Tek-Tips Forums:

Register now while it's still free!

Already a member? Close this window and log in.

Join Us             Close