Download authorized pages via LWP

[yerman01]

Hi everyone!
I've developed a little perl program using the module LWP to download italian stock pages,
to make real time statistics.
The program works for delayed data, but it doesn't work for real time data because of the
authentication.
I mean the authentication is made at session level, so I need to authenticate in some
way in the same program.
I've tried to use $ua->credentials and $req->authorization_basic, but without results.
I confess I'm novice in perl programming and in using LWP, and surely it's my fault.
Can anyone help me?

My simple code is:

$protected_page="

http://www.24oreborsaonline.ilsole24ore.com/fc?sezId=10026&cmd=miafinanza&page=ListinoPersonale";

use LWP::UserAgent;
$ua = LWP::UserAgent->new;

#
# Authentication block ???
#

$req = HTTP::Request->new(GET => "<$protected_page>");

$res = $ua->request($req);
if ($res->is_success)
{
$out="c:/mypage.txt";
open OUT, ">$out" or die "Cannot open $out for write :$!";
print OUT $res->content;
close OUT;
}
else
{
print "Time out: fault in download \n";
}

If you would like to test if it works I can leave the logging credentials.

Site logging page:

http://www.24oreborsaonline.ilsole24ore.com/fc?cmd=index&chId=43

username: yer
password: man

I thank you in advance for whatever help you can give me.

 

[tchatzi]

You dont need a user name nor a password for this page.

http://www.24oreborsaonline.ilsole24ore.com/fc?cmd=index&amp;chId=43

To fetch a page with needs credent.. the link has to be of the page that opens after you login. Not the page where you put your username & password.
Is that understandable?

 

[yerman01]

Thank you perluserpengo for your interest.

I understand what you say, but the link

http://www.24oreborsaonline.ilsole24ore.com/fc?cmd=index&amp;chId=43

was just to allow an interactive login to try the credentials.

The protected page was specified in the code with the variable $protected_page.

Thank you again.

 

[threetone]

I recently wrote a script to access a site I have a membership to. It was a secure site with a login, but didn't use simple authentication that credentials would work for. The login page sent a session cookie that was appended to the login form action when you log in. So, every time I access the site, I get the login page first, parse out the cookie from the header and add it manually to the login form action, then I make a post to get the info I really want. I activated a cookie jar file, and caching, and all of that together let me access secure pages once I had first logged in.

here is my code:
[tt]
use LWP 5.803;
use LWP::ConnCache;
use HTTP::Cookies;

# Create a new UserAgent
my $browser = LWP::UserAgent->new;
$browser->agent('Name your user agent whatever you want');
$browser->conn_cache(LWP::ConnCache->new()); # Set up the cache
$browser->conn_cache->total_capacity(undef); # cache unlimited
# create a file for cookiejar
$browser->cookie_jar(HTTP::Cookies->new(
'file' => 'e:/web/whatever_folder/cookies.lpw', 'autosave' => 1
));

# Get the login page
my $resp = $browser->get($login_url);

# Pull the session cookie out of the headers (this gets any and all cookies)
my $headers = $resp->headers_as_string;
my %cookies;
while($headers =~ m/Set-Cookie: (.*?)=(.*?);/g) {
my $cname = $1;
$cookies{$cname} = $2;
}
# Add the sessionid to the login action url (login action set previeously)
$login_action .= ';jsessionid=' . $cookies{'jsessionid'};

# Log in
sleep 3;
$resp = $browser->post($login_action,[%login_data]); # %login_data setup previously

# Now, access whatever protected content with $browser
sleep 3;
$resp = $browser->post($get_report_url,[%form_data]); # %form_data setup previously
my $report = $resp->content;
[/tt]

 

[yerman01]

Dear threetone,
thank you for your contribution.
I've almost understood your approach (unfortunately programming is not my field, being mainly interested in statistics and economy), and I've customized my url.
About the variables [%login_data] and [%form_data] I haven't understood how to specify them.
I know I should debug and go in deep of LWP, but it's not so simple, knowing perl in very novice way.

If it comes easy for you, I would be very grateful if you could drop some lines for my case.
As I wrote:
- the protected url is

http://www.24oreborsaonline.ilsole24ore.com/fc?sezId=10026&cmd=miafinanza&page=ListinoPersonale

- the login page is

http://du.ilsole24ore.com/du/login.aspx?

- user = yer
- pass = man

In any case I thank you very much for your support.