perl browser 1

[SirCharles]

script to interface with web server. I'd like to interface with the web server via a perl script rather than a browser. So, browser calls perl cgi script, then perl makes http call to another browser, passing get and post vars along with a url. Then perl script receives response from web server. I guess what this means is writing a mini- kind of - web browser type functionality in perl. Would this require xml or is there some perl module (http module) that could do this type of interface?

I'd really like to keep it very simple stupid kind of thing here.

 

[SirCharles]

Oops I meant to say above that " then perl makes http call to a web server ..." not another browser.

 

[siberian]

lib

http://www is

your thing for this, it works great.

http://search.cpan.org/~gaas/libwww-perl-5.72/lib/LWP.pm

 

[goBoating]

#!perl
use LWP::Simple;
my $page = get('[URL unfurl="true"]http://www.csc.noaa.gov/');[/URL]
print $page;

'hope this helps

If you are new to Tek-Tips, please use descriptive titles, check the FAQs, and beware the evil typo.

 

[SirCharles]

Thanks for your prompt replies. Looks like the LWP module is the ticket, so far.

 

[SirCharles]

How would someone post to a url where a password and username is required?

 

[siberian]

http://username:password@www.yahoo.com/goehad/and/login/

This is ASSUMING that normal http authentication is being used.

If the page uses custom login code you may have to post/get the arguments the custom code is expecting to receive. This is unique to the implementation.

The way to tell this is if you login on a webpage or via the pop-up box. If its the webpage you need to view source, get the FORM URL, find any INPUT fields and then construct a request that emulates that form based login.

Even then it may not work if the person who authored the login code did a remotely competent job and is checking the HTTP_REFERER variable.

So, if its standard http auth the first option works. if its login via a custom form and software then its totally situational.

 

[SirCharles]

I'm getting a 401 authorization error. However,I can put the http ref on my url in the browser and it gets to the page just fine.
#!/usr/bin/perl

use Time::localtime;
use CGI::Enurl;
use LWP::Simple;

# Create a user agent object
use LWP::UserAgent;
$ua = LWP::UserAgent->new;
$ua->agent("MyApp/0.1 "

# Create a request
# This required authorization, but next error said url must
# be absolute.
$req = HTTP::Request->new(GET => '

http://username:password@site/path/to/cgi');

# send request
$res = $ua->request($req);

# check the outcome
if ($res->is_success) {
#print $res->content;
print $res->as_string;
} else {
print "Error: " . $res->status_line . "\n";
}

 

[SirCharles]

Apache displays authentication window for login to happen.

 

[siberian]

Try this (from the manpage)

$ua->credentials($netloc, $realm, $uname, $pass)
Set the user name and password to be used for a realm.
It is often more useful to specialize the
get_basic_credentials() method instead.

 

[SirCharles]

Thanks, that worked for a simple get and post.
How would you post multipart form data such as a file?
Can assume we've already got the post data in the %in hash.
So, how would we post multipart form data to another site?

Also, how's the stuff look below? It apears to work ok.

Reference:
# *************** Get example:
#!/usr/bin/perl

use Time::localtime;
use CGI::Enurl;
use LWP::Simple;

# Create a user agent object
use LWP::UserAgent;
$ua = LWP::UserAgent->new;
$ua->agent("MyApp/0.1 "

# Create a request

$req = HTTP::Request->new(GET => '

http://www.site.com');

# establish authentication credentials
# NOTE:
# To locate needed info for call to credentials, from
# command-line:
#lwp-request -e -a

http://sitename.com

#
$ua->credentials(
'desintationSite.com:80',
'Ream name goes here - ie. name of site appearing as realm in Apache authentication window.',
'username' => 'password'
);

# send request
$res = $ua->request($req);

# check the outcome
if ($res->is_success) {
print $res->content;
#print $res->as_string;
} else {
#print $res->content;
print "Error: " . $res->status_line . "\n";

# For a post:
print '**************Post with LWP*********************\n';
my ($content, $messasge, $is_success) = do_POST('

http://sitename.ext/path/to/cgi',

[ 'key1' => "value1", 'key2' => "value2" ],);
print "2:".$content if $is_success;
print "3:".$message;

#==========================
# do_POST
#==========================
sub do_POST {
use LWP;
# Parameters:
# the URL,
# an arrayref or hashref for the key/value pairs,
# and then, optionally, any header lines: (key,value, key,value)
my $browser; # good to have this as local var here?
$browser = LWP::UserAgent->new( ) unless $browser;
$browser->credentials(
'sitename.com:80',
'Realm goes here again',
'username' => 'password'
);

my $resp = $browser->post(@_);
return ($resp->content, $resp->status_line, $resp->is_success, $resp)
if wantarray;
return unless $resp->is_success;
return $resp->content;
}

 

[siberian]

Try using HTML::Form. This has a method for setting the enctype which will enable your file upload.

http://search.cpan.org/~gaas/libwww-perl-5.73/lib/HTML/Form.pm

 

[SirCharles]

I need to load an array and pass it to POST request as in
$res = $ua->post($urlHref, split (/&/, $envArray));

How to programatically load the array?
If the string I want to split to an array looks like this
ke1=value1&k2=value2&k3=value3

 

[siberian]

It looks like you have a standard CGI Query string you want to split up?

If thats the case use CGI.pm

my $query = new CGI( $string ) ;
# string is ke1=value1&k2=value2&k3=value3

Then

my @names = $query->param

Then create your hashref

my $hashref ;

foreach(@names){
$hashref->{ $_ } = $query->param( $_ } ;
}

Then pass in the $hashref.

Maybe.. I did not actually run this..

 

[SirCharles]

Thanks Siberian. I didn't try that but found that following appears to work:
@getParams = split /&/, $envArray;
$res = $ua->post($urlHref, \@getParams);

I was leaving out the leading '\'.

Since you're here and talking about hash refs. I've been able to get a hash ref with Php for a fetch statement using dbi, but have not seen where I can get the same sort of things using PERL. So far, I've had to use an array reference in fetching rows with DBI. Got any ideas on syntax to use hashref instead? Or should I post another question on this.

I've still got to figure out how to post files using LWP and PERL. Am having a dickens of a time figuring this stuff out, working into all hours of the morning. Appreciate any advice.

 

[siberian]

You can fetch out of DBI row by row with a hashref but the only way to get all results in a single fetch is with an arrayref as you describe.

Here are is a google URL that I would use as an example

http://ftp.ics.uci.edu/pub/websoft/libwww-perl/archive/1997h2/0479.html

 

[SirCharles]

Thanks again.
Correction to previous post. I should have substituted
= for & before the split. This works!! finally.
$tmpString =~ s/=/&/g;
@getParams = split /&/, $tmpString;
$res = $ua->post($urlHref, \@getParams);

Now on to how to post files with LWP. Looks like I may have a reference to look at, but any help here would be great.

 

[siberian]

Try here, this is a straight out example:

http://kochi.etla.org/london.pm/html/2000/08/msg00576.html

Here is another more comprehensive method

http://perlmonks.thepen.com/33048.html

 

[SirCharles]

In one scenario, I'll redirect the user to another page using LWP. So, I'd like to get the user credentials in order to authenticate the user. Does PERL have a way to get the credentials from an HTTP_SERVER_VARS hash or from the httpd password file (decrypted password)?

Also, is there a way to tell if user is using http or https before making call to LWP?

 

[siberian]

You can do all this and more with ModPerl but not CGI based perl.

This is not perl's fault, CGI/Perl/PHP/Java/WHatever runs during a specific phase of the HTTP request that occurs AFTER the authentication phase.

ModPerl allows you to intercept any phase and make it do nifty stuff.

BTW : This statement above

"Thanks again.
Correction to previous post. I should have substituted
= for & before the split. This works!! finally.
$tmpString =~ s/=/&/g;
@getParams = split /&/, $tmpString;
$res = $ua->post($urlHref, \@getParams);
"

Is a glowing example of why its not good to roll your own common code, just use CGI.pm!