INTELLIGENT WORK FORUMS
FOR COMPUTER PROFESSIONALS

Log In

Come Join Us!

Are you a
Computer / IT professional?
Join Tek-Tips Forums!
  • Talk With Other Members
  • Be Notified Of Responses
    To Your Posts
  • Keyword Search
  • One-Click Access To Your
    Favorite Forums
  • Automated Signatures
    On Your Posts
  • Best Of All, It's Free!

*Tek-Tips's functionality depends on members receiving e-mail. By joining you are opting in to receive e-mail.

Posting Guidelines

Promoting, selling, recruiting, coursework and thesis posting is forbidden.

Jobs

LWP::Useragent causing HTTP 501 error

LWP::Useragent causing HTTP 501 error

(OP)

Hi,

The following block of code is causing a "501 Protocol scheme 'http' is not supported" error. Any suggestions as to why this might be happening, the version of perl is pretty old (v5.003 i think) but it's being hosted on a remote server.

CODE

my $ua = LWP::UserAgent->new;
$ua->agent('Mozilla/5.0');
my $response = HTTP::Request->new('GET', 'http://uk.shopping.com/');
my $response = $ua->get('http://uk.shopping.com/');
print $response;
if ($response->is_success) {
    my $text = $response->content;  # or whatever
}  else  {
    die $response->status_line;
}
I tried using Mechanize but couldn't get it working using a library from my own version of perl (not compatible with v5.003) so have gone for the Useragent approach, which again works fine on my own server just not on the remotely hosted site.

Rob Waite
http://www.hostpipe.co.uk?src=TT

RE: LWP::Useragent causing HTTP 501 error

Not really sure what you are doing here, beyond me I'm afraid but you are declaring $response twice, is that meant to be?

Keith
www.studiosoft.co.uk

RE: LWP::Useragent causing HTTP 501 error

(OP)

In it's simplest fashion, it is literally just getting a page and assigning it to the $text variable.

One of the $response declarations should not have been there, although it still refuses to work no matter which one I use!

Rob Waite
http://www.hostpipe.co.uk?src=TT

RE: LWP::Useragent causing HTTP 501 error

I'd check the dependencies of LWP::UserAgent, though you're getting a 501, might be a firewall issue, in that the server can accept connections, but not create them

Check with your host

--Paul

cigless ...

RE: LWP::Useragent causing HTTP 501 error

can you use this?

CODE

#!/usr/bin/perl

print "Content-type: text/html\n\n";

use LWP::Simple;

$url = get("http://uk.shopping.com");

print "$url";


Kind Regards
Duncan

RE: LWP::Useragent causing HTTP 501 error

(OP)

Hi Duncan/Paul,

Tried the script you suggested and recieved:

'd:\h\o\hostpipe\www\ecaps\temp\test.pl' script produced no output

So, I think you may well be onto something with regards to the firewall issues. Do you know of any hosting companies who allow the creation of connections?

Rob Waite
http://www.hostpipe.co.uk?src=TT

RE: LWP::Useragent causing HTTP 501 error

I'd ask the hosting company, could they check for outbound dropped packets coming from the machine, and does it pose an actual threat or is the firewall just over protective.

Many admins lock stuff down until a legitimate request is made to free up the resource.

Are you on a shared server?
Your use of LWP, is it totally, or quasi legit?
Is there only one IP for your server?

If you answered No, Totally, Yes then they should be able to create a rule for you to create outbound connections from that specific IP

HTH
--Paul

RE: LWP::Useragent causing HTTP 501 error

(OP)

Paul,

Many thanks, I have posed the question to our hosting company, and will leave a post here with their answer. The use of LWP is totally legit, we are hoping eventually move it onto a dedicated server where we can set up the system as we wish, but for the time being it's purely a proof of concept project.

Anyhoo, thanks again, will keep you informed.

Best regards

Rob

Rob Waite
http://www.hostpipe.co.uk?src=TT

RE: LWP::Useragent causing HTTP 501 error

I can get the page with no probs with the tiny script i posted so i am sure Paul is correct


Kind Regards
Duncan

RE: LWP::Useragent causing HTTP 501 error

(OP)

Hi,

I now don't think it's a firewall issue, I've tried the following script:

CODE

#!/usr/bin/perl
print "Content-type: text/html\n\n";
use LWP::Simple;
$url = get("http://uk.shopping.com");
print "$url";
and it appears to work fine. So I now think there is an issue with my creation of the useragent using LWP::UserAgent.

Are there any major differences between the latest version of activeperl and the old version running on the remote server (v5_003)??

Rob Waite
http://www.hostpipe.co.uk?src=TT

RE: LWP::Useragent causing HTTP 501 error

To rule out the firewall issue, request a page from the localhost on the server, using "http://localhost"

Let us know how you get on

--Paul

cigless ...

RE: LWP::Useragent causing HTTP 501 error

(OP)
Hiya Paul,

Many thanks, I've managed to get around the problem. I was using a library from my newer version of perl and linking to it for Mechanize which caused an initial problem, then my UserAgent script wasn't quite right. It now appears to be successfully requesting and processing the page now using the following script:

CODE

#!/usr/bin/perl
print "Content-Type: text/html\n\n";

use CGI::Carp qw(fatalsToBrowser);
use LWP::UserAgent;

$ua = LWP::UserAgent->new;
$ua->agent("$0/0.1 " . $ua->agent);
$ua->agent("Mozilla/8.0");
$url = 'webaddressgoesinhere';
$req = HTTP::Request->new(GET => $url );
$req->header('Accept' => 'text/html');
$res = $ua->request($req);
if ($res->is_success) {
    print $url."<br />";
    $text = $res->content;
} else {
    print "8::" . $res->status_line . "\n";
}

It seems a little bit silly that it should default to the 'lib' directory I had created as opposed to the fully compiled version of Perl on my host's server, but hey, at least it works now!

Rob Waite
http://www.hostpipe.co.uk?src=TT

Red Flag This Post

Please let us know here why this post is inappropriate. Reasons such as off-topic, duplicates, flames, illegal, vulgar, or students posting their homework.

Red Flag Submitted

Thank you for helping keep Tek-Tips Forums free from inappropriate posts.
The Tek-Tips staff will check this out and take appropriate action.

Reply To This Thread

Posting in the Tek-Tips forums is a member-only feature.

Click Here to join Tek-Tips and talk with other members!

Resources

Close Box

Join Tek-Tips® Today!

Join your peers on the Internet's largest technical computer professional community.
It's easy to join and it's free.

Here's Why Members Love Tek-Tips Forums:

Register now while it's still free!

Already a member? Close this window and log in.

Join Us             Close