Pleae test this LWP example

[Travistee2]

I can't get this example using LWP to work.

Can someone test this and let me know if it actually works
the way it is.
Then I can try to narrow down why it wont work for me.
I understand that the version number for LWP should be the same as my PERL installation.

thanks

###########

use strict;
use warnings;
use LWP 5.64;
my $browser = LWP::UserAgent->new;

my $word = 'tarragon';

my $url = '

http://www.altavista.com/sites/search/web';

my $response = $browser->post( $url,
[ 'q' => $word, # the Altavista query string
'pg' => 'q', 'avkw' => 'tgz', 'kl' => 'XX',
]
);
die "$url error: ", $response->status_line
unless $response->is_success;
die "Weird content type at $url -- ", $response->content_type
unless $response->content_type eq 'text/html';

if( $response->content =~ m{AltaVista found ([0-9,]+) results} ) {
# The substring will be like "AltaVista found 2,345 results"
print "$word: $1\n";
} else {
print "Couldn't find the match-string in the response\n";
}

 

[fishiface]

Tried it and it timed out. I added a call to env_proxy (do you use a proxy?) and it (correctly) reported that the page had been 301d.

What happens to yours? ("can't get it to work" is famously unhelpful to debuggers).

fish

"As soon as we started programming, we found to our surprise that it wasn't as easy to get programs right as we had thought. Debugging had to be discovered. I can remember the exact instant when I realized that a large part of my life from then on was going to be spent in finding mistakes in my own programs."
--Maurice Wilkes

 

[Travistee2]

I print a line at the start of the script to
see that it started ok.

That line prints ok.

Then the LWP above is next.

Nothing happens for the LWP example I gave above.

Can you give me the code for the call to env_proxy
that you added.

I don't know what you mean by using a proxy.

What about getting the results back
# The substring will be like "AltaVista found 2,345 results"


I found the example on another site, exactly as it is, and I'm trying to see if it actually works the way it was given before I try to
modify it for what I need.

 

[fishiface]

fish@spider:~$ cat  t
#!/usr/bin/perl

use strict;
use warnings;
use LWP 5.64;
my $browser = LWP::UserAgent->new;
$browser->env_proxy;

my $word = 'tarragon';

my $url = '[URL unfurl="true"]http://www.altavista.com/sites/search/web';[/URL]
my $response = $browser->post( $url,
                [ 'q' => $word,  # the Altavista query string
                'pg' => 'q', 'avkw' => 'tgz', 'kl' => 'XX',
                ]
        );
die "$url error: ", $response->status_line
        unless $response->is_success;
die "Weird content type at $url -- ", $response->content_type
        unless $response->content_type eq 'text/html';

if( $response->content =~ m{AltaVista found ([0-9,]+) results} ) {
# The substring will be like "AltaVista found 2,345 results"
        print "$word: $1\n";
} else {
        print "Couldn't find the match-string in the response\n";
}
fish@spider:~$ perl t
[URL unfurl="true"]http://www.altavista.com/sites/search/web[/URL] error: 301 Moved Permanently at t line 17.
fish@spider:~$ wget [URL unfurl="true"]http://www.altavista.com/sites/search/web[/URL]
--17:43:58--  [URL unfurl="true"]http://www.altavista.com/sites/search/web[/URL]
           => `web'
Connecting to 193.112.233.111:3128... connected.
Proxy request sent, awaiting response... 301 Moved Permanently
Location: /web/results [following]
--17:43:59--  [URL unfurl="true"]http://www.altavista.com/web/results[/URL]
           => `results'
Connecting to 193.112.233.111:3128... connected.
Proxy request sent, awaiting response... 302 Moved Temporarily
Location: / [following]
--17:43:59--  [URL unfurl="true"]http://www.altavista.com/[/URL]
           => `index.html.1'
Connecting to 193.112.233.111:3128... connected.
Proxy request sent, awaiting response... 200 OK
Length: unspecified [text/html]

    [ <=>                                 ] 8,724         --.--K/s

17:43:59 (78.06 KB/s) - `index.html.1' saved [8724]

so it's telling the truth.

f

"As soon as we started programming, we found to our surprise that it wasn't as easy to get programs right as we had thought. Debugging had to be discovered. I can remember the exact instant when I realized that a large part of my life from then on was going to be spent in finding mistakes in my own programs."
--Maurice Wilkes

 

[Travistee2]

Sorry if this is too basic, I haven't been using LWP

I got this compile error:

Can't locate object method "post" via package "LWP::UserAgent" (perhaps you forg
ot to load "LWP::UserAgent"?) at postxav.cgi line 41.

I tried changing this:
use strict;
use warnings;
use LWP 5.53; #<compile says I need this version
use LWP::UserAgent; #<------ added this line
my $browser = LWP::UserAgent->new;
$browser->env_proxy;

Do you know what it needs?

 

[fishiface]

That looks like an installation problem. Can you try this stripped-down and instrumented version and let me know what output or errors you get?

#!/usr/bin/perl

use strict;
use warnings;
use LWP;
print "LWP version $LWP::VERSION\n";
my $browser = LWP::UserAgent->new or die "No ua: $!";
print 'The user agent is an ', ref $browser, "\n";
print "LWP::UserAgent version $LWP::UserAgent::VERSION\n";
$browser->env_proxy();
my $response = $browser->post( '[URL unfurl="true"]http://www.google.com'[/URL] );
print 'The response is an ', ref $response, "\n";

f

&quot;As soon as we started programming, we found to our surprise that it wasn't as easy to get programs right as we had thought. Debugging had to be discovered. I can remember the exact instant when I realized that a large part of my life from then on was going to be spent in finding mistakes in my own programs.&quot;
--Maurice Wilkes

 

[Travistee2]

This printed in the browser:

before form was sentLWP version 5.53 The user agent is an LWP::UserAgent LWP::UserAgent version 1.80

This is from telnet:
perl postxe1.cgi
Content-type: text/html

<HTML><HEAD>
<TITLE></TITLE>
</HEAD>
<BODY TEXT=#DD00DD LINK=#00FFCC VLINK=#0040ff >before form was sentLWP version 5
.53
The user agent is an LWP::UserAgent
LWP::UserAgent version 1.80
Can't locate object method "post" via package "LWP::UserAgent" (perhaps you forg
ot to load "LWP::UserAgent"?) at postxe1.cgi line 40.

 

[fishiface]

The line

use LWP 5.64;

means that you are insisting on version 5.64 or higher of LWP but you only have version 5.53 installed. You should probably update your LWP library (the current version is 5.8).

f


&quot;As soon as we started programming, we found to our surprise that it wasn't as easy to get programs right as we had thought. Debugging had to be discovered. I can remember the exact instant when I realized that a large part of my life from then on was going to be spent in finding mistakes in my own programs.&quot;
--Maurice Wilkes

 

[Travistee2]

I tried it with
use LWP 5.53; #<--- thats the perl versiion

before form was sent LWP version 5.53 The user agent is an LWP::UserAgent LWP::UserAgent version 1.80

Then compile gets this:
perl postxe1.cgi
Content-type: text/html

<HTML><HEAD>
<TITLE></TITLE>
</HEAD>
<BODY TEXT=#DD00DD LINK=#00FFCC VLINK=#0040ff >before form was sentLWP version 5
.53
The user agent is an LWP::UserAgent
LWP::UserAgent version 1.80
Can't locate object method "post" via package "LWP::UserAgent" (perhaps you forg
ot to load "LWP::UserAgent"?) at postxe1.cgi line 40.

 

[fishiface]

I'm pretty stumped by this. Try the one-liner below - it prints out all the symbols that LWP::UserAgent knows about (including the methods). You should hope to see "post" in the output. Here's the command

perl -MLWP -e 'print join( "\n", sort keys %LWP::UserAgent:: ),
"\n";'

and here's the output that I see

BEGIN
ISA
VERSION
_agent
_need_proxy
_new_response
_process_colonic_headers
_request_sanity_check
agent
clone
conn_cache
cookie_jar
credentials
default_header
default_headers
env_proxy
from
get
get_basic_credentials
head
is_protocol_supported
max_redirect
max_size
mirror
new
no_proxy
parse_head
post
prepare_request
protocols_allowed
protocols_forbidden
proxy
redirect_ok
request
requests_redirectable
send_request
simple_request
timeout
use_alarm
use_eval

What do you see?

f

&quot;As soon as we started programming, we found to our surprise that it wasn't as easy to get programs right as we had thought. Debugging had to be discovered. I can remember the exact instant when I realized that a large part of my life from then on was going to be spent in finding mistakes in my own programs.&quot;
--Maurice Wilkes

 

[duncdude]

this worked fine for me...

[b]#!/usr/bin/perl[/b]

print "Content-type: text/html\n\n";

use strict;
use warnings;

use LWP::UserAgent;

my $browser = LWP::UserAgent->new;

my $word = 'tarragon';
my $url  = '[URL unfurl="true"]http://www.altavista.com/sites/search/web';[/URL]

my $response = $browser->post( $url,
  [
       'q' => $word,  # the Altavista query string
      'pg' => 'q',
    'avkw' => 'tgz',
      'kl' => 'XX',
  ]
);

die "$url error: ", $response->status_line                    unless $response->is_success;
die "Weird content type at $url -- ", $response->content_type unless $response->content_type eq 'text/html';

if ( $response->content =~ m{AltaVista found ([0-9,]+) results} ) {
  # The substring will be like "AltaVista found 2,345 results"
  print "$word: $1\n";
} else {
  print "Couldn't find the match-string in the response\n";
}

Kind Regards
Duncan

 

[Travistee2]

It seems the ISP I was on originally did not have the complete LWP install. I tried it on another ISP and this is what I got:


before form was sent

LWP version 5.803 The user agent is an LWP::UserAgent LWP::UserAgent version 2.033 The response is an HTTP::Response form was sent

Using this:
use strict;
use warnings;
use LWP 5.8.6;
print "LWP version $LWP::VERSION\n";
my $browser = LWP::UserAgent->new or die "No ua: $!";
print 'The user agent is an ', ref $browser, "\n";
print "LWP::UserAgent version $LWP::UserAgent::VERSION\n";
$browser->env_proxy();
my $response = $browser->post( '

http://www.google.com'

);
print 'The response is an ', ref $response, "\n";

 

[fishiface]

That looks great - I'm glad my diagnosis was correct. It's now time to start again with your original code. Remember that the URL you are requesting has moved, so you probably want to point directly at the real one or (while you have the chance) extend it to cover 301 responses and follow the redirect.

Have fun,

f

&quot;As soon as we started programming, we found to our surprise that it wasn't as easy to get programs right as we had thought. Debugging had to be discovered. I can remember the exact instant when I realized that a large part of my life from then on was going to be spent in finding mistakes in my own programs.&quot;
--Maurice Wilkes

 

[Travistee2]

I tried this, with the altavista form variables updated from what I had originally.

I'm still getting nothing after it prints

print "before form was sent";
#######################


use strict;
use warnings;
use LWP 5.8.6;
use LWP::UserAgent;

my $browser = LWP::UserAgent->new;

my $word = 'tarragon';
my $url = '

http://www.altavista.com';

my $response = $browser->post( $url,
[
'q' => $word,
'kgs' => '0',
'kls' => '1',
]
);

die "$url error: ", $response->status_line unless $response->is_success;
die "Weird content type at $url -- ", $response->content_type unless $response->content_type eq 'text/html';

if ( $response->content =~ m{AltaVista found ([0-9,]+) results} ) {
# The substring will be like "AltaVista found 2,345 results"
print "$word: $1\n";
} else {
print "Couldn't find the match-string in the response\n";
}

 

[fishiface]

The code runs correctly on my system if I add a call to env_proxy(). The returned page doesn't contain the required string, so it prints the error message as required.

The phrase "before form was sent" doesn't occur in your code so I can't tell how far it gets.

This probably brings us to my very first point. Proxies are used in corporate environments and by some ISPs. If you have no proxy, you have a direct connection. You can check by looking at the configuration of a working browser on the same machine on which your code is running. If you've got "use a proxy" ticked or proxy settings (host and port, or a URL like

http://proxy.me.com:3128)

set anywhere then you need to tell your code about this.

If you don't have a working browser, try telnetting to port 80 on www.altavista.com

telnet altavista.com 80

from a command line. If you have no proxy and a direct connection, then you should connect within a second or so - don't expect a prompt, though. If, instead, you get an error message or just a timeout and no connection, then we need to investigate further. Try it and let me know.

Yours,

f

&quot;As soon as we started programming, we found to our surprise that it wasn't as easy to get programs right as we had thought. Debugging had to be discovered. I can remember the exact instant when I realized that a large part of my life from then on was going to be spent in finding mistakes in my own programs.&quot;
--Maurice Wilkes

 

[Travistee2]

Now that the example appears to work, I want to see if I can actually submit a form, so I tried this:

#######################

print "before form was sent";
use strict;
use warnings;
use LWP 5.8.6;
use LWP::UserAgent;

my $browser = LWP::UserAgent->new;
$browser->env_proxy;

my $url = '

http://www.somewhere.com/wwwform/';

my $response = $browser->post( $url,
[
'name' => 'cindy',
'email' => 'cindy@nowhere.com',
'subject' => 'newvideo',
'body' => 'whatever I wrote',
]
);

die "$url error: ", $response->status_line unless $response->is_success;
die "Weird content type at $url -- ", $response->content_type unless $response->content_type eq 'text/html';

print "form was sent";
##########################

It prints "form was sent" then nothing.
The form does not show the submitted result?
Am I close?

 

[fishiface]

Very close. You're retrieving results and you can access them as $response->content.

Glad it's going ok now,

f

&quot;As soon as we started programming, we found to our surprise that it wasn't as easy to get programs right as we had thought. Debugging had to be discovered. I can remember the exact instant when I realized that a large part of my life from then on was going to be spent in finding mistakes in my own programs.&quot;
--Maurice Wilkes

 

[Travistee2]

I added:
$result="$response->content";
print "<br>$result";

at the end of the script.

I doesn't print anything and the form that the post was
to shows nothing updated.

Thanks for helping with this.

 

[duncdude]

Travistee2

It certainly should print something - although i have not really been following this thread - as you have made a very basic error. The assignment of $result is within double speech marks??? That won't work! Think about it - the -> has no special meaning within speech marks!!! What will happen here is $response will be evaluated and then output - and then ->content (literally) will be output after that:-

#!/usr/bin/perl

$result="$response->content";
print "<br>$result";

would output the following if $response was null:-

<br>->content

i think you meant $result = $response->content; - with no speech marks


Kind Regards
Duncan

 

[fishiface]

I'm on holiday for the next week and a half so I'll not be able to follow this further. I hope that it works out for you.

Yours,

fish

[&quot;]As soon as we started programming, we found to our surprise that it wasn't as easy to get programs right as we had thought. Debugging had to be discovered. I can remember the exact instant when I realized that a large part of my life from then on was going to be spent in finding mistakes in my own programs.[&quot;]
--Maur