Perl using LWP

[pong123]

Hi
I tried to use LWP to retrieve information from mapquest.com Basically I want to input street addresses, and scrap distance from the page. The problem is that the information I want to scrap from the web page is dynamic (embedded in JavaScript), so I can't process them using regular expression (although the source codes are viewable via my web browser, but can't process them). Below is my code (doesn't work), and basically I want to extract distance from the web page. Any suggestion on how to fix the code? Thanks...

#!/usr/bin/perl
use strict;
use URI;
use LWP::UserAgent;
use LWP::Simple;

my $url = URI->new("

http://www.mapquest.com/maps");

$url->query_form(
"1c"=>"Palo Alto",
"1s"=>"CA",
"1a"=>"1322 Harker Ave",
"1z"=>"94301",

"2c"=>"Palo Alto",
"2s"=>"CA",
"2a"=>"3543 Louis Rd",
"2z"=>"94303");

#print $url;
my $resp=LWP::UserAgent->new->get($url);
#my $resp = get($url);
print $resp->content;

if ($resp=~m/dirDistanceSummary/) {print "$1 \n";}
else {print "No Match!\n"};

 

[PinkeyNBrain]

An example of what is contained your var $resp would help. However I'm going to suspect that $resp consists of multiple lines (assuming you haven't changed $/). If so, try adding the 's' pattern modifier

$resp =~ /dirDistanceSummary/s
#  note the 's' is at the end not the beginning.

Also, if you're going to {print "$1\n";}, you'll probably want to use ()'s to match something within the RE.

$resp =~ /dirDistanceSummary\s*([\d\.]+)/s

 

[pong123]

Thanks,

the difficulty I have is that the page is dynamic, so I can't save the entire source code (only parts of it). Even if I don't do any text processing, I still can't view save the entire source code. Is there a way to do it?


For example, when I print $resp, the distance between two points and directions are not available for me to grab.

 

[KevinADC]

Check these modules out:

http://search.cpan.org/search?query=mapquest&mode=all

------------------------------------------
- Kevin, perl coder unexceptional!

 

[PinkeyNBrain]

Ran the code listed above with one addition at the very end

foreach $key (keys %{$resp}) {
  print "key>$key<  val>$resp->{$key}<\n";
}

One of the output values indicated 'Unauthorized'. If you're getting the same, it may suggest something.

FWIW - During testing, I'll put ><'s I put around vars to help show if it has an empty value.

 

[KevinADC]

No extraneous semi-colons! I think we're finally getting you trained Pinky!

------------------------------------------
- Kevin, perl coder unexceptional!

 

[PinkeyNBrain]

Back at the ranch they're still in there. However for this forum I'm trying to flush them out. There is an extra one though.

 

[pong123]

Thanks...
I ran the code you suggested but it seemed the dynamic part still doesn't show. For example, the distance between two addresses are not availabe for grabbing. Below is the code I ran. Any help is greatly appreciated.

#!/usr/bin/perl
use strict;
use URI;
use LWP::UserAgent;
use LWP::Simple;

my $url = URI->new("

http://www.mapquest.com/maps");

$url->query_form(
"1c"=>"Palo Alto",
"1s"=>"CA",
"1a"=>"1320 Harker Ave",
"1z"=>"94301",

"2c"=>"Palo Alto",
"2s"=>"CA",
"2a"=>"3580 Louis Rd",
"2z"=>"94303");

my $resp=LWP::UserAgent->new->get($url);

foreach my $key (keys %{$resp}) {
print "key>$key< val>$resp->{$key}<\n";
}

 

[KevinADC]

If you can use Windows and IE check out Win32::IE::Mechanize

------------------------------------------
- Kevin, perl coder unexceptional!