sending requests to HTTPS server and receiving HTML

sad69 · Jan 30, 2004

Hi all,

I would like to know how I might be able to do something using PHP (or some other technology that will allow this...).

What I need to do, and I know that it's possible because this is what web browsers do all the time, is send an HTTP request (to an HTTPS server) and receive the HTML file from that server replying to my request.

I've got an HTML file, and it contains a form with all hidden input fields and automatically (using <body onLoad=form.submit()>) submits itself to an asp page on the web.

Basically I'd like to POST all the information from that form to some ASP page on the web, and download the HTML response file so that I can parse its HTML code (using PHP's PERL abilities).

Any and all help would be appreciated!

Thanks,
Sadiq.

DRJ478 · Jan 31, 2004

I usually open a socket connection and send a GET request header to load the HTML form any URL into a string.

Code:

<?php
$fp = fsockopen(&quot;[URL unfurl="true"]www.example.com&quot;,[/URL] 80, $errno, $errstr, 30);
if (!$fp) {
   echo &quot;$errstr ($errno)<br />\n&quot;;
} else {
   # construct and send the GET header
   $out = &quot;GET / HTTP/1.1\r\n&quot;;
   $out .= &quot;Host: [URL unfurl="true"]www.example.com\r\n&quot;;[/URL]
   $out .= &quot;Connection: Close\r\n\r\n&quot;;

   fputs($fp, $out);
   # read the returned stream
   while (!feof($fp)) {
       $content .= fgets($fp, 128);
   }
   fclose($fp);
}
?>

After that you can use regular expressions to manipulate the HTML.
Happy screen-scraping! Also, please respect Copyright etc.

sleipnir214 · Feb 1, 2004

fsockopen() is good for HTTP, but for HTTPS the cryptography inherent in HTTPS is tough.

I generally use PHP's cURL functions to interact programmatically with foreign web servers. cURL can give you full cookie support, browser identification, etc. All of which is doable with fsockpen(), but a lot easier with cURL.

This page will tell you how to get cURL working with your PHP installation:

http://us2.php.net/manual/en/ref.curl.php

See my FAQ in this forum for example code which uses cURL: faq434-2502

Want the best answers? Ask the best questions:

http://www.catb.org/~esr/faqs/smart-questions.html

TANSTAAFL!!

Tek-Tips is the largest IT community on the Internet today!

Members share and learn making Tek-Tips Forums the best source of peer-reviewed technical information on the Internet!

sending requests to HTTPS server and receiving HTML

sad69

Programmer

DRJ478

IS-IT--Management

sleipnir214

Programmer

Similar threads

Part and Inventory Search

Sponsor