Automating html processing and posting HTTPS

[SnyAc]

This is a continuation of my previous post. As it turns out the web pages that I need to interact with are SSL secured... therefore HTTPS pages. Anyone have any info on how to access these pages thru the API interfaces?

Thanks

Andy Snyder
SnyAc Software Services

http://www.snysoft.com

 

[wgcs]

Sure... it's just like HTTP, only start the url with HTTPS ... the Windows API handles everything else:

http://fox.wikis.com/wc.dll?Wiki~ReadUrl

faq184-3447

 

[wgcs]

The POST is more difficult.

 

[SnyAc]

I was already using ReadURL.... and it says the URL is unavailable... so there is some other flag or something that I need to set for it to read a secured page.....



Andy Snyder
SnyAc Software Services

http://www.snysoft.com

 

[wgcs]

I just tested it on an HTTPS page using the ReadUrl just posted on the wiki.... it worked with no adjustment.

Have you figured out how to POST on HTTP? I've implemented a manual POST operation, but if there is a WinApi POST, then it shouldn't be too hard to get it to use the windows implementation of HTTPS.

 

[wgcs]

Take a look at the ww Internet tools:

http://www.west-wind.com/presentations/wchttp.htm

This page explains: "But the low level mechanism pays off by providing many other options such as allowing access to Authentication of username and password, secure connections via SSL, the HTTP headers and most importantly the POST buffer."

This seems to mean that Rick has found out how to make Windows do the SSL layer.

 

[wgcs]

Aha! I found the flag you're talking about: it is INTERNET_FLAG_ SECURE

Adjust the connect portion of ReadUrl, and your POST code, to be something like:

hService = InternetConnect(hSession, "aarons.axtech.com", 
   INTERNET_DEFAULT_HTTPS_PORT, NULL, NULL, 
   INTERNET_SERVICE_HTTP, 0, 0);
if (hService)
{
 HINTERNET hHttpRequest;
 hHttpRequest = HttpOpenRequest( hService, "POST", 
   "purchase.cgi", NULL, NULL, NULL, INTERNET_FLAG_SECURE, 0);

This is, of course, is C++ code... you'll have to convert.

The complete article + source is here:

http://www.microsoft.com/mind/1297/inet/inet.asp

http://www.microsoft.com/mind/1297/inet/inettextfigs.htm#fig17

 

[SnyAc]

Still returns 'URL cannot be opened' The site I'm trying to navigate also requires a security certificate which I have installed. I can access the site thru explorer manually with no problem.


Andy Snyder
SnyAc Software Services

http://www.snysoft.com

 

[wgcs]

What mechanism are you using to access the site?

the ReadUrl example does not implement the POST operation.

I'm going to be using the HTTPS Post in the future, so I figured I'd do the C++ to VFP conversion... this does not yet parse the URL into HTTP/HTTPS, Server, Path ...

  FUNCTION HTTPPost2(plHttps,pcSvr,pcObjName,pcData,pcUser,pcPwd)
    * This is the 'proper' way to use the WinINet functions to POST
    *   The INTERNET_FLAG_SECURE makes it an HTTS connection  
    DECLARE INTEGER InternetOpen IN wininet; 
      STRING  sAgent,; 
      INTEGER lAccessType,; 
      STRING  sProxyName,; 
      STRING  sProxyBypass,; 
      STRING  lFlags     
    DECLARE INTEGER InternetConnect IN wininet; 
      INTEGER hInternetSession,; 
      STRING  sServerName,; 
      INTEGER nServerPort,; 
      STRING  sUsername,; 
      STRING  sPassword,; 
      INTEGER lService,; 
      INTEGER lFlags,; 
      INTEGER lContext         
    DECLARE INTEGER HttpOpenRequest IN wininet; 
      INTEGER hConnect,; 
      STRING  lpszVerb,; 
      STRING  lpszObjectName,; 
      STRING  lpszVersion,; 
      STRING  lpszReferer,; 
      INTEGER lpszAcceptTypes,; 
      INTEGER dwFlags,; 
      INTEGER dwContext      
    DECLARE INTEGER HttpSendRequest IN wininet; 
      INTEGER hRequest,; 
      STRING @ lpszHeaders,; 
      INTEGER dwHeadersLength,; 
      STRING @ lpOptional,; 
      INTEGER dwOptionalLength 
    DECLARE INTEGER InternetQueryDataAvailable IN wininet; 
      INTEGER   hFile,; 
      INTEGER @ lpdwBytesAvailable,; 
      INTEGER   dwFlags,; 
      INTEGER   dwContext     
    DECLARE INTEGER InternetReadFile IN wininet; 
      INTEGER   hFile,; 
      STRING  @ sBuffer,; 
      INTEGER   lNumBytesToRead,; 
      INTEGER @ dwNumberOfBytesRead
    DECLARE INTEGER InternetCloseHandle IN wininet; 
      INTEGER hInet     
      
    LOCAL hSession, hService, hHttpRequest
    LOCAL lsDomain, lsUser, lsPwd, lsUrlPath, lpData, lnFlags
    LOCAL lnBytesAvailable, lnBytesRead, lpBuffer

    * plHttps,pcSvr,pcObjName,pcData,pcUser,pcPwd
    lsDomain  = pcSvr
    lsUrlPath = pcObjName
    lpData    = pcData
    lnFlags   = iif( plHtpps, INTERNET_FLAG_SECURE, 0 )
    lsUser    = pcUser
    lsPwd     = pcPwd

    lpBuffer  = 'FAILED'
    
    hSession = InternetOpen("aasVfp", INTERNET_OPEN_TYPE_DIRECT, '', '', 0)
    if (hSession#0)
      hService = InternetConnect(hSession, (lsDomain), ;
                                 INTERNET_DEFAULT_HTTPS_PORT, (lsUser), (lsPwd), ;
                                 INTERNET_SERVICE_HTTP, 0, 0)
      if (hService#0)
        hHttpRequest = HttpOpenRequest(hService, "POST", (lsUrlPath), '', ;
                                       '', '*/*', (lnFlags), 0)
        if (hHttpRequest#0)
          if (0=HttpSendRequest(hHttpRequest, 0, 0, @lpData, len(lpData)) )
            lnBytesAvailable = 0
            lnBytesRead      = 0
            InternetQueryDataAvailable(hHttpRequest, @lnBytesAvailable, 0, 0)
            lpBuffer = Space(lnBytesAvailable+1)
            InternetReadFile(hHttpRequest, @lpBuffer, lnBytesAvailable, @lnBytesRead)
            lpBuffer = left(lpBuffer,lnBytesRead)
          endif
        endif
      endif
    endif   
    InternetCloseHandle(hSession)
    RETURN lpBuffer
  ENDFUNC

 

[SnyAc]

What I'm trying to do is to navigate a site that has a login page (which has a form... at least I think it's a form) and then fill in the form values, click the submit button and go on the the next page.....

I can do this with my own site's email form page, but so far ReadURL will not open the SSL page.



Andy Snyder
SnyAc Software Services

http://www.snysoft.com

 

[wgcs]

It's possible the server looks at the 'referer' field to be sure the login is coming from its own login page.

ReadUrl won't submit form results through POST unless you modify it significantly, in accord with the HttpPost example above (establishing the HttpRequest, filling the buffer, etc)

 

[SnyAc]

I'm still at the stage of finding out what the input field names are..... haven't even tried to actually login thru the program yet.

And I'm not so sure that POST is what I need to do.....

I need to be able to see the HTML of the page in order to determine how to proceed next.

Andy Snyder
SnyAc Software Services

http://www.snysoft.com

 

[wgcs]

<i>I need to be able to see the HTML of the page in order to determine how to proceed next. </I>

Load the page in Explorer, then go to View->Source.

Then, search for < FORM> and < /FORM>, delete everything out side of that section.

Then, find all &quot;<INPUT&quot; tags, and save them. Also, keep any &quot;<TEXTAREA&quot; tags, though a login form shouldn't have any.

Delete everything but these tags (<FORM, <INPUT, </FORM, <TEXTAREA ) and leave a brief label for each. Save the result as a temp.htm file, then load it in Explorer, and try it. If it works, then you can automated it in VFP.

If you post the URL you want to use, I can tell you just what the essential form tags are.

 

[SnyAc]

Thanks for the suggestion.... I already knew how to do that.... I was referring to accessing the source programmatically.

I have had some success this afternoon without using ReadURL... and have managed to login and navigate the site, so I'm well on my way to finishing this additional feature for my application this weekend.

Thanks for all the help. (I will probably post my solution in a faq in the next week or so)

Andy Snyder
SnyAc Software Services

http://www.snysoft.com