INTELLIGENT WORK FORUMS
FOR COMPUTER PROFESSIONALS

Log In

Come Join Us!

Are you a
Computer / IT professional?
Join Tek-Tips Forums!
  • Talk With Other Members
  • Be Notified Of Responses
    To Your Posts
  • Keyword Search
  • One-Click Access To Your
    Favorite Forums
  • Automated Signatures
    On Your Posts
  • Best Of All, It's Free!

*Tek-Tips's functionality depends on members receiving e-mail. By joining you are opting in to receive e-mail.

Posting Guidelines

Promoting, selling, recruiting, coursework and thesis posting is forbidden.

Jobs

Problem reading data from internet

Problem reading data from internet

(OP)
I am having problems reading data from a URL.
This data is details and GPS locations of a number of vehicles (Buses)
I can read it with Firefox opening with Notepad and see it is binary with some printable text.
If I open with Internet Transfer or Webbrowser I get a blank screen
If I open with IExplorer (ver 6 in my XP machine) I get the "Cant open" notice.

Is there a way to open it with Winsock and read to a stream in Vb6, erasing the stream as I read each vehicle? What port would I use?

Do I have to send it some code once connected to start it downloading?

This is the URL: https://gtfsrt.api.translink.com.au/feed
This is the sender's description "The feed is an actual continuous stream which is in Protocol Buffer (Googles simplified XML platform) and has to be queried continuously"

I would have no trouble decoding and handling the data in vb6 once I could receive it.

I am not very familiar with HTML so any help would be appreciated.

RE: Problem reading data from internet

Add a referencwe to the Microsoft WinHTTP Services library, then use this code as a starting point:

CODE

Option Explicit

Private Sub Command1_Click()
    Dim binaryresult() As Byte
    Dim stringresult As String
    binaryresult = GetDataFromURL("https://gtfsrt.api.translink.com.au/feed")
    stringresult = StrConv(binaryresult, vbUnicode)
    ' Do what you will with the data ...
End Sub

' returns binary data
Function GetDataFromURL(strURL As String) As Byte()
    With New WinHttpRequest
        .Open "GET", strURL
        .Send 'send an HTTP request
        GetDataFromURL = .ResponseBody  ' get the binary data
    End With
End Function 

RE: Problem reading data from internet

(OP)
Thanks.
That works OK in my Windows 7 machine with reference added to winhttp.dll. Looks very neat and simple.
I also was able to read data using the wininet.dll This works in my Win7 machine using IExplorer8 but not in XP (which is not really a problem in this case)
It looks a bit cludgy having to assemble the string the way it does but it works at the same speed (limited by the internet anyway to .05 seconds.)

CODE

I first declared InternetOpen InternetOpenUrl InternetReadFile and InternetCloseHandle
Private Function GetHTMLFromURL(sUrl As String) As String
Dim s As String
Dim hOpen As Long
Dim hOpenUrl As Long
Dim bDoLoop As Boolean
Dim bRet As Boolean
Dim sReadBuffer As String * 2048
Dim lNumberOfBytesRead As Long
hOpen = InternetOpen(scUserAgent, INTERNET_OPEN_TYPE_PRECONFIG, vbNullString, vbNullString, 0)
hOpenUrl = InternetOpenUrl(hOpen, sUrl, vbNullString, 0, INTERNET_FLAG_RELOAD, 0)
bDoLoop = True
While bDoLoop
    sReadBuffer = vbNullString
    bRet = InternetReadFile(hOpenUrl, sReadBuffer, Len(sReadBuffer), lNumberOfBytesRead)
    s = s & Left$(sReadBuffer, lNumberOfBytesRead)
    If Not CBool(lNumberOfBytesRead) Then bDoLoop = False
Wend
If hOpenUrl <> 0 Then InternetCloseHandle (hOpenUrl)
If hOpen <> 0 Then InternetCloseHandle (hOpen)
GetHTMLFromURL = s
End Function 

RE: Problem reading data from internet

(OP)
On Windows XP, with the GetDataFromUrl method I get an error message when the .send line tries to execute.

CODE -->

Run time error '-2147012739(80072f7d)
An error occurred in the secure channel support 

This does not happen with Windows 7.

The Reference is the same version 5.1

Is it because XP has IE version 6 while my Win7 has version 8?
Is there anyway to make it run in XP?
The client's machine where it will run has Server2003 installed. Will it run in that I wonder?

RE: Problem reading data from internet

Unlike your wininet.dll approach above, WinHTTP is completely independent of Internet Explorer and its various component DLLs. It is a separate and lighter weight HTTP stack of its own.


Caution, wild guess:

Microsoft has patched a number of Schannel security holes recently along with replacing several of the SLL/TLS ciphers.

While Windows Server 2003 is approaching end of extended support (July 2015), the server in question should have been patched this month. However if this customer has disabled Windows Update they may not have the patches installed and thus may be using ciphers no longer supported by the server at the remote end.

One of these patches: Vulnerability in Schannel Could Allow Remote Code Execution (2992611)

Getting caught up on patches might do the trick, but it would be better to get off this nearly dead OS and onto something with a longer support horizon.

As for Windows XP, it is past dead and such machines have not been safe to connect to the Internet for many months now. Retiring these spam relay zombied machines is a big favor to the entire Internet community.

RE: Problem reading data from internet

(OP)
Unfortunately both versions do not work on my old Xp machine. Neither will Ie6 open or save the file. Only Mozilla will do this on the old machine
It is something to do with the request transmitted by my Ie6 which seems to be the same as the inet version. When the same code is executed in a Win7 machine (with Ie8 installed) both versions work.

Alternatively I wondered if there was any way of finding out the characters that need to be sent, it could be done using a winsock. That way it would be OS independent.

I am probably naive but I was trying to get the subject to the most basic level but can't find any info on that level in plain English that does that. Info seems to refer to higher level solutions using a language like C or python or whatever.

RE: Problem reading data from internet

Try reregistering rsabase.dll

RE: Problem reading data from internet

>Alternatively I wondered if there was any way of finding out the characters that need to be sent, it could be done using a winsock

Well, sure. You could write your own HTTP user agent. But that isn't quite as straightforward as it sounds. And then you'd have to write an SSL/TLS layer in order to establish an encryption tunnel between your agent and the host (you may note they are delivering their data over HTTPS). It's really quite a lot of work, writing it all from 'the most basic level'. It's certainly not something I'd try and do.

RE: Problem reading data from internet

(OP)
Strongm you are a genius.
My computer didn't even have rsabase.dll in it so I downloaded it. (hopefully the genuine one)

So Internet transfer control works OK however WinHTTP give an error on send "an error occurred in the secure channel support" -2147012739(80072f7d)

RE: Problem reading data from internet

(OP)
I experimented with using Winsock by adding ,80 and get a response OK in the form of a small text notice but no data. I tried an old example for HTTP1 but of course got no data either.
Is there a simple code that enables the encryption I should be sending to receive HTTPS data?
Where would I find out what code is required or is that not available to the public?

RE: Problem reading data from internet

(OP)
2hours later -
I am SO frustrated! My test of Inet now does not work again!
I know it definitely did because I still have the file it saved!
I wondered if perhaps I was really reading my cache instead of the real file?

RE: Problem reading data from internet

If it is a failing hard drive I'd expect you to see many other issues. Or perhaps you had a shutdown error? That leaves malware as the next likely suspect.

Files like that don't just randomly disappear.

RE: Problem reading data from internet

>Is there a simple code that enables the encryption

Well ... yes - telling an HTTP user agent (in my example that would be winHTTP) that the protocol is HTTPS, which is just HTTP tunnelled through an SSL/TLS encrypted tunnel.

Drop the HTTP user agent and you are pretty much on your own - you pretty much have to start here: http://tools.ietf.org/html/rfc5246 and https://tools.ietf.org/html/rfc6101. Some additional related links can be found here: http://www.rfc-base.org/rfc-6101.html (as you can see there's no simple 'give me an encrypted tunnel, please' option - that's what the HTML user agent provides)

You might want to look at the OpenSSL Toolkit project, which provides an SDK to implement "the Secure Sockets Layer (SSL v2/v3) and Transport Layer Security (TLS) protocols". There is a Windows release available. Not designed for use from VB, though (and, although based on Open Source code, there is a commercial license for this release. An older completely free version is available here) All this is pretty hard-core stuff; good luck. I can't guarantee you'll ever get it working. I've never tried ...

RE: Problem reading data from internet

(OP)
Thanks, I'll look at it later (or maybe give up!)

It wasn't any of my files that disappeared, it is my app that no longer will read the data. The file I have is the file I made using my app which proves it did "work" originally.

RE: Problem reading data from internet

I suppose we can always try another HTTP user agent (but, like dilettante, I rather suspect that it is XP that is the problem). Add a reference to Microsoft XML library (latest version on your XP machine should be 6). Then this following modification:

CODE

' returns binary data
Function GetDataFromURL(strURL As String) As Byte()
    With New XMLHTTP60 ' could also try ServerXMLHTTP60
        .Open "GET", strURL
        .Send 'send an HTTP request
        GetDataFromURL = .ResponseBody  ' get the binary data
    End With
End Function 

RE: Problem reading data from internet

The XMLHTTPxx classes are wrappers on the IE-related WinInet stack. ServerXMLHTTPxx classes wrap WinHTTP.

RE: Problem reading data from internet

Has anyone considered that this is fallout from POODLE

i.e translink.com.au have turned off SSLv3

Take Care

Matt
I have always wished that my computer would be as easy to use as my telephone.
My wish has come true. I no longer know how to use my telephone.

RE: Problem reading data from internet

(OP)
Forgive me but can you explain POODLE and why it still works on machines with ie8 installed?

RE: Problem reading data from internet

You've not heard of poodle?

http://en.wikipedia.org/wiki/POODLE

In short, A vulnerability was found in SSL V3.0 which potentially allows the encryption to be broken. SSLv3 is quite old and has been superceded so most system administrators have remeditated the security risk by turning off / preventing SSLv3 encryption at the server.

I think the main people who this affects are XP users, because XP does not support the newer encryption protocols. for example XP only supported certain certificate types from SP3.

hence, it is entirely independent of browser,that is IE8 would connet if the OS is W7, but not if the OS is XP

Honest, time to ditch XP!

Take Care

Matt
I have always wished that my computer would be as easy to use as my telephone.
My wish has come true. I no longer know how to use my telephone.

RE: Problem reading data from internet

Ah yes. Good point.

>It is something to do with the request transmitted by my Ie6

No, I think it is something to do with what is returned by the webserver, and how IE6 responds to that. Specifically a pragma controlling caching. You can bypass this by adding a registry setting:

  • Start Registry Editor.
  • For a per-user setting, locate the following registry key:
  • HKEY_CURRENT_USER\SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings

  • For a per-computer setting, locate the following registry key:
  • HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings

  • On the Edit menu, click New>DWORD Value and add "BypassSSLNoCacheCheck" with a value of 1
  • RE: Problem reading data from internet

    That's weird. Posted itself ...

    OK, that should fix the IE "cannot download error", but probably won't deal with the secure channel support issue. I suppose it might be feasible to try and ignore SSL errors:

    CODE

    Function GetDataFromURL(strURL As String) As Byte()
        With New WinHttpRequest
            .Open "GET", strURL
            .Option(WinHttpRequestOption_SslErrorIgnoreFlags) = &H3300
            .Send
            GetDataFromURL = .ResponseBody 
        End With
    End Function 

    RE: Problem reading data from internet

    >Honest, time to ditch XP!

    The secure channel error might well be as a result of this - except it works fine on my XP laptop ...

    RE: Problem reading data from internet

    (OP)
    I can't ditch XP for this installation. There are so many computers involved and they will opnlyt be replaced as they fail with old age.
    Besides Firefox on an XP machine downloads the file to disk without the slightest problem so I doubt whether XP is the problem. You can't install anything later then IE6 on XP anyway according to Ms.
    I'm experimenting further.

    RE: Problem reading data from internet

    (OP)
    Trying the examples you have kindly posted I get "No reference" messages when trying XMLHTTP60 or ServerXMLHTTP60.
    When trying the last one I get a message to say message was badly formatted on the .SEND line.

    RE: Problem reading data from internet

    (OP)
    I see my Firefox is set to 'use system proxy'.
    I can use Firefos to download the file in question whether proxy is set or not.
    As there are Inet proxy settings, maybe I need to set it to a proxy to get Inet to work?
    Problem is then how do I find out what proxy address to use as Firefox doesn't tell you what they use?

    RE: Problem reading data from internet

    >You can't install anything later then IE6 on XP anyway according to Ms.

    You most certainly can. Don't know where you got that info. IE8 happily runs on XP. Here it is, direct from Microsoft. BTW, is it safe to assume that your XP boxes are running Service Pack 3?

    >without the slightest problem so I doubt whether XP is the problem

    Firefox, being multiplatform, uses its own crypto and certificate libraries rather than using XPs. Unlike IE (and the other user agents I suggested)

    >I get "No reference" messages when trying XMLHTTP60 or ServerXMLHTTP60.

    Ok, note that my instruction was:
    "Add a reference to Microsoft XML library (latest version on your XP machine should be 6)"

    The "60" in the references relates to the version number of the library you are using. So if the latest XML library you have available is 4, then the references would be XMLHTTP40 and ServerXMLHTTP40.

    Mind you, this suggests one other possibility - a 'simple' mismatch in (default) security protocols between XP and the server. Later versions of IE change the default protocols in use (which may explain why IE8 - and winHHTP and XMLHTTP - on my XP SP3 box works fine with the site). So we'll try forcing a specific protocol:

    CODE

    Function GetDataFromURL(strURL As String) As Byte()
        With New WinHttpRequest
            .Open "GET", strURL
            .Option(WinHttpRequestOption_SecureProtocols) = SecureProtocol_TLS1
            .Send 
            GetDataFromURL = .ResponseBody  
        End With
    End Function 



    RE: Problem reading data from internet

    (OP)
    No actually I am running service pack 2.
    I remember trying to install it some time ago but it wouldn't install at that time. I think I got it from some dodgy third party site trying to see you some other junk software that was rejected by my virus detector because it contained malware.
    I'll try it again.

    RE: Problem reading data from internet

    (OP)
    Panic over!
    Strongm's first simple version works now I have installed XP Service Pack 3
    Thanks once again for all your trouble.
    Hopefully it will continue to work and there is not some other bug.

    Red Flag This Post

    Please let us know here why this post is inappropriate. Reasons such as off-topic, duplicates, flames, illegal, vulgar, or students posting their homework.

    Red Flag Submitted

    Thank you for helping keep Tek-Tips Forums free from inappropriate posts.
    The Tek-Tips staff will check this out and take appropriate action.

    Reply To This Thread

    Posting in the Tek-Tips forums is a member-only feature.

    Click Here to join Tek-Tips and talk with other members!

    Resources

    Close Box

    Join Tek-Tips® Today!

    Join your peers on the Internet's largest technical computer professional community.
    It's easy to join and it's free.

    Here's Why Members Love Tek-Tips Forums:

    Register now while it's still free!

    Already a member? Close this window and log in.

    Join Us             Close