Winsock problem

[fenris]

I am trying to create a program that downloads webpages from a server on the internet. I can get it to download the webpages with out a problem. But If I download too many webpages in a row, the server sends me a webpage complaining that I have too many connections to it.

I don't understand what is going on. I open a connection to that webserver using the winsock control.Then download the webpage and close the connection. I repeat the process for the next pages....

Any ideas on what is going wrong? Or how I could fix this....



'======================================================
Private Sub downLoadHTML()
'this is the part that downloads the html page

lblUrl.Caption = myServer & urlDir
lblStatus.Caption = "Begining Download"
strHttpResponse = ""


If dlHTML.State = sckClosed Then
dlHTML.Connect myServer, 80
End If
End Sub

'winsock control events
Private Sub dlHTML_Connect()
sendHTMLRequest
End Sub

Private Sub sendHTMLRequest()
Dim strHttpRequest As String

lblStatus.Caption = "Connected to: " & myServer
'
'create the HTTP Request
'
'build request line that contains the HTTP method,
'path to the file to retrieve,
'and HTTP version info. Each line of the request
'must be completed by the vbCrLf
strHttpRequest = "GET " & urlDir & " HTTP/1.1" & vbCrLf
'
'add HTTP headers to the request
'
'add required header - "Host", that
'contains the remote host name
strHttpRequest = strHttpRequest & _
"Host: " & myServer & vbCrLf
'add optional header "Accept"
strHttpRequest = strHttpRequest & "Accept: */*" & vbCrLf
'add the "Connection" header to force
'the server to close the connection
strHttpRequest = strHttpRequest & "Connection: close" & vbCrLf
'
'add other optional headers
'
'strHttpRequest = strHttpRequest & <Header Name> & _
<Header Value> & vbCrLf
'. . .
'
'add a blank line that indicates the end of the request
strHttpRequest = strHttpRequest & vbCrLf


'
'send the request

'txtLog.Text = txtLog.Text & &quot;====Request====&quot; & vbNewLine
'txtLog.Text = txtLog.Text & strHttpRequest & vbNewLine
'txtLog.Text = txtLog.Text & &quot;===============&quot; & vbNewLine

'====================

dlHTML.SendData strHttpRequest
End Sub

Private Sub dlHTML_DataArrival(ByVal bytesTotal As Long)
Dim strData As String

dlHTML.GetData strData
strHttpResponse = strHttpResponse & strData
lblBytes.Caption = &quot;Bytes Received: &quot; & bytesTotal
End Sub



Private Sub dLHTML_Close()
'to cut of the header info, we must find
'a blank line (vbCrLf & vbCrLf)
'that separates the message body from the header
Dim sTemp() As String
lblStatus.Caption = &quot;Connection Closed&quot;
dlHTML.Close

sTemp = Split(strHttpResponse, vbNewLine & vbNewLine)
httpHeader = sTemp(0)
httpResponse = sTemp(1)

'txtLog.Text = txtLog.Text & &quot;====Server Header Respose====&quot; & vbNewLine
'txtLog.Text = txtLog.Text & httpHeader & vbNewLine
'txtLog.Text = txtLog.Text & &quot;===============&quot; & vbNewLine


Select Case GetHttpResponseCode(httpHeader)
'
Case 300, 301, 302, 303, 307
'handle the redirection
lblStatus.Caption = &quot;Redirecting to another URL&quot;
redirection
downLoadHTML
Exit Sub
Case 200 'ok
processHTML

'case whatever number is returned to indicate a file download
Case Else
'Nothing to do yet
End Select Troy Williams B.Eng.
fenris@hotmail.com

 

[strongm]

Is using Winsock a prerequisite of this project?

 

[fenris]

I would like to stick with winsock as I am writing this project to learn how to use it.

I have it working for one download. But as soon as I try and download a few more webpages, the web server complains that I have too many instances open. This is what I don't understand. Basically I am downloading the webpages serially (for now) and am only opening one connection to the server then closing it. How can I have multiple sessions opened up on the server.
Troy Williams B.Eng.
fenris@hotmail.com

 

[JohnStep]

Why dont you try to put a DSR in the program to delay each connection request for a couple of seconds. After a connection closes wait a second before requestion another fom the same IP (your).

John Stephens

 

[comaboy]

fenris,

I think JohnStep's on the right line. Web servers don't like too many concurrent connections because you're basically flooding their system in sort of the same way that a DOS attack works.

Real humans can't read pages as fast as an automated algorithm'll ask for them, so try to slow down your request rate a little so that Resquest Number 1 can complete before Request Number X is requested.

 

[fenris]

Thanks for the input.


I was thinking along the same lines, but I wasn't sure that web servers behaved in such a manner.


What about redirects? the browser seems to handle these things seamlessly...

Troy Williams B.Eng.
fenris@hotmail.com