download webpage api hangs

Question

download webpage api hangs

simian336 (Programmer)

(OP)

3 Dec 15 16:07

I am trying to download a web page from a local program accessed thru an api. It works thru a browser and it works in python but it appears to hang and just sits there like the request never closes. It actually prints the contents to shell.

Any ideas?

Simi

import urllib.request

link="""http://172.17.3.18:7780/dataservices/cdk?SP=md_cst...('mj0001','1','5000','1','',?,?,?int status)"""
print(link)

x = urllib.request.urlopen(link)
print(x.read())

simian336 (Programmer) · Answer 1 · 2015-12-03 16:09:20

Also a similar string is processed to open the connection and it opens it and returns in less then a second.

Simi

simian336 (Programmer) · Answer 2 · 2015-12-03 16:44:46

This appears to fix the problem. It is not super fast but it takes are of the problem.

response = urllib.request.urlopen(link, timeout=.5).read().decode('utf-8')

It still takes 5 seconds to retrieve 3000 rows.

Thanks

Simi

ChrisHirst (IS/IT--Management) · Answer 3 · 2015-12-03 18:03:16

HTTP connections do not 'close' themselves. Because HTTP is a 'disconnected protocol', a http user agent should read the amount of bytes that the server reports it is going to send (sec 14.13 Content-Length) from the HTTP: Response Header, then disconnect itself when that number has been received.

Chris.

Indifference will be the downfall of mankind, but who cares?
Time flies like an arrow, however, fruit flies like a banana.
Webmaster Forum

simian336 (Programmer) · Answer 4 · 2015-12-18 20:54:59

Here are 2 nearly identical pieces of code. the first call rund fine, the second one fails. But if I have it print the url and I paste the second one in the browser it runs fine?

Any ideas?

Thanks Simi

CODE --> phython

cust='http://192.17.3.17:7780/dataservices/cdk?SP=md_cst.get_customer_list(%270003%27,%271%27,%275000%27,%271%27,%27%27,?,?,?int%20status)'
print(cust)
req=urllib.request.Request(cust)
resp=urllib.request.urlopen(req)
respData=resp.read()
print(respData)

#########  This fails  ###############################
cust="http://192.17.3.17:7780/dataservices/cdk?SP=md_dev.get_device_list('0003','2','','n','1','1000',?,?,?int status)"
print(cust)
req=urllib.request.Request(cust)
resp=urllib.request.urlopen(req)
respData=resp.read()
print(respData)

ChrisHirst (IS/IT--Management) · Answer 5 · 2015-12-18 23:31:00

Probably because the second one is not URL encoded and the single quotes 'break' the receiving end.

Chris.

Indifference will be the downfall of mankind, but who cares?
Time flies like an arrow, however, fruit flies like a banana.
Webmaster Forum

simian336 (Programmer) · Answer 6 · 2015-12-22 18:46:16

I have tried it both ways and get the same result.

Here is the latest error messages.

Traceback (most recent call last):
File "C:\Program Files\python33\url_test.py", line 32, in <module>
resp=urllib.request.urlopen(req)
File "C:\Program Files\python33\lib\urllib\request.py", line 160, in urlopen
return opener.open(url, data, timeout)
File "C:\Program Files\python33\lib\urllib\request.py", line 479, in open
response = meth(req, response)
File "C:\Program Files\python33\lib\urllib\request.py", line 591, in http_response
'http', request, response, code, msg, hdrs)
File "C:\Program Files\python33\lib\urllib\request.py", line 517, in error
return self._call_chain(*args)
File "C:\Program Files\python33\lib\urllib\request.py", line 451, in _call_chain
result = func(*args)
File "C:\Program Files\python33\lib\urllib\request.py", line 599, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 400: Bad Request

Thanks

Simi

Come Join Us!

Posting Guidelines

download webpage api hangs