INTELLIGENT WORK FORUMS
FOR COMPUTER PROFESSIONALS

Log In

Come Join Us!

Are you a
Computer / IT professional?
Join Tek-Tips Forums!
  • Talk With Other Members
  • Be Notified Of Responses
    To Your Posts
  • Keyword Search
  • One-Click Access To Your
    Favorite Forums
  • Automated Signatures
    On Your Posts
  • Best Of All, It's Free!

*Tek-Tips's functionality depends on members receiving e-mail. By joining you are opting in to receive e-mail.

Posting Guidelines

Promoting, selling, recruiting, coursework and thesis posting is forbidden.

Jobs

download webpage api hangs

download webpage api hangs

(OP)
I am trying to download a web page from a local program accessed thru an api. It works thru a browser and it works in python but it appears to hang and just sits there like the request never closes. It actually prints the contents to shell.

Any ideas?

Simi


import urllib.request

link="""http://172.17.3.18:7780/dataservices/cdk?SP=md_cst...('mj0001','1','5000','1','',?,?,?int status)"""
print(link)

x = urllib.request.urlopen(link)
print(x.read())

RE: download webpage api hangs

(OP)
Also a similar string is processed to open the connection and it opens it and returns in less then a second.

Simi

RE: download webpage api hangs

(OP)
This appears to fix the problem. It is not super fast but it takes are of the problem.

response = urllib.request.urlopen(link, timeout=.5).read().decode('utf-8')

It still takes 5 seconds to retrieve 3000 rows.

Thanks

Simi

RE: download webpage api hangs

HTTP connections do not 'close' themselves. Because HTTP is a 'disconnected protocol', a http user agent should read the amount of bytes that the server reports it is going to send (sec 14.13 Content-Length) from the HTTP: Response Header, then disconnect itself when that number has been received.

Chris.

Indifference will be the downfall of mankind, but who cares?
Time flies like an arrow, however, fruit flies like a banana.
Webmaster Forum

RE: download webpage api hangs

(OP)
Here are 2 nearly identical pieces of code. the first call rund fine, the second one fails. But if I have it print the url and I paste the second one in the browser it runs fine?

Any ideas?

Thanks Simi


CODE --> phython

cust='http://192.17.3.17:7780/dataservices/cdk?SP=md_cst.get_customer_list(%270003%27,%271%27,%275000%27,%271%27,%27%27,?,?,?int%20status)'
print(cust)
req=urllib.request.Request(cust)
resp=urllib.request.urlopen(req)
respData=resp.read()
print(respData)

#########  This fails  ###############################
cust="http://192.17.3.17:7780/dataservices/cdk?SP=md_dev.get_device_list('0003','2','','n','1','1000',?,?,?int status)"
print(cust)
req=urllib.request.Request(cust)
resp=urllib.request.urlopen(req)
respData=resp.read()
print(respData) 

RE: download webpage api hangs

Probably because the second one is not URL encoded and the single quotes 'break' the receiving end.

Chris.

Indifference will be the downfall of mankind, but who cares?
Time flies like an arrow, however, fruit flies like a banana.
Webmaster Forum

RE: download webpage api hangs

(OP)
I have tried it both ways and get the same result.

Here is the latest error messages.

Traceback (most recent call last):
File "C:\Program Files\python33\url_test.py", line 32, in <module>
resp=urllib.request.urlopen(req)
File "C:\Program Files\python33\lib\urllib\request.py", line 160, in urlopen
return opener.open(url, data, timeout)
File "C:\Program Files\python33\lib\urllib\request.py", line 479, in open
response = meth(req, response)
File "C:\Program Files\python33\lib\urllib\request.py", line 591, in http_response
'http', request, response, code, msg, hdrs)
File "C:\Program Files\python33\lib\urllib\request.py", line 517, in error
return self._call_chain(*args)
File "C:\Program Files\python33\lib\urllib\request.py", line 451, in _call_chain
result = func(*args)
File "C:\Program Files\python33\lib\urllib\request.py", line 599, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 400: Bad Request

Thanks

Simi

Red Flag This Post

Please let us know here why this post is inappropriate. Reasons such as off-topic, duplicates, flames, illegal, vulgar, or students posting their homework.

Red Flag Submitted

Thank you for helping keep Tek-Tips Forums free from inappropriate posts.
The Tek-Tips staff will check this out and take appropriate action.

Reply To This Thread

Posting in the Tek-Tips forums is a member-only feature.

Click Here to join Tek-Tips and talk with other members!

Resources

Close Box

Join Tek-Tips® Today!

Join your peers on the Internet's largest technical computer professional community.
It's easy to join and it's free.

Here's Why Members Love Tek-Tips Forums:

Register now while it's still free!

Already a member? Close this window and log in.

Join Us             Close