Source From Here
Question
I'm gathering statistics on a list of websites and I'm using requests for it for simplicity. Here is my code:
Now, I want
requests.get to timeout after 10 seconds so the loop doesn't get stuck.
This question has been of interest before too but none of the answers are clean. I will be putting some bounty on this to get a nice answer. I hear that maybe not using requests is a good idea but then how should I get the nice things requests offer. (the ones in the tuple)
How-To
What about using eventlet? If you want to timeout the request after 10 seconds, even if data is being received, this snippet will work for you:
More articles and discussion on
eventlet:
* Python——eventlet
* Eventlet official documentation
If it’s your first time to Eventlet, you may find the illuminated examples in the Design Patterns document to be a good starting point. Eventlet is built around the concept of green threads (i.e. coroutines, we use the terms interchangeably) that are launched to do network-related work. Green threads differ from normal threads in two main ways:
Question
I'm gathering statistics on a list of websites and I'm using requests for it for simplicity. Here is my code:
- import requests
- data=[]
- websites=['http://google.com', 'http://bbc.co.uk']
- for w in websites:
- r= requests.get(w, verify=False)
- data.append( (r.url, len(r.content), r.elapsed.total_seconds(),
- str([(l.status_code, l.url) for l in r.history]),
- str(r.headers.items()), str(r.cookies.items())) )
This question has been of interest before too but none of the answers are clean. I will be putting some bounty on this to get a nice answer. I hear that maybe not using requests is a good idea but then how should I get the nice things requests offer. (the ones in the tuple)
How-To
What about using eventlet? If you want to timeout the request after 10 seconds, even if data is being received, this snippet will work for you:
- import requests
- import eventlet
- eventlet.monkey_patch(all=False, socket=True)
- with eventlet.Timeout(10):
- requests.get("http://ipv4.download.thinkbroadband.com/1GB.zip", verify=False)
* Python——eventlet
* Eventlet official documentation
If it’s your first time to Eventlet, you may find the illuminated examples in the Design Patterns document to be a good starting point. Eventlet is built around the concept of green threads (i.e. coroutines, we use the terms interchangeably) that are launched to do network-related work. Green threads differ from normal threads in two main ways:
沒有留言:
張貼留言