2018年11月8日 星期四

[ Python 常見問題 ] Timeout for python requests.get entire response

Source From Here
Question
I'm gathering statistics on a list of websites and I'm using requests for it for simplicity. Here is my code:
  1. import requests  
  2.   
  3. data=[]  
  4. websites=['http://google.com''http://bbc.co.uk']  
  5. for w in websites:  
  6.     r= requests.get(w, verify=False)  
  7.     data.append( (r.url, len(r.content), r.elapsed.total_seconds(),   
  8.                          str([(l.status_code, l.url) for l in r.history]),   
  9.                          str(r.headers.items()), str(r.cookies.items())) )  
Now, I want requests.get to timeout after 10 seconds so the loop doesn't get stuck.

This question has been of interest before too but none of the answers are clean. I will be putting some bounty on this to get a nice answer. I hear that maybe not using requests is a good idea but then how should I get the nice things requests offer. (the ones in the tuple)

How-To
What about using eventlet? If you want to timeout the request after 10 seconds, even if data is being received, this snippet will work for you:
  1. import requests  
  2. import eventlet  
  3. eventlet.monkey_patch(all=False, socket=True)  
  4.   
  5. with eventlet.Timeout(10):  
  6.     requests.get("http://ipv4.download.thinkbroadband.com/1GB.zip", verify=False)  
More articles and discussion on eventlet:
Python——eventlet
所謂”綠化”是指綠化後的 Python 環境支持綠色線程的運行模式。Python 原生的標準庫不支持 eventlet 這種綠色線程之間互相讓渡CPU控制權的執行模型,為此 eventlet 開發者改寫了部分 Python 標準庫(自稱”補丁“)。如果想在應用中使用 eventlet,需要顯式地綠化自己要引入的模塊...

Eventlet official documentation
If it’s your first time to Eventlet, you may find the illuminated examples in the Design Patterns document to be a good starting point. Eventlet is built around the concept of green threads (i.e. coroutines, we use the terms interchangeablythat are launched to do network-related work. Green threads differ from normal threads in two main ways:
1. Green threads are so cheap they are nearly free. You do not have to conserve green threads like you would normal threads. In general, there will be at least one green thread per network connection.
2. Green threads cooperatively yield to each other instead of preemptively being scheduled. The major advantage from this behavior is that shared data structures don’t need locks, because only if a yield is explicitly called can another green thread have access to the data structure. It is also possible to inspect primitives such as queues to see if they have any pending data.

沒有留言:

張貼留言

[ Python 常見問題 ] How to shift a datetime object by 12 hours in python

Source From   Here   Question   Datetime   objects hurt my head for some reason. I am writing to figure out   how to shift a date time obje...