beautifulsoup - python requests 高频率刷新时卡顿

Question

1. 我用requests.get()读取固定网页上的信息（网页非常简单，不超过十个字符），然后用beautifulsoup解析，我设定的是1秒读取一次，但是发现运行时非常不稳定，有时会隔十几秒才能读到内容。这是网站服务器端的问题还是我的程序问题，能够解决吗？ （服务器是我自己的，用的是nginx和uwsgi，没有刷新频率限制）

2. python程序如下：

import requests
import time
from bs4 import BeautifulSoup

while True:

html = requests.get("http://121.42.159.21")
soup = BeautifulSoup(html.content,"html.parser")

d1=soup.find(id="d1")
d2=soup.find(id="d2")

d1_state_new = d1.string
d2_state_new = d2.string

if d1_state_new != d1_state_old:
    print('d1 open')
else:
    print('d1 remain close')

if d2_state_new != d2_state_old:
    print('d2 open')
else:
    print('d2 remain close')

d1_state_old = d1_state_new
d2_state_old = d2_state_new
print(time.ctime())
time.sleep(1)

运行结果：

d1 remain close
d2 remain close
Wed Jun 15 18:05:35 2016
d1 remain close
d2 remain close
Wed Jun 15 18:05:36 2016
d1 remain close
d2 remain close
Wed Jun 15 18:05:39 2016
d1 remain close
d2 remain close
Wed Jun 15 18:05:40 2016
d1 remain close
d2 remain close
Wed Jun 15 18:05:44 2016
d1 remain close
d2 remain close
Wed Jun 15 18:05:45 2016

可以看到很大概率会隔两三秒才读到信息

Answer 1

It’s blocked, so it should be done asynchronously.

Answer 2

Restrict the timeout of requests to see if it is a network problem.
There is also this lightweight extraction, which is enough for regular use.

Answer 3

Is it blocked while waiting for a response?

Answer 4

This is not a problem with requests, it is blocked. You can use an asynchronous method. In addition, I personally feel that bs is actually a bit slow. You can use etree in lxml to parse it directly.