我在用python监控一个网页 这个网页不定时的会更新 我要寻找需要匹配的关键词比如‘ABC’ 大概的程序框架如下
基本方法就是 用 selenium 获取源码 然后beautifulsoup解析 然后再去结果里面match 每2秒循环一次
while true:
html = browser.page_source
soup = BeautifulSoup(html)
abc=soup.find_all(text=re.compile("(ABC)"))
if not abc:
.....
else:
.....
browser.refresh()
time.sleep(2.0 - ((time.time() - starttime) % 2.0))
现在问题就是这个程序很依赖网速,browser.refresh() 刷新一次有可能就会用1秒钟
有没有什么办法 不需要刷新网页 就能知道网页有变化
或者有没有其他办法能让我这个程序 不被网速拖累
Http
Last-Modified
headers
'If-Modified-Since'
Status Code:304 Not Modified
Status code 304 means the page has not been changed
Time changed to yesterday (4th)
The server returns status code 200
and there are
'Last-Modified': 'Sun, 05 Feb 2017 06:00:03 GMT'
indicates the time of last modification.
No matter what, you have to visit the source site to get the data. If you don’t capture the data, how will you know if there are changes?
This kind of update may be updated using ajax. Personally, I think you can look at the js code of the website to find the request URL and parameters. If possible, go to the request directly?