python - 爬虫抓取数据页一直返回HTTP code 417

Question

同一份代码在本地测试请求数据页返回200，

但是在线上之前请求的步骤都可以成功请求到，到请求数据页这步一直返回417

用Advanced REST client在本地模拟这段请求也是可以通过的

查了一下原因，有可能是服务器被识别被封了，如果是这样请问如何解决？谢谢

Answer 1

Two points are very important for crawlers. Control the frequency of crawling and use an agent to crawl. It doesn’t matter if it is slower. The goal is to be long-lasting and effective.

Answer 2

It is very likely that the server has been blocked due to collecting too much data, so you can only use HTTP proxy