python - 爬虫在爬取网页的时候出现了302要如何解决?
ringa_lee
ringa_lee 2017-04-18 09:48:26
0
3
876

在抓取360问答的时候,前面抓取得好好的,但是每次抓到大约40条的时候,就打不开了,所有的问题都打不开了,直接全部跳转到首页!我抓包了看了下,是被302了,请问下,除了换ip,还有什么比较好的方法来突破这种限制?

我设置了时间间隔,现在也不行了

ringa_lee
ringa_lee

ringa_lee

reply all(3)
小葫芦

sleep(10)
Pause, don’t climb too fast~

迷茫

Floor 1 is right, set the crawl delay

阿神

The simple solution is to use an agent.

Setting the time interval is generally not very useful. In the case of large-scale crawling, if an IP is accessed hundreds of times, the verification code will usually be redirected.

Latest Downloads
More>
Web Effects
Website Source Code
Website Materials
Front End Template