python - Use scrapy framework to crawl Baidu pictures and get blocked

Question

The request address url is the address of json obtained through firefox. It can be opened with a browser, but it was banned when crawling with scrapy. Please solve it.

https://image.baidu.com/searc...

Answer 1

Try it at settings.py 将 ROBOTSTXT_OBEY = False.

Answer 2

Try without adding hearingers

Answer 3

I agree with the upstairs, if there will still be a wall. The method of scrapy+selenium+phantomjs can be used.