python - scrapy crawls many more pages than actually enters items?

Question

2017-05-12 23:39:52 [scrapy.extensions.logstats] INFO:
Crawled 3000 pages (at 61 pages/min), scraped 454 items (at 10 items/min)

Why do I visit more pages than actually enter items when I use scrapy to crawl?
It’s just that after crawling for a long time, there are not many pieces of data in the items.
Is there any way to solve it?

Answer 1

Redirected, as you can see, redirection requires two pages to be requested. Also check to see if there are any page parsing errors.