python - Scrapy's spider is convenient for running and debugging in the IDE

Question

I have many spiders, so I can’t manually create a debugger every time I debug (scrapy crawl spider_name)

I would like to right-click the spider file to run/debug directly.

Answer 1

Reference article: Summary of common problems with Scrapy crawlers

spider is easy to run/debug

In the spider file, add the cmdline calling method

import scrapy.cmdline

#Your Spider Class...

def main():
    scrapy.cmdline.execute(['scrapy', 'crawl', 'your_spider_name'])

if __name__ == '__main__':
    main()

Answer 2

Create a new py file and use this to debug

from scrapy.cmdline import execute

import sys
import os


sys.path.append(os.path.dirname(os.path.abspath(__file__)))

execute(["scrapy","crawl","YOUR_SPIDER"])