What is the reason why pipeline files cannot be written when using Scapy crawler?

Analysis and solution for persistent storage of Scapy crawler data

When writing crawlers using Scapy, persistent data storage to pipeline files often encounters write failures. This article will analyze the causes of the problem and provide solutions for a practical case.

Question description:

The user tries to use a pipeline to store the crawl data, but the file is always empty and cannot be written.

Code example:

spider file (biedou.py):

 import scrapy
import sys
sys.path.append(r'd:\project_test\pydemo\demo1\xunlian\myspider\qiubai')
from ..items import qiubaiitem

class biedouspider(scrapy.Spider):
    name = "biedou"
    start_urls = ["https://www.biedoul.com/wenzi/"]

    def parse(self, response):
        dl_list = response.xpath('/html/body/div[4]/div[1]/div[1]/dl')

        for dl in dl_list:
            title = dl.xpath('./span/dd/a/strong/text()')[0].extract()
            content = dl.xpath('./dd//text()').extract()
            content = ''.join(content)

            item = qiubaiitem()
            item['title'] = title
            item['content'] = content
            yield item
            break

item file (item.py):

 import scrapy

class qiubaiitem(scrapy.Item):
    title = scrapy.Field()
    content = scrapy.Field()

pipeline file (pipelines.py): (The original code has a typo)

 class qiubaipipeline(object):
    def __init__(self):
        self.fp = None

    def open_spider(self, spider): #The original code is spelled incorrectly here print("Start crawler")
        self.fp = open('./biedou.txt', 'w', encoding='utf-8')

    def close_spider(self, spider):
        print("End Crawler")
        self.fp.close()

    def process_item(self, item, spider):
        title = str(item['title'])
        content = str(item['content'])
        self.fp.write(title ':' content '\n')
        return item

error message:

 <code>... typeerror: object of type qiubaiitem is not json serializable结束爬虫... attributeerror: 'nonetype' object has no attribute 'close'</code>

Problem analysis:

The error message prompts 'nonetype' object has no attribute 'close' , indicating that self.fp is None , causing the file to be closed. This is because the spelling error of the open_spider method in the pipelines.py file (original code is open_spdier ), which causes the method not to be called by the Scrapy framework and self.fp is not initialized.

Solution:

Correct the open_spdier method name in the pipelines.py file to open_spider :

 class QiubaiPipeline(object): # It is also recommended to use the camel nomenclature def __init__(self):
        self.fp = None

    def open_spider(self, spider):
        print("Start crawler")
        self.fp = open('./biedou.txt', 'w', encoding='utf-8')

    def close_spider(self, spider):
        print("End Crawler")
        self.fp.close()

    def process_item(self, item, spider):
        title = str(item['title'])
        content = str(item['content'])
        self.fp.write(title ':' content '\n')
        return item

After correcting the spelling error, the open_spider method will be called correctly by the Scrapy framework, and self.fp will be initialized, thus solving the problem of file writing failure. In addition, it is recommended to use more standardized class names and variable names, such as changing qiubaipipeline to QiubaiPipeline .

Through the above modifications, the data of the Scapy crawler can be correctly written to the pipeline file. Remember to check for spelling errors in your code, which is often the source of many problems.

The above is the detailed content of What is the reason why pipeline files cannot be written when using Scapy crawler?. For more information, please follow other related articles on the PHP Chinese website!