python - Scrapy有記憶體洩漏的問題。

Question

再寫爬蟲的時候，總是跑了一段時間（不會超過12小時）就會被OOM掉。很是無奈！ ！ ！
根據官方的文檔， 使用這個prefs()但是實在找不出問題的所在。

Live References

HtmlResponse                       42   oldest: 753s ago
MySuteSpider                        1   oldest: 2964s ago
Request                         32412   oldest: 2920s ago
Selector                           42   oldest: 751s ago
TripItem                           37   oldest: 751s ago

爬蟲的處理是取得所有頁面的a標籤的連結：

#获取域名的后缀
def get_domain_suffix(domain):
    if 'com' in tldextract.extract(domain).suffix:
        return True
    return False
#拼接域名。只存主域名
def save_domain(domain):
    domain_name = tldextract.extract(domain).domain
    suffix_name = tldextract.extract(domain).suffix

    return domain_name + '.' + suffix_name

#获取域名ip
def get_domain_ip(domain):
    try:
        ip = socket.gethostbyname(domain)
        return ip
    except:
        return '114.114.114.114'

# 获取域名所在的国家
def get_domain_ct_iso(ip):
    GEO = geoip2.database.Reader(
        '/var/test/geodb/GeoLite2-City.mmdb')
    r = GEO.city(ip)
    return r.country.iso_code

class MyDomainSpider(scrapy.Spider):
    name = 'my_domain'
    start_urls = [
        'http://xxx.com
    ]

    def parse_items(self, response):
        item = TripItem()
        for url in response.xpath('//a/@href').extract():
            if url.startswith('http'):
                    domain = urlparse.urlparse(url).netloc
                    if get_domain_tw(domain) or get_domain_ct_iso(get_domain_ip(domain)) == 'US':
                        item['domain'] = save_domain(domain)
                        item['ip'] = get_domain_ip(domain)
                        item['datetime'] = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")
                        yield item


    def parse(self, response):
        for url in response.xpath('//a/@href').extract():
            if url.startswith('http'):
                domain = urlparse.urlparse(url).netloc
                if get_domain_tw(domain) or get_domain_ct_iso(get_domain_ip(domain)) == 'US':
                     yield scrapy.Request(url, callback=self.parse_items)

請指教謝謝

Answer 1

yield item 是不是得落地，存檔案或db，不然就一直存內存了