python - About TwistedPipline error in Scrapy
给我你的怀抱
给我你的怀抱 2017-05-18 10:56:26
0
1
596
> python3.5 / pycharm 

crawlspider There is absolutely no problem with the returned item. If you connect it to TwistedPipline, an error will be reported. Can any teacher help analyze the error code

# TwistedPipline
class MysqlTwistedPipline(object):
    def __init__(self, dbpool):
        self.dbpool = dbpool

    @classmethod
    def from_settings(cls, settings):
        dbparms = dict(
            host = settings["MYSQL_HOST"],
            db = settings["MYSQL_DBNAME"],
            user = settings["MYSQL_USER"],
            passwd = settings["MYSQL_PASSWORD"],
            charset='utf8',
            cursorclass=MySQLdb.cursors.DictCursor,
            use_unicode=True,
        )
        dbpool = adbapi.ConnectionPool("MySQLdb", **dbparms)

        return cls(dbpool)

    def process_item(self, item, spider):
        query = self.dbpool.runInteraction(self.do_insert, item)
        query.addErrback(self.handle_error, item, spider) 

    def handle_error(self, failure, item, spider):
        print (failure)

    def do_insert(self, cursor, item):
        insert_sql, params = item.insert_values()
        print (insert_sql, params)
        cursor.execute(insert_sql, params)
        return item
        

Error code

2017-05-01 00:06:16 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.lagou.com/jobs/2108656.html> (referer: https://www.lagou.com/zhaopin/Python/)
2017-05-01 00:06:16 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.lagou.com/jobs/2108656.html>
None
[Failure instance: Traceback: <class 'TypeError'>: connect() argument 3 must be str, not None
D:\Python35\lib\threading.py:914:_bootstrap_inner
D:\Python35\lib\threading.py:862:run
D:\Python35\lib\site-packages\twisted\_threads\_threadworker.py:46:work
D:\Python35\lib\site-packages\twisted\_threads\_team.py:190:doWork
--- <exception caught here> ---
D:\Python35\lib\site-packages\twisted\python\threadpool.py:250:inContext
D:\Python35\lib\site-packages\twisted\python\threadpool.py:266:<lambda>
D:\Python35\lib\site-packages\twisted\python\context.py:122:callWithContext
D:\Python35\lib\site-packages\twisted\python\context.py:85:callWithContext
D:\Python35\lib\site-packages\twisted\enterprise\adbapi.py:464:_runInteraction
D:\Python35\lib\site-packages\twisted\enterprise\adbapi.py:36:__init__
D:\Python35\lib\site-packages\twisted\enterprise\adbapi.py:76:reconnect
D:\Python35\lib\site-packages\twisted\enterprise\adbapi.py:431:connect
D:\Python35\lib\site-packages\MySQLdb\__init__.py:86:Connect
D:\Python35\lib\site-packages\MySQLdb\connections.py:204:__init__
]
 
给我你的怀抱
给我你的怀抱

reply all(1)
大家讲道理

It can be seen from the prompt that one parameter in your settings is None, not a string

Latest Downloads
More>
Web Effects
Website Source Code
Website Materials
Front End Template