网页爬虫 - python requests 爬取网页出现 IncompleteRead(2048 bytes read))

Question

requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(2048 bytes read)', IncompleteRead(2048 bytes read))

这个错误出现的时间不固定，有时几分钟就出现了 有时几个小时才出现

Answer 1

It may be caused by incomplete chunked encoding.
Please see here for details: http://blog.csdn.net/wangzuxi/article/details/40377467

Answer 2

Typical size of a buffer. In fact, IncompleteRead is an exception in the python native library httplib.

It is normal for the HTTP channel to return 0 bytes, or to get stuck after returning a certain number of bytes. If the reliability of the returned data is important, be careful to set reasonable timeouts and catch relevant errors. If necessary, you can even consider manually validating the Content-Length field of the HTTP header data.