python - pyspider爬取了接近1G的数据,无法导出csv?
大家讲道理
大家讲道理 2017-04-18 10:31:22
0
4
647

已经增加虚拟机内存至2g,依然无法导出。

大家讲道理
大家讲道理

光阴似箭催人老,日月如移越少年。

reply all(4)
伊谢尔伦

Generally, it is exported in files. If the data of one unit is about 1M, it will be divided into 1024 files.

In addition, it is impractical to synthesize a 1G CSV, because a 1G CSV is almost impossible to open. If you really want to synthesize 1G, you can synthesize more than a thousand exported files into a CSV. File synthesis is easy to achieve in Python.

黄舟

Monthly CSV processing of 1G data is not reasonable. Try sqlite3.

阿神

8G is not enough. It is recommended to choose a database as a storage solution. A 1G CSV has no meaning.

洪涛

Downloading is very fast using ftp intranet. I uploaded 8g two days ago. It feels like it downloaded in just a few minutes

Latest Downloads
More>
Web Effects
Website Source Code
Website Materials
Front End Template