python - pyspider爬取了接近1G的数据，无法导出csv？

Question

已经增加虚拟机内存至2g，依然无法导出。

伊谢尔伦 · Answer

Generally, it is exported in files. If the data of one unit is about 1M, it will be divided into 1024 files.

In addition, it is impractical to synthesize a 1G CSV, because a 1G CSV is almost impossible to open. If you really want to synthesize 1G, you can synthesize more than a thousand exported files into a CSV. File synthesis is easy to achieve in Python.

黄舟 · Answer

Monthly CSV processing of 1G data is not reasonable. Try sqlite3.

阿神 · Answer

8G is not enough. It is recommended to choose a database as a storage solution. A 1G CSV has no meaning.

PHP中文网 · Answer

Downloading is very fast using ftp intranet. I uploaded 8g two days ago. It feels like it downloaded in just a few minutes

Php8, I'm coming too

Learn website layout in 30 minutes

Shangguan Oracle Beginner to Proficient Video Tutorial

Your first line of UNI-APP code

Flutter from scratch to app launch

Brother Lian New Linux Video Tutorial

AXURE 9 Video Tutorial (Suitable for Product Manager Interactive Product Design UI)

Zero Basic Proficiency PS Video Tutorial

16 day UI video tutorial to get you started

PS Techniques and Slicing Techniques Video Tutorial

Alibaba Cloud Environment Construction and Project Launch Video Tutorial

Overview of Computer Networks - Basic Knowledge that Programmers Must Master

Essential Tutorial for Programmers - HTTP Protocol Explanation

Websocket Video Tutorial