python adds columns to 8000 rows csv

Question

I want to add more than 2000 columns to the existing csv with more than 8000 columns. Because the file is too large and cannot be loaded into the memory at once, I want to write it into the existing csv column by column. I have tried many times. None of the methods work. How can I solve it?

Answer 1

Read in rows and then add these columns. CSV is generally a comma-delimited text file, and it can be processed according to the processing method of text files. The general process is:
1. Read a line
2. Split the string into an array with commas
3. Add the column elements you want to the array
4. Connect the arrays with commas as separation
5. Write this line Enter a new file
6. Just go to the end of the file.

Answer 2

pandas has block reading, sample code

import pandas as pd

reader = pd.read_csv('a.csv', iterator=True)

header = True
try:
    df = reader.get_chunk(10000)

    #循环加添新列到df
    df['新列'] = '值'

    #把记录追加到新csv
    df.to_csv('b.csv', mode='a', index=False, header=header)
    
    #文件头只写一次
    header = False

except StopIteration:
    pass