How to solve the problem of text file conversion encoding in Python?

Test data

Note: The test text is encoded in UTF-8, and Chinese characters usually occupy three bytes. Chinese characters in GBK usually occupy 2 bytes.

Encoding conversion code

import os

# 对于单个文件进行操作的函数，如果需要对文件夹进行操作，可以使用一个函数包装它，这样不用修改本函数，即达到扩展的目的了。
def transfer_encode(source_path, target_path, source_encode='GBK', target_encode='UTF-8'):
    with open(source_path, mode='r', errors='ignore', encoding=source_encode) as source_file:  # 读取文件时，如果直接忽略报错，则程序正常执行，但是文件已经损坏了。
        with open(target_path, mode='w', encoding=target_encode) as target_file:               # 所以，应该捕获异常，停止程序执行。
            line = source_file.readline()
            while line != '':
                target_file.write(line)
                line = source_file.readline()
    print("Execute End!")


# 这个函数的功能和上面是一样的，区别在于它是以二进制读取的，然后解码、转码再写入的
def transfer_encode2(source_path, target_path, source_encode='GBK', target_encode='UTF-8'):
    with open(source_path, mode='rb') as source_file:
        with open(target_path, mode="wb") as target_file:
            bs = source_file.read(1024)
            while len(bs) != 0:
                target_file.write(bs.decode(source_encode).encode(target_encode))
                bs = source_file.read(1024)
    print("Execute End!")



source_path = r'C:\Users\Alfred\Desktop\test_data\test\data.txt'
target_path = r'C:\Users\Alfred\Desktop\test_data\test\data1.txt'

transfer_encode(source_path=source_path, target_path=target_path, source_encode="UTF-8", target_encode="GBK")

# transfer_encode2(source_path=source_path, target_path=target_path)

# 在cmd中使用 type命令，可以查看文件的内容，并且使用cmd默认的编码。
# 使用 chcp 命令可以查看当前使用的编码的数字编号

Execution result

Console output The output of this function execution is meaningless, but I want to know whether it has been executed. , so printed.

Test folder data1.txt is the converted and encoded text.

Judging from the generated file, because it only contains one word, you can know whether the conversion is successful by just comparing the size. Of course, it is also possible to open it directly for viewing, but if you open it directly for viewing, it will have no effect and the Chinese character 龙 will be displayed. So, here we take a different approach and use a different viewing method!

Note: data.txt is encoded in UTF-8, while data1.txt is encoded in GBK. Because Windows used in China adopts China's encoding method by default, it cannot display UTF-8 encoded text. The third output is to view the currently used encoding. It returns the encoding code. See the figure below for details:

Note: GBK is an encoding compatible with GB2312.

Instructions

If you use python, you only need 7 lines of code to convert a single file! I wrote two functions above, but the functions are the same. The difference is that the first function reads text information in a specific encoding and then writes it directly in another encoding. The second function reads the file content in binary form, then decodes and transcodes it for writing. Its principle is the same, that is, it must include sequential decoding and transcoding operations.

Encoding, decoding, and character sets themselves are very complicated, and I don’t know how to explain them in depth. The understanding here can be simplified like this. Two different encoding character sets have the same characters, so the purpose of reading the UTF-8 encoded file is to get the characters it maps, and then writing it again to map it to another A coded character set, so the character is similar to the function of a transfer station. If you directly use one character set to read the content of another character set, the garbled characters displayed in the cmd above will appear.

PS: Therefore, it can also explain a problem, that is, why opening a large text file will cause the program to freeze! Because a large text file contains many characters that need to be decoded. This is somewhat similar to queuing. Each character is waiting to be decoded. Although processing one character is fast, a large text file contains a large number of characters. For example, Notepad can open large texts without any pressure. But when I opened this very large text, it still got stuck! (The queuing here is just a metaphor, and I don’t know the actual situation, but it must be processed one by one.)

We estimate it, assuming that all The characters are all Chinese (actually, some English are included, but overall Chinese is still the majority.) It is shown here that there are about 50 million characters that need to be decoded, so it is still very difficult for the computer to process, you can view it on notepad Summary, but it gets stuck when I open it directly, so I won’t try it here.

The above is the detailed content of How to solve the problem of text file conversion encoding in Python?. For more information, please follow other related articles on the PHP Chinese website!