Python solves the problem of Chinese character encoding: Unicode Decode Error_python

Preface

Recently, due to project needs, I need to read a txt document containing Chinese, and then save the file. The document was previously encoded by base64, causing all Chinese characters to be read and displayed as garbled characters. After the project team abandoned base64, two errors occurred successively:

ascii codec can't encode characters in position ordinal not in range 128
UnicodeDecodeError: ‘utf8' codec can't decode byte 0x。

If you don’t know about ascii, unicode and utf-8, you can read This previous article is about strings and encoding

Then you must understand the following three concepts:

1. ascii can only represent Numbers, English letters and some special symbols cannot represent Chinese characters

2. Both unicode and utf-8 can represent Chinese characters. Unicode is a fixed length and utf-8 is a variable length

3. The memory storage method is generally unicode, while the disk file storage method is generally utf-8, because utf-8 can save storage space

So what is python's default encoding?

>>> import sys
>>> sys.getdefaultencoding()
'ascii'
>>> reload(sys)
<module &#39;sys&#39; (built-in)>
>>> sys.setdefaultencoding(&#39;utf-8&#39;)
>>> sys.getdefaultencoding()
&#39;utf-8&#39;

The default encoding of python is ascii, which can be set by the sys.setdefaultencoding('utf-8') function the default encoding.

In python, you can change the encoding of data through encode and decode, for example:

>>> u'汉字'
u'\u6c49\u5b57'
>>> u'汉字'.encode('utf-8')
'\xe6\xb1\x89\xe5\xad\x97'
>>> u'汉字'.encode('utf-8').decode('utf-8')
u'\u6c49\u5b57'

We can set the encoding through these two functions .

So, what type is str in python?

>>> import binascii
>>> '汉字'
'\xba\xba\xd7\xd6'
>>> type('汉字')
<type &#39;str&#39;>
>>> print binascii.b2a_hex(&#39;汉字&#39;)
babad7d6
>>> print binascii.b2a_hex(u&#39;汉字&#39;)
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
UnicodeEncodeError: &#39;ascii&#39; codec can&#39;t encode characters in
position 0-1: ordinal not in range(128)
>>> print binascii.b2a_hex(u&#39;汉字&#39;.encode(&#39;utf-8&#39;))
e6b189e5ad97
>>> print binascii.b2a_hex(u&#39;汉字&#39;.encode(&#39;gbk&#39;))
babad7d6

binascii converts the binary of data into ascii. The above explanation is: the type of 'Chinese characters' is str, and the binary is babad7d6, u 'Chinese characters' cannot be converted into ascii, so the first error at the beginning is reported. The solution is to .encode('utf-8') it into str type. Because my command line uses the default GBK encoding of Windows, when all u'Chinese characters'.encode('gbk') is used, the output results are the same as the 'Chinese character' results.

To summarize, python's str is actually a type of unicode. The default encoding of python is ascii. When converting non-ascii to ascii, an error will be reported. Keep the following rules in mind:

1. unicode => encode('suitable encoding') => str

2. ##str => decode('suitable encoding') => unicode

   
   

There is also a simple way, which is to set the encoding in the file header, which can save a lot of trouble:

import sys
reloads(sys)
sys.setdefaultencoding('utf-8')

For the second problem, there is an error when reading the file. UTF-8 files have two modes: BOM and no BOM. The difference between the two seems to be that the BOM file has one more header than the BOM-less file, causing an error when reading the file in UTF-8 mode. When I tried to read the file before, First, judge whether there is a BOM and skip the header of the BOM file. Then it failed. It was really embarrassing~~.

You have to go to Google for help. The specific operation method is to use the codecs library to read the file (I guess this library is to detect the header of the file).

import codecs
codecs.open(file_name, "r",encoding='utf-8', errors='ignore')

For encoding issues, you must understand the working principles of ascii, unicode and utf-8.

For more python solutions to Chinese character encoding problems: Unicode Decode Error_python related articles, please pay attention to the PHP Chinese website!