Python solves the problem of Chinese character encoding: Unicode Decode Error_python

Preface

Recently, due to project needs, I need to read a txt document containing Chinese, and then save the file. The document was previously encoded by base64, causing all Chinese characters to be read and displayed as garbled characters. After the project team abandoned base64, two errors occurred successively:

ascii codec can't encode characters in position ordinal not in range 128
UnicodeDecodeError: ‘utf8' codec can't decode byte 0x。

If you don’t know about ascii, unicode and utf-8, you can read This previous article is about strings and encoding

Then you must understand the following three concepts:

1. ascii can only represent Numbers, English letters and some special symbols cannot represent Chinese characters

2. Both unicode and utf-8 can represent Chinese characters. Unicode is a fixed length and utf-8 is a variable length

3. The memory storage method is generally unicode, while the disk file storage method is generally utf-8, because utf-8 can save storage space

So what is python's default encoding?

>>> import sys
>>> sys.getdefaultencoding()
'ascii'
>>> reload(sys)
<module &#39;sys&#39; (built-in)>
>>> sys.setdefaultencoding(&#39;utf-8&#39;)
>>> sys.getdefaultencoding()
&#39;utf-8&#39;

The default encoding of python is ascii, which can be set by the sys.setdefaultencoding('utf-8') function the default encoding.

In python, you can change the encoding of data through encode and decode, for example:

>>> u'汉字'
u'\u6c49\u5b57'
>>> u'汉字'.encode('utf-8')
'\xe6\xb1\x89\xe5\xad\x97'
>>> u'汉字'.encode('utf-8').decode('utf-8')
u'\u6c49\u5b57'

We can set the encoding through these two functions .

So, what type is str in python?

>>> import binascii
>>> '汉字'
'\xba\xba\xd7\xd6'
>>> type('汉字')
<type &#39;str&#39;>
>>> print binascii.b2a_hex(&#39;汉字&#39;)
babad7d6
>>> print binascii.b2a_hex(u&#39;汉字&#39;)
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
UnicodeEncodeError: &#39;ascii&#39; codec can&#39;t encode characters in
position 0-1: ordinal not in range(128)
>>> print binascii.b2a_hex(u&#39;汉字&#39;.encode(&#39;utf-8&#39;))
e6b189e5ad97
>>> print binascii.b2a_hex(u&#39;汉字&#39;.encode(&#39;gbk&#39;))
babad7d6

binascii converts the binary of data into ascii. The above explanation is: the type of 'Chinese characters' is str, and the binary is babad7d6, u 'Chinese characters' cannot be converted into ascii, so the first error at the beginning is reported. The solution is to .encode('utf-8') it into str type. Because my command line uses the default GBK encoding of Windows, when all u'Chinese characters'.encode('gbk') is used, the output results are the same as the 'Chinese character' results.

To summarize, python's str is actually a type of unicode. The default encoding of python is ascii. When converting non-ascii to ascii, an error will be reported. Keep the following rules in mind:

1. unicode => encode('suitable encoding') => str

2. ##str => decode('suitable encoding') => unicode

   
   

import sys
reloads(sys)
sys.setdefaultencoding('utf-8')

import codecs
codecs.open(file_name, "r",encoding='utf-8', errors='ignore')