Encoding and decoding strings solves the problem of garbled characters

Why is the error "UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-1: ordinal not in range(128)" reported? This article will study this issue.

The internal representation of strings in Python is unicode encoding. Therefore, when doing encoding conversion, it is usually necessary to use unicode as the intermediate encoding, that is, first decode strings in other encodings into unicode, and then encode them from unicode. ) into another encoding.

The function of decode is to convert other encoded strings into unicode encoding, such as str1.decode('gb2312'), which means converting the gb2312 encoded string str1 into unicode encoding.

The function of encode is to convert unicode encoding into other encoded strings, such as str2.encode('gb2312'), which means converting unicode encoded string str2 into gb2312 encoding.

Therefore, when transcoding, you must first understand what encoding the string str is, then decode it into unicode, and then encode it into other encodings

The default encoding of the string in the code is consistent with the encoding of the code file itself.

For example: s='Chinese'

If it is in a utf8 file, the string is utf8 encoded. If it is in a gb2312 file, its encoding is gb2312. In this case, to perform encoding conversion, you need to first use the decode method to convert it to unicode encoding, and then use the encode method to convert it to other encodings. Usually, when no specific encoding method is specified, code files created using the system default encoding are used.

If the string is defined like this: s=u'Chinese'

The encoding of the string is specified as unicode, which is python's internal encoding, regardless of the encoding of the code file itself. Therefore, for encoding conversion in this case, you only need to directly use the encode method to convert it to the specified encoding.

If a string is already unicode, an error will occur when decoding it, so it is usually necessary to judge whether the encoding method is unicode:

isinstance(s, unicode)  #用来判断是否为unicode

Using str in non-unicode encoding form to encode will result in an error

How to obtain the system the default encoding?

#!/usr/bin/env python
#coding=utf-8
import sys
print sys.getdefaultencoding()

The output of this program on English Windows , rather than a problem with the program itself.

If you run the following code in UliPad:

s=u"中文"
print s

it will prompt: UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-1: ordinal not in range(128). This is because UliPad's console information output window on English Windows

Change the last sentence to: print s.encode('gb2312')

The word "Chinese" can be correctly output.

If the last sentence is changed to: print s.encode('utf8')

, then the output is: xe4xb8xadxe6x96x87, which is the result of the console information output window outputting the utf8-encoded string according to ascii encoding.

unicode(str,'gb2312') is the same as str.decode('gb2312'), both convert gb2312 encoded str into unicode encoding

Use str.__class__ to view the encoding form of str

Principle After talking for a long time, let’s finally come up with a cure-all:)

The code is as follows:

#!/usr/bin/env python 
#coding=utf-8 
s="中文" 
if isinstance(s, unicode): 
#s=u"中文" 
print s.encode('gb2312') 
else: 
#s="中文" 
print s.decode('utf-8').encode('gb2312')