Detailed explanation of the solution to the problem of encoding and decoding garbled strings

Why is the error "UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-1: ordinal not in range(128) "? This article will study this issue.

StringThe internal representation of Python is unicode encoding. Therefore, when doing encoding conversion, it is usually necessary to use unicode as the intermediate encoding, that is, convert other encodings first The string is decoded into unicode, and then encoded from unicode into another encoding.

The function of decode is to convert other encoded strings into unicode encoding, such as str1.decode('gb2312'), which means converting the gb2312 encoded string str1 into unicode encoding.

The function of encode is to convert unicode encoding into other encoded strings, such as str2.encode('gb2312'), which means converting unicode encoded string str2 into gb2312 encoding.

Therefore, when transcoding, you must first understand what encoding the string str is, then decode it into unicode, and then encode it into other encodings

The default encoding of the string in the code is the same as The code files themselves are encoded consistently.

For example: s='Chinese'

If it is in a utf8 file, the string is utf8 encoded. If it is in a gb2312 file, its encoding is gb2312. In this case, to perform encoding conversion, you need to first use the decode method to convert it to unicode encoding, and then use the encode method to convert it to other encodings. Usually, when no specific encoding method is specified, code files created using the system default encoding are used.

If the string is defined like this: s=u'Chinese'

The encoding of the string is specified as unicode, which is python's internal encoding, and is different from the code file itself. Encoding has nothing to do with it. Therefore, for encoding conversion in this case, you only need to directly use the encode method to convert it to the specified encoding.

If a string is already unicode, an error will occur when decoding it. Therefore, it is usually necessary to judge whether the encoding method is unicode:

isinstance(s, unicode)  #用来判断是否为unicode

Use str in non-unicode encoding form. encode will report an error

How to obtain the system's default encoding?

#!/usr/bin/env python
#coding=utf-8
import sys
print sys.getdefaultencoding()

The output of this program on English Windows The station itself cannot display the encoding of the string, and it is not a problem with the program itself.

If you run the following code in UliPad:

s=u"中文"
print s

will prompt: UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-1: ordinal not in range(128). This is because UliPad's console information output window on English Windows

Change the last sentence to: print s.encode('gb2312')

The word "Chinese" can be correctly output.

If the last sentence is changed to: print s.encode('utf8')

, the output will be: \xe4\xb8\xad\xe6\x96\x87, this is the console information output window Output the result of UTF8 encoded string according to ASCII encoding.

unicode(str,'gb2312') and str.decode('gb2312') are the same, they both convert gb2312 encoded str into unicode encoding

Use str.

class

You can check the encoding form of str After talking about the principle for a long time, let’s finally come up with a cure-all:)

The code is as follows:

#!/usr/bin/env python 
#coding=utf-8 
s="中文" 
if isinstance(s, unicode): 
#s=u"中文" 
print s.encode('gb2312') 
else: 
#s="中文" 
print s.decode('utf-8').encode('gb2312')

The above is the detailed content of Detailed explanation of the solution to the problem of encoding and decoding garbled strings. For more information, please follow other related articles on the PHP Chinese website!