Detailed explanation of the difference and usage analysis between str and Unicode in Python encoding processing

Use python to process Chinese, When reading files or messages, if garbled characters are found (String processing, read and write files, print), what most people do is to call encode/decode for debugging without thinking clearly about why garbled characters appear. Today we will discuss how to deal with encoding problems.

Note: The following discussion is for the Python2.x version and has not been tested under Py3k

The most common errors during debugging

Error 1

Traceback (most recent call last): File "<stdin>", line 1, in <module> UnicodeDecodeError: ‘ascii‘ codec can‘t decode byte 0xe6 in position 0: ordinal not in range(128)

Error 2

Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/encodings/utf_8.py", line 16, in decode     return codecs.utf_8_decode(input, errors, True) UnicodeEncodeError: ‘ascii‘ codec can‘t encode characters in position 0-1: ordinal not in range(128)

First of all

We must have a general concept, understandCharacter set, character encoding

ASCII | Unicode | UTF-8 | etc.

Character encoding notes: ASCII, Unicode and UTF-8

str and unicode

str and unicode are both subclasses of basestring

So there is a way to determine whether it is a string

def is_str(s): return isinstance(s, basestring)

Str and unicode conversion

str -> decode('the_coding_of_str') -> unicode unicode -> encode('the_coding_you_want') -> str

Difference

str is a byte string, encoded by unicode

Declaration method composed of the following bytes

>>> s = ‘中文‘ s = u‘中文‘.encode(‘utf-8‘)  
>>> type(‘中文‘) <type ‘str‘>

Find the length (return the number of bytes)

>>> u‘中文‘.encode(‘utf-8‘) ‘\xe4\xb8\xad\xe6\x96\x87‘ 
>>> len(u‘中文‘.encode(‘utf-8‘)) 
6

Unicode is the real string, composed of characters

Declaration method

>>> s = u‘中文‘ 
>>> s = ‘中文‘.decode(‘utf-8‘) 
>>> s = unicode(‘中文‘, ‘utf-8‘)  
>>> type(u‘中文‘) <type ‘unicode‘>

Find the length (return the number of characters), what you really want to use in the logic

>>> u‘中文‘ u‘\u4e2d\u6587‘ 
>>> len(u‘中文‘) 
2

Conclusion

Understand what you need to deal with str is still unicode, use the right processing method (str.decode/unicode.encode)

The following is the method to determine whether it is unicode/str

>>> isinstance(u‘中文‘, unicode) True 
>>> isinstance(‘中文‘, unicode) False  
>>> isinstance(‘中文‘, str) True 
>>> isinstance(u‘中文‘, str) False

Simple principle: do not use encode for str. Do not use decode for unicode (in fact, you can encode str. See the end for details. To ensure simplicity, it is not recommended)

>>> ‘中文‘.encode(‘utf-8‘) 
Traceback (most recent call last): File "", line 1, in  UnicodeDecodeError: ‘ascii‘ codec can‘t decode byte 0xe4 in position 0: ordinal not in range(128)  
>>> u‘中文‘.decode(‘utf-8‘) 
Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/encodings/utf_8.py", line 16, in decode     return codecs.utf_8_decode(input, errors, True) UnicodeEncodeError: ‘ascii‘ codec can‘t encode characters in position 0-1: ordinal not in range(128)

For different encoding conversions, use unicode as the intermediate encoding

#s是code_A的str s.decode(‘code_A‘).encode(‘code_B‘)

File processing, IDE and console

Processing process, you can use it like this, think of python as a pool, an entrance, and an exit

At the entrance, all are converted to unicode , all in the pool are processed with unicode, and then converted to the target encoding at the exit (of course, there are exceptions, and specific encodings are used in the processing logic)

Read the file, external input encoding, decode into unicode for processing (Internal encoding, unified unicode) Encode is converted into the required target encoding and written to the target output (file or console)

The IDE and console report errors because the encoding is inconsistent with the IDE's own encoding when printing

When outputting, convert the encoding to a consistent one and you can output normally

>>> print u‘中文‘.encode(‘gbk‘) ???? 
>>> print u‘中文‘.encode(‘utf-8‘) 中文

Recommendations

Standardize the encoding

Uniform encoding to prevent garbled codes caused by a certain link

Environment coding, IDE/textEditor, file coding, database data table coding

Ensure code source file coding

This is very important

The default encoding of py files is ASCII. In the source code file, if non-ASCII characters are used, the encoding declaration needs to be made in the header of the file.

If not declared, errors will be encountered when inputting non-ASCII characters. , must be placed on the first or second line of the file

File "XXX.py", line 3 SyntaxError: Non-ASCII character ‘\xd6‘ in file c.py on line 3, but no encoding declared; see http://www.python.org/peps/pep-0263.html for details

Declaration method

# -*- coding: utf-8 -*- 或者 #coding=utf-8

If the header declares coding=utf-8, a = 'Chinese', its encoding is utf-8

If the header declares coding=gb2312, a = 'Chinese', its encoding is gbk

so, all source file headers in the same project have the same encoding, and the declared encoding must be the same as the source file The saved encoding is consistent (editor related)

is used as a hard-coded string for processing in the source code, and uniformly uses unicode

将其类型和源文件本身的编码隔离开, 独立无依赖方便流程中各个位置处理

if s == u‘中文‘:  #而不是 s == ‘中文‘     pass #注意这里 s到这里时，确保转为unicode

以上几步搞定后，你只需要关注两个 unicode和 你设定的编码(一般使用utf-8)

处理顺序

1. Decode early 2. Unicode everywhere 3. Encode later 

相关模块及一些方法

获得和设置系统默认编码

>>> import sys 
>>> sys.getdefaultencoding() ‘ascii‘  
>>> reload(sys) <module ‘sys‘ (built-in)> 
>>> sys.setdefaultencoding(‘utf-8‘) 
>>> sys.getdefaultencoding() ‘utf-8‘ 
>>> str.encode(‘other_coding‘)

在python中，直接将某种编码的str进行encode成另一种编码str

#str_A为utf-8 str_A.encode(‘gbk‘)  执行的操作是 str_A.decode(‘sys_codec‘).encode(‘gbk‘) 这里sys_codec即为上一步 sys.getdefaultencoding() 的编码 

‘获得和设置系统默认编码‘和这里的str.encode是相关的，但我一般很少这么用，主要是觉得复杂不可控,还是输入明确decode，输出明确encode来得简单些

chardet

文件编码检测，下载

>>> import chardet 
>>> f = open(‘test.txt‘,‘r‘) 
>>> result = chardet.detect(f.read()) 
>>> result {‘confidence‘: 0.99, ‘encoding‘: ‘utf-8‘}

\u字符串转对应unicode字符串

>>> u‘中‘ u‘\u4e2d‘  
>>> s = ‘\u4e2d‘ 
>>> print s.decode(‘unicode_escape‘) 中  
>>> a = ‘\\u4fee\\u6539\\u8282\\u70b9\\u72b6\\u6001\\u6210\\u529f‘ 
>>> a.decode(‘unicode_escape‘) u‘\u4fee\u6539\u8282\u70b9\u72b6\u6001\u6210\u529f‘

The above is the detailed content of Detailed explanation of the difference and usage analysis between str and Unicode in Python encoding processing. For more information, please follow other related articles on the PHP Chinese website!