Python encoding processing-str and Unicode

A good article about STR and UNICODE

Organize the content related to python coding

Note: The following discussion is for the Python2.x version, Py3k is to be tried

Start

When using python to process Chinese, read files or messages, http parameters, etc.

As soon as I run it, garbled characters are found (string processing, reading Write a file, print)

Then, what most people do is to call encode/decode for debugging, without clearly thinking about why garbled characters appear

So the most common errors during debugging

Error 1

Traceback (most recent call last): File "", line 1, in UnicodeDecodeError: 'ascii' codec can't decode byte 0xe6 in position 0: ordinal not in range(128)

Error 2

##Traceback (most recent call last): File "", line 1, in File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/encodings/utf_8.py" , line 16, in decode return codecs.utf_8_decode(input, errors, True) UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-1: ordinal not in range(128)

First of all

You must have a general concept, understand the character set, character encoding

ASCII | Unicode | UTF-8 | etc.

Character encoding notes: ASCII, Unicode and UTF-8

Taobao Search Technology Blog-Chinese Encoding Talk

str and unicode

both str and unicode It is a subclass of basestring

So there is a method to determine whether it is a string

def is_str(s): return isinstance(s, basestring)

str and unicode conversion

decode document

encode document

str -> decode('the_coding_of_str') -> unicode unicode -> ; encode('the_coding_you_want') -> str

Difference

str is a byte string, which is encoded by unicode

Declaration method consisting of bytes

s = 'Chinese' s = u'中文'.encode('utf-8') >>> type( 'Chinese')

Find the length (return the number of bytes)

>>> u'中文'.encode('utf-8') '\xe4\xb8\xad\xe6\x96\x87' >>> len(u'中文'.encode('utf-8')) 6

unicode is the real string, composed of characters

Declaration method

s = u'Chinese ' s = '中文'.decode('utf-8') s = unicode('中文', 'utf-8') >>> type(u'中文')

Find the length (return the number of characters), what you really want to use in the logic

>>> u'中文' u'\ u4e2d\u6587' >>> len(u'中文') 2

##Conclusion

Understand that str is what needs to be processed Or unicode, use the right processing method (str.decode/unicode.encode)

The following is the method to determine whether it is unicode/str

>>> isinstance(u'中文', unicode) True >>> isinstance('中文', unicode) False >>> isinstance('中文', str) True >>> isinstance(u'中文', str) False

Simple principle: do not use encode for str and do not use decode for unicode (in fact, str can be encoded, see the end for details. To ensure simplicity, it is not recommended)

>>> '中文'.encode('utf-8') Traceback (most recent call last): File "", line 1, in UnicodeDecodeError: 'ascii' codec can 't decode byte 0xe4 in position 0: ordinal not in range(128) >>> u'中文'.decode('utf-8') Traceback (most recent call last): File "" , line 1, in File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/encodings/utf_8.py", line 16, in decode return codecs.utf_8_decode(input , errors, True) UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-1: ordinal not in range(128)

Different encoding conversions, use unicode as the intermediate encoding

#s is the str s.decode('code_A').encode('code_B') of code_A

File processing, IDE and console

The processing process can be used like this. Think of python as a pool, an entrance, and an exit.

At the entrance, all are converted to unicode, and all in the pool are processed with unicode. At the exit, they are converted to the target encoding. (Of course, there are exceptions, and specific encoding needs to be used in the processing logic)

Read the file, external input encoding, decode into unicode, process (internal encoding, unified unicode), encode into the required target encoding, write to the target output (file or console)

IDE and control The reason for the error is that when printing, the encoding is inconsistent with the IDE's own encoding.

Convert the encoding to a consistent one during output and the output will be normal.

##>>> print u '中文'.encode('gbk') ???? >>> print u'中文'.encode('utf-8') 中文

It is recommended that

standardize the coding

unify the coding to prevent garbled codes caused by a certain link

Environment coding, IDE/text editor, file coding, database data table coding

Ensure code source file coding

This is very important

The default encoding of py files is ASCII. In the source code file, if non-ASCII characters are used, the encoding declaration document needs to be declared in the header of the file.

If not declared, inputting non-ASCII will cause Errors encountered must be placed on the first or second line of the file

File "XXX.py", line 3 SyntaxError: Non-ASCII character '\xd6' in file c.py on line 3, but no encoding declared; see http://www.php.cn/ for details

Declaration method

# -*- coding: utf-8 -*- or #coding=utf- 8

If the header declares coding=utf-8, a = 'Chinese' its encoding is utf-8

If the header declares coding=gb2312, a = 'Chinese' is encoded as gbk

so. All source file headers in the same project have the same encoding, and the declared encoding must be consistent with the encoding saved in the source file (editor related)

For hard-coded strings used for processing in the source code, use unicode

to isolate their type from the encoding of the source file itself. It is independent and has no dependencies to facilitate processing at various locations in the process.

if s == u'中文': #Instead of s == '中文' pass #Note that when s here comes here, make sure to convert it to unicode

After completing the above steps, you only need to pay attention to two Unicode and the encoding you set (usually use utf-8)

Processing order

1. Decode early 2. Unicode everywhere 3. Encode later

Related modules and some methods

Get and set the system default encoding

##>>> import sys >>> sys.getdefaultencoding() 'ascii' >>> reload(sys) >>> sys.setdefaultencoding('utf-8') >>> sys. getdefaultencoding() 'utf-8'

str.encode('other_coding')
In python, directly encode a certain encoding str into another encoding str

#str_A is utf-8 str_A.encode('gbk') The operation performed is str_A.decode('sys_codec').encode('gbk') Here sys_codec is the encoding of sys.getdefaultencoding() in the previous step

'Getting and setting the system default encoding' is related to str.encode here, but I rarely use it this way. Mainly because it feels complicated and uncontrollable, and it is better to input clear decode and output clear encode. Simpler (personal opinion)

chardet

File encoding detection, download

>>> import chardet >>> f = open ('test.txt','r') >>> result = chardet.detect(f.read()) >>> result {'confidence': 0.99, 'encoding': 'utf- 8'}

\uString to corresponding unicode string

>>> u'中' u'\u4e2d' > >> s = '\u4e2d' >>> print s.decode('unicode_escape') in >>> a = '\u4fee\\u6539\\u8282\\u70b9\\u72b6 \\u6001\\u6210\\u529f' >>> a.decode('unicode_escape') u'\u4fee\u6539\u8282\u70b9\u72b6\u6001\u6210\u529f'

The above is the collection of information on Python coding processing. We will continue to add relevant information in the future. Thank you for your support of this site!

For more Python encoding processing-str and Unicode related articles, please pay attention to the PHP Chinese website!