Table of Contents
str and unicode
Conclusion
Recommendations
Home Backend Development Python Tutorial Detailed explanation of the difference and usage analysis between str and Unicode in Python encoding processing

Detailed explanation of the difference and usage analysis between str and Unicode in Python encoding processing

Mar 16, 2017 pm 04:23 PM

Use python to process Chinese, When reading files or messages, if garbled characters are found (String processing, read and write files, print), what most people do is to call encode/decode for debugging without thinking clearly about why garbled characters appear. Today we will discuss how to deal with encoding problems.

Note: The following discussion is for the Python2.x version and has not been tested under Py3k

The most common errors during debugging

Error 1

Traceback (most recent call last): File "<stdin>", line 1, in <module> UnicodeDecodeError: ‘ascii‘ codec can‘t decode byte 0xe6 in position 0: ordinal not in range(128)
Copy after login

Error 2

Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/encodings/utf_8.py", line 16, in decode     return codecs.utf_8_decode(input, errors, True) UnicodeEncodeError: ‘ascii‘ codec can‘t encode characters in position 0-1: ordinal not in range(128)
Copy after login

First of all

We must have a general concept, understandCharacter set, character encoding

ASCII | Unicode | UTF-8 | etc.

Character encoding notes: ASCII, Unicode and UTF-8

str and unicode

str and unicode are both subclasses of basestring

So there is a way to determine whether it is a string

def is_str(s): return isinstance(s, basestring)

Str and unicode conversion

str -> decode('the_coding_of_str') -> unicode unicode -> encode('the_coding_you_want') -> str

Difference

str is a byte string, encoded by unicode

Declaration method composed of the following bytes

>>> s = ‘中文‘ s = u‘中文‘.encode(‘utf-8‘)  
>>> type(‘中文‘) <type ‘str‘>
Copy after login

Find the length (return the number of bytes)

>>> u‘中文‘.encode(‘utf-8‘) ‘\xe4\xb8\xad\xe6\x96\x87‘ 
>>> len(u‘中文‘.encode(‘utf-8‘)) 
6
Copy after login

Unicode is the real string, composed of characters

Declaration method

>>> s = u‘中文‘ 
>>> s = ‘中文‘.decode(‘utf-8‘) 
>>> s = unicode(‘中文‘, ‘utf-8‘)  
>>> type(u‘中文‘) <type ‘unicode‘>
Copy after login

Find the length (return the number of characters), what you really want to use in the logic

>>> u‘中文‘ u‘\u4e2d\u6587‘ 
>>> len(u‘中文‘) 
2
Copy after login

Conclusion

Understand what you need to deal with str is still unicode, use the right processing method (str.decode/unicode.encode)

The following is the method to determine whether it is unicode/str

>>> isinstance(u‘中文‘, unicode) True 
>>> isinstance(‘中文‘, unicode) False  
>>> isinstance(‘中文‘, str) True 
>>> isinstance(u‘中文‘, str) False
Copy after login

Simple principle: do not use encode for str. Do not use decode for unicode (in fact, you can encode str. See the end for details. To ensure simplicity, it is not recommended)

>>> ‘中文‘.encode(‘utf-8‘) 
Traceback (most recent call last): File "", line 1, in  UnicodeDecodeError: ‘ascii‘ codec can‘t decode byte 0xe4 in position 0: ordinal not in range(128)  
>>> u‘中文‘.decode(‘utf-8‘) 
Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/encodings/utf_8.py", line 16, in decode     return codecs.utf_8_decode(input, errors, True) UnicodeEncodeError: ‘ascii‘ codec can‘t encode characters in position 0-1: ordinal not in range(128)
Copy after login

For different encoding conversions, use unicode as the intermediate encoding

#s是code_A的str s.decode(‘code_A‘).encode(‘code_B‘)
Copy after login

File processing, IDE and console

Processing process, you can use it like this, think of python as a pool, an entrance, and an exit

At the entrance, all are converted to unicode , all in the pool are processed with unicode, and then converted to the target encoding at the exit (of course, there are exceptions, and specific encodings are used in the processing logic)

Read the file, external input encoding, decode into unicode for processing (Internal encoding, unified unicode) Encode is converted into the required target encoding and written to the target output (file or console)

The IDE and console report errors because the encoding is inconsistent with the IDE's own encoding when printing

When outputting, convert the encoding to a consistent one and you can output normally

>>> print u‘中文‘.encode(‘gbk‘) ???? 
>>> print u‘中文‘.encode(‘utf-8‘) 中文
Copy after login

Recommendations

Standardize the encoding

Uniform encoding to prevent garbled codes caused by a certain link

Environment coding, IDE/textEditor, file coding, database data table coding

Ensure code source file coding

This is very important

The default encoding of py files is ASCII. In the source code file, if non-ASCII characters are used, the encoding declaration needs to be made in the header of the file.

If not declared, errors will be encountered when inputting non-ASCII characters. , must be placed on the first or second line of the file

File "XXX.py", line 3 SyntaxError: Non-ASCII character ‘\xd6‘ in file c.py on line 3, but no encoding declared; see http://www.python.org/peps/pep-0263.html for details
Copy after login

Declaration method

# -*- coding: utf-8 -*- 或者 #coding=utf-8
Copy after login

If the header declares coding=utf-8, a = 'Chinese', its encoding is utf-8

If the header declares coding=gb2312, a = 'Chinese', its encoding is gbk

so, all source file headers in the same project have the same encoding, and the declared encoding must be the same as the source file The saved encoding is consistent (editor related)

is used as a hard-coded string for processing in the source code, and uniformly uses unicode

将其类型和源文件本身的编码隔离开, 独立无依赖方便流程中各个位置处理

if s == u‘中文‘:  #而不是 s == ‘中文‘     pass #注意这里 s到这里时,确保转为unicode
Copy after login

以上几步搞定后,你只需要关注两个 unicode和 你设定的编码(一般使用utf-8)

处理顺序

1. Decode early 2. Unicode everywhere 3. Encode later 

相关模块及一些方法

获得和设置系统默认编码

>>> import sys 
>>> sys.getdefaultencoding() ‘ascii‘  
>>> reload(sys) <module ‘sys‘ (built-in)> 
>>> sys.setdefaultencoding(‘utf-8‘) 
>>> sys.getdefaultencoding() ‘utf-8‘ 
>>> str.encode(‘other_coding‘)
Copy after login

在python中,直接将某种编码的str进行encode成另一种编码str

#str_A为utf-8 str_A.encode(‘gbk‘)  执行的操作是 str_A.decode(‘sys_codec‘).encode(‘gbk‘) 这里sys_codec即为上一步 sys.getdefaultencoding() 的编码 

‘获得和设置系统默认编码‘和这里的str.encode是相关的,但我一般很少这么用,主要是觉得复杂不可控,还是输入明确decode,输出明确encode来得简单些

chardet

文件编码检测,下载

>>> import chardet 
>>> f = open(‘test.txt‘,‘r‘) 
>>> result = chardet.detect(f.read()) 
>>> result {‘confidence‘: 0.99, ‘encoding‘: ‘utf-8‘}
Copy after login

\u字符串转对应unicode字符串

>>> u‘中‘ u‘\u4e2d‘  
>>> s = ‘\u4e2d‘ 
>>> print s.decode(‘unicode_escape‘) 中  
>>> a = ‘\\u4fee\\u6539\\u8282\\u70b9\\u72b6\\u6001\\u6210\\u529f‘ 
>>> a.decode(‘unicode_escape‘) u‘\u4fee\u6539\u8282\u70b9\u72b6\u6001\u6210\u529f‘
Copy after login

The above is the detailed content of Detailed explanation of the difference and usage analysis between str and Unicode in Python encoding processing. For more information, please follow other related articles on the PHP Chinese website!

Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

AI Hentai Generator

AI Hentai Generator

Generate AI Hentai for free.

Hot Article

R.E.P.O. Energy Crystals Explained and What They Do (Yellow Crystal)
2 weeks ago By 尊渡假赌尊渡假赌尊渡假赌
Repo: How To Revive Teammates
4 weeks ago By 尊渡假赌尊渡假赌尊渡假赌
Hello Kitty Island Adventure: How To Get Giant Seeds
3 weeks ago By 尊渡假赌尊渡假赌尊渡假赌

Hot Tools

Notepad++7.3.1

Notepad++7.3.1

Easy-to-use and free code editor

SublimeText3 Chinese version

SublimeText3 Chinese version

Chinese version, very easy to use

Zend Studio 13.0.1

Zend Studio 13.0.1

Powerful PHP integrated development environment

Dreamweaver CS6

Dreamweaver CS6

Visual web development tools

SublimeText3 Mac version

SublimeText3 Mac version

God-level code editing software (SublimeText3)

How to Use Python to Find the Zipf Distribution of a Text File How to Use Python to Find the Zipf Distribution of a Text File Mar 05, 2025 am 09:58 AM

This tutorial demonstrates how to use Python to process the statistical concept of Zipf's law and demonstrates the efficiency of Python's reading and sorting large text files when processing the law. You may be wondering what the term Zipf distribution means. To understand this term, we first need to define Zipf's law. Don't worry, I'll try to simplify the instructions. Zipf's Law Zipf's law simply means: in a large natural language corpus, the most frequently occurring words appear about twice as frequently as the second frequent words, three times as the third frequent words, four times as the fourth frequent words, and so on. Let's look at an example. If you look at the Brown corpus in American English, you will notice that the most frequent word is "th

How to Download Files in Python How to Download Files in Python Mar 01, 2025 am 10:03 AM

Python provides a variety of ways to download files from the Internet, which can be downloaded over HTTP using the urllib package or the requests library. This tutorial will explain how to use these libraries to download files from URLs from Python. requests library requests is one of the most popular libraries in Python. It allows sending HTTP/1.1 requests without manually adding query strings to URLs or form encoding of POST data. The requests library can perform many functions, including: Add form data Add multi-part file Access Python response data Make a request head

Image Filtering in Python Image Filtering in Python Mar 03, 2025 am 09:44 AM

Dealing with noisy images is a common problem, especially with mobile phone or low-resolution camera photos. This tutorial explores image filtering techniques in Python using OpenCV to tackle this issue. Image Filtering: A Powerful Tool Image filter

How Do I Use Beautiful Soup to Parse HTML? How Do I Use Beautiful Soup to Parse HTML? Mar 10, 2025 pm 06:54 PM

This article explains how to use Beautiful Soup, a Python library, to parse HTML. It details common methods like find(), find_all(), select(), and get_text() for data extraction, handling of diverse HTML structures and errors, and alternatives (Sel

How to Work With PDF Documents Using Python How to Work With PDF Documents Using Python Mar 02, 2025 am 09:54 AM

PDF files are popular for their cross-platform compatibility, with content and layout consistent across operating systems, reading devices and software. However, unlike Python processing plain text files, PDF files are binary files with more complex structures and contain elements such as fonts, colors, and images. Fortunately, it is not difficult to process PDF files with Python's external modules. This article will use the PyPDF2 module to demonstrate how to open a PDF file, print a page, and extract text. For the creation and editing of PDF files, please refer to another tutorial from me. Preparation The core lies in using external module PyPDF2. First, install it using pip: pip is P

How to Cache Using Redis in Django Applications How to Cache Using Redis in Django Applications Mar 02, 2025 am 10:10 AM

This tutorial demonstrates how to leverage Redis caching to boost the performance of Python applications, specifically within a Django framework. We'll cover Redis installation, Django configuration, and performance comparisons to highlight the bene

Introducing the Natural Language Toolkit (NLTK) Introducing the Natural Language Toolkit (NLTK) Mar 01, 2025 am 10:05 AM

Natural language processing (NLP) is the automatic or semi-automatic processing of human language. NLP is closely related to linguistics and has links to research in cognitive science, psychology, physiology, and mathematics. In the computer science

How to Perform Deep Learning with TensorFlow or PyTorch? How to Perform Deep Learning with TensorFlow or PyTorch? Mar 10, 2025 pm 06:52 PM

This article compares TensorFlow and PyTorch for deep learning. It details the steps involved: data preparation, model building, training, evaluation, and deployment. Key differences between the frameworks, particularly regarding computational grap

See all articles