python黑魔法之编码转换
我们在使用其他语言的库做编码转换时，对于无法理解的字符，通常的处理也只有两种（或三种）：
- 抛异常
- 替换成替代字符
-
跳过
但是在复杂的现实世界中，由于各种不靠谱，我们处理的文本总会出现那么些不和谐因素，比如混合编码。在这种情况下，又回到了上面的处理办法。
那么问题来了，python有没有更好地办法呢？
答案是，有！
python的编码转换流程实际上是两段式转换：
source -> unicode -> dest
首先将字符串从原始编码转换成unicode。再将unicode转换成目标编码。
第一步我们一般采用decode()或者 unicode() 这两个函数完成。
第二步我们使用encode()函数完成。
在这里我们说的黑魔法就是在第一步实现。
decode和unicode函数都有一个叫做errors的可选参数。看看官方的描述：
- errors may be given to set a different error
- handling scheme. Default is 'strict' meaning that encoding errors raise
- a UnicodeDecodeError. Other possible values are 'ignore' and 'replace'
- as well as any other name registered with codecs. register_error that is
- able to handle UnicodeDecodeErrors.
这个参数通常有三种值：
- strict 默认值。如果出现编码错误，则会抛出UnicodeDecodeError。
- ignore 跳过。
- replace 用?替换。
好了，看到最后一句话了吗？好戏上演了！
模块codec有一个函数叫做register_error。他的作用让用户可以注册自定义的errors处理方法。
用来处理UnicodeDecodeError。
我们看看函数原型：
codecs.register_error(name, error_handler)
name: 错误处理的名称。用以填写在decode函数的error参数中。
error_handler: 处理函数。该函数接受一个异常参数。
返回一个tuple，该tuple有2个元素，第一个是纠错后的字符串，第二个是继续decode的起始位置
有了上面的基本概念。我们看下具体实现：
def cjk_error(e):
if not isinstance(e, UnicodeDecodeError):
raise TypeError("don't know how to handle %r" % exc)
if exc.end + 1 > len(exc.object):
raise TypeError('unknown codec ,the object too short!')
ch1 = ord(exc.object[exc.start:exc.end])
newpos = exc.end + 1
ch2 = ord(exc.object[exc.start + 1:newpos])
sk = exc.object[exc.start:newpos]
if 0x81<=ch1<=0xFE and (0x40<=ch2<=0x7E or 0x7E<=ch2<=0xFE): # GBK
return (unicode(sk,'cp936'), newpos)
if 0x81<=ch1<=0xFE and (0x40<=ch2<=0x7E or 0xA1<=ch2<=0xFE): # BIG5
return (unicode(sk,'big5'), newpos)
raise TypeError('unknown codec !')
codecs.register_error("cjk_replace", cjk_replace)
上面这个是我从网上copy的。开始我觉得很不错，但是后来发现是个很不经推敲的算法。
比如utf8和gbk在前两个字节就有交集的部分。当一个utf8的字符串以gbk编码decode的时候，出现错误是从第三个字节开始（前两个字节也能够在gbk编码范围中对应到一个汉字）。
如：
a = "你" # utf8编码：'\xe4\xbd\xa0' c = unicode(a[:2],'gbk') # 正常返回 c = unicode(a, 'gbk') # UnicodeDecodeError 。错误发生在第三个字节
所以针对这种情况，做了下改进：
import codec
def cjk_replace(e):
if not isinstance(e, UnicodeDecodeError):
raise TypeError("invalid exception type %s" e)
src = e.encoding
if src in ('gbk','gb18030', 'big5'):
beg = e.start - 2
if beg >= 0:
try:
return unicode(e.object[beg:e.end], 'utf8'), e.end + 1
except:
pass
if exc.end + 1 > len(exc.object):
raise TypeError('unknown codec ,the object too short!')
ch1 = ord(exc.object[exc.start:exc.end])
newpos = exc.end + 1
ch2 = ord(exc.object[exc.start + 1:newpos])
sk = exc.object[exc.start:newpos]
if src != 'gbk' and 0x81<=ch1<=0xFE and (0x40<=ch2<=0x7E or 0x7E<=ch2<=0xFE): # GBK
return (unicode(sk,'cp936'), newpos)
if src != 'big5' and 0x81<=ch1<=0xFE and (0x40<=ch2<=0x7E or 0xA1<=ch2<=0xFE): # BIG5
return (unicode(sk,'big5'), newpos)
raise TypeError('unknown codec !')
codecs.register_error("cjk_replace", cjk_replace)
当然，这个逻辑其实还是不够严谨的。虽然对于这种混合编码这种畸形活处理有点较真儿。
不过既然python提供这样的能力，大家可以一起来讨论下，我们怎么可以做的更好?
Hot AI Tools
Undresser.AI Undress
AI-powered app for creating realistic nude photos
AI Clothes Remover
Online AI tool for removing clothes from photos.
Undress AI Tool
Undress images for free
Clothoff.io
AI clothes remover
AI Hentai Generator
Generate AI Hentai for free.
Hot Article
Hot Tools
Notepad++7.3.1
Easy-to-use and free code editor
SublimeText3 Chinese version
Chinese version, very easy to use
Zend Studio 13.0.1
Powerful PHP integrated development environment
Dreamweaver CS6
Visual web development tools
SublimeText3 Mac version
God-level code editing software (SublimeText3)
Hot Topics
How to solve the permissions problem encountered when viewing Python version in Linux terminal?
Apr 01, 2025 pm 05:09 PM
Solution to permission issues when viewing Python version in Linux terminal When you try to view Python version in Linux terminal, enter python...
How to efficiently copy the entire column of one DataFrame into another DataFrame with different structures in Python?
Apr 01, 2025 pm 11:15 PM
When using Python's pandas library, how to copy whole columns between two DataFrames with different structures is a common problem. Suppose we have two Dats...
Python hourglass graph drawing: How to avoid variable undefined errors?
Apr 01, 2025 pm 06:27 PM
Getting started with Python: Hourglass Graphic Drawing and Input Verification This article will solve the variable definition problem encountered by a Python novice in the hourglass Graphic Drawing Program. Code...
Python Cross-platform Desktop Application Development: Which GUI Library is the best for you?
Apr 01, 2025 pm 05:24 PM
Choice of Python Cross-platform desktop application development library Many Python developers want to develop desktop applications that can run on both Windows and Linux systems...
Do Google and AWS provide public PyPI image sources?
Apr 01, 2025 pm 05:15 PM
Many developers rely on PyPI (PythonPackageIndex)...
How to efficiently count and sort large product data sets in Python?
Apr 01, 2025 pm 08:03 PM
Data Conversion and Statistics: Efficient Processing of Large Data Sets This article will introduce in detail how to convert a data list containing product information to another containing...
How to optimize processing of high-resolution images in Python to find precise white circular areas?
Apr 01, 2025 pm 06:12 PM
How to handle high resolution images in Python to find white areas? Processing a high-resolution picture of 9000x7000 pixels, how to accurately find two of the picture...
How to solve the problem of file name encoding when connecting to FTP server in Python?
Apr 01, 2025 pm 06:21 PM
When using Python to connect to an FTP server, you may encounter encoding problems when obtaining files in the specified directory and downloading them, especially text on the FTP server...
