Python code optimization tips

Code Optimization Part 1

Share some tips on code optimization that I have seen recently.

Short-circuit characteristics of if judgment

For and, the conditions that meet the fewest conditions should be placed first, so that when a large number of judgments are made, the conditions that satisfy the fewest conditions will directly cause other subsequent expressions not to be calculated, thus saving time (because False and True or False)

import timeit

s1 = """
a = range(2000)
[i for i in a if i % 2 ==0 and i > 1900]
"""

s2 = """
a = range(2000)
[i for i in a if  i > 1900 and i % 2 ==0]
"""

print timeit.timeit(stmt=s1, number=1000)
print timeit.timeit(stmt=s2, number=1000)

The operation results are as follows:

➜  python test6.py
0.248532056808
0.195827960968

# 可以看到s2 表达式计算更快， 因为大部分情况都不满足 i>1900, 所以这些情况下, i % 2 == 0 也没有计算，从而节约了时间

Similarly for or, put the one that meets the most conditions first.

import timeit

s1 = """
a = range(2000)
[i for i in a if 10 < i <20 or 1000 < i < 2000]
"""

s2 = """
a = range(2000)
[i for i in a if 1000 < i < 2000 or 10 < i <20]
"""

print timeit.timeit(stmt=s1, number=1000)
print timeit.timeit(stmt=s2, number=1000)

Run results:

0.253124952316
0.202992200851

join merge strings

join merge strings faster than looping + to merge.

import timeit

s1 = """
a = [str(x) for x in range(2000)]
s = ''
for i in a:
    s += i
"""

s2 = """
a = [str(x) for x in range(2000)]
s = ''.join(a)
"""

print timeit.timeit(stmt=s1, number=1000)
print timeit.timeit(stmt=s2, number=1000)

The running results are as follows:

python test6.py

0.558945894241
0.422435998917

while 1 and while True

In python2.x, True and False are not reserved keywords, but a global variable, which means you can do this

>>> True = 0
>>> True
0
>>> if not True:
...   print '1'
...
1

So the following two In this case:

import timeit

s1 = """
n = 1000000
while 1:
    n -= 1
    if n <= 0: break
"""

s2 = """
n = 1000000
while True:
    n -= 1
    if n <= 0: break
"""

print timeit.timeit(stmt=s1, number=100)
print timeit.timeit(stmt=s2, number=100)

The operation result is as follows:

➜  python test6.py
5.18007302284
6.84624099731

Because every time when judging "while True", we must first find the value of True.

In python3.x, True becomes a keyword argument, so the above two situations are the same.

cProfile, cStringIO and cPickle

Extensions written using the C version are faster than the native ones. cPickle vs pickle is as follows:

import timeit

s1 = """
import cPickle
import pickle
n = range(10000)
cPickle.dumps(n)
"""

s2 = """
import cPickle
import pickle
n = range(10000)
pickle.dumps(n)
"""

print timeit.timeit(stmt=s1, number=100)
print timeit.timeit(stmt=s2, number=100)

The running results are as follows:

➜ python test6.py
0.182178974152
1.70917797089

Use the generator appropriately

Difference

Using () to get a generator object, the memory space required has nothing to do with the size of the list, so the efficiency will be higher .

import timeit

s1 = """
[i for i in range (100000)]
"""

s2 = """
(i for i in range(100000))
"""

print timeit.timeit(stmt=s1, number=1000)
print timeit.timeit(stmt=s2, number=1000)

Result:

➜  python test6.py
5.44327497482
0.923446893692

But for situations where loop traversal is required: using iterators is not efficient, as follows:

import timeit

s1 = """
ls = range(1000000)
def yield_func(ls):
    for i in ls:
        yield i+1
for x in yield_func(ls):
    pass
"""

s2 = """
ls = range(1000000)
def not_yield_func(ls):
    return [i+1 for i in ls]
for x in not_yield_func(ls):
    pass
"""

print timeit.timeit(stmt=s1, number=10)
print timeit.timeit(stmt=s2, number=10)

The result is as follows:

➜  python test6.py
1.03186702728
1.01472687721

So using a generator is a trade-off, for memory and speed. Consider the results.

xrange

在python2.x里xrange 是纯C实现的生成器，相对于range来说，它不会一次性计算出所有值在内存中。但它的限制是只能和整型一起工作：你不能使用long或者float。

import 语句的开销

import语句有时候为了限制它们的作用范围或者节省初始化时间，被卸载函数内部，虽然python的解释器不会重复import同一个模块不会出错，但重复导入会影响部分性能。有时候为了实现懒加载（即使用的时候再加载一个开销很大的模块），可以这么做：

email = None

def parse_email():
    global email
    if email is None:
        import email
    ...

# 这样一来email模块仅会被引入一次，在parse_email()被第一次调用的时候。