Python: How Pandas operates efficiently

This article conducts a comparative test on the operating efficiency of Pandas to explore which methods can make the operating efficiency better.

The test environment is as follows:

• windows 7, 64-bit

• python 3.5

• pandas 0.19.2

• numpy 1.11.3

• ##jupyter notebook

Required It should be noted that different systems, different computer configurations, and different software environments may have different operating results. Even if it is the same computer, the results will not be exactly the same every time it is run.

1 Test content

The test content is to use three methods to calculate a simple operation process, namely a*a+b*b.

The three methods are:

1. Python’s for loop

2. Pandas’ Series

3. Numpy's ndarray

First construct a DataFrame. The size of the data amount, that is, the number of rows of the DataFrame, are 10, 100, 1000, ..., until 10,000,000 (one millions).

Then in jupyter notebook, use the following codes to test respectively to check the running time of different methods and make a comparison.

import pandas as pdimport numpy as np# 100分别用 10，100，...，10,000,000来替换运行list_a = list(range(100))# 200分别用 20，200，...，20,000,000来替换运行list_b = list(range(100,200))
print(len(list_a))
print(len(list_b))

df = pd.DataFrame({'a':list_a, 'b':list_b})
print('数据维度为：{}'.format(df.shape))
print(len(df))
print(df.head())

100
100
数据维度为：(100, 2)
100
   a    b
0  0  100
1  1  101
2  2  102
3  3  103
4  4  104

• Perform the operation, a*a + b*b

• Method 1: for loop

%%timeit# 当DataFrame的行数大于等于1000000时，请用 %%time 命令for i in range(len(df)):
    df['a'][i]*df['a'][i]+df['b'][i]*df['b'][i]

100 loops, best of 3: 12.8 ms per loop

• Method 2: Series

type(df['a'])

pandas.core.series.Series

%%timeit
df['a']*df['a']+df['b']*df['b']

The slowest run took 5.41 times longer than the fastest. This could mean that an intermediate result is being cached.
1000 loops, best of 3: 669 µs per loop

• Method 3: ndarray

type(df['a'].values)

numpy.ndarray

%%timeit
df['a'].values*df['a'].values+df['b'].values*df['b'].values

10000 loops, best of 3: 34.2 µs per loop

2 Test results

The running results are as follows:

It can be seen from the running results , the for loop is obviously much slower than Series and ndarray, and the larger the amount of data, the more obvious the difference.

When the amount of data reaches 10 million rows, the performance of the for loop is more than 10,000 times worse. The difference between Series and ndarray is not that big.

PS: When there are 10 million rows, the for loop takes a very long time to run. If you want to test it, you need to pay attention. Please use the

%%time command (only test once).

The following chart compares the performance between Series and ndarray.

As can be seen from the above figure, when the data is less than 100,000 rows, ndarray performs better than Series. When the number of data rows is greater than 1 million rows, Series performs slightly better than ndarray. Of course, the difference between the two is not particularly obvious.

So under normal circumstances, I personally recommend that

for loops be used if possible. When the number is not particularly large, it is recommended to use ndarray (i.e. df['col'].values) To perform calculations, the operating efficiency is relatively better.

The above is the detailed content of Python: How Pandas operates efficiently. For more information, please follow other related articles on the PHP Chinese website!