A brief introduction to preprocessing and heatmaps in python

This article brings you a brief introduction to preprocessing and heat maps in python. It has certain reference value. Friends in need can refer to it. I hope it will be helpful to you.

There are still a lot of things in data analysis. I am just giving a heuristic introduction here. After understanding this aspect, I can find solutions faster when using them. I hope that Helpful to everyone.

This time, we still use the iris data set in sklearn and display it through a heat map.

Preprocessing

sklearn.preprocessing is the preprocessing module in the machine learning library. It can standardize, regularize, etc. the data and use it according to needs. Here its standardized method is used to organize the data. Other methods can be queried by yourself.

Standardization: Adjust the distribution of feature data to a standard normal distribution, also called Gaussian distribution, which means that the mean of the data is 0 and the variance is 1.

The reason for standardization is that if the variance of some features is too large, it will dominate the objective function and prevent the parameter estimator from learning other features correctly.

The standardization process is two steps: decentralization of the mean (the mean becomes 0); scaling of the variance (the variance becomes 1).

A scale method is provided in sklearn.preprocessing to achieve the above functions.

Let’s take an example:

from sklearn import preprocessing
import numpy as np
# 创建一组特征数据，每一行表示一个样本，每一列表示一个特征
xx = np.array([[1., -1., 2.],
              [2., 0., 0.],
              [0., 1., -1.]])
# 将每一列特征标准化为标准正太分布，注意，标准化是针对每一列而言的
xx_scale = preprocessing.scale(xx)
xx_scale

The result after normalizing the data in each column is:

array([[ 0.        , -1.22474487,  1.33630621],
       [ 1.22474487,  0.        , -0.26726124],
       [-1.22474487,  1.22474487, -1.06904497]])

You can see that the data inside has occurred The changes, the numerical values ​​are relatively small, maybe someone can see it at a glance, but it doesn’t matter if they can’t see it. Python can easily calculate some of their statistics.

 # 测试一下xx_scale每列的均值方差
 print('均值：', xx_scale.mean(axis=0))  # axis=0指列，axis=1指行
 print('方差：', xx_scale.std(axis=0))

The above has introduced what standardization is to be converted into, and the results are indeed consistent. The results of calculating the mean and variance by column are:

均值： [0. 0. 0.]
方差： [1. 1. 1.]

Of course, for standardization, the variance and mean It doesn’t have to be done together. For example, sometimes you just want to benefit from one of the methods. There is also a way:

with_mean,with_std. Both of these are boolean parameters, and they are both true by default, but It can also be customized to false. That is, do not want mean centering or variance scaling to 1.

heatmap

I will only briefly mention the heatmap here. Because there is already a lot of detailed information about it on the Internet.

In a heat map, the data exists in the form of a matrix, and the attribute range is represented by a gradient of color. Here, pcolor is used to draw the heat map.

小 Lizi

Still start from the import library, then load the data set, process the data, then draw the image, make some annotations and decorations on the image, etc. I am used to making comments in the code. If there is anything you don’t understand, you can leave a message and I will reply in time.

# 导入后续所需要的库
from sklearn.datasets import load_iris
from sklearn.preprocessing import scale
import numpy as np
import matplotlib.pyplot as plt
# 加载数据集
data = load_iris()
x = data['data']
y = data['target']
col_names = data['feature_names']
# 数据预处理
# 根据平均值对数据进行缩放
x = scale(x, with_std=False)
x_ = x[1:26,] # 选取其中25组数据
y_labels = range(1, 26)
# 绘制热图
plt.close('all')
plt.figure(1)
fig, ax = plt.subplots()
ax.pcolor(x_, cmap=plt.cm.Greens, edgecolors='k')
ax.set_xticks(np.arange(0, x_.shape[1])+0.5) # 设置横纵坐标
ax.set_yticks(np.arange(0, x_.shape[0])+0.5)
ax.xaxis.tick_top() # x轴提示显示在图形上方
ax.yaxis.tick_left() # y轴提示显示在图形的左侧
ax.set_xticklabels(col_names, minor=False, fontsize=10) # 传递标签数据
ax.set_yticklabels(y_labels, minor=False, fontsize=10)
plt.show()

So what does the drawn image look like:

Just follow these simple steps The data draws an intuitive image. Of course, it will not be so simple when it is actually used, and more knowledge needs to be expanded.

The above is the detailed content of A brief introduction to preprocessing and heatmaps in python. For more information, please follow other related articles on the PHP Chinese website!