How to use Python to discover patterns in data

1. Preparation

Before you start, you must ensure that Python and pip have been successfully installed on your computer.

(Optional 1) If you use Python for data analysis, you can install Anaconda directly, which has Python and pip built-in.

(optional Choose 2) In addition, it is recommended that you use the VSCode editor, which has many advantages

Please choose any of the following methods to enter the command to install dependencies :

1. Open Cmd (Start-Run-CMD) in Windows environment.

2. MacOS environment Open Terminal (command space and enter Terminal).

3. If you are using VSCode editor or Pycharm, you can directly use the Terminal at the bottom of the interface.

pip install pandas
pip install numpy
pip install scipy
pip install seaborn
pip install matplotlib

# 机器学习部分
pip install scikit-learn

2. Statistical description and discovery patterns

Use Python for statistics The description can use some built-in libraries such as Numpy and Pandas.

The following are some basic statistical description functions:

Mean (mean): Calculate the average of a set of data.

import numpy as np

data = [1, 2, 3, 4, 5]
mean = np.mean(data)
print(mean)

The output result is: 3.0

Median (median): Calculate the median of a set of data.

import numpy as np

data = [1, 2, 3, 4, 5]
median = np.median(data)
print(median)

The output result is: 3.0

Mode (mode): Calculate the mode of a set of data.

import scipy.stats as stats

data = [1, 2, 2, 3, 4, 4, 4, 5]
mode = stats.mode(data)
print(mode)

The output result is: ModeResult(mode=array([4]), count=array([3]))

Variance (variance): Calculate the variance of a set of data.

import numpy as np

data = [1, 2, 3, 4, 5]
variance = np.var(data)
print(variance)

The output result is: 2.0

Standard deviation (standard deviation): Calculate the standard deviation of a set of data.

import numpy as np

data = [1, 2, 3, 4, 5]
std_dev = np.std(data)
print(std_dev)

The output result is: 1.4142135623730951

The above are some basic statistical description functions. There are other functions that can be used. For specific usage methods, please view the corresponding documents.

3. Data visualization analysis rules

Python has many libraries that can be used for data visualization, the most commonly used of which are Matplotlib and Seaborn. The following are some basic data visualization methods:

Line plot (line plot): can be used to show trends over time or a certain variable.

import matplotlib.pyplot as plt

x = [1, 2, 3, 4, 5]
y = [2, 4, 6, 8, 10]

plt.plot(x, y)
plt.show()

Scatter plot: Can be used to show the relationship between two variables.

import matplotlib.pyplot as plt

x = [1, 2, 3, 4, 5]
y = [2, 4, 6, 8, 10]

plt.scatter(x, y)
plt.show()

Histogram: can be used to show the distribution of numerical data.

import matplotlib.pyplot as plt

data = [1, 2, 2, 3, 4, 4, 4, 5]

plt.hist(data, bins=5)
plt.show()

Box plot (box plot): can be used to display information such as the median, quartiles, and outliers of numerical data.

import seaborn as sns

data = [1, 2, 2, 3, 4, 4, 4, 5]

sns.boxplot(data)
plt.show()

Bar chart: Can be used to show differences or comparisons between categorical variables.

import matplotlib.pyplot as plt

categories = ['A', 'B', 'C', 'D']
values = [10, 20, 30, 40]

plt.bar(categories, values)
plt.show()

The above are some basic data visualization methods. Both Matplotlib and Seaborn provide richer functions that can be used to create more complex charts and graphics.

4. Grouping and aggregation analysis to discover patterns

In Python, you can use the pandas library to easily group and aggregate data to discover patterns in the data. Here is a basic grouping and aggregation example:

Suppose we have a data set containing sales dates, sales amounts, and salesperson names, and we want to know the total sales for each salesperson. We can group by salesperson name and apply aggregate functions like sum, average, etc. to each group. The following is a sample code:

import pandas as pd

# 创建数据集
data = {'sales_date': ['2022-01-01', '2022-01-02', '2022-01-03', '2022-01-04', '2022-01-05', '2022-01-06', '2022-01-07', '2022-01-08', '2022-01-09', '2022-01-10'],
        'sales_amount': [100, 200, 150, 300, 250, 400, 350, 450, 500, 600],
        'sales_person': ['John', 'Jane', 'John', 'Jane', 'John', 'Jane', 'John', 'Jane', 'John', 'Jane']}

df = pd.DataFrame(data)

# 按销售员名称分组，并对每个组的销售金额求和
grouped = df.groupby('sales_person')['sales_amount'].sum()

print(grouped)

The output result is:

sales_person
Jane 2200
John 1800
Name: sales_amount, dtype: int64

As you can see, we successfully grouped by salesperson name and summed the sales amount of each group. In this way, we can find the total sales of each salesperson and understand the pattern of the data.

5. Machine learning algorithm analysis and discovery of patterns

You can use the scikit-learn library to implement machine learning algorithms and discover patterns in data. The following is a basic example showing how to use the decision tree algorithm to classify data and discover patterns in the data:

import pandas as pd
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

# 创建数据集
data = {'age': [22, 25, 47, 52, 21, 62, 41, 36, 28, 44],
        'income': [21000, 22000, 52000, 73000, 18000, 87000, 45000, 33000, 28000, 84000],
        'gender': ['M', 'F', 'F', 'M', 'M', 'M', 'F', 'M', 'F', 'M'],
        'bought': ['N', 'N', 'Y', 'Y', 'N', 'Y', 'Y', 'N', 'Y', 'Y']}

df = pd.DataFrame(data)

# 将文本数据转换成数值数据
df['gender'] = df['gender'].map({'M': 0, 'F': 1})
df['bought'] = df['bought'].map({'N': 0, 'Y': 1})

# 将数据集分成训练集和测试集
X = df[['age', 'income', 'gender']]
y = df['bought']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

# 创建决策树模型
model = DecisionTreeClassifier()

# 训练模型
model.fit(X_train, y_train)

# 在测试集上进行预测
y_pred = model.predict(X_test)

# 计算模型的准确率
accuracy = accuracy_score(y_test, y_pred)
print("Accuracy: {:.2f}%".format(accuracy*100))

The output result is:

Accuracy: 50.00%

As you can see, we used the decision tree algorithm to classify the data and calculated the accuracy of the model on the test set. In this way, we can discover patterns in the data, such as which factors affect purchasing decisions. It should be noted that this is just a simple example. In actual applications, appropriate machine learning algorithms and feature engineering methods need to be selected based on specific problems.

The above is the detailed content of How to use Python to discover patterns in data. For more information, please follow other related articles on the PHP Chinese website!