Machine learning hyperparameter tuning tips in Python

With the widespread popularity of machine learning applications, more and more data scientists and machine learning developers have begun to pay attention to model performance optimization, of which hyperparameter tuning is an indispensable part. In machine learning, hyperparameters represent parameters of a model rather than weights learned from training data. Improper hyperparameter settings may lead to poor model performance during training and testing, so hyperparameter tuning is a key step.

Python provides many popular machine learning libraries, such as Scikit-learn, TensorFlow, etc. These libraries provide many tools to help us with hyperparameter tuning. In this article, we will discuss some machine learning hyperparameter tuning tips in Python.

1. Grid search

Grid search is a simple and effective hyperparameter tuning method. Its core idea is to try different hyperparameter combinations and perform cross-validation on each combination to find the best performing hyperparameter combination. In Scikit-learn, we can use the GridSearchCV class to implement grid search.

The following is the general process of grid search:

1) Define the parameters and parameter combinations that need to be tuned

2) Use the GridSearchCV class to perform cross-validation and grid Search

3) Output the best hyperparameter combination

For example, when using Support Vector Machine (SVM) in Scikit-learn for classification, you can use the following code to perform grid search ：

from sklearn.model_selection import GridSearchCV
from sklearn import svm, datasets

iris = datasets.load_iris()
X = iris.data[:, :2]
y = iris.target

parameters = {'kernel':('linear', 'rbf'), 'C':[0.1, 1, 10]}
svc = svm.SVC()
clf = GridSearchCV(svc, parameters)
clf.fit(X, y)

print("Best parameters:", clf.best_params_)

2. Random search

Although grid search is simple and reliable, when the hyperparameter exhaustive space is large, the amount of calculation will be extremely huge. Random search methods alleviate this situation by randomly sampling a set of parameters in the parameter space and evaluating their performance. This method explores a wider parameter space, especially when the parameters have a wide range and are independent of each other. Random search may be more efficient than grid search under the same computing resources.

The following is the general process of random search:

1) Define the parameters and parameter ranges that need to be tuned

2) Use the RandomizedSearchCV class to perform cross-validation and random search

3) Output the best hyperparameter combination

For example, in the Random Forest model, you can use the following code to perform a random search:

from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import RandomizedSearchCV
import numpy as np

# Load data
iris = datasets.load_iris()
X = iris.data[:, :2]
y = iris.target

# Define parameter grid
param_grid = {'n_estimators': np.arange(10, 200, 10),
              'max_features': ['auto', 'sqrt', 'log2'],
              'max_depth' : np.arange(1, 10),
              'criterion' :['gini', 'entropy']
             }

# Random search using 10-fold cross validation
rf = RandomForestClassifier()
rf_random = RandomizedSearchCV(estimator=rf,
                               param_distributions=param_grid,
                               n_iter=100,
                               cv=10,
                               random_state=42)

# Fit the model
rf_random.fit(X, y)

# Print the best parameters
print("Best parameters:", rf_random.best_params_)

3. Bayesian Optimization

Bayesian optimization is an efficient hyperparameter tuning method. This method searches the hyperparameter space by selecting the hyperparameter combination most likely to improve performance in each iteration, gradually converging in the process. This approach uses the results of early runs for inference when trying different parameters, and because the prior probability D is used to model the probability distribution of the parameters of the model, it can optimize the situation where only a small amount of exploration can be performed, and can work well It can handle discrete or continuous parameters, different types of objective functions and noise, and automatically adjust preset search rules. It has significant advantages in multi-dimensional hyperspace optimization.

The following is the general process of Bayesian optimization:

1) Define the parameters and parameter ranges that need to be tuned

2) Define the objective function and use Bayesian Optimize the optimization algorithm

3) Output the best hyperparameter combination

For example, when using the Gradient Boosting Machine (GBM) in Scikit-learn for classification, you can use the following code Bayesian optimization:

from sklearn.datasets import load_iris
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import cross_val_score
from bayes_opt import BayesianOptimization

# Load data
iris = load_iris()
X = iris.data[:, :2]
y = iris.target

# Define objective function
def gbmler_learning_rate(learning_rate):
    gb = GradientBoostingClassifier(learning_rate=learning_rate)
    return cross_val_score(gb, X, y, cv=5).mean()

# Define parameter space
param_space = {'learning_rate': (0.01, 1.0)}

# Initialize optimizer
optimizer = BayesianOptimization(
    f=gbmler_learning_rate,
    pbounds=param_space,
    random_state=42
)

# Optimize
optimizer.maximize(
    init_points=5,
    n_iter=25
)

# Print the best parameters
print("Best parameters:", optimizer.max['params'])

4. Interpretability parameter search

Usually hyperparameter search is to put the classifier idea before the hyperparameter search, for all possible The hyperparameter settings perform some encoding, such as arranging each hyperparameter setting into an array during the search process, and using the array as input. The main problem with performing a search with this approach is that the hyperparameters will appear as individuals and will lose information about their interactive effects. Rather, this approach is part of engaging in simple and interpretable statistical modeling. Another benefit of using Bayesian optimization is the ability to search for relationships between cameras and constrained hyperparameters when using probabilistic model inference.

To summarize, Python provides many techniques for machine learning hyperparameter tuning, including grid search, random search, Bayesian optimization and interpretable parameter search. Choosing a method that fits your dataset can help you achieve better results in hyperparameter tuning.

The above is the detailed content of Machine learning hyperparameter tuning tips in Python. For more information, please follow other related articles on the PHP Chinese website!