Introduce the definition, usage scenarios, algorithms and techniques of ensemble learning

Ensemble learning is a method that achieves consensus by integrating the salient features of multiple models. By combining predictions from multiple models, ensemble learning frameworks can improve the robustness of predictions and thereby reduce prediction errors. By integrating the different advantages of multiple models, ensemble learning can better adapt to complex data distribution and uncertainty, and improve the accuracy and robustness of predictions.

To understand simply, ensemble learning captures complementary information from different models.

In this article, let’s take a look at the situations in which integrated learning will be used, and what algorithms and techniques are available for integrated learning?

Application of ensemble learning

1. Unable to choose the best model

Different models are Certain distributions in the data set perform better, and an ensemble of models may draw more discriminative decision boundaries between all three classes of data.

2. Excess/insufficiency of data

When a large amount of data is available, we can divide the classification between different classifiers tasks and integrate them within prediction time, rather than trying to train a classifier with large amounts of data. And in cases where the available data set is smaller, a guided integration strategy can be used.

3. Confidence estimation

The core of the ensemble framework is based on the confidence of different model predictions.

4. High problem complexity

A single classifier may not be able to generate appropriate boundaries. An ensemble of multiple linear classifiers can generate any polynomial decision boundary.

5. Information fusion

The most common reason for using ensemble learning models is information fusion to improve classification performance. That is, use a model that has been trained on different data distributions belonging to the same set of categories during prediction time to obtain more robust decisions.

Ensemble learning algorithms and techniques

Bagging integration algorithm

As the earliest proposed ensemble One of the methods. Subsamples are created from a data set and they are called "bootstrap sampling". Simply put, random subsets of the dataset are created using replacement, which means the same data points may exist in multiple subsets.

These subsets are now treated as independent datasets to which multiple machine learning models will be fit. During testing, the predictions of all such models trained on different subsets of the same data are taken into account. Finally there is an aggregation mechanism used to calculate the final prediction.

Parallel processing flows occur in the mechanism of Bagging, whose main purpose is to reduce the variance in ensemble predictions. , therefore, the selected ensemble classifier usually has high variance and low bias.

Therefore, the selected ensemble classifier usually has high variance and low bias.

Boosting integration algorithm

Different from the Bagging integration algorithm, the Boosting integration algorithm does not process data in parallel, but processes the data set sequentially. The first classifier takes in the entire dataset and analyzes the predictions. Instances that fail to produce correct predictions are fed to a second classifier. The ensemble of all these previous classifiers is then computed to make the final prediction on the test data.

The main purpose of the Boosting algorithm is to reduce bias in ensemble decision-making. Therefore, the classifier selected for the ensemble usually needs to have low variance and high bias, i.e. a simpler model with fewer trainable parameters.

stacking ensemble algorithm

The output of this algorithm model is used as the input of another classifier (meta-classifier), and finally Prediction sample. The purpose of using a two-layer classifier is to determine whether the training data has been learned, helping the meta-classifier to correct or improve before making the final prediction.

Mixture of Experts

This method trains multiple classifiers, and then the output is integrated using generalized linear rules. The weights assigned to these combinations are further determined by the "Gating Network", which is also a trainable model, usually a neural network.

Majority Voting

Majority Voting is one of the earliest and simplest integration schemes in the literature. In this method, an odd number of contributing classifiers are selected and the predictions from the classifiers are calculated for each sample. Then, most of the predicted classes considered as sets are obtained from the pool of classifiers.

This method is suitable for binary classification problems because only two candidate classifiers can be voted on. However, methods based on confidence scores are more reliable for now.

Max rule

The "Max rule" ensemble method relies on the probability distribution generated by each classifier. This method uses the concept of "prediction confidence" of the classifier, and for the class predicted by the classifier, the corresponding confidence score is checked. Consider the prediction of the classifier with the highest confidence score as the prediction of the ensemble framework.

Probability average

In this ensemble technique, the probability scores of multiple models are first calculated. Then, the scores of all models across all classes in the dataset are averaged. The probability score is the confidence level in a particular model's prediction. Therefore, the confidence scores of several models are pooled to generate the final probability score of the ensemble. The class with the highest probability after the averaging operation is assigned as the prediction.

Weighted Probability Averaging

Similar to the method of probability averaging, the probability or confidence scores are extracted from different contributing models. But the difference is that a weighted average of the probabilities is calculated. The weight in this method refers to the importance of each classifier, that is, a classifier whose overall performance on the data set is better than another classifier is given a higher importance when calculating the ensemble, thus giving the ensemble framework Better predictive capabilities.

The above is the detailed content of Introduce the definition, usage scenarios, algorithms and techniques of ensemble learning. For more information, please follow other related articles on the PHP Chinese website!