Generative and discriminative models

Classification models can be divided into two categories: generative models and discriminative models. This article explains the differences between these two model types and discusses the pros and cons of each approach.

Discriminative model

The discriminative model is a model that can learn the relationship between input data and output labels by learning the characteristics of the input data to predict the output label. In a classification problem, our goal is to assign each input vector x to a label y. Discriminative models attempt to directly learn a function f(x) that maps input vectors to labels. These models can be further divided into two sub-types:

Classifiers try to find f(x) without using any probability distribution. These classifiers directly output a label for each sample without providing a probability estimate of the class. These classifiers are often called deterministic classifiers or distribution-free classifiers. Examples of such classifiers include k-nearest neighbors, decision trees, and SVM.

The classifier first learns the posterior class probabilities P(y = k|x) from the training data, and assigns a new sample x to one of the classes (usually the class with the highest posterior probability) based on these probabilities ).

These classifiers are often called probabilistic classifiers. Examples of such classifiers include logistic regression and neural networks using sigmoid or softmax functions in the output layer.

All things being equal, I generally use a probabilistic classifier rather than a deterministic classifier because this classifier provides additional information about the confidence in assigning a sample to a specific class.

General discriminant models include:

• Logistic Regression (LR)
• Support Vector Machine (SVM)
• Decision Tree (DT)

Generative model

Generative models learn the distribution of inputs before estimating class probabilities. A generative model is a model that can learn the data generation process. It can learn the probability distribution of input data and generate new data samples.

More specifically, the generative model first estimates the conditional density P(x|y = k) of the category and the prior category probability P(y = k) from the training data. They try to understand how the data for each category is generated.

Then use Bayes’ theorem to estimate the posterior class probability:

The denominator of Bayes’ rule can be expressed by the variables appearing in the numerator:

Generative models can also first learn the joint distribution of input and label P(x, y), and then normalize it to get the posterior probability P(y = k |x). Once we have the posterior probabilities, we can use them to assign a new sample x to one of the classes (usually the class with the highest posterior probability).

For example, consider an image classification task where we need to distinguish between images of dogs (y = 1) and cats (y = 0). The generative model will first build a model of dog P(x|y = 1) and a model of cat P(x|y = 0). Then when it classifies a new image, it matches it against both models to see if the new image looks more like a dog or more like a cat.

The generative model allows us to generate new samples from the learned input distribution P(x|y). So we call it a generative model. The simplest example is that for the above model we can generate new dog images by sampling from P(x|y = 1).

General generative models include

• Naive Bayes(Naïve Bayes)
• Gaussian Mixture Models (GMMs)
• Hidden Markov Model (hmm)
• Linear discriminant analysis (LDA)

Deep generative models (DGMs) combine generative models and deep neural networks:

• Since Encoder (Autoencoder, AE)
• Generative Adversarial Network (GAN)
• Autoregressive model, such as GPT (Generative Pre-trained Transformer) is a model that contains billions Parametric autoregressive language models.

Differences, advantages and disadvantages

The main difference between generative models and discriminative models lies in their different learning goals. Generative models learn the distribution of input data and can generate new data samples. Discriminative models learn the relationship between input data and output labels and can predict new labels.

Generative models:

Generative models give us more information because they learn input distributions and class probabilities simultaneously. New samples can be generated from the learned input distribution. And can handle missing data because they can estimate the input distribution without using missing values. But most discriminative models require that all features are present.

Training complexity is high because the generative model requires a large amount of computing and storage resources to establish a joint distribution between input data and output data. The assumption of data distribution is relatively strong, because the generative model needs to establish a joint distribution between the input data and the output data, and needs to assume and model the distribution of the data. Therefore, for complex data distribution, the generative model is suitable for small-scale Not applicable to computing resources.

Generative models can handle multimodal data because generative models can establish multivariate joint distributions between input data and output data, allowing them to handle multimodal data.

Discriminative model:

It is computationally difficult for a generative model to learn the input distribution P(x|y) without making some assumptions about the data. For example, if x consists of m binary Feature Composition,To model P(x|y), we need to estimate 2 ᵐ parameters from the data for each class (these parameters represent the conditional probability of each of the 2 ᵐ combinations of m features). Models such as Naïve Bayes assume conditional independence of features to reduce the number of parameters that need to be learned, so the training complexity is low. But such assumptions often result in generative models performing worse than discriminative models.

It has good performance for complex data distribution and high-dimensional data, because the discriminative model can flexibly model the mapping relationship between input data and output data.

The discriminative model is sensitive to noisy data and missing data, because the model only considers the mapping relationship between input data and output data, and does not use the information in the input data to fill in missing values ​​and remove noise.

Summary

Generative models and discriminative models are both important model types in machine learning. They each have their own advantages and applicable scenarios. In practical applications, it is necessary to select an appropriate model according to the needs of specific tasks, and combine hybrid models and other technical means to improve the performance and effect of the model.

The above is the detailed content of Generative and discriminative models. For more information, please follow other related articles on the PHP Chinese website!