Application of attention mechanism in NLP in natural language processing

The concept of attention

The concept of attention is well known in the seq2seq model of neural machine translation. The amount of information passed from the encoder to the decoder is limited, limiting the performance of the model. However, the introduction of attention can overcome this bottleneck and enable the model to better handle long sentences and complex semantics.

Simply put, the model utilizes all hidden states of the encoder during the decoding stage and feeds the final hidden state to the decoder as the initial hidden state. The benefit of this is that the model is able to utilize more information during decoding and is able to "pay attention" to the most relevant parts of the input sequence, making more accurate predictions when generating different parts of the output sequence.

General framework ideas of attention

Although attention mechanisms differ in different architectures and implementation details, they also have Something in common. For example, consider a scenario where we want to make predictions using a neural network with a certain architecture. In this case, we get a vector containing some encoded information. We can use this vector for prediction, such as inputting it into a fully connected layer, and then processing it through a softmax layer. Although the specific processing steps may vary, the basic idea is similar across different architectures.

However, its prediction results proved unsatisfactory. There may be many reasons, the following is a possible idea:

1. The vector used does not contain all the information useful for achieving good predictions.

Often, the information required to make good predictions will be distributed among many vectors, such as in the case of NLP tasks, where there is a sequence of vectors encoded by a marker. Although all distributed information has been accounted for, some information will inevitably be lost as it flows deeper within the neural network.

2. What is useful is not only the individual pieces of information contained in these vectors, but also their relationship to the current vector.

The current vector may need to interact and communicate with other vectors and help determine what information to pass. Therefore, a smarter way is needed to combine all the potentially useful vectors you have and allow the model to learn what to pay attention to in order to make better predictions.

After considering these two points, assume that there is now such a vector and other qualified vectors. These vectors are important for making predictions, and the method of processing this information is A general framework for attention.

In this framework, accept the query and let it interact with the keys one by one:

1. The specific relationship between the query and each key Interactions, which can be inner products or additions or combinations of connections and feeding into small neural networks etc. Each different key of the query is processed using the same operation with the same parameters trained from backpropagation. Furthermore, it is required that the final output after these operations is a single value. These output values ​​are called energy. After repeating this process on the query and all key vectors, a series of energies will be obtained.
2. Use softmax layer to normalize all energies.
3. Perform a weighted sum of the value vectors, and the weight is the normalized energy. This produces a context vector with the same dimensions as a value vector containing information about all elements combined in an intelligent way.
4. Context vectors and query vectors can be used together to make predictions, for example, concatenating them and feeding them to the neural network as needed, followed by a softmax layer.

The above is the process of the general framework of attention. Let’s take a look at how this framework is applied to different tasks.

Emotional analysis task

The emotional analysis type task is a classification task, the input is a piece of text, and the output is corresponding to one of the Labels for possible emotions. Before text is fed into a neural network, it needs to be cleaned, formatted, tokenized, and converted into a series of vocabulary-based indexes. Although this is not seq2seq but seq2one, the attention mechanism still applies and helps improve performance.

Ordinary one-way or two-way LSTM-based networks can be used to perform this task. In this case, only the final hidden state of the last layer (unidirectional) or the two final hidden states (bidirectional, one from forward and one from backward) are used for prediction by passing to the classification head, e.g. fully connected layers and softmax. The limited information carried only by the final hidden state sets a bottleneck for the model's performance.

Date translation task

Date translation is an example of a character-level seq2seq task. The goal of this task is to take a human-readable date as input, such as "December 27, 2022", and output a machine-readable date representing the same date as the input, such as "2022-12-27".

Attention-based models have an attention block before the LSTM unit in the decoder. At each loop step, the output context vector of the attention block and the output of the last step are concatenated and then fed to the LSTM unit. Another implementation of attention is to concatenate the attention block with the output vector of the current step after the LSTM unit and the output context vector, which is then fed to a fully connected layer to predict the next token. The attention block here follows the general framework, the key and value vectors are the same set of vectors, i.e. the hidden states of the last layer of the encoder, and the interaction between the query and each key is a small neural network.

During the decoding stage, the one-way LSTM predicts one token at a time, so the input to each step has two choices: the token predicted by the current step from the previous step output or the ground truth. Here a hyperparameter can be defined to control what percentage of the input tokens used during training are ground truth and can be experimented with to optimize model performance.

The above is the detailed content of Application of attention mechanism in NLP in natural language processing. For more information, please follow other related articles on the PHP Chinese website!