The most complete collection of Transformers in history! LeCun recommends: Create a catalog for 60 models. Which paper have you missed?

If there is something that has supported the development of large-scale models in the past few years, it must be Transformer!

Based on Transformer, a large number of models are springing up in various fields. Each model has a different architecture, different details, and a name that is not easy to explain.

Recently, an author conducted a comprehensive classification of all popular Transformer models released in recent years. And index, try to provide a comprehensive but simple catalog . The article includes an introduction to Transformer innovation and a review of the development process.

Paper link: ​https://arxiv.org/pdf/2302.07730.pdf​

Turing Award winner Yann LeCun expressed his approval.

##The author of the article, Xavier (Xavi) Amatriain, graduated with a PhD from Pompeu Fabra University in Spain in 2005 and is currently an engineer at LinkedIn Vice President of the Department, mainly responsible for product artificial intelligence strategy.

What is Transformer?

Transformer is a type of deep learning model with some unique architectural features. It first appeared in the famous "Attention is All you Need" paper published by Google researchers in 2017. The paper was published in In just 5 years, it has accumulated an astonishing 38,000 citations.

The Transformer architecture also belongs to the encoder-decoder model (encoder-decoder), but in the previous models, attention was only one of the mechanisms, and most of them were based on LSTM. (Long Short-Term Memory) and other variants of RNN (Recurrent Neural Network).

One of the key insights of the paper that proposes Transformer is that the attention mechanism can be used as the only mechanism to derive the dependency between input and output. This paper does not intend to To delve into all the details of the Transformer architecture, interested friends can search the "The Illustrated Transformer" blog.

Blog link: ​https://jalammar.github.io/illustrated-transformer/​

Only the most important components are briefly described below.

Encoder-Decoder Architecture​

a The general encoder/decoder architecture consists of two models, the encoder takes the input and encodes it into a fixed-length vector; the decoder takes the vector and decodes it into an output sequence.

The encoder and decoder are jointly trained to minimize the conditional log-likelihood. Once trained, the encoder/decoder can generate an output based on a given input sequence, or it can score a pair of input/output sequences.

Under the original Transformer architecture, both the encoder and the decoder have 6 identical layers. In each of these 6 layers, the encoder has two sub-layers: a multi-head attention layer, and a Simple feedforward network with one residual connection and one layer normalization for each sub-layer.

The output size of the encoder is 512, and the decoder adds a third sub-layer, another multi-head attention layer on the encoder output. In addition, another multi-head layer in the decoder is masked out to prevent information leakage from applying attention to subsequent positions.

Attention mechanism

From the above description As can be seen, the only "strange" element in the model's structure is the attention of the bulls, and this is where the model's entire power lies.

The attention function is a mapping between query and a set of key-value pairs to the output. The output is calculated as a weighted sum of values, where the weight assigned to each value is given by Calculated by the compatibility function between query and corresponding key.

Transformer uses multi-head attention, which is the parallel calculation of a set of attention functions, also known as scaling dot product attention.

Compared with recurrent and convolutional networks, the attention layer has several advantages. The more important ones are its lower computational complexity and higher connectivity, which is good for learning sequences. Particularly useful for long-term dependencies in .

What can Transformer do? Why did it become popular? ​

The original Transformer was designed for language translation, mainly from English to German, but the first version of the paper Experimental results have shown that the architecture generalizes well to other language tasks.

This particular trend was quickly noticed by the research community.

In the next few months, the rankings of any language-related ML task will be completely occupied by some version of the Transformer architecture, such as the question and answer task Squad soon It was killed by various Transformer models.

One of the key reasons why Transofrmer can occupy most NLP rankings so quickly is: their ability to quickly adapt to other tasks, that is, transfer learning; pre-trained Transformer models can be very Easily and quickly adapt to tasks for which they have not been trained, a huge advantage over other models.

As an ML practitioner, you no longer need to train a large model from scratch on a huge data set, just reuse the pretrained model on the task at hand, maybe Just tweak it slightly with a much smaller data set.

The specific technique used to adapt pre-trained models to different tasks is so-called fine-tuning.

It turns out that Transformers are so adaptable to other tasks that although they were originally developed for language-related tasks, they quickly became useful for other tasks, From visual or audio and music applications all the way to playing chess or doing math.

Of course, none of these applications would be possible if it weren't for the myriad of tools readily available to anyone who can write a few lines of code.

Transformer was not only quickly integrated into major artificial intelligence frameworks (i.e. Pytorch and TensorFlow), but there were also some companies that were entirely built for Transformer.

Huggingface, a startup that has raised over $60 million to date, was built almost entirely around the idea of ​​commercializing their open source Transformer library.

GPT-3 is a Transformer model launched by OpenAI in May 2020. It is a subsequent version of their earlier GPT and GPT-2. The company created a lot of buzz by introducing the model in a preprint, claiming that the model was so powerful that they were not qualified to release it to the world.

Moreover, OpenAI not only did not release GPT-3, but also achieved commercialization through a very large partnership with Microsoft.

Today, GPT-3 provides underlying technical support for more than 300 different applications and is the foundation of OpenAI’s business strategy. That's significant for a company that has received more than $1 billion in funding.

RLHF

From human feedback (or preferences ), also known as RLHF (or RLHP), has recently become a huge addition to the artificial intelligence toolbox.

This concept first came from the 2017 paper "Deep Reinforcement Learning from Human Preferences", but recently it has been applied to ChatGPT and similar conversational agents, and has achieved quite good results. The effect has attracted public attention again.

The idea in the article is very simple. Once the language model is pre-trained, it can produce different effects on the dialogue. responses and have humans rank the results, these rankings (also known as preferences or feedback) can be used to train rewards using reinforcement learning mechanisms.

Diffusion modelDiffusion##​

The diffusion model has become The new SOTA for image generation has a tendency to replace GANs (Generative Adversarial Networks).

The diffusion model is a type of trained latent variable model of variational inference. In practice, it means training a deep neural network to use a certain noise function. Blurred images are denoised.

A network trained in this way is actually learning the latent space represented by these images.

After reading the introduction, let’s start the retrospect journey of Transformer!

The above is the detailed content of The most complete collection of Transformers in history! LeCun recommends: Create a catalog for 60 models. Which paper have you missed?. For more information, please follow other related articles on the PHP Chinese website!