Home > Technology peripherals > AI > body text

ReLU replaces softmax in visual Transformer, DeepMind's new trick reduces costs rapidly

PHPz
Release: 2023-09-20 20:53:02
forward
1301 people have browsed it

The Transformer architecture has been widely used in the field of modern machine learning. The key point is to focus on one of the core components of transformer, which contains a softmax, which is used to generate a probability distribution of tokens. Softmax has a higher cost because it performs exponential calculations and summing sequence lengths, which makes parallelization difficult to perform.

Google DeepMind thought of a new idea: Replace the softmax operation with a new method that does not necessarily output a probability distribution. They also observed that using ReLU divided by the sequence length can approach or rival traditional softmax when used with a visual Transformer.

ReLU replaces softmax in visual Transformer, DeepMinds new trick reduces costs rapidly

Paper link: https://arxiv.org/abs/2309.08586

This result Brings new solutions to parallelization, because ReLU can be parallelized in the sequence length dimension, and requires fewer gather operations than traditional ones

Method

The key point is to concentrate

The key point is to concentrate on the function Convert d-dimensional queries, keys and values ​​{q_i, k_i, v_i} through a two-step process

In the first step, it is important to focus on getting the key points by Force weight ReLU replaces softmax in visual Transformer, DeepMinds new trick reduces costs rapidly

ReLU replaces softmax in visual Transformer, DeepMinds new trick reduces costs rapidly

##where ϕ is usually softmax.

The next step, using this focus is to focus on weights to calculate the output This paper explores the use of point-wise calculations as an alternative to ϕ. ReLU replaces softmax in visual Transformer, DeepMinds new trick reduces costs rapidly

The key point of ReLU is to focus on

DeepMind observed that for ϕ = softmax in Eq. 1, ReLU replaces softmax in visual Transformer, DeepMinds new trick reduces costs rapidly is a better alternative. They will use ReLU replaces softmax in visual Transformer, DeepMinds new trick reduces costs rapidlyfocus is called ReLU.

Expanded point-by-point focus is to focus

The researchers also experimentally explored more A wide range of ReLU replaces softmax in visual Transformer, DeepMinds new trick reduces costs rapidly choices, where α ∈ [0, 1] and h ∈ {relu,relu², gelu,softplus, identity,relu6,sigmoid}.

What needs to be rewritten is: the extension of sequence length

They also found that if using a Expanding items with sequence length L can improve accuracy. Previous research work trying to remove softmax has not used this extension scheme

Among the Transformers currently designed to focus on using softmax, there is ReLU replaces softmax in visual Transformer, DeepMinds new trick reduces costs rapidly , which means ReLU replaces softmax in visual Transformer, DeepMinds new trick reduces costs rapidly although this is unlikely to be A necessary condition, but ReLU replaces softmax in visual Transformer, DeepMinds new trick reduces costs rapidly can ensure that the complexity of ReLU replaces softmax in visual Transformer, DeepMinds new trick reduces costs rapidly during initialization is ReLU replaces softmax in visual Transformer, DeepMinds new trick reduces costs rapidly , retain this Conditions may reduce the need to change other hyperparameters when replacing softmax.

At the time of initialization, the elements of q and k are O (1), so ReLU replaces softmax in visual Transformer, DeepMinds new trick reduces costs rapidly will also be O (1). Activation functions like ReLU maintain O (1), so a factor of ReLU replaces softmax in visual Transformer, DeepMinds new trick reduces costs rapidly is needed to make ReLU replaces softmax in visual Transformer, DeepMinds new trick reduces costs rapidly have a complexity of ReLU replaces softmax in visual Transformer, DeepMinds new trick reduces costs rapidly.

Experiments and results

Main results

Figure 1 Description In terms of ImageNet-21k training, ReLU focuses on focusing and softmax focuses on the scaling trend. The x-axis shows the total kernel computation time required for the experiment in hours. A big advantage of ReLU is that it can be parallelized in the sequence length dimension, requiring fewer gather operations than softmax.

ReLU replaces softmax in visual Transformer, DeepMinds new trick reduces costs rapidly

The content that needs to be rewritten is: the effect of extending the sequence length

Figure 2 compares what needs to be rewritten: the results of the sequence length extension method and various other point-by-point solutions that replace softmax. Specifically, it is to use relu, relu², gelu, softplus, identity and other methods to replace softmax. The X-axis is α. The Y-axis is the accuracy of the S/32, S/16, and S/8 Vision Transformer models. The best results are usually obtained when α is close to 1. Since there is no clear optimal nonlinearity, they used ReLU in their main experiments because it is faster.

ReLU replaces softmax in visual Transformer, DeepMinds new trick reduces costs rapidly

## The effect of qk-layernorm can be restated as follows:

The main experiments used qk-layernorm, where queries and keys are passed through LayerNorm before calculating weights. DeepMind states that the reason for using qk-layernorm by default is that it is necessary to prevent instability when scaling model sizes. Figure 3 shows the impact of removing qk-layernorm. This result indicates that qk-layernorm has little impact on these models, but the situation may be different when the model size becomes larger.

ReLU replaces softmax in visual Transformer, DeepMinds new trick reduces costs rapidly

##Redescription: The additional effect of the door

Previous research on removing softmax has adopted the method of adding a gating unit, but this method cannot scale with the sequence length. Specifically, in the gated attention unit, there is an additional projection that produces an output that is obtained by an element-wise multiplicative combination before the output projection. Figure 4 explores whether the presence of gates eliminates the need for rewriting what is: an extension of the sequence length. Overall, DeepMind observes that the best accuracy is achieved with or without gates, with and without gates, by requiring rewriting: Sequence length extensions. Also note that for the S/8 model using ReLU, this gating mechanism increases the core time required for the experiment by approximately 9.3%.

ReLU replaces softmax in visual Transformer, DeepMinds new trick reduces costs rapidly

The above is the detailed content of ReLU replaces softmax in visual Transformer, DeepMind's new trick reduces costs rapidly. For more information, please follow other related articles on the PHP Chinese website!

Related labels:
source:51cto.com
Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn
Popular Tutorials
More>
Latest Downloads
More>
Web Effects
Website Source Code
Website Materials
Front End Template