Google reuses classic algorithms from 30 years ago, and CV introduces reinforcement learning. Netizens: Is visual RLHF coming?

The popularity of ChatGPT is obvious to all, and for the technology behind its success, supervised instruction fine-tuning and reinforcement learning based on human feedback are crucial. These techniques are also gradually expanding into other AI fields, including computer vision (CV). ​

We know that when dealing with complex output in computer vision, the main criterion for success is not how well the model optimizes the training target, but how well the prediction ability matches the task. That is, how well the model performs for its intended use.

In order to pursue this consistency, some researchers have made some improvements in model architecture, data, optimization, sampling, post-processing, etc. For example, in the object detection task, researchers used NMS (non-maximum suppression), set-based global loss, and changing the input data to obtain models with improved behavior at test time. Although these methods bring significant benefits, they are often only useful for specific tasks and only indirectly optimize mission risks. ​

Not only CV, but also natural language processing (NLP), reinforcement learning (RL) and other fields are also extensively studying this phenomenon. In these domains, it is very difficult to formulate optimization goals for less well-defined tasks, such as translation or summarization generation. When dealing with this type of problem, A popular approach is to learn to imitate the output of the example and then perform reinforcement learning to align the model with the reward function. The field of NLP has produced exciting results using this approach, which uses large pre-trained language models and rewards defined by human feedback to tackle otherwise difficult-to-specify tasks.

Furthermore, the same method is widely used in image captioning tasks, where CIDEr (proposed by Vedantam et al. 2015) is used as reward. Nonetheless, to the best of our knowledge, reward optimization has not been previously explored in (non-textual) computer vision tasks.

Recently, researchers from the Google Brain team demonstrated in the paper "Tuning computer vision models with task rewards" that uses the REINFORCE algorithm (proposed by Williams in 1992) Tuning pre-trained models with reward functions can be used out of the box for a variety of computer vision tasks.

In fact, many studies on reinforcement learning tasks will mention Williams' REINFORCE algorithm, which shows the importance of this algorithm. It can be said that the REINFORCE algorithm is a typical representative of policy gradient and even reinforcement learning.

Paper address: https://arxiv.org/pdf/2302.08242v1.pdf​

Figure 1 Some key results are presented, mainly including quantitative and qualitative improvements resulting from reward optimization in object detection, panoramic segmentation, and image colorization. The method proposed in this study is simple and effective in handling various CV tasks, demonstrating its versatility and adaptability. Although this paper mainly uses rewards in the form of evaluation metrics, these preliminary results show that this method can also be an effective way to optimize computer vision models that have more complex and difficult to specify rewards, such as human feedback or overall systems. performance.

Twitter netizens gave a relatively comprehensive summary of this article, that is, the function implemented in this article is to use RL to adjust the pre-trained visual model. The motivation for the research was inspired by the success of LLM reinforcement learning; the effect was a significant improvement in performance in target detection, panoramic segmentation, etc. He also stated that this research may be an effective way to achieve visual RLHF (Reinforcement Learning from Human Feedback).

## Source: https://twitter.com/johnjnay/status /1627009121378598912

Reward​

Without losing generalization, this study describes the CV task as the process of learning a function that maps the input x (i.e., the image) to the output y = [y_1, y_1,..., y_n] (text token sequence, bounding box sequence, etc.). This research aims to learn a conditional distribution P (y|x, θ) with θ as a parameter to maximize the reward function R. To use an abstract formula to describe it, this article will solve the following optimization problems.

The problem is there, and the next step is how to solve it. This article is divided into two steps: first, use maximum likelihood estimation Pre-train the model; then use the REINFORCE algorithm to tune the model. Let’s take a look at the specific process of these two steps:

##Maximum likelihood pre-training​

The parameters θ are first estimated using the maximum likelihood principle and the distribution of the training data is captured. This is accomplished using the gradient descent algorithm, which works by maximizing the log-likelihood

of the training data. Algorithm 1 and Figure 2 describe the MLE (maximum likelihood estimation) optimization step, which is the most common method for training models. Completing this step will result in an MLE model.

##REINFORC algorithm maximizes rewards ​

In order to better optimize the MLE model to adapt to task risks, it is also necessary to maximize the reward function. For a given input x, this study uses the REINFORCE algorithm to estimate the gradient of the expected reward for a given x. The formula is as follows:

Algorithm 2 Pseudocode is provided and Figure 3 illustrates the process:

##Experimental results

Next, let’s take a look at the performance of the method proposed in this article on visual tasks.

Panoramic Segmentation

As shown in Table 1 below, the Tuning process significantly improves the MLE model. Results after visual inspection show that the model after Tuning is better at avoiding incoherent predictions, especially for small-scale objects, see Figure 1.

Target Detection

Table 2 shows that through optimization, this study significantly improved the mAP score of the original MLE model from 39.2% to 54.3%. In Pix2seq, the same-sized ViT-B model with a slightly larger 1333×1333 resolution and many heuristics achieved 47.1%. When using the larger ViT-L backbone, Pix2seq reported the best target detection result of 50.0%.

Coloring

​

The qualitative results presented in Figure 4 clearly show that the new model consistently produces more colorful images.

Image description

Table 3 results show that applying the proposed method can improve the MLE model, which is consistent with previous observations in the literature, demonstrating the effectiveness of this method for tuning for specific task risks.

The above is the detailed content of Google reuses classic algorithms from 30 years ago, and CV introduces reinforcement learning. Netizens: Is visual RLHF coming?. For more information, please follow other related articles on the PHP Chinese website!