Google proposes a new RLHF method: eliminating reward models and eliminating the need for adversarial training

The effect is more stable and the implementation is simpler.

The success of large language models (LLM) is inseparable from "reinforcement learning based on human feedback (RLHF)". RLHF can be roughly divided into two stages. First, given a pair of preferred and unpreferred behaviors, a reward model is trained to assign a higher score to the former by classifying the target. This reward function is then optimized through some kind of reinforcement learning algorithm. However, key elements of the reward model may have some undesirable effects.

Researchers from Carnegie Mellon University (CMU) and Google Research jointly proposed a simple, theoretically rigorous, and experimentally effective new RLHF Method - Self-Play Preference Optimization (SPO). This approach eliminates reward models and does not require adversarial training.

Paper: A Minimaximalist Approach to Reinforcement Learning from Human Feedback

Paper address: https://arxiv.org/abs/2401.04056

Method Introduction

The SPO method mainly includes two aspects. First, this research truly eliminates the reward model by constructing RLHF as a zero-sum game, making it more capable of handling noisy, non-Markovian preferences that often appear in practice. Second, by exploiting the symmetry of the game, this study demonstrates that a single agent can simply be trained in a self-game manner, thereby eliminating the need for unstable adversarial training.

In practice, this is equivalent to sampling multiple trajectories from the agent, asking the evaluator or preference model to compare each pair of trajectories, and setting the reward to the winning rate of the trajectory .

#SPO avoids reward modeling, compound errors, and adversarial training. By establishing the concept of minmax winner from social choice theory, this study constructs RLHF as a two-person zero-sum game and exploits the symmetry of the game's payoff matrix to demonstrate that a single agent can be simply trained to against itself.

The study also analyzed the convergence characteristics of SPO and proved that when the potential reward function does exist, SPO can be compared with The fast rate of convergence to the optimal policy is comparable to standard methods.

Experiment

This study performed a series of continuous control tasks with realistic preference functions On the above, it is proved that SPO performs better than methods based on reward models. SPO is able to learn samples more efficiently than reward model-based methods in various preference settings, as shown in Figure 2 below.

This study combines SPO with the iterative reward modeling (RM) method from multiple dimensions A comparison is made to answer 4 questions:

1. Can SPO calculate MW when facing intransitive preferences?
2. Can SPO match or exceed RM sample efficiency on problems with unique Copeland Winners/optimal strategies?
3. How robust is SPO to random preferences?
4. Can SPO handle non-Markovian preferences?

In terms of maximum reward preference, noise preference, and non-Markov preference, the experimental results of this study are shown in Figures 6, 7, and 8 below respectively. Show:

Interested readers can read the original text of the paper to learn more about the research content.

The above is the detailed content of Google proposes a new RLHF method: eliminating reward models and eliminating the need for adversarial training. For more information, please follow other related articles on the PHP Chinese website!