ICML 2024 Oral | Is DPO more suitable for LLM than PPO? Tsinghua Wuyi team's latest revelation

The AIxiv column is a column where this site publishes academic and technical content. In the past few years, the AIxiv column of this site has received more than 2,000 reports, covering top laboratories from major universities and companies around the world, effectively promoting academic exchanges and dissemination. If you have excellent work that you want to share, please feel free to contribute or contact us for reporting. Submission email: liyazhou@jiqizhixin.com; zhaoyunfeng@jiqizhixin.com

Wu Yi, assistant professor at the Institute of Interdisciplinary Information, Tsinghua University, former full-time researcher at OpenAI, research areas include reinforcement learning, large model alignment, human Computer interaction, robot learning, etc. Obtained a PhD from the University of California, Berkeley, in 2019, studying under Professor Stuart Russell; graduated from Tsinghua University's Institute of Interdisciplinary Information (Yao Class) in 2014. His representative works include: NIPS2016 best paper, Value Iteration Network; the most cited paper in the field of multi-agent deep reinforcement learning, MADDPG algorithm; OpenAI hide-and-seek project, etc.

How to make large models better comply with human instructions and intentions? How to make large models have better reasoning capabilities? How to avoid hallucinations in large models? Whether these problems can be solved is the most critical technical challenge to make large models truly widely available and even achieve super intelligence. These most difficult challenges are also the long-term research focus of Wu Yi’s team, and are problems to be overcome by large model alignment technology (Alignment).

In alignment technology, the most important algorithm framework is reinforcement learning based on human feedback (RLHF, Reinforcement Learning from Human Feedback). RLHF learns a reward function (Reward Model) based on human feedback based on human preference feedback for the output of the large model, and further performs reinforcement learning training on the large model, so that the large model can learn to distinguish the quality of responses in repeated iterations, and achieve Improved model capabilities. Currently, the world's most powerful language models, such as OpenAI's GPT model and Anthropic's Claude model, place great emphasis on the importance of RLHF training. OpenAI and Anthropic have also internally developed RLHF training systems based on large-scale PPO algorithms for large model alignment.

However, due to the complex process of the PPO algorithm and the high consumption of computing power, the large-scale RLHF training system of the American AI company has never been open source. Therefore, although the PPO algorithm is very powerful, alignment work in academia has rarely used complex methods. The PPO algorithm is used for RLHF research, and alignment algorithms such as SFT (supervised fine-tuning) or DPO (Direct Policy Optimization) are generally used that are simpler, more direct, and have lower requirements on the training system.

So, does a simple alignment algorithm definitely work better? The work "Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study" published by Wu Yi's team at ICML 2024 carefully discussed the characteristics of the DPO and PPO algorithms and pointed out the key points to improve the effect of the RLHF algorithm. In this work, based on the self-developed large-scale RLHF training system, Wu Yi's team used the PPO algorithm and an open source model with fewer parameters for the first time, surpassing the closed-source large-scale code generation task CodeContest in the recognized most difficult challenge. Model AlphaCode 41B.

The relevant results were accepted as Oral Presentation by ICML 2024, and will be publicly reported at the first Oral session Alignment-1 of ICML 2024 on July 23, together with the work of well-known institutions such as OpenAI and Anthropic.

Next, let us compare the code generation effects of GPT-4 and the CodeLlama 34B model trained by the PPO algorithm. In Example 1, the CodeLlama 34B model trained by the PPO algorithm and the GPT-4 model are generated Code of comparable quality.

In Example 2, you can see that both the CodeLlama 34B model and the GPT-4 model trained by the PPO algorithm can generate complete and runnable python code. However, in this example, GPT-4 generated incorrect code and failed to output correctly on the test data. The code generated by the CodeLlama 34B model trained by the PPO algorithm can pass the test.

In this paper from ICML 2024, the research team discussed the characteristics of DPO and PPO algorithms in detail and pointed out the key points to improve the capabilities of DPO and PPO.

• Paper title: Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study
• Paper address: https://arxiv.org/pdf/2404.10719

DPO algorithm Limitations

Compared with PPO, DPO uses offline data instead of online sampling data for training. After analysis, the DPO algorithm will cause the trained model to have a preference for outputs outside the training data distribution, and in some cases produce unpredictable responses.

So, in order to improve the capabilities of the DPO algorithm, the research team summarized two key technologies: Carry out additional SFT training before RLHF training, and use online sampling data instead of offline data.

Experiments show that using additional SFT training can make the base model and reference model more biased towards the distribution within the data set, greatly improving the effect of the DPO algorithm; on the other hand, the DPO algorithm using online sampling data for iterative training can It has been steadily improved and its performance is far better than the basic DPO algorithm.

The key points of the PPO algorithm

Excluding DPO, the paper also summarizes three key points to maximize the capabilities of PPO:

• Use a large batch size (large batch size)
• advantage normalization
• and update the reference model using exponential moving average (exponential moving average for the reference model).

The research team successfully used the PPO algorithm to achieve SOTA results on the dialogue tasks Safe-RLHF/HH-RLHF and the code generation task APPS/CodeContest.

On the dialogue task, the research team found that the PPO algorithm that combines three key points is significantly better than the DPO algorithm and the online sampling DPO algorithm DPO-Iter.

In the code generation tasks APPS and CodeContest, based on the open source model Code Llama 34B, the PPO algorithm has also reached the strongest level, surpassing the previous SOTA, AlphaCode 41B in CodeContest.

To achieve large model alignment with better results, a highly efficient training system is indispensable. Wu Yi’s team has long-term accumulation in realizing large-scale reinforcement learning training and has built it since 2021 A proprietary distributed reinforcement learning framework.

• NeurIPS 2022 The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games [1]: Proposed and open sourced the reinforcement learning parallel training framework MAPPO for multi-agent to support multi-agent training in cooperative scenarios. This work was It has been used in a large number of works in the field of multi-agent, and the current number of citations in papers has exceeded 1k.
• ICLR 2024 Scaling Distributed Reinforcement Learning to Over Ten Thousand Cores [2]: Proposed a distributed training framework for reinforcement learning, which can be easily expanded to tens of thousands of cores, and the acceleration ratio exceeds OpenAI's large-scale reinforcement learning system Rapid .
• ReaLHF: Optimized RLHF Training for Large Language Models through Parameter Reallocation [3]: Recently, Wu Yi’s team further implemented the distributed RLHF training framework ReaLHF. The ICML Oral paper of Wu Yi's team was produced based on the ReaLHF system. The ReaLHF system has been developed for a long time and has undergone a lot of detail polishing to achieve optimal performance. Compared with previous open source work, ReaLHF can achieve near-linear scalability in RLHF, a scenario that is more complex than pre-training. It also has higher resource utilization and can perform RLHF stably and quickly on 128 A100 GPUs. Training, related work has been open sourced: https://github.com/openpsi-project/ReaLHF

In addition to improving the coding capabilities of large language models, Wu Yi’s team also uses a variety of reinforcement learning algorithms and large The combination of models realizes a variety of complex LLM Agents and can carry out complex interactions with humans.

Use reinforcement learning in MiniRTS to create a language agent that can both follow human instructions and make optimal decisions [4].

Train strategy-diversified reinforcement learning strategies in Werewolf to improve the decision-making capabilities of large models [5].

In the Overcooked game, small models and large models are combined to implement a cooperative Language Agent that can provide real-time feedback [6].

Combining the robot control strategy trained by reinforcement learning and the reasoning capabilities of the large language model allows the robot to perform a series of complex tasks [7].

In order for large models to truly enter thousands of households, alignment technology is crucial. For academics and large model practitioners, good open source work and papers will undoubtedly greatly reduce experimental costs and development It is also expected that with the development of technology, more large models that serve human beings will appear.

^{[1] Yu, Chao, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. "The surprising effectiveness of ppo in cooperative multi-agent games."}

^{[2] Mei, Zhiyu, Wei Fu, Guangju Wang, Huanchen Zhang, and Yi Wu. "SRL: Scaling Distributed Reinforcement Learning to Over Ten Thousand Cores."}

^{[3] Mei, Zhiyu, Wei Fu, Kaiwei Li, Guangju Wang, Huanchen Zhang, and Yi Wu. "ReaLHF: Optimized RLHF Training for Large Language Models through Parameter Reallocation."}

^{[4] Xu, Shusheng, Huaijie Wang, Jiaxuan Gao, Yutao Ouyang, Chao Yu, and Yi Wu. "Language-guided generation of physically realistic robot motion and control."}

^{[5] Xu, Zelai, Chao Yu, Fei Fang, Yu Wang, and Yi Wu. "Language agents with reinforcement learning for strategic play in the werewolf game."}

^{[6] Liu, Jijia, Chao Yu, Jiaxuan Gao, Yuqing Xie, Qingmin Liao, Yi Wu, and Yu Wang. "Llm- powered hierarchical language agent for real-time human-ai coordination."}

^{[7] Ouyang, Yutao, Jinhan Li, Yunfei Li, Zhongyu Li, Chao Yu, Koushil Sreenath, and Yi Wu. "Long-horizon Locomotion and Manipulation on a Quadrupedal Robot with Large Language Models."}

The above is the detailed content of ICML 2024 Oral | Is DPO more suitable for LLM than PPO? Tsinghua Wuyi team's latest revelation. For more information, please follow other related articles on the PHP Chinese website!