Posthumous work of the OpenAI Super Alignment Team: Two large models play a game, and the output becomes more understandable

If the answer given by the AI ​​model is incomprehensible at all, do you dare to use it?

As machine learning systems are used in more important areas, it becomes increasingly important to demonstrate why we can trust their output, and to make it clear when we should not trust them.

One possible way to gain trust in the output of a complex system is to require the system to produce an interpretation of its output that is readable to a human or another trusted system, that is, completely Understand so that any possible errors can be caught. For example, to build trust in the judicial system, we require courts to provide clear and readable written opinions that explain and support their decisions.

For large language models, we can also adopt a similar approach.

However, when using this approach, it is very important to ensure that the language model generates understandable text, especially when dealing with complex tasks such as mathematics and coding.

As shown in the figure below, you ask AI to write a quick sort algorithm. AI writes it quickly, and the answer is very concise. But if you don’t know how to write code, how can you judge whether the AI ​​is written correctly?

OpenAI studied this problem in a recent paper.

• Paper title: PROVER-VERIFIER GAMES IMPROVE LEGIBILITY OF LLM OUTPUTS
• Paper link: https://cdn.openai.com/prover-verifier-games-improve-legibility-of-llm- outputs/legibility.pdf

They found that if large language models are asked to generate answers with the goal of "getting the correct answer", the answers they give may be difficult to understand. Human evaluators are twice as likely to make mistakes when judging these answers.

So their goal is to train high-level language models so that they can generate text that simpler models can easily verify. They found that such text was not only friendly to simple models but also more readable to humans.

So, how is it achieved? This also mentions a 2021 paper - "Learning to Give Checkable Answers with Prover-Verifier Games". In this paper, Cem Anil et al. proposed a game theory framework called "Prover-Verifier Games". The framework consists of two competing learners: a network of trusted verifiers trying to choose the correct answer, and a more powerful but untrustworthy network of provers trying to convince the verifiers to accept a specific answer. , regardless of its correctness. The goal of this framework is to produce a reliable proof protocol from the game.

OpenAI borrowed this framework in this new research and let two large models (the stronger one is the prover and the weaker one is the verifier) ​​play against each other until the two reach a balance. At this time, the answer given by the prover is both readable and reasonably correct.

OpenAI said that by improving the readability of model output, it can make the application of artificial intelligence more effective and trustworthy, which is beneficial to various fields that require precise communication.

It is worth mentioning that this paper comes from OpenAI’s Super Alignment team, several of whose authors left OpenAI earlier this year, and the team has also fallen apart. One of the authors revealed that this was the team's last paper.

An intuitive example

In order to improve the readability of model output, OpenAI optimized the chain of thinking (CoT) of some elementary school mathematics problems so that it can be understood by weaker models. Here are some examples:

Question: Shawna’s father is five times older than Shawna. Shauna is now three times Aliya's age. If Aaliyah is 3, how old is Shauna's father?

这么做的目的是为了验证经过这种训练的模型，它的输出是否更加直接易懂。

这种方法在只对训练集做了一些优化的情况下，让模型的性能提升了 50%。同时，模型输出的答案也更好懂了，这也帮助需要评估这些答案正确性的人类用户更准确地做出判断。OpenAI 认为，进一步缩小可读性强的模型和能力强的模型之间的性能差距，是未来对齐研究中一个关键的目标。

「证明者」和「验证者」是怎么在训练中博弈的？

为了激励强力模型（证明者）生成可读性更强的文本，研究者采用了一种训练方案，要求强力模型生成的答案能够被较弱的模型（验证者）轻松验证。具体来说，他们使用了 GPT-4 系列中的大模型和小模型。它们在预训练数据量上相差约三个数量级。它们都接受了一些基于人工标注的数学推导训练，但未经过广泛的后训练。

首先，研究团队让证明者模型尝试解决一系列数学问题。由于证明者模型自身可能出错，它生成的一些答案可能不正确。接下来，他们将这些答案交由验证者模型进行检验，并在以下两个阶段之间进行多轮的交替验证：

• 「验证者」训练：研究团队训练较弱的「验证者」模型，验证「证明者」给出的答案是否正确。
• 「证明者」训练：研究团队指示性能较强的「证明者」模型扮演「有用」或「狡猾」的戏份。
• 当「证明者」被设定为「有用」时，研究团队将奖励「验证者」打分高的正确答案。
• 当「证明者」被设定为「狡猾」时，研究团队反而奖励「验证者」打分高的错误答案。

研究团队发现，最初，狡猾的「证明者」能够轻松生成让「验证者」信服的错误答案。然而，这些错误答案将会进入下一轮的「验证者」训练中，相同的漏洞在下一轮中，将不再具有说服力。因此，「证明者」将被迫在每一轮中探索不同的策略。

当狡猾的「证明者」再也无法生成让「验证者」信服的错误解决方案时，训练宣告结束。

研究者都发现了什么？

性能与可读性权衡：

• 一个仅在人类编写的推导上进行监督式微调的模型，虽然具有较高的可读性，但准确性较差。
• 当模型的优化目标仅为正确性，即只关注生成正确答案时，虽然能够达到高准确性，但生成的解决方案的可读性却较差。
• OpenAI 提出的可检查博弈方法平衡了这两者，在保持高可读性的同时保持适度的准确性。

三种有用的模型：

• 鲁棒验证者：即使解决方案具有误导性，仍能够有效区分正确和错误的解决方案。
• 有用的证明者：生成对人类来说易于理解的解决方案，从而减少人类评估者的错误。
• 狡猾证明者：产生微妙的、错误的解决方案，这些解决方案最初会混淆人类评估者，从而突出需要进一步改进的领域。

研究团队使用了同样的测试问题，来展示狡猾证明者（Sneaky Prover）的生成内容和训练过程。

问题：肖娜（Shawna）的父亲的年龄是肖娜的五倍。肖娜现在的年龄是阿莉娅（Aliya）的三倍。如果阿莉娅 3 岁，那么肖娜的父亲多少岁？

结论

随着大语言模型（LLMs）在关键领域和复杂应用中的使用日益增加，了解其所生成的内容是否值得信赖变得至关重要。通过要求模型生成清晰且可验证的理由，可以增强对其生成内容的信任。

此方法的一个显着优势是它减少了对人类示范或可读性判断的依赖。这种自主性对于未来超智能 AI 系统的对齐尤为重要，其最终目标是在没有人类进行直接监督的情况下，可靠地将 AI 系统与人类的价值观和期望对齐。

尽管这项工作仅在一个数据集上进行了实验，并且仍然需要真值标签（ground truth labels），但研究团队仍预计在开发正确、透明及可验证的AI 系统中，此类方法会起到关键作用，并增强其在现实应用中的可信任性和安全性。

更多详情，请参考原论文。

^{参考链接：}

^{https://openai.com/index/prover-verifier-games-improve-legibility/}

The above is the detailed content of Posthumous work of the OpenAI Super Alignment Team: Two large models play a game, and the output becomes more understandable. For more information, please follow other related articles on the PHP Chinese website!