Quickly learn the key technical points of the InstructGPT paper: follow Li Mu to master the technology behind ChatGPT

After ChatGPT became popular, many students who pay attention to technology are asking a question: Are there any learning materials that can allow us to systematically understand the principles behind ChatGPT? This problem becomes tricky because OpenAI has not released a paper related to ChatGPT.

However, we know from OpenAI’s blog about ChatGPT that the method used by ChatGPT is the same as its brother model-InstructGPT, except that InstructGPT is fine-tuned on GPT-3 , while ChatGPT is based on GPT-3.5. There are also some differences between the two in terms of data collection.

Blog link: https://openai.com/blog/chatgpt/

The InstructGPT paper was released in March 2022, but OpenAI published a related blog as early as January (see "What to do with GPT-3 nonsense? OpenAI: We re-trained it" , the new version is more "obedient"). At that time, OpenAI clearly mentioned that InstructGPT used the reinforcement learning method of human feedback (RLHF) to fine-tune GPT-3, making the output of the model more consistent with human preferences. This has been continued in the training of ChatGPT.

Paper link: https://arxiv.org/pdf/2203.02155.pdf

In addition, there are many similarities between InstructGPT and ChatGPT. Therefore, a thorough understanding of the InstructGPT paper will be of great benefit to students who want to do some work in the direction of ChatGPT. This is why we highly recommend Li Mu’s lectures.

Course address: https://jmq.xet.tech/s/2lec6b (Click "Read Original text" can be accessed directly)

Dr. Li Mu is the senior chief scientist of Amazon. He previously co-authored "Hands-on Deep Learning" with Aston Zhang and others. In the past two years, he has been introducing various AI knowledge to everyone through videos and produced intensive reading courses on dozens of papers. Many students have developed the habit of reading papers intensively with Li Mu.

Dr. Li Mu’s account on Station B is “Learn AI from Li Mu”.

This interpretation course for InstructGPT lasts 67 minutes in total and is basically introduced in the order of writing the paper.

Students who have read the ChatGPT blog know that its technical principles can basically be summarized in one picture. This It is also a picture that has already appeared in the InstructGPT paper (there are subtle differences between the two). When interpreting the abstract and introduction of the paper, Li Mu introduced the three steps in the diagram in detail.

Technical schematics from the ChatGPT blog.

InstructGPT Technical schematic from the paper.

In the third chapter of the paper, the authors of InstructGPT first introduced their data acquisition method and process, and Li Mu also took everyone to read it in detail. This part is very valuable in engineering. As Li Mu said, if you have never done anything like this before (data labeling, etc.) and need to find someone to help you label data, then you can look at its appendix, which contains many templates that can be used directly. The author of the paper It even describes what the UI of their annotated website looks like, which is worth learning from.

Next, Li Mu focused on the three models written in Chapter 3 (see 3.5 Models) - SFT (Supervised fine-tuning) model, RM (Reward modeling) model and RL (Reinforcement learning) ) models, including details such as parameters and objective functions involved in these models.

Finally, Li Mu concluded that technically speaking, InstructGPT is still a very practical technology. It tells everyone a method: given a large language model, how can you quickly improve its performance in a field you care about through some annotated data to make it practical. Therefore, it provides an operational idea for people who want to use generative models to make products.

Of course, as Dr. Li Mu said, scientific research work is step-by-step, and InstructGPT is also based on previous research, so students who want to thoroughly understand ChatGPT will inevitably have to go back and read it. More papers. In previous courses, Li Mu also interpreted the papers of GPT, GPT-2, and GPT-3 in detail:

## Course address: https://jmq.xet.tech/s/2lec6b

The above is the detailed content of Quickly learn the key technical points of the InstructGPT paper: follow Li Mu to master the technology behind ChatGPT. For more information, please follow other related articles on the PHP Chinese website!