Fudan releases Chinese version of ChatGPT: MOSS starts testing and hits hot searches, servers are overcrowded

ChatGPT is the most advanced AI and the most popular application - since its release at the end of November last year, its monthly active users have exceeded 100 million in two months, easily ranking first in user growth in the history of the global Internet. one.

It is also a technology with a high threshold. Due to the large computing resources and high labeling costs required for the training process of ChatGPT, there are currently no similar products open to the public in China. Major Internet companies such as Baidu, Alibaba, and JD.com have all released news that they are building "domestic ChatGPT" and will release it in the near future.

Before the products of major manufacturers were in place, the academic community first had news. On the evening of February 20, the Natural Language Processing Laboratory of Fudan University released MOSS, a language model with ChatGPT capabilities, and invited the public for internal testing.

• MOSS experience link: https://moss.fastnlp.top/
• MOSS project homepage: https://txsun1997.github.io/blogs/moss.html

##MOSS's name comes from the movie "The Wandering Earth". As popular as the movie, the news released by MOSS quickly reached the top of the hot search list on Zhihu and other platforms.

However, unlike science fiction, AI in the real world does not yet have the blessing of quantum computers, and it is not 24 years away from being open to the public. For hours, due to excessive instant access pressure, the MOSS server was overcrowded last night, which shows how high everyone's expectations are for generating language models.

According to researchers from Fudan University, it is currently in internal testing and iterative optimization of user interaction is not suitable for public testing.

We know that natural language processing is one of the biggest challenges in the field of AI. Although breakthroughs have already occurred, this month The new Bing search launched online, as well as the competing product BARD released by Google, have problems from time to time during testing. What is the MOSS level of Fudan University?

Conversation MOSS, how is your level?

The basic functions of MOSS are similar to ChatGPT. It can complete various natural language processing tasks according to the instructions entered by the user, including text generation, text summarization, translation, code generation, chatting, etc. During the preview period, use of MOSS is free.

MOSS is the same as ChatGPT. The construction process includes natural language basic model training and dialogue ability training to understand human intentions. Two stages.

##According to the project homepage, the main difference between MOSS and ChatGPT is:

MOSS has much fewer parameters than ChatGPT.
MOSS learns by talking to humans and other AI models, while ChatGPT is trained through reinforcement learning with human feedback (RLHF).
MOSS will be open source to facilitate future research, but ChatGPT may not.

# Let’s look at a few examples of how conversational MOSS is. The following are some interaction records generated by MOSS:

In this example, the user first asked MOSS to recommend five science fiction movies, then asked MOSS to generate a table to display these movies and their directors, and finally asked MOSS to insert a new column in the table to display these movies. year of release. Completing this task requires the language model to have strong multi-round interaction capabilities and instruction understanding capabilities. MOSS obviously performs well in these two aspects.

Similar to ChatGPT, MOSS sometimes outputs examples of factual errors.

In addition to multiple rounds of dialogue, MOSS generation code is also a piece of cake. In the following example, MOSS can not only provide users with Python code to implement quick sorting, but also provide explanations and usage examples for this code at the user's request. It can be said that it is a programmer who teaches step by step.

##In addition to asking MOSS to help write the code, you can also ask MOSS questions about the code details to make it more accurate. Good help in understanding the code. In the example below, the user asked MOSS about the programming language and functions of a piece of code, and further asked about the role of one of the functions. MOSS gave satisfactory replies.

In addition, MOSS also has human values. When asked to answer unreasonable questions, MOSS will refuse to answer. and give correct advice.

It is understood that MOSS uses a self-developed model with tens of billions of parameters for training. During the dialogue ability training phase, OpenAI collected at least hundreds of thousands of human instructions - allowing professional annotators from various industries to write instructions and then input them into the model base to help ChatGPT gradually understand various instructions. The Fudan team adopted a different technical route. By allowing MOSS to interact with humans and other AI models, it significantly improved learning efficiency and R&D efficiency, and efficiently completed dialogue ability training in a short time.

The R&D team stated that although MOSS has implemented some functions of ChatGPT, there are still many limitations. Due to the lack of high-quality data, computing resources and model capacity, MOSS is still far behind. on ChatGPT.

• Due to the limited multilingual corpus in the training data, MOSS performs poorly at understanding and generating text in languages ​​other than English. The team is currently developing an improved version to improve their Chinese language skills.
• Due to the relatively small model capacity, MOSS does not contain sufficient world knowledge. Therefore, some responses generated by MOSS may contain misleading or false information.
• Sometimes MOSS performs in a roundabout way or even fails to follow instructions. In this case, the user may need to regenerate or modify the prompt several times to get a satisfactory response. The team is actively improving its ability to follow instructions and its productivity.
• Sometimes MOSS may generate unethical or harmful reactions due to prompts. Users can help reduce this behavior by clicking "Dislike" and the team will update the model in the next release.

The research team pointed out that the performance of the current version of MOSS is still unstable and is also affected by data set issues: "The English answer level of MOSS is higher than that of Chinese because of its model The base has learned more than 300 billion English words, and only about 30 billion Chinese words."

After the release, the team will continue to provide an accessible interface for MOSS, based on valuable User feedback (with permission) continuously improves the model.

In the future, the researchers also plan to combine Fudan’s research results in artificial intelligence and related interdisciplinary subjects to give MOSS multi-modal capabilities such as drawing, speech, and music composition, and strengthen it to assist scientists The ability to conduct efficient scientific research, etc.

We hope that MOSS can make a good start for the development of large-scale domestic dialogue models.

Team Introduction

There are two main authors of MOSS: Professor Qiu Xipeng of Fudan University and his doctoral student Sun Tianxiang. In addition, several members contributed to the project.

##Qiu Xipeng is a professor and doctoral supervisor at the School of Computer Science and Technology, Fudan University. A National Outstanding Young Scholar, he received his Bachelor of Science and Doctorate degrees from Fudan University. Mainly engaged in research in natural language processing, deep learning and other directions, published more than 70 CCF Category A/B papers, and won the ACL 2017 Outstanding Paper Award (CCF Category A), CCL 2019 Best Paper Award, "Science in China: Technical Sciences" In the 2021 High Impact Paper Award, 5 papers were selected as the most influential papers of IJCAI/ACL/EMNLP published by PaperDigest (the number of citations entered the top 20 of the current conference). Published the open source monograph "Neural Network and Deep Learning", with 15,000 Github followers and a Douban score of 9.4. He presided over the development of the open source frameworks FudanNLP and FastNLP, which have been used by hundreds of units at home and abroad. In 2015, he was selected into the first China Association for Science and Technology Youth Talent Promotion Project. In 2018, he won the first prize of the Youth Innovation Award of the Qian Weichang Chinese Information Processing Science and Technology Award. In 2020, he won the Excellence Award in the 4th Shanghai University Young Teacher Teaching Competition. In 2021, he won the The first prize of the first Shanghai Computer Society Teaching Achievement Award (the first person to complete it), etc. The students have been trained to win multiple first-level academic scholarships, Microsoft Scholars, Baidu Scholarships, etc.

Sun Tianxiang is a doctoral student in the School of Computer Science at Fudan University. His supervisors are Professor Qiu Xipeng and Professor Huang Xuanjing. Graduated from Xi'an University of Electronic Science and Technology in 2019. Research interests focus on machine learning and its application in natural language processing, especially efficient fine-tuning and inference of pre-trained language models, multi-task learning, knowledge representation learning, etc. Published multiple papers as the first author in conferences such as ICML, ACL, NAACL, AAAI, COLING, etc.

The above is the detailed content of Fudan releases Chinese version of ChatGPT: MOSS starts testing and hits hot searches, servers are overcrowded. For more information, please follow other related articles on the PHP Chinese website!