Moving towards the digital world AGI! The agent has started playing 'Red Dead Redemption 2' from scratch

Universal computer control

The information revolution produced the digital world, digital The world provides data for the birth of large models, and it is also the easiest to implement general artificial intelligence (AGI).

Towards AGI in the digital world, Beijing Zhiyuan Artificial Intelligence Research Institute, Nanyang Technological University of Singapore, and Peking University jointly proposed

General Computer Control (GCC)

, that is, the agent needs to see the screen like a human being, and complete all tasks on the computer through the keyboard and mouse. For a long time in the past, artificial intelligence research was based on games, and GCC will provide a scenario for general artificial intelligence research, and will further promote the implementation and industrialization of large models and AI Agents .

To this end, the research team proposed a

universal computer-controlled agent framework Cradle

, which enables the agent to directly control the keyboard, mouse and any other functions without relying on any internal API. Software interaction, whether open source or closed source, can even play commercial AAA game masterpieces such as "Red Dead Redemption 2"!

Paper title: Towards General Computer Control: A Multimodal Agent for Red Dead Redemption II as a Case Study
Paper link: https://arxiv.org/abs/2403.03186
Project homepage: https://baai-agents.github.io/Cradle/
Code link: https://github.com/BAAI-Agents/Cradle

With the development of large models With the development of AI, more and more research on AI Agents focuses on computer control, including browsing the web, operating smartphones, playing games, etc. However, existing research relies on internal APIs to obtain input and output predefined actions. To build a

universal agent

that can complete all tasks on a computer, you must use the most common and standard input and output to interact with the computer. Therefore, universal computer control uses unified inputs and outputs, making the universality of agents possible.

But the versatility brings operational difficulties: (1) Using the computer screen as input puts higher requirements on the agent’s video understanding ability, for example due to There is no internal API, and visual information is needed to determine whether the action is successfully executed; (2) Using keyboard and mouse operations as output requires the agent to require higher spatiotemporal operation accuracy. For example, keyboard keystrokes and mouse clicks usually involve additional time dimensions. How to solve these problems is the challenge of building

General Computer Controlled Agents (GCC Agents)

!

Cradle: Control all software

"Computer refers to any user-centered Computing devices, including PCs, smartphones, tablets, etc. Although Cradle focuses on keyboard and mouse operations, it can be easily extended to control handles and touch screens, etc."

General The computer-controlled agent framework Cradle is mainly composed of 6 modules: information collection, self-reflection, task inference, skill management, action planning and memory modules. Cradle's high degree of versatility comes from its reasonable encapsulation and abstraction of the original input and output during interaction with the computer. It takes the video displayed on the screen as input, extracts the text and visual information for decision-making, and outputs the keyboard and mouse control signals in the underlying operating system to interact with the computer, allowing it to interact with all software without relying on any assumptions. .

"Cradle is mainly composed of 6 modules including information collection, self-reflection, task inference, skill management, action planning and memory module. Its powerful decision-making reasoning comes from "reflecting on the past, summarizing the present, and planning for the future""

At the same time, Cradle's powerful decision-making reasoning module allows it to spontaneously interact with the software and complete tasks. This process can be simply summarized as: Reflect on the past, summarize the present, Planning for the future.

• Reflect on the past: Use videos of past action processes as input to extract key textual and visual information respectively. , use reflection to determine whether the previous action was successfully executed, whether the task was completed, and how to improve.
• Summary Now: After reflection, summarize the current situation, and use this as a basis to decide whether to change the task objective or modify the task content.
• Planning for the future: Finally, generate or update skills based on the current task and current situation, and retrieve skills related to the current task from the learned skills as alternatives. Then select the appropriate skill and instantiate it as an action to execute.

While making decisions and reasoning, Cradle will periodically summarize and maintain historical information stored in contextual memory and long-term memory. Skill. The brain of this process is a large multi-modal model, such as GPT-4V, but Cradle adds functions such as summary, reflection and memory to it, forming a complete intelligent agent framework for general computer control, effectively solving the problems of universality. problems brought about.

Cradle: Take you to explore "Red Dead Redemption 2" from the beginning

For Proving the framework's versatility and powerful decision-making capabilities, the research team chose to deploy Cradle to the most difficult and rarely explored commercial AAA game masterpiece "Red Dead Redemption 2". They believe that as the most difficult software to operate, if Cradle can freely explore and even complete the main storyline on AAA games, it shows that the framework has great potential to be generalized to other games and software.

"Unlike open source games like Minecraft, most commercial games, especially 3A games, do not provide internal API interfaces, making games like Voyager rely on internal APIs to obtain input and The framework that outputs predefined actions cannot be migrated to other games"

Based on GPT-4V, Cradle can directly generate the corresponding executable based on in-game prompts and tutorials Use code as a skill to enrich your skill library step by step and reuse these skills in subsequent games.

After performing an incorrect action, Cradle can effectively discover and correct the error through reflection.

Cradle can not only follow the game guidance from scratch to generate corresponding skills and complete the 40-minute main story, but also can freely explore, ride horses, hunt, and fight in the open world , talking to NPCs, using props, operating maps, and even shopping in stores are all a breeze. This is the first robot that can play commercial AAA games for a long time.

#Conclusion

The open source Cradle code can be easily extended to other software and games. The research team stated that in order to achieve true universal computer control, Cradle will be ported to more software and games in the future, and it also encourages relevant research teams/industry to conduct further research and exploration. The goal is to allow intelligent agents to interact with all software, whether open source or closed source, and continuously improve themselves to achieve universality, and ultimately become the cradle of the birth of general

artificial intelligence.

"GCC is a cradle for AGI."

—The Cradle team

One more thing: Cradle technical interpretation live broadcast

March 14th 14:30- At 15:30, the first author of the paper, Tan Weihao, a doctoral student at Nanyang Technological University in Singapore, gave an online interpretation report. Scan the QR code below to register.

The above is the detailed content of Moving towards the digital world AGI! The agent has started playing 'Red Dead Redemption 2' from scratch. For more information, please follow other related articles on the PHP Chinese website!