Tencent's robot dog evolves: mastering autonomous decision-making capabilities through deep learning

On June 14, Tencent Robotics Decision-making ability has been greatly improved.

Letting robot dogs move as flexibly and stably as humans and animals is a long-term goal in the field of robotics research. The continuous advancement of deep learning technology allows machines to master relevant abilities through "learning" and learn to deal with complex changes. environment becomes feasible.

Introducing pre-training and reinforcement learning: making the robot dog more agile

By introducing pre-training models and reinforcement learning technology, Tencent Robotics Instead of re-learning, you can reuse the multi-level knowledge of posture, environmental perception, and strategic planning that you have learned, and make inferences about other cases from one instance to flexibly respond to complex environments

This series of learning is divided into three stages:

In the first stage, through the motion capture system commonly used in game technology, the researcher collected the movement posture data of real dogs, including walking, running, jumping, standing and other actions, and used these data to construct an imitation learning task in the simulator. , and then abstract and compress the information in these data into a deep neural network model. These models can not only accurately cover the collected animal movement posture information, but also have high interpretability.

Tencent Robotics These technologies and data play a certain auxiliary role in physical simulation-based agent training and real-world robot strategy deployment.

The neural network model only accepts the proprioceptive information of the robot dog (such as motor status) as input and is trained in an imitation learning manner. In the next step, the model incorporates sensory data from the surrounding environment, such as using other sensors to detect obstacles underfoot.

In the second stage, additional network parameters are used to connect the smart posture of the robot dog mastered in the first stage with external perception, so that the robot dog can respond to the external environment through the smart posture that has been learned. When the robot dog adapts to a variety of complex environments, the knowledge that links smart postures with external perception will also be solidified and stored in the neural network structure.

In the third stage, using the neural network obtained in the above two pre-training stages, the robot dog has the prerequisite and opportunity to focus on solving the top-level policy learning problem, and finally has the ability to solve complex tasks end-to-end. In the third phase, additional networks will be added to collect data related to complex tasks, such as obtaining information about opponents and flags in the game. In addition, by comprehensively analyzing all information, the neural network responsible for strategy learning will learn high-level strategies for the task, such as which direction to run, predicting the opponent's behavior to decide whether to continue chasing, etc.

The knowledge learned at each stage above can be expanded and adjusted without re-learning, so it can be continuously accumulated and learned continuously.

Robot Dog Obstacle Chase Competition: Has autonomous decision-making and control capabilities

In order to test these new skills mastered by Max, the researchers were inspired by the obstacle chasing game "World Chase Tag" and designed a two-dog obstacle chasing game. World Chase Tag is a competitive obstacle chasing organization founded in the United Kingdom in 2014. It is standardized from folk children's chasing games. Generally speaking, each round of the obstacle chasing competition involves two athletes competing against each other. One is the pursuer (called the attacker) and the other is the dodger (called the defender). When an athlete competes throughout the entire The team will receive one point when they successfully evade their opponent (i.e. no contact occurs) during the pursuit round (i.e. 20 seconds). The team that scores the most points in the predetermined number of chasing rounds wins the game.

The venue size of the robot dog obstacle chasing competition is 4.5 meters x 4.5 meters, with some obstacles scattered on it. At the beginning of the game, two MAX robot dogs will be placed at random locations in the field, and one robot dog will be randomly assigned the role of a pursuer and the other a dodger. At the same time, a flag will be placed at a random location in the field.

The goal of the dodger is to get as close to the flag as possible without getting caught by the pursuer. The pursuer's task is to catch the evader. If the dodger successfully touches the flag before being caught, the roles of the two robot dogs will instantly switch, and the flag will reappear in another random location. The game ends when the dodger is caught by the current pursuer and the robot dog playing the role of the pursuer wins. In all games, the average forward speed of the two robot dogs is limited to 0.5m/s.

Judging from this game, based on the pre-trained model, the robot dog already has certain reasoning and decision-making capabilities through deep reinforcement learning:

For example, when the pursuer realizes that he can no longer catch up with the dodger before it touches the flag, the pursuer will give up the pursuit and instead wander away from the dodger in order to wait for the next important step. The set flag appears.

In addition, when the pursuer is about to catch the dodger at the last moment, it likes to jump up and make a "pounce" action towards the dodger, which is very similar to the behavior of animals when catching prey, or when the dodger is about to touch the flag. will exhibit the same behavior at times. These are all proactive acceleration measures taken by the robot dog to ensure its victory.

According to reports, all control strategies of the robot dog in the game are neural network strategies. They are learned in simulation and through zero-shot transfer (zero adjustment transfer), allowing the neural network to simulate human reasoning methods to identify never See new things and deploy this knowledge to real robot dogs. For example, as shown in the figure below, the knowledge of avoiding obstacles that the robot dog learned in the pre-training model is used in the game, even though the scenes with obstacles are not trained in the virtual world of Chase Tag Game (only in the virtual world After training in game scenes on flat ground), the robot dog can also successfully complete the task.

Tencent Robotics Learning technology is introduced into the field of robots to improve the control capabilities of robots and make them more flexible. This also lays a solid foundation for robots to enter real life and serve human beings.

The above is the detailed content of Tencent's robot dog evolves: mastering autonomous decision-making capabilities through deep learning. For more information, please follow other related articles on the PHP Chinese website!