Google RT-1 model asks a robot to do several jobs, with a success rate of 97% for 700 instructions

Many recent advances in multiple subfields of machine learning (ML) research, such as computer vision and natural language processing, are built on leveraging large, diverse data sets and representations that can efficiently absorb all the data. Model.

However, this high-performance model method has relatively few applications in the field of robotics.

The reason is simple. First, the lack of large-scale and diverse robot data limits the model’s ability to absorb a wide range of robot experience.

Second, there is a lack of expressive, scalable, and fast enough real-time inference models that can learn from such datasets and generalize effectively.

This time, Google’s Robotics Transformer 1 (RT-1 for short) is a multi-task model that can label robot input and output actions (e.g., camera images, tasks instructions and motor commands) to enable efficient inference at runtime and enable real-time control.

RT-1 absorbs a large amount of data to make the robot competent in a variety of tasks in different environments, thereby improving machine performance and generalization Ability

Simply put, it means letting a robot do several jobs at the same time.

The model was trained on a large real-world robotics dataset containing 130k episodes, covering more than 700 tasks, using 13 machines from Everyday Robots (EDR) The robots were collected over a period of 17 months.

The results show that RT-1 can significantly improve zero-shot generalization to new tasks, environments, and objects compared to existing techniques.

The editor has thoughtfully placed the Github link below. If you are interested, please go and have a look.

##​https://github.com/google-research/robotics_transformer​

RT-1 model specific principle

RT-1 is built on the Transformer architecture. It can obtain the image history from the robot's camera and take the task description expressed in natural language as input, and output it directly at the same time. Tokenized actions.

The architecture of RT-1 is similar to that of a decoder-only sequence model trained against a standard categorical cross-entropy objective with causal masking.

The model takes as input text instructions and a set of images, encoding them into tokens through the pre-trained FiLM EfficientNet model, And compress it through TokenLearner, and then output the action tag through Transformer.

Its main functions include: Image Tokenization, Action Tokenization and Token Compression.

• Image tokenization: We pass the image through the EfficientNet-B3 model pre-trained on ImageNet, and then flatten the resulting 9×9×512 spatial feature map into 81 tokens. The image tokenizer is conditioned on natural language task instructions and extracts task-relevant image features early on using a FiLM layer initialized to identity. ​
• Action tokenization: The robot’s action dimension is the 7 variables of arm movement (x, y, z, roll, pitch, yaw, clamp open) , 3 basic motion variables (x, y, yaw), and an additional discrete variable to switch between the three modes. ​
• Token compression: The model adaptively selects soft combinations of image tokens based on their impact on learning using the element attention module TokenLearner compression, resulting in more than 2.4x faster inference. ​

We use demonstrations provided by humans via remote operation and textual descriptions of the robot’s execution instructions for each Annotate one episode.

This robot performs tasks by relying on "an arm with 7 degrees of freedom, a two-finger gripper and a mobile base."

The set of high-level skills represented in the dataset include picking and placing items, opening and closing drawers, putting items in and out of drawers, holding long, thin items upright, and knocking objects apart. Turn over and wait for the operation.

Integrating heterogeneous data sources

To further advance RT-1, we trained it using data collected from another robot to test the (1) model whether its performance on the original task is maintained when new data sources are presented, and (2) whether the model improves in generalization with new and different data.

We transform the collected data to match the action specifications and boundaries of the original datasets we collected using EDR, and label each dataset with task instructions.

Kuka data is then mixed with EDR data in a 1:2 ratio in each training batch to control the regression of the original EDR skills.

The picture shows the training method when collecting data from multiple robots

The results show that RT-1 is able to acquire new skills through the experience of observing other robots.

When RT-1 was trained on Kuka’s bin picking data and existing EDR data from the robotics classroom, there was a 22% accuracy jump when trained using only EDR data. "nearly doubled" to 39%.

Accuracy when RT-1 is trained using picking data from Kuka alone and evaluated using picking data from EDR robots is 0%.

Experimental results

To better understand the generalization ability of RT-1, we studied its performance against three baselines: Gato, BC-Z and BC-Z XL (i.e. BC-Z with the same number of parameters as RT-1).

and divide it into four categories:

The picture shows the test environment Performance of RT-1 and control group

• Seen tasks performance (Seen tasks performance): task performance observed during training;
• Unseen tasks performance (Unseen tasks performance): observed during training Performance on invisible tasks where skills and objects are separated in the training set;
  
• Robustness: during the intervention of distractors Performance and performance of background changes (new kitchen, lighting, background scenes)
• Long-horizon scenarios: Execution of SayCan-like natural language instructions in a real kitchen

RT-1’s high performance and generalization capabilities enable long-distance, mobile operation tasks through SayCan.

SayCan works by placing a language model within the robot's affordances and using a small number of prompts to break down the long-term task of expressing natural language into a series of low-level skills.

We evaluate SayCan in two real kitchens using RT-1 and two other baselines (SayCan with Gato and SayCan with BC-Z).

Below, "Kitchen2" poses a more challenging generalization scenario than "Kitchen1". The simulated kitchen used to collect most of the training data was modeled after Kitchen1.

It can be seen that the execution success rate of SayCan with RT-1 in Kitchen1 is 67%, which is better than other baselines.

The performance of SayCan with Gato and SayCan with BCZ decreased due to the generalization difficulties caused by the new invisible kitchen, but the success rate of RT-1 did not decrease accordingly.

The above is the detailed content of Google RT-1 model asks a robot to do several jobs, with a success rate of 97% for 700 instructions. For more information, please follow other related articles on the PHP Chinese website!