At the GTC conference held in September, NVIDIA product manager Gautham Sholingar gave a complete introduction to NVIDIA's latest developments in long-tail scenario training over the past year under the title "Synthetic Data Generation: Accelerating the Development and Verification of Self-Driving Vehicles" Progress and related experiences, specifically exploring how developers can use DRIVE Replicator to generate diverse synthetic datasets with accurate ground-truth data labels to accelerate the development and validation of autonomous vehicles. The lecture was full of useful information and aroused widespread attention and discussion in the industry. This article summarizes and organizes the essence of this sharing to help everyone better understand DRIVE Replicator and the synthetic data generation of autonomous driving perception algorithms.
Figure 1
Over the past year, NVIDIA has been using DRIVE Replicator to generate data for training autonomous driving perception algorithms. Positive progress has been made on synthetic datasets. Figure 1 shows some of the long-tail scenario challenges that NVIDIA is currently solving:
The above functions are being implemented through NVIDIA DRIVE Replicator.
DRIVE Replicator is part of the DRIVE Sim tool suite and can be used for autonomous driving simulation.
DRIVE Sim is NVIDIA's leading self-driving car simulator built on Omniverse, which can perform physically accurate sensor simulation on a large scale. Developers can run repeatable simulations on a workstation and then scale to batch mode in the data center or cloud. DRIVE Sim is a modular platform built on powerful open standards such as USD, allowing users to introduce their own functionality through Omniverse extensions.
DRIVE Sim includes multiple apps including DRIVE Replicator. DRIVE Replicator mainly provides a series of functions focused on the generation of synthetic data for training and algorithm verification of autonomous vehicles. DRIVE Sim and DRIVE Constellation also support full-stack autonomous driving simulation at all levels, including software-in-the-loop, hardware-in-the-loop, and other in-the-loop simulation tests (models, plants, humans, and more).
The difference between DRIVE Sim and traditional autonomous driving simulation tools is that when creating synthetic data sets, traditional autonomous driving simulation tools are often combined with professional game engines to restore sufficiently realistic scenes. However, for autonomous driving simulation, this is far from enough, and core requirements including physical accuracy, repeatability, and scale need to be addressed.
Figure 2
Before further introducing DRIVE Replicator, let me first introduce several related concepts (Figure 2). Especially Omniverse, to help everyone better understand the underlying technical support related to DRIVE Replicator.
First, learn about Omniverse, NVIDIA’s engine for large-scale simulations. Omniverse is built on USD (Universal Scene Description, an extensible universal language for describing virtual worlds) developed by Pixar. USD is a single source of truth-value data for the entire simulation and all aspects of the simulation (including sensors, 3D environment). These scenes built entirely through USD allow developers to have hierarchical access to every element in the simulation, and generate diverse data for subsequent generation. lay the foundation for a specialized synthetic data set.
Second, Omniverse provides real-time ray tracing effects to support sensors in DRIVE Sim. RTX is one of NVIDIA's important advances in computational graphics, leveraging an optimized ray tracing API that focuses on physical accuracy to ensure complex behavior of cameras, lidar, millimeter-wave radar and ultrasonic sensors such as multiple Reflections, multipath effects, rolling shutter, and lens distortion) are natively modeled.
Third, NVIDIA Omniverse is an easily scalable open platform designed for virtual collaboration and physically accurate real-time simulation. It can run workflows in the cloud or data center and achieve multi-GPU and node parallelism. Rendering and data generation.
Fourth, Omniverse and DRIVE Sim adopt open and modular designs, and a huge partner ecosystem has been formed around this platform. These partners can provide 3D materials, sensors, vehicle and traffic models, verification tools, etc.
Fifth, the core of Omniverse collaboration is Nucleus. Nucleus has data storage and access control functions, and it can serve as a backend for multiple users. Centralized content repository with DRIVE Sim to decouple runtime from content, improve version control, and create a single reference point for all footage, scenes, and metadata.
DRIVE Sim is a platform. NVIDIA adopts an ecological cooperation approach to build the platform, allowing partners to contribute to this universal platform. At present, DRIVE Sim has established a huge partner ecosystem, covering 3D assets, environmental sensor models, verification and other fields. With the DRIVE Sim SDK, partners can easily introduce their own sensor, traffic and vehicle dynamics models and extend their core simulation capabilities. Developers can not only write extensions in Omniverse and easily add new features, but also enjoy the benefits of developing on a common platform - Omniverse has connected several key partners who provide important work related to autonomous driving development flow.
Next, I will explain how the above contents are combined, and the five main tasks of DRIVE Replicator to generate synthetic data. Steps (Figure 3): Content - DRIVE Sim Runtime - Sensing - Randomization - Data Writers.
Figure 3
The first step in the simulation process is the 3D content and materials stored on the Nucleus server. These assets are passed to DRIVE Sim Runtime, the core technology for executing scenarios, traffic models, vehicle dynamics and behavior. DRIVE Sim Runtime can be used with RTX ray tracing-based cameras, lidar, millimeter wave radar and perception technology from the USS. The next step is to introduce variety into the data through randomization of motion, behavior, lighting, and appearance. For closed-loop simulation, the next step is to connect the simulation to the autonomous driving stack through vehicle I/O, which typically consists of sensor protocols, CAN messages, and a virtual ECU (which sends important information to the autonomous driving stack to close the loop).
For synthetic data generation, this is an open loop process that sends randomized sensor data to data writers, and these data writers can output for training autonomous driving perception The truth label of the algorithm. The above steps represent a complete workflow for synthetic data generation.
As mentioned above, the first step in the simulation process is the 3D content and materials stored on the Nucleus server . Where does this content come from? How to get it? What are the standards or requirements?
Over the past few years, NVIDIA has worked with multiple content partners to build a vast ecosystem of 3D asset providers, including vehicles, props, pedestrians, vegetation and 3D environments, ready to be used in DRIVE Sim used in.
One thing to note is that even if you obtain these assets from the market, it does not mean that you can start simulation work. You also need to make these assets ready for simulation, and this is SimReady Useful place.
An important part of the expansion is working with 3D asset providers and providing them with the tools they need to ensure that certain conventions, naming, asset rigging, Semantic labels and physical properties.
SimReady Studio helps content providers convert their existing assets into simulation-ready USD assets that can be loaded onto DRIVE Sim, including 3D environments, dynamic assets and static props.
So, what is SimReady? You can think of it as a converter that helps ensure that 3D assets in DRIVE Sim and Replicator are ready to support end-to-end simulation workflows. SimReady has several key elements, including:
Based on the above understanding, let’s take a look at the process of how to use SimReady studio to obtain assets that can be used in DRIVE Sim.
Assume that this process starts with purchasing assets from the 3D market. The first step is to import this asset into SimReady Studio. This can also be done in bulk, or by batch importing multiple assets to complete this step.
After importing, the material names of these content assets will be updated, and their material properties will also be updated, expanding to include properties such as reflectivity and roughness.
This is important to ensure physically realistic rendering data quality, and to ensure that the material system interacts with all RTX sensor types, not just those operating in the visible spectrum.
The next step involves updating semantic tags and tags. Why is this step important? Having the right labels means the data generated using the asset can be used to train AV algorithms. Additionally, DRIVE Sim and Omniverse use Nucleus as a central asset repository. There are thousands of content assets on Nucleus, and having searchable tags with associated thumbnails will help new users find an asset more easily.
Next, start defining the collision volume and geometry of the object and observe how the content asset behaves from a physical perspective. The object's physical and mass properties are then modified to create the desired behavior.
The final step in the entire process is to validate assets to ensure that these content assets adhere to the correct conventions. Simulation-ready USD assets can now be saved and re-imported into NVIDIA Omniverse and DRIVE Sim. The biggest advantage of building a scene via USD is that all metadata created in the previous steps is transferred with the final asset and is hierarchically linked to the USD of the main object, laying the foundation for subsequent generation of diverse synthetic datasets.
Figure 4
Back to self-driving car simulation content creation, there are usually several ways to create an environment from map data (see Figure 4). One option is to use MathWorks' Roadrunner tool to create a 3D environment on an open NVIDIA DRIVE map (multimodal map platform). The output of this step, along with semantic map information, signal timing, etc., is then transferred to SimReady Studio, where the 3D environment can be converted into a USD asset that can be loaded onto DRIVE Sim.
Another option is to use voxel map data from a self-driving fleet and extract semantic map information such as lanes, road signs, and other metadata. This information is created via a digital twin, resulting in a USD asset that can be loaded onto DRIVE Sim.
The above two types of USD environments will be used to support end-to-end (E2E) simulation testing of autonomous vehicles and synthetic data generation workflows.
Next I will introduce you to the second step of simulation - DRIVE Sim Runtime, which lays the foundation for our DRIVE Replicator The basis for all features in . used to generate synthetic datasets.
DRIVE Sim Runtime is an open, modular and extensible component. What does this mean in practice (see Figure 5)?
First, it is built on scenes, where developers can define the specific position, movement, and interaction of objects in the scene. These scenarios can be defined in Python or using the scenario editor UI and saved for later use.
Second, it supports integration with custom vehicle dynamics packages via the DRIVE Sim SDK, either as a step in the process or as a co-simulation with DRIVE Sim 2.0.
Third, traffic model. DRIVE Sim has a rich vehicle model interface, and with the help of the Runtime, developers can introduce their own vehicle dynamics or configure existing rules-based traffic models.
Fourth, the action system, which contains a rich library of predefined actions (such as lane changes), time triggers that can be used to create scenes of interactions between different objects, etc.
Figure 5
Let’s briefly review the previous content: the first step of the simulation process, after SimReady conversion, simulation Ready 3D content and assets are stored on the Nucleus server. In the second step, these materials are passed to DRIVE Sim Runtime, which is the core technology for executing scenarios, traffic models, vehicle dynamics and behavior, laying the foundation for all subsequent functions to generate synthetic data sets.
Before generating data, sensors need to be used to set up the target test vehicle. Using the Ego Configurator tool, developers can select specific vehicles and add them to the scene.
In addition, developers can also move vehicles in the scene and add sensors to the vehicles. The Ego configurator tool supports universal and Hyperion 8 sensors.
After adding the sensor to the vehicle, developers can also change parameters such as FOV, resolution, sensor name, and intuitively configure sensor locations on the vehicle.
Users can also view previews from the sensor POV and visualize the field of view in a 3D environment before creating data generation scenes.
This tool can help developers quickly prototype different configurations and visualize the coverage achieved by sensing tasks.
Now let’s briefly introduce the fourth step of the simulation process, domain randomization, how to pass motion and behavior , randomization of lighting and appearance to introduce variety into the data.
This involves another way of creating scenes, using Python. DRIVE Replicator's Python API allows developers to query the open NVIDIA DRIVE Map and place a range of static and dynamic assets in a context-aware manner. Some randomizers will focus on how to teleport an autonomous vehicle from one point to the next, how to generate objects around the autonomous vehicle, and generate different synthetic datasets from this. These complex-sounding operations are easily accomplished because the user has direct control over the USD scene and all objects in that environment.
Another important step when creating synthetic datasets for training is the ability to introduce changes in the appearance of the 3D scene. The powerful functions of USD were also mentioned above. For example, scenarios built through USD allow developers to have hierarchical access to every element in the simulation. SimReady's API uses USD to quickly set up features in a scene.
Let’s look at an example (see Figure 6): the road surface is a little wet, but when we set different parameters, the wetness level of the road surface will change. We can make similar changes to aspects such as solar azimuth and solar altitude to generate realistic datasets under a range of environmental conditions.
Another takeaway is the ability to enable lighting and appearance changes, all of which are available through the SimReady API and USD.
Figure 6
#One of the main advantages of DRIVE Sim is the RTX sensor workflow, which supports a variety of sensors (see Figure 7 ), including universal and off-the-shelf models for cameras, lidar, conventional radar and USS. In addition, DRIVE Sim offers full support for the NVIDIA DRIVE Hyperion sensor suite, allowing users to begin algorithm development and validation work in a virtual environment.
In addition, DRIVE Sim has a powerful and versatile SDK that enables partners to use NVIDIA’s ray tracing API to implement complex sensor models while protecting their IP and proprietary algorithms. This ecosystem has grown over the years, and NVIDIA is working with partners to bring new types of sensors, such as imaging radar, FMCW lidar, and more, into DRIVE Sim.
Figure 7
Now, the focus shifts to generating ground truth data and how to visualize this information. This involves the last step in the simulation flow, the data writer. In this process, randomized sensor data is sent to data writers, which output ground truth labels used to train autonomous driving perception algorithms.
The data writer is a Python script used to generate ground truth labels required for training autonomous driving perception algorithms.
NVIDIA DRIVE Replicator comes with template writers such as base writer and KITTI writer.
The base writer covers a wide range of real-world data labels, including object classes, tight and loose bounding boxes in 2D and 3D, semantic and instance masks, depth output, occlusions, normals, and more.
Similarly, there are lidar/normal radar writers that can be used to export laser point cloud data to numpy arrays, or to any relevant custom format along with bounding boxes, semantics and object labels .
These writers provide examples for developers to configure their own writers based on custom markup formats and extend their data generation efforts.
Finally, I would like to introduce to you an exciting software, Replicator Insight created by the Omniverse team.
Replicator Insight is a standalone application built on Omniverse Kit that can be used to inspect rendered synthetic datasets and overlay various ground truth labels for training.
Replicator Insight supports all synthetic data generation use cases, including DRIVE, Isaac, and Omniverse Replicator.
Let’s look at an example (see Figure 8): Users can load data generated by DRIVE Replicator in this visualization tool and turn on and off different truth labels for different object categories in the scene.
Figure 8
With this visualization tool, users can play videos, comb through data sets, and even view depth and RGB data and compare between different views.
Users can also change parameters such as playback frame rate and depth range, or quickly visualize data sets before self-driving car training.
This will help developers easily understand new truth label types and parse complex data sets.
Overall, this is a powerful tool that allows users to gain new insights every time they look at data, whether it is real or synthetic.
The above summarizes the latest development of DRIVE Replicator in the past year, and shares how developers can use DRIVE Replicator to generate diverse synthetic data sets and accurate ground truth data tags to accelerate the development and validation of autonomous vehicles. NVIDIA has made exciting progress in generating high-quality sensor data sets for a variety of real-world use cases, and we look forward to further communication with you!
The above is the detailed content of Accelerating autonomous vehicle development and verification: A closer look at DRIVE Replicator synthetic data generation technology. For more information, please follow other related articles on the PHP Chinese website!
![[Web front-end] Node.js quick start](https://img.php.cn/upload/course/000/000/067/662b5d34ba7c0227.png)
