Table of Contents
Thesis idea:
Main contribution:
Network design:
Experimental results:
Home Technology peripherals AI Advanced driving simulation: Driving scene reconstruction with realistic surround data

Advanced driving simulation: Driving scene reconstruction with realistic surround data

Jan 01, 2024 pm 12:58 PM
data Autopilot

Original title: DrivingGaussian: Composite Gaussian point rendering for ambient dynamic autonomous driving scenes

Please click here to view the paper: https://arxiv.org/pdf/2312.07920.pdf

Code link: https://pkuvdig.github.io/DrivingGaussian/

Author affiliation: Peking University Google Research University of California, Merced

Thesis idea:

This article proposes DrivingGaussian, a highly efficient and cost-effective framework for dynamic autonomous driving scenarios. For complex scenes with moving objects, this paper first uses incremental static 3D Gaussians to sequentially and progressively model the static background of the entire scene. Then, this paper uses a composite dynamic Gaussian graph to process multiple moving objects, reconstruct each object individually and restore their accurate position and occlusion relationship in the scene. This paper further uses LiDAR priors for Gaussian Splatting to reconstruct the scene with more details and maintain panoramic consistency. DrivingGaussian outperforms existing methods in driving scene reconstruction and enables realistic surround-view synthesis with high fidelity and multi-camera consistency.

Main contribution:

According to the understanding of this article, DrivingGaussian is the first framework to use composite Gaussian splash technology for large-scale dynamic driving scene representation and modeling

Introduced two novel modules, including incremental static 3D Gaussian and composite dynamic Gaussian map. The former incrementally reconstructs a static background, while the latter models multiple dynamic objects using Gaussian maps. Aided by lidar priors, the method helps recover complete geometries in large-scale driving scenarios

Comprehensive experiments demonstrate that DrivingGaussian outperforms previous approaches on challenging autonomous driving benchmarks method, and is able to simulate various extreme situations for downstream tasks

Network design:

This article introduces a new framework called DrivingGaussian for representing look-around Dynamic autonomous driving scenarios. The key idea of ​​this framework is to hierarchically model complex driving scenarios using sequential data from multiple sensors. By using Composite Gaussian Splatting technology, the entire scene is decomposed into static backgrounds and dynamic objects, and each part is reconstructed separately. Specifically, a synthetic scene is first constructed sequentially from look-around multi-camera views using an incremental static 3D Gaussian method. Then, a composite dynamic Gaussian map is employed to reconstruct each moving object individually and dynamically integrate them into the static background through the Gaussian map. On this basis, global rendering is performed through Gaussian Splatting to capture occlusion relationships in the real world, including static backgrounds and dynamic objects. In addition, this paper also introduces a LiDAR prior into the GS representation, which is able to recover more accurate geometries and maintain better multi-view consistency compared to point clouds generated using random initialization or SfM

Extensive experiments show that our method achieves state-of-the-art performance on public autonomous driving datasets. Even without lidar beforehand, our method still shows good performance, demonstrating its versatility in reconstructing large-scale dynamic scenes. In addition, the framework of this article supports dynamic scene construction and corner case simulation, which helps to verify the safety and robustness of the autonomous driving system.

Advanced driving simulation: Driving scene reconstruction with realistic surround data

Figure 1. DrivingGaussian achieves realistic rendering performance for surround-view dynamic autonomous driving scenes. Naive methods [13, 49] either produce unpleasant artifacts and blurring in large-scale backgrounds or have difficulty reconstructing dynamic objects and detailed scene geometry. DrivingGaussian first introduced Composite Gaussian Splatting to effectively represent static backgrounds and multiple dynamic objects in complex surround-view driving scenes. DrivingGaussian enables high-quality synthesis of surround views across multiple cameras and facilitates long-term dynamic scene reconstruction.

Advanced driving simulation: Driving scene reconstruction with realistic surround data

Figure 2. The overall process of this article’s method. Left: DrivingGaussian acquires continuous data from multiple sensors, including multi-camera images and LiDAR. Center: To represent large-scale dynamic driving scenarios, this paper proposes Composite Gaussian Splatting, which consists of two parts. The first part incrementally reconstructs a broad static background, while the second part uses Gaussian maps to construct multiple dynamic objects and dynamically integrate them into the scene. Right: DrivingGaussian demonstrates good performance across multiple tasks and application scenarios.

Advanced driving simulation: Driving scene reconstruction with realistic surround data

Figure 3. Composite Gaussian Splatting with incremental static 3D Gaussian and dynamic Gaussian plots. This article uses Composite Gaussian Splatting to decompose the entire scene into static backgrounds and dynamic objects, reconstruct each part separately and integrate them for global rendering.

Experimental results:

Advanced driving simulation: Driving scene reconstruction with realistic surround data

Advanced driving simulation: Driving scene reconstruction with realistic surround data

Advanced driving simulation: Driving scene reconstruction with realistic surround data

Advanced driving simulation: Driving scene reconstruction with realistic surround data

Advanced driving simulation: Driving scene reconstruction with realistic surround data

Advanced driving simulation: Driving scene reconstruction with realistic surround data

Advanced driving simulation: Driving scene reconstruction with realistic surround data

Summary:

This article introduces DrivingGaussian, A novel framework to represent large-scale dynamic autonomous driving scenarios based on the proposed Composite Gaussian Splatting. DrivingGaussian progressively models a static background using incremental static 3D Gaussians and captures multiple moving objects using composite dynamic Gaussian maps. This paper further exploits LiDAR priors to achieve accurate geometry and multi-view consistency. DrivingGaussian achieves state-of-the-art performance on two autonomous driving datasets, enabling high-quality surround view synthesis and dynamic scene reconstruction

Citation:

Zhou, X., Lin, Z., Shan, X., Wang, Y., Sun, D., & Yang, M. (2023). DrivingGaussian: Composite Gaussian Splatting for Surrounding Dynamic Autonomous Driving Scenes. ArXiv. /abs/2312.07920

Advanced driving simulation: Driving scene reconstruction with realistic surround data

Original link:

https://www.php.cn/link/a878dbebc902328b41dbf02aa87abb58

The above is the detailed content of Advanced driving simulation: Driving scene reconstruction with realistic surround data. For more information, please follow other related articles on the PHP Chinese website!

Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

AI Hentai Generator

AI Hentai Generator

Generate AI Hentai for free.

Hot Article

Repo: How To Revive Teammates
1 months ago By 尊渡假赌尊渡假赌尊渡假赌
R.E.P.O. Energy Crystals Explained and What They Do (Yellow Crystal)
2 weeks ago By 尊渡假赌尊渡假赌尊渡假赌
Hello Kitty Island Adventure: How To Get Giant Seeds
1 months ago By 尊渡假赌尊渡假赌尊渡假赌

Hot Tools

Notepad++7.3.1

Notepad++7.3.1

Easy-to-use and free code editor

SublimeText3 Chinese version

SublimeText3 Chinese version

Chinese version, very easy to use

Zend Studio 13.0.1

Zend Studio 13.0.1

Powerful PHP integrated development environment

Dreamweaver CS6

Dreamweaver CS6

Visual web development tools

SublimeText3 Mac version

SublimeText3 Mac version

God-level code editing software (SublimeText3)

Use ddrescue to recover data on Linux Use ddrescue to recover data on Linux Mar 20, 2024 pm 01:37 PM

DDREASE is a tool for recovering data from file or block devices such as hard drives, SSDs, RAM disks, CDs, DVDs and USB storage devices. It copies data from one block device to another, leaving corrupted data blocks behind and moving only good data blocks. ddreasue is a powerful recovery tool that is fully automated as it does not require any interference during recovery operations. Additionally, thanks to the ddasue map file, it can be stopped and resumed at any time. Other key features of DDREASE are as follows: It does not overwrite recovered data but fills the gaps in case of iterative recovery. However, it can be truncated if the tool is instructed to do so explicitly. Recover data from multiple files or blocks to a single

Open source! Beyond ZoeDepth! DepthFM: Fast and accurate monocular depth estimation! Open source! Beyond ZoeDepth! DepthFM: Fast and accurate monocular depth estimation! Apr 03, 2024 pm 12:04 PM

0.What does this article do? We propose DepthFM: a versatile and fast state-of-the-art generative monocular depth estimation model. In addition to traditional depth estimation tasks, DepthFM also demonstrates state-of-the-art capabilities in downstream tasks such as depth inpainting. DepthFM is efficient and can synthesize depth maps within a few inference steps. Let’s read about this work together ~ 1. Paper information title: DepthFM: FastMonocularDepthEstimationwithFlowMatching Author: MingGui, JohannesS.Fischer, UlrichPrestel, PingchuanMa, Dmytr

How to solve the long tail problem in autonomous driving scenarios? How to solve the long tail problem in autonomous driving scenarios? Jun 02, 2024 pm 02:44 PM

Yesterday during the interview, I was asked whether I had done any long-tail related questions, so I thought I would give a brief summary. The long-tail problem of autonomous driving refers to edge cases in autonomous vehicles, that is, possible scenarios with a low probability of occurrence. The perceived long-tail problem is one of the main reasons currently limiting the operational design domain of single-vehicle intelligent autonomous vehicles. The underlying architecture and most technical issues of autonomous driving have been solved, and the remaining 5% of long-tail problems have gradually become the key to restricting the development of autonomous driving. These problems include a variety of fragmented scenarios, extreme situations, and unpredictable human behavior. The "long tail" of edge scenarios in autonomous driving refers to edge cases in autonomous vehicles (AVs). Edge cases are possible scenarios with a low probability of occurrence. these rare events

Google is ecstatic: JAX performance surpasses Pytorch and TensorFlow! It may become the fastest choice for GPU inference training Google is ecstatic: JAX performance surpasses Pytorch and TensorFlow! It may become the fastest choice for GPU inference training Apr 01, 2024 pm 07:46 PM

The performance of JAX, promoted by Google, has surpassed that of Pytorch and TensorFlow in recent benchmark tests, ranking first in 7 indicators. And the test was not done on the TPU with the best JAX performance. Although among developers, Pytorch is still more popular than Tensorflow. But in the future, perhaps more large models will be trained and run based on the JAX platform. Models Recently, the Keras team benchmarked three backends (TensorFlow, JAX, PyTorch) with the native PyTorch implementation and Keras2 with TensorFlow. First, they select a set of mainstream

The vitality of super intelligence awakens! But with the arrival of self-updating AI, mothers no longer have to worry about data bottlenecks The vitality of super intelligence awakens! But with the arrival of self-updating AI, mothers no longer have to worry about data bottlenecks Apr 29, 2024 pm 06:55 PM

I cry to death. The world is madly building big models. The data on the Internet is not enough. It is not enough at all. The training model looks like "The Hunger Games", and AI researchers around the world are worrying about how to feed these data voracious eaters. This problem is particularly prominent in multi-modal tasks. At a time when nothing could be done, a start-up team from the Department of Renmin University of China used its own new model to become the first in China to make "model-generated data feed itself" a reality. Moreover, it is a two-pronged approach on the understanding side and the generation side. Both sides can generate high-quality, multi-modal new data and provide data feedback to the model itself. What is a model? Awaker 1.0, a large multi-modal model that just appeared on the Zhongguancun Forum. Who is the team? Sophon engine. Founded by Gao Yizhao, a doctoral student at Renmin University’s Hillhouse School of Artificial Intelligence.

Slow Cellular Data Internet Speeds on iPhone: Fixes Slow Cellular Data Internet Speeds on iPhone: Fixes May 03, 2024 pm 09:01 PM

Facing lag, slow mobile data connection on iPhone? Typically, the strength of cellular internet on your phone depends on several factors such as region, cellular network type, roaming type, etc. There are some things you can do to get a faster, more reliable cellular Internet connection. Fix 1 – Force Restart iPhone Sometimes, force restarting your device just resets a lot of things, including the cellular connection. Step 1 – Just press the volume up key once and release. Next, press the Volume Down key and release it again. Step 2 – The next part of the process is to hold the button on the right side. Let the iPhone finish restarting. Enable cellular data and check network speed. Check again Fix 2 – Change data mode While 5G offers better network speeds, it works better when the signal is weaker

The U.S. Air Force showcases its first AI fighter jet with high profile! The minister personally conducted the test drive without interfering during the whole process, and 100,000 lines of code were tested for 21 times. The U.S. Air Force showcases its first AI fighter jet with high profile! The minister personally conducted the test drive without interfering during the whole process, and 100,000 lines of code were tested for 21 times. May 07, 2024 pm 05:00 PM

Recently, the military circle has been overwhelmed by the news: US military fighter jets can now complete fully automatic air combat using AI. Yes, just recently, the US military’s AI fighter jet was made public for the first time and the mystery was unveiled. The full name of this fighter is the Variable Stability Simulator Test Aircraft (VISTA). It was personally flown by the Secretary of the US Air Force to simulate a one-on-one air battle. On May 2, U.S. Air Force Secretary Frank Kendall took off in an X-62AVISTA at Edwards Air Force Base. Note that during the one-hour flight, all flight actions were completed autonomously by AI! Kendall said - "For the past few decades, we have been thinking about the unlimited potential of autonomous air-to-air combat, but it has always seemed out of reach." However now,

The first robot to autonomously complete human tasks appears, with five fingers that are flexible and fast, and large models support virtual space training The first robot to autonomously complete human tasks appears, with five fingers that are flexible and fast, and large models support virtual space training Mar 11, 2024 pm 12:10 PM

This week, FigureAI, a robotics company invested by OpenAI, Microsoft, Bezos, and Nvidia, announced that it has received nearly $700 million in financing and plans to develop a humanoid robot that can walk independently within the next year. And Tesla’s Optimus Prime has repeatedly received good news. No one doubts that this year will be the year when humanoid robots explode. SanctuaryAI, a Canadian-based robotics company, recently released a new humanoid robot, Phoenix. Officials claim that it can complete many tasks autonomously at the same speed as humans. Pheonix, the world's first robot that can autonomously complete tasks at human speeds, can gently grab, move and elegantly place each object to its left and right sides. It can autonomously identify objects

See all articles