iPhone renders a 300-square-meter room in real time, reaching centimeter-level accuracy! Google's latest research: NeRF is not bankrupt yet

3D real-time rendering of large scenes can be completed with a computer or even a mobile phone.

Every blind corner from the living room to the master bedroom, storage room, kitchen, and bathroom can be realistically rendered on the computer, just like shooting a video of the real thing.

Moreover, you can also complete complex scene rendering on an iPhone.

Researchers from Google, Google DeepMind and the University of Tübingen recently proposed a new technology SMERF.

It can render large view scenes in real time on various devices such as smartphones and laptops.

Paper address: https://arxiv.org/pdf/2312.07541.pdf

Essence Technically speaking, SMERF is a method based on NeRFs and relies on MERF (Memory-Efficient Radiance Fields), which is more memory efficient.

NeRF is dead?

Currently, Radiance fields have become a powerful and easy-to-optimize representation for reconstructing and re-rendering photorealistic real-world 3D scenes.

In contrast to explicit representations such as meshes and point clouds, radiation fields are typically stored as neural networks and rendered using volumetric ray travel.

Given a large enough computational budget, neural networks can concisely represent complex geometries and view-dependent effects.

#As a volumetric representation, the number of operations required to render an image is measured in the number of pixels rather than the number of primitives (e.g. triangles), giving the best performance The best models require tens of millions of network evaluations.

Therefore, real-time methods of radiation fields make concessions in terms of quality, speed, or representation size, and whether such representations can compete with alternatives such as Gaussian Splatting , remains an open question.

In the latest research, the author proposes a scalable method to achieve higher fidelity real-time large-space rendering than ever before.

SMERF renders in real time with centimeter-level accuracy

SMERF is specially designed for learning large 3D representations, such as the rendering of houses.

Google and other researchers combined a hierarchical model partitioning scheme, in which different parts of the space and learning parameters are represented by different MERFs.

This not only increases model capacity, but also limits computational and memory requirements. Because large 3D representations like this cannot be rendered in real time with classic NERF.

Coordinate system of a scene in SMERF with K=3 coordinate space partitions and P=4 delayed appearance network sub-partitions

In order to improve the rendering quality of SMERF, the research team also used a "teacher-student" distillation method.

In this method, the already trained high-quality Zip-Nerf model (teacher) is used to train a new MERF model (student).

As shown below, the overall process of "teacher supervision". The teacher model provides photometric supervision by rendering colors and geometric supervision by volumetric weighting along camera rays. Both teacher and student operate on the same set of light intervals.

This approach allows researchers to transfer the detail and image quality of powerful Zip-Nerf models to more efficient and faster structures.

This is especially useful for apps on less powerful devices such as smartphones and laptops.

Experimental evaluation

The researchers first evaluated the method on the four major scenarios introduced by Zip-NeRF: Berlin, Alameda, and London and New York.

Each of these scenes were taken from 1,000-2,000 photos using a 180° fisheye lens. For a comprehensive comparison with 3DGS, the researchers cropped the photos to 110° and used COLMAP to re-estimate the camera parameters.

The results shown in Table 1 show that for moderate spatial subdivisions K, the accuracy of the state-of-the-art methods significantly exceeds MERF and 3DGS.

As K increases, the reconstruction accuracy of the model improves and is close to the accuracy of its Zip-NeRF teacher. When K=5, the difference is less than 0.1 PSNR and 0.01 SSIM.

The researchers also found that these quantitative improvements underestimated the qualitative improvements in reconstruction accuracy, as shown in Figure 5.

In large scenes, the SMERF method consistently models thin geometry, high-frequency textures, specular highlights, and distant content beyond the reach of real-time baselines.

#At the same time, the researchers found that increasing sub-model resolution naturally improves quality, especially in terms of high-frequency textures.

In fact, the researchers found that the latest rendering method is almost indistinguishable from Zip-NeRF, as shown in Figure 8.

Additionally, the researchers further evaluated the state-of-the-art method on the mip-NeRF 360 dataset of indoor and outdoor scenes.

These scenes are much smaller than those in the Zip-NeRF dataset, so no spatial subdivision is required to obtain high-quality results. As shown in Table 2, the K=1 version of the model outperforms all previous real-time models in this benchmark in terms of image quality and is comparable in rendering speed to 3DGS.

Figures 6 and 8 qualitatively illustrate this improvement. The method proposed by the researchers is much better at representing high-frequency geometry and textures. , while eliminating distracting flotsam and fog.

Web pages can transmit realistic 3D spaces

Once trained, SMERF can be used in browsers Rollup enables full 6 degrees of freedom navigation with real-time rendering on popular smartphones and laptops.

Everyone knows that the ability to render large 3D scenes in real time is important for a variety of applications, including video games, virtual augmented reality, and professional design and architecture applications.

For example, in Google Immersive Maps, real-time navigation is possible.

However, the latest methods proposed by teams such as Google also have certain limitations. Although SMERF has excellent reconstruction quality and storage efficiency, it has high storage cost, long loading time, and heavy training workload.

However, this study shows that NeRFs and similar radiation fields may still have advantages in the future compared with three-dimensional Gaussian stitching methods.

The above is the detailed content of iPhone renders a 300-square-meter room in real time, reaching centimeter-level accuracy! Google's latest research: NeRF is not bankrupt yet. For more information, please follow other related articles on the PHP Chinese website!