High-gloss reflective terminator? Google NeRF-Casting: Ray tracing can do it!

NeRF is no longer "afraid" of near specular reflections

Early NeRF variants used multilayer perceptrons (MLPs) to map from 3D coordinates to volume density and viewpoints Related colors, but the large MLPs required to represent detailed 3D geometry and color are extremely slow to train and evaluate. Recent work has focused on making NeRF more efficient by replacing large MLPs with voxel grid-like data structures or a combination of grids and small MLPs. While it can be extended to represent detailed large-scale scenes, its advantages are limited to three-dimensional geometry and predominantly diffuse color. Expanding NeRF’s ability to model realistic viewpoint-dependent appearance remains a challenge. The current advanced model for view synthesis of shiny objects has limitations in two aspects:

can only synthesize accurate reflections of distant ambient lighting, while rendering close Poor performance in realistic reflections of scene content.

Relying on large MLPs to represent the viewpoint-related outgoing radiation of any point is difficult to scale to larger realistic scenes with detailed reflections.

• NeRF-Casting is a method to solve these problems by introducing ray tracing into the NeRF rendering model. It mainly involves 3 areas:

Reflection modeling

: Traditional reflection modeling methods use physical laws and image-based techniques to represent surface reflection properties. In recent years, neural networks have been used to learn reflection properties, especially under complex materials and lighting conditions.

• Ray Tracing: Ray tracing is a widely used computer graphics technique that produces realistic images by simulating the interaction of light with object surfaces. Ray tracing techniques have been used to generate high-quality reflection and refraction effects, but with high computational complexity.
• Three-dimensional imaging: Three-dimensional imaging technology involves generating three-dimensional representations from multi-view data. NeRF and other neural network methods have made significant progress in the field of 3D imaging by learning the 3D geometry and color distribution of a scene to generate new views.

Instead of querying an expensive MLP at the points of each camera ray to obtain a viewpoint-dependent appearance, NeRF-Casting casts reflection lines from these points to In NeRF geometry, we sample correctly anti-aliased reflection in-scene content features and use a small MLP to decode these features into reflection colors. Casting rays into the recovered NeRF naturally synthesizes consistent reflections of near and far interior content. Computing appearance via ray tracing reduces the burden of representing highly detailed viewpoint-related functions with large MLPs at every point in the scene.

Interested friends can watch the video effect: https://nerf-casting.github.io

Model details

NeRF-Casting’s Three main goals:

Want to model accurate, detailed reflections without relying on computationally expensive MLP evaluation.

Want to cast only a small amount of reflected light.

• Hopefully minimize the amount of computation required to look up our representation at each point of these reflected rays.
• Based on Zip-NeRF[2]: Utilize a multi-scale hash grid to store 3D features, a small MLP (1 layer, width 64) to decode these features into densities, a larger MLP (3 layers, width 256) decodes these features into colors. This means that it is relatively cheap to query the density and characteristics of a sample along a ray. Given these constraints, the following process is followed to render a specular appearance:

Query the volume density along each camera ray to calculate the ray's expected end point and surface normal.

Project a reflection cone through the desired end point in the direction of reflection.

• Use a small MLP to combine the accumulated reflection features with other sampled quantities (such as diffuse color features and blending weights for each sample) to generate a color value for each sample along the ray.
• Alpha composite these samples and densities to get the final color.

Reflection cone tracking

Then construct a new reflected light direction by reflecting the initial light about the surface normal

##Conical reflection feature

Now that a vMF distribution has been defined covering the reflection rays, the goal is to estimate the expected volume rendering characteristics on the vMF distribution and then decode it into reflection colors. This expected characteristic can be written as:

Using Monte Carlo methods to perform integral estimates on randomly sampled rays is very expensive because each sample requires a volume along the ray render. Inspired by Zip-NeRF, this integral is approximated using a small set of representative samples combined with feature subtraction. However, unlike Zip-NeRF, we perform both operations in the two-dimensional direction domain rather than in the three-dimensional Euclidean space.

Directional Sampling

##Reflection Feature Reduction

The direction sampling described above helps select a small representative set of rays to average. However, for surfaces with high roughness, the sampled rays may be far apart relative to the underlying 3D grid cells. This means that the features in Equation 9 may be subject to artifacts, and small changes in the direction of the reflected ray may cause large changes in appearance.

In order to prevent this from happening, adjust the "feature weight reduction" technology in Zip-NeRF to the direction setting. This is achieved by multiplying features corresponding to voxels that are smaller compared to the vMF cone by a small multiplier, reducing their impact on the rendered color. According to Zip-NeRF, the weight-reducing feature at point x is defined as:

##Color decoder

The role of the color decoder is to assign a color to each sampling point along the ray, using the convexity of the two color components Combination:

The first color component Cv is similar to the typical NeRF perspective-dependent appearance model:

The second A component Cr, intended to simulate a glossy appearance, is calculated as follows:

Geometric representation and regularization

Appreciation of the effect

##To summarize

NeRF-Casting is a method of rendering scenes containing specular objects using Neural Radiation Fields (NeRF).

Method: Reflection cones are reflected from surfaces in the scene and tracked via NeRF, combined with a novel set of techniques to anti-alias these reflections, enabling synthesis of distant and near Accurately detailed reflections of field content that move consistently and smoothly across surfaces.

Discussion: Quantitatively outperforms existing view synthesis techniques, especially for smooth surfaces showing detailed specular reflections. Qualitative visual improvements are far outweighed by quantitative improvements in image metrics. Of particular note is the smooth and consistent motion of reflections synthesized by this method, which is more realistic than the view-dependent appearance presented by baseline methods. This shows that standard image error metrics (PSNR, SSIM, etc.) are insufficient to assess the quality of view-dependent appearance.

The above is the detailed content of High-gloss reflective terminator? Google NeRF-Casting: Ray tracing can do it!. For more information, please follow other related articles on the PHP Chinese website!