Home > Technology peripherals > AI > body text

A must-have for beginners, NeRF study notes provide insight into everything!

WBOY
Release: 2023-12-26 13:05:33
forward
978 people have browsed it

What exactly is the neural radiation field?

Radiation field: The light emitted by the light source is formed during the propagation and reflection process in the scene Energy distribution. In layman's terms, it is a function that records the radiation information in a certain direction at a certain location in space. The radiation information (or energy distribution) is actually color, brightness, shadow and other information. The direction here requires extra attention, it is one of the important factors for NeRF to achieve real reconstruction!

A must-have for beginners, NeRF study notes provide insight into everything!

This leads to the concept of neural radiation field.
Neural Radiation Field: Use neural network to store space position radiation in any direction . The description in the original text is as follows:

A must-have for beginners, NeRF study notes provide insight into everything!

The more standardized formula is expressed as follows:

A must-have for beginners, NeRF study notes provide insight into everything!

Input 3D position (x,y ,z) and 2D viewing angle direction (), the output is color and volume density

NeRF specific network results refer to the original text as follows:

A must-have for beginners, NeRF study notes provide insight into everything!

  • The 3D coordinates d is sent to fc (one layer, 128 channels, ReLU activation) to predict RGB;

A must-have for beginners, NeRF study notes provide insight into everything!It can be seen from the above that NeRF is implicit modeling because the model is stored In MLP, the model is the parameter of MLP, which is different from the previous point cloud and mesh modeling (point cloud/mesh can directly see the model). NeRF must query three-dimensional points one by one and then render them into an image. This

viewing method

or rendering method is called volume rendering. Before looking at volume rendering. Let’s first take a look at the effect of the network:

A must-have for beginners, NeRF study notes provide insight into everything!It can be seen that the colors are different under different viewing angles! This is one of the very important advantages of NeRF over traditional reconstruction~

The core of NeRF: volume rendering

Let’s enter the second core point of NeRF-volume render. Volume rendering is a method used to render color and density into 2D images!

A must-have for beginners, NeRF study notes provide insight into everything!The schematic diagram is as follows: Figure a shows that a ray is emitted from the optical center position of the camera. There are sampling points on the ray. The sampling points and directions are sent to the MLP to obtain the color and volume density. Figure c shows the volume density distribution curve along the ray, which is obtained through sampling. The color of the pixel can be obtained by integrating the curve. This process is volume rendering

A must-have for beginners, NeRF study notes provide insight into everything!To summarize the volume rendering steps:

A ray penetrating each pixel is emitted from the optical center of the camera. Take three-dimensional sampling points;
  • Send the sampling point coordinates and viewing angle direction to MLP to calculate color and volume density;
  • Integrate (stack) color information according to volume density to form a 2D image

A must-have for beginners, NeRF study notes provide insight into everything!The formula for volume rendering is as follows:

A must-have for beginners, NeRF study notes provide insight into everything! Of course, the discrete version of the formula is actually used:

A must-have for beginners, NeRF study notes provide insight into everything!

Implicit reconstruction process

After talking about the neural radiation field and volume rendering, now we start the complete reconstruction process~

In Before forming a completed pipeline, there are still two problems that need to be solved:

A must-have for beginners, NeRF study notes provide insight into everything!In order to solve the above two problems, NeRF proposed the position encoding and stratified sampling process

Position coding:

A must-have for beginners, NeRF study notes provide insight into everything!The paper visually shows the comparison of the effects of position coding:

It can be seen that without position coding, the model cannot express high-frequency geometric and texture information~

Multi-layer adoption:

A must-have for beginners, NeRF study notes provide insight into everything!

The training process is as follows :

A must-have for beginners, NeRF study notes provide insight into everything!

Experiment and summary

Evaluation indicators:

A must-have for beginners, NeRF study notes provide insight into everything!

Experiment Settings:

A must-have for beginners, NeRF study notes provide insight into everything!

The experimental results show that the astigmatism on balls of various materials can also be well expressed~

A must-have for beginners, NeRF study notes provide insight into everything!

A must-have for beginners, NeRF study notes provide insight into everything!

Ablation experiment:

A must-have for beginners, NeRF study notes provide insight into everything!

Summary

The neural radiation proposed in the article Field, a sampling ray is emitted from the camera optical center through the pixel, a point is taken on the ray, its three-dimensional position and viewing direction are mapped to volume density and color using an MLP, and then volume rendering is used to stack the volume density and color on the sampling ray. , get the pixel value. The error between the pixel value and the GT image is calculated and then back-propagated to optimize the MLP parameters. This paper uses such an implicit reconstruction method to achieve photorealistic model reconstruction and rendering.

Defects:

  • Slow rendering and training speed;
  • Have high requirements on the number and distribution of perspectives;
  • It is difficult to expand to have backgrounds or larger scenes.

A must-have for beginners, NeRF study notes provide insight into everything!

Original link: https://mp.weixin.qq.com/s/ctDBTaLWuHTM9MONrAor4g

The above is the detailed content of A must-have for beginners, NeRF study notes provide insight into everything!. For more information, please follow other related articles on the PHP Chinese website!

Related labels:
source:51cto.com
Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn
Popular Tutorials
More>
Latest Downloads
More>
Web Effects
Website Source Code
Website Materials
Front End Template