Table of Contents
Part 2 The algorithm principle of NeRFFaceEditing
Part 4 Conclusion and Acknowledgments
Home Technology peripherals AI NeRFFaceEditing, a mask editing method for facial neural radiation fields, can edit three-dimensional faces without 3D modeling.

NeRFFaceEditing, a mask editing method for facial neural radiation fields, can edit three-dimensional faces without 3D modeling.

Apr 13, 2023 am 11:16 AM
number method

Want to personally design a highly realistic three-dimensional face, but find that you are not familiar with professional design software? The 3D face editing method NeRFFaceEditing provides a new solution. Even if you don’t know 3D modeling, you can freely edit highly realistic three-dimensional faces and model personalized digital portraits in the metaverse!

NeRFFaceEditing was completed by researchers from the Institute of Computing Technology, Chinese Academy of Sciences and City University of Hong Kong. Related technical papers were published at ACM SIGGRAPH Asia 2022, the top conference on computer graphics.

NeRFFaceEditing, a mask editing method for facial neural radiation fields, can edit three-dimensional faces without 3D modeling.

##Project homepage: http://geometrylearning.com/NeRFFaceEditing/

NeRFFaceEditing uses two-dimensional semantic masks as a bridge for three-dimensional geometry editing. The semantic editing performed by users in one perspective can be propagated to the entire three-dimensional face geometry while keeping the material unchanged. Furthermore, given an image representing a reference style, the user can easily change the material style of the entire 3D face while keeping the geometry unchanged.

Based on the 3D face editing system based on this method, even users who are not familiar with professional 3D design can easily carry out personalized face design and customize face shape and appearance. Let’s first look at two amazing effects using NeRFFaceEditing!

NeRFFaceEditing, a mask editing method for facial neural radiation fields, can edit three-dimensional faces without 3D modeling.

​Figure 1 Geometric editing effect: the edit on the two-dimensional semantic mask is propagated to the entire three-dimensional geometric space

NeRFFaceEditing, a mask editing method for facial neural radiation fields, can edit three-dimensional faces without 3D modeling.

Figure 2 Style transfer effect: Apply a given style to the entire three-dimensional face space. Keeping geometry unchanged

Part I Background

In recent years, with the development of neural radiation fields[1] and adversarial generative networks[2 ], various high-quality, fast-rendering 3D face generation networks have been proposed, including EG3D [3].

NeRFFaceEditing, a mask editing method for facial neural radiation fields, can edit three-dimensional faces without 3D modeling.

##Figure 3 Generation effects and geometric representation of different perspectives of EG3D

The three-plane representation of this method combines the traditional two-dimensional generative adversarial network and the latest three-dimensional implicit representation, so it inherits the powerful generation ability of StyleGAN [4] and the representation ability of neural radiation fields. However, these generative models cannot provide decoupled control of the geometry and material of the human face, which is an indispensable feature for applications such as 3D character design.

Existing work, such as DeepFaceDrawing [5] and DeepFaceEditing [6], can realize the decoupling control of geometry and material based on line drawing and the generation and editing of two-dimensional face images. DeepFaceVideoEditing [7] applies line drawing editing to face videos, which can generate rich editing effects in time series.

However, image decoupling and editing methods are difficult to directly apply to three-dimensional space. However, existing geometric and material decoupling methods for three-dimensional faces often require retraining network parameters, and the spatial representation methods used have major limitations and lack the good properties of three-plane representation. In order to solve the above problems, NeRFFaceEditing is based on the pre-trained model parameters of the three-dimensional generative adversarial network represented by the three planes, and uses the two-dimensional semantic mask from any perspective as a medium to realize geometric editing of the three-dimensional face and the solution of the material. coupling control.

Part 2 The algorithm principle of NeRFFaceEditing

After the three-plane generator generates the three-plane, it is inspired by AdaIN [8], that is, for the two-dimensional feature map (Feature Map), its Statistics can represent its style, and NeRFFaceEditing decomposes the three planes into mean and standard deviation (a) that express spatially invariant high-level material features, and normalized three planes that express spatially varying geometric features. Combining the standardized three-plane and the decomposed material characteristics (a) can restore the original three-plane. Therefore, given different material characteristics, the same geometry can be given different materials.

Further, in order to achieve decoupled control of geometry and materials, NeRFFaceEditing decomposes the original single decoder into a geometry decoder and a material decoder. The geometry decoder inputs features, predicted densities and semantic labels obtained from normalized three-plane sampling, and is used to express the geometric and semantic mask volume (Volume) of the 3D face. After the geometric features and material features (a) are combined through the controllable material module (CAM) module, the sampled features are then input into the material decoder to predict the color. Finally, through volume rendering, the face image and the corresponding semantic mask from a certain perspective are obtained. Given a different material feature (b), the geometric feature and material feature (b) can be used to obtain another face image with unchanged geometry and changed material through the CAM module and volume rendering. The overall network structure is shown in the figure below:

NeRFFaceEditing, a mask editing method for facial neural radiation fields, can edit three-dimensional faces without 3D modeling.

##Figure 4 NeRFFaceEditing network architecture

In addition, in order to constrain the rendering results of samples with the same material features but different geometries to be similar in material, NeRFFaceEditing uses the generated semantic masks and uses histogram features to represent these respectively. The color distribution of different facial components, such as hair, skin, etc., in samples with the same material characteristics but different geometry. The distance sum of the color distribution of these samples over the individual components is then optimized. As shown in the figure below:

NeRFFaceEditing, a mask editing method for facial neural radiation fields, can edit three-dimensional faces without 3D modeling.

Figure 5 Material similarity constraint training strategy

Part 3 Effect display and experimental comparison

Using NeRFFaceEditing, you can use two-dimensional semantic masks to perform geometric editing on the three-dimensional face space:

NeRFFaceEditing, a mask editing method for facial neural radiation fields, can edit three-dimensional faces without 3D modeling.

Figure 6 Three-dimensional face geometry editing

In addition, you can also edit based on Refer to the picture to perform material style migration in a three-dimensional consistent three-dimensional space:

NeRFFaceEditing, a mask editing method for facial neural radiation fields, can edit three-dimensional faces without 3D modeling.

Figure 7 Three-dimensional face style migration

On this basis, decoupled face interpolation deformation application can be realized. As shown in the figure below, the upper left corner and lower right corner are used as the starting and ending points, and the camera, geometry, and material are Perform linear interpolation:

NeRFFaceEditing, a mask editing method for facial neural radiation fields, can edit three-dimensional faces without 3D modeling.

Figure 8 Decoupling face deformation effect display

Using PTI [9] to back-project real images into the latent space of NeRFFaceEditing, editing and style transfer of real images can also be achieved. Through this, NeRFFaceEditing was also compared with other open source methods for face editing that can control the viewing angle, namely SofGAN [10], proving the superiority of the method.

NeRFFaceEditing, a mask editing method for facial neural radiation fields, can edit three-dimensional faces without 3D modeling.

Figure 9 Example of three-dimensional geometry editing of real images. It can be seen that the authenticity of NeRFFaceEditing is better than SofGAN, and SofGAN has certain changes in identity from other perspectives.

NeRFFaceEditing, a mask editing method for facial neural radiation fields, can edit three-dimensional faces without 3D modeling.

Figure 10 Example of real image style transfer. It can be seen that SofGAN has certain flaws and there are certain changes in identity.

Part 4 Conclusion and Acknowledgments

Digital content generation is widely used in the fields of industrial production and digital media, especially for virtual digital people Generation and editing have received widespread attention recently, and the decoupled editing of 3D face geometry and materials is a possible solution for personalized shaping of real virtual images.

NeRFFaceEditing system, by decoupling the design of the three-dimensional face generation network, can transform the user's modification of the semantic mask from the two-dimensional perspective into the geometry of the entire three-dimensional space Modify and ensure that the material does not change. In addition, with the help of training strategies that enhance the style transfer effect, effective material style transfer in three-dimensional space can be achieved. NeRFFaceEditing's paper has been accepted by ACM SIGGRAPH ASIA 2022, a top computer graphics conference.

The research team of this project includes Jiang Kaiwen (first author), an undergraduate student in the elite class of the Institute of Computing Technology, Chinese Academy of Sciences, Associate Researcher Gao Lin (corresponding author of this article), Dr. Chen Shuyu and City University of Hong Kong Professor Fu Hongbo, etc. For more details about the paper, please visit the project homepage:

​http://geometrylearning.com/NeRFFaceEditing/​

The above is the detailed content of NeRFFaceEditing, a mask editing method for facial neural radiation fields, can edit three-dimensional faces without 3D modeling.. For more information, please follow other related articles on the PHP Chinese website!

Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

AI Hentai Generator

AI Hentai Generator

Generate AI Hentai for free.

Hot Article

R.E.P.O. Energy Crystals Explained and What They Do (Yellow Crystal)
2 weeks ago By 尊渡假赌尊渡假赌尊渡假赌
Hello Kitty Island Adventure: How To Get Giant Seeds
1 months ago By 尊渡假赌尊渡假赌尊渡假赌
Two Point Museum: All Exhibits And Where To Find Them
1 months ago By 尊渡假赌尊渡假赌尊渡假赌

Hot Tools

Notepad++7.3.1

Notepad++7.3.1

Easy-to-use and free code editor

SublimeText3 Chinese version

SublimeText3 Chinese version

Chinese version, very easy to use

Zend Studio 13.0.1

Zend Studio 13.0.1

Powerful PHP integrated development environment

Dreamweaver CS6

Dreamweaver CS6

Visual web development tools

SublimeText3 Mac version

SublimeText3 Mac version

God-level code editing software (SublimeText3)

How to write a novel in the Tomato Free Novel app. Share the tutorial on how to write a novel in Tomato Novel. How to write a novel in the Tomato Free Novel app. Share the tutorial on how to write a novel in Tomato Novel. Mar 28, 2024 pm 12:50 PM

Tomato Novel is a very popular novel reading software. We often have new novels and comics to read in Tomato Novel. Every novel and comic is very interesting. Many friends also want to write novels. Earn pocket money and edit the content of the novel you want to write into text. So how do we write the novel in it? My friends don’t know, so let’s go to this site together. Let’s take some time to look at an introduction to how to write a novel. Share the Tomato novel tutorial on how to write a novel. 1. First open the Tomato free novel app on your mobile phone and click on Personal Center - Writer Center. 2. Jump to the Tomato Writer Assistant page - click on Create a new book at the end of the novel.

How to enter bios on Colorful motherboard? Teach you two methods How to enter bios on Colorful motherboard? Teach you two methods Mar 13, 2024 pm 06:01 PM

Colorful motherboards enjoy high popularity and market share in the Chinese domestic market, but some users of Colorful motherboards still don’t know how to enter the bios for settings? In response to this situation, the editor has specially brought you two methods to enter the colorful motherboard bios. Come and try it! Method 1: Use the U disk startup shortcut key to directly enter the U disk installation system. The shortcut key for the Colorful motherboard to start the U disk with one click is ESC or F11. First, use Black Shark Installation Master to create a Black Shark U disk boot disk, and then turn on the computer. When you see the startup screen, continuously press the ESC or F11 key on the keyboard to enter a window for sequential selection of startup items. Move the cursor to the place where "USB" is displayed, and then

How to recover deleted contacts on WeChat (simple tutorial tells you how to recover deleted contacts) How to recover deleted contacts on WeChat (simple tutorial tells you how to recover deleted contacts) May 01, 2024 pm 12:01 PM

Unfortunately, people often delete certain contacts accidentally for some reasons. WeChat is a widely used social software. To help users solve this problem, this article will introduce how to retrieve deleted contacts in a simple way. 1. Understand the WeChat contact deletion mechanism. This provides us with the possibility to retrieve deleted contacts. The contact deletion mechanism in WeChat removes them from the address book, but does not delete them completely. 2. Use WeChat’s built-in “Contact Book Recovery” function. WeChat provides “Contact Book Recovery” to save time and energy. Users can quickly retrieve previously deleted contacts through this function. 3. Enter the WeChat settings page and click the lower right corner, open the WeChat application "Me" and click the settings icon in the upper right corner to enter the settings page.

Quickly master: How to open two WeChat accounts on Huawei mobile phones revealed! Quickly master: How to open two WeChat accounts on Huawei mobile phones revealed! Mar 23, 2024 am 10:42 AM

In today's society, mobile phones have become an indispensable part of our lives. As an important tool for our daily communication, work, and life, WeChat is often used. However, it may be necessary to separate two WeChat accounts when handling different transactions, which requires the mobile phone to support logging in to two WeChat accounts at the same time. As a well-known domestic brand, Huawei mobile phones are used by many people. So what is the method to open two WeChat accounts on Huawei mobile phones? Let’s reveal the secret of this method. First of all, you need to use two WeChat accounts at the same time on your Huawei mobile phone. The easiest way is to

The secret of hatching mobile dragon eggs is revealed (step by step to teach you how to successfully hatch mobile dragon eggs) The secret of hatching mobile dragon eggs is revealed (step by step to teach you how to successfully hatch mobile dragon eggs) May 04, 2024 pm 06:01 PM

Mobile games have become an integral part of people's lives with the development of technology. It has attracted the attention of many players with its cute dragon egg image and interesting hatching process, and one of the games that has attracted much attention is the mobile version of Dragon Egg. To help players better cultivate and grow their own dragons in the game, this article will introduce to you how to hatch dragon eggs in the mobile version. 1. Choose the appropriate type of dragon egg. Players need to carefully choose the type of dragon egg that they like and suit themselves, based on the different types of dragon egg attributes and abilities provided in the game. 2. Upgrade the level of the incubation machine. Players need to improve the level of the incubation machine by completing tasks and collecting props. The level of the incubation machine determines the hatching speed and hatching success rate. 3. Collect the resources required for hatching. Players need to be in the game

How to set font size on mobile phone (easily adjust font size on mobile phone) How to set font size on mobile phone (easily adjust font size on mobile phone) May 07, 2024 pm 03:34 PM

Setting font size has become an important personalization requirement as mobile phones become an important tool in people's daily lives. In order to meet the needs of different users, this article will introduce how to improve the mobile phone use experience and adjust the font size of the mobile phone through simple operations. Why do you need to adjust the font size of your mobile phone - Adjusting the font size can make the text clearer and easier to read - Suitable for the reading needs of users of different ages - Convenient for users with poor vision to use the font size setting function of the mobile phone system - How to enter the system settings interface - In Find and enter the "Display" option in the settings interface - find the "Font Size" option and adjust it. Adjust the font size with a third-party application - download and install an application that supports font size adjustment - open the application and enter the relevant settings interface - according to the individual

The difference between Go language methods and functions and analysis of application scenarios The difference between Go language methods and functions and analysis of application scenarios Apr 04, 2024 am 09:24 AM

The difference between Go language methods and functions lies in their association with structures: methods are associated with structures and are used to operate structure data or methods; functions are independent of types and are used to perform general operations.

How to choose a mobile phone screen protector to protect your mobile phone screen (several key points and tips for purchasing mobile phone screen protectors) How to choose a mobile phone screen protector to protect your mobile phone screen (several key points and tips for purchasing mobile phone screen protectors) May 07, 2024 pm 05:55 PM

Mobile phone film has become one of the indispensable accessories with the popularity of smartphones. To extend its service life, choose a suitable mobile phone film to protect the mobile phone screen. To help readers choose the most suitable mobile phone film for themselves, this article will introduce several key points and techniques for purchasing mobile phone film. Understand the materials and types of mobile phone films: PET film, TPU, etc. Mobile phone films are made of a variety of materials, including tempered glass. PET film is relatively soft, tempered glass film has good scratch resistance, and TPU has good shock-proof performance. It can be decided based on personal preference and needs when choosing. Consider the degree of screen protection. Different types of mobile phone films have different degrees of screen protection. PET film mainly plays an anti-scratch role, while tempered glass film has better drop resistance. You can choose to have better

See all articles