


CVPR 2024 | New framework CustomNeRF accurately edits 3D scenes with only text or image prompts
Meitu Imaging Research Institute (MT Lab), together with the Institute of Information Engineering of the Chinese Academy of Sciences, Beijing University of Aeronautics and Astronautics, and Sun Yat-sen University, jointly proposed a 3D scene editing method - CustomNeRF. The research results have been accepted by CVPR 2024. CustomNeRF not only supports text descriptions and reference pictures as editing tips for 3D scenes, but also generates high-quality 3D scenes based on information provided by users.
Neural Radiance Field (NeRF) Since the Neural Radiance Field (NeRF) was proposed in 2020, it has pushed implicit expression to a new level. As one of the most cutting-edge technologies currently, NeRF has been rapidly generalized and applied in fields such as computer vision, computer graphics, augmented reality, and virtual reality, and continues to receive widespread attention. NeRF enables high-quality image synthesis by modeling the radiation and density of each point in the scene, which makes it widely attractive for applications in fields such as computer vision, computer graphics, augmented reality, and virtual reality. NeRF is unique in its ability to generate high-quality images from input scenes without the need for complex 3D scans or dense perspective images. This feature makes NeRF have broad application prospects in many fields, including computer vision, computer graphics, augmented reality, and virtual reality, and continues to receive widespread attention. NeRF enables high-quality image synthesis by modeling the radiance and density of every point in the scene. NeRF can also be used to generate high-quality 3D renderings, which makes it very promising for applications in areas such as virtual reality and augmented reality. The rapid development and widespread application of NeRF will continue to receive widespread attention, and it is expected that more innovations and applications based on NeRF will emerge in the future.
NeRF (Neural Radiation Field) is a feature used for optimization and continuous representation that has many applications in 3D scene reconstruction. It has even driven research in the field of 3D scene editing, such as texture redrawing and stylization of 3D objects or scenes. In order to further improve the flexibility of 3D scene editing, NeRF editing methods based on pre-trained models are also being extensively explored recently. Due to the implicit representation of NeRF and the geometric characteristics of 3D scenes, editing results that comply with text prompts can be obtained. These are Something very easy to implement.
In order to enable text-driven 3D scene editing to achieve precise control, Meitu Imaging Research Institute (MT Lab), the Institute of Information Engineering of the Chinese Academy of Sciences, Beijing University of Aeronautics and Astronautics, and Sun Yat-sen University jointly proposed a method to Text descriptions and reference images are unified into the CustomNeRF framework provided by the editor. This framework has a built-in perspective-specific subject V∗, which is embedded into the hybrid representation to meet general and customized 3D scene editing requirements. The research results have been recorded in CVPR 2024 and the code has been open source.
Paper link: https://arxiv.org/abs/2312.01663
Code link: https://github.com/hrz2000/customnerf
Two major challenges solved by CustomNeRF
Currently, 3D scene editing based on pre-trained diffusion models is the mainstream Methods are mainly divided into two categories.
One is to use the image editing model to iteratively update the images in the data set. However, it is limited by the ability of the image editing model and will fail in some editing situations. Second, the fractional distillation sampling (SDS) loss is used to edit the scene. However, due to the alignment problem between the text and the scene, this method cannot be directly adapted in the real scene and will cause unnecessary distortion in the non-editing area. Modification often requires explicit intermediate expressions such as mesh or voxel. In addition, the current two types of methods mainly focus on text-driven 3D scene editing tasks. Text descriptions are often difficult to accurately express the user's editing needs and cannot customize specific concepts in the image to the 3D scene. , only general editing can be performed on the original 3D scene, so it is difficult to obtain the editing results expected by the user. In fact, the key to obtaining the desired editing results is to accurately identify the image foreground area, which can promote geometrically consistent image foreground editing while maintaining the image background. Therefore, in order to achieve accurate editing of only the foreground area of the image, this paper proposes a training scheme of local-global iterative editing (LGIE), which alternates between image foreground area editing and full image editing. . This solution can accurately locate the image foreground area and only operate on the image foreground while retaining the image background.In addition, in image-driven 3D scene editing, there is a problem of geometric inconsistency in the editing results caused by the fine-tuned diffusion model overfitting to the reference image perspective. In this regard, the paper designs a class-guided regularization, using only class words to represent the subject of the reference image in the local editing stage, and leveraging a general class prior in a pre-trained diffusion model to promote geometrically consistent editing.
The overall process of CustomNeRF
As shown in Figure 2, CustomNeRF uses 3 steps to accurately edit and reconstruct 3D scenes under the guidance of text prompts or reference images. this goal.
# 图 2 Customnerf's overall flowchart
This is first. When reconstruction of the original 3D scene,, when reconstruction of the original 3D scene, CustomNeRF introduces an additional mask field to estimate edit probabilities in addition to regular color and density. As shown in Figure 2(a), for a set of images that need to reconstruct a 3D scene, the paper first uses Grouded SAM to extract the mask of the image editing area from the natural language description, and combines the original image set to train foreground-aware NeRF. After NeRF reconstruction, the edit probabilities are used to distinguish image regions to be edited (i.e., image foreground regions) from irrelevant image regions (i.e., image background regions) to facilitate decoupled rendering during image editing training.
Secondly, in order to unify image-driven and text-driven 3D scene editing tasks, as shown in Figure 2(b), this paper uses the Custom Diffusion method to fine-tune the reference image under image-driven conditions to Learn the key characteristics of a specific agent. After training, the special word V* can be used as a regular word tag to express the subject concept in the reference image, thus forming a hybrid cue, such as “a photo of a V* dog”. In this way, CustomNeRF enables consistent and efficient editing of adaptive types of data, including images or text.
In the final editing stage, due to the implicit expression of NeRF, optimizing the entire 3D area using the SDS loss will result in significant changes in the background areas that should remain consistent with the original scene after editing. As shown in Figure 2(c), the paper proposes a local-global iterative editing (LGIE) scheme for decoupled SDS training, enabling it to preserve background content while editing the layout area.
Specifically, this paper divides the editing training process of NeRF into a more fine-grained manner. With foreground-aware NeRF, CustomNeRF can flexibly control the rendering process of NeRF during training, that is, under a fixed camera perspective, it can choose to render foreground, background, and regular images containing foreground and background. During training, the current NeRF scene can be edited at different levels using the SDS loss by iteratively rendering the foreground and background, combined with corresponding foreground or background cues. Among them, local foreground training allows you to focus only on the area that needs to be edited during the editing process, simplifying the difficulty of editing tasks in complex scenes; while global training takes the entire scene into consideration and can maintain the coordination of the foreground and background. In order to further keep the non-edited area unchanged, the paper also uses the newly rendered background during the background supervision training process before editing training to maintain the consistency of the background pixels.
Additionally, there are exacerbating geometric inconsistencies in image-driven 3D scene editing. Because the diffusion model that has been fine-tuned with the reference image tends to produce images with a perspective similar to that of the reference image during the inference process, causing multiple perspectives of the edited 3D scene to be front-view geometric problems. To this end, the paper designs a class-guided regularization strategy, using special descriptors V* in global cues and only class words in local cues to take advantage of the class priors included in the pre-trained diffusion model, using more Inject new concepts into the scene in a geometrically consistent manner.
Experimental results
Figure 3 and Figure 4 show the comparison of the 3D scene reconstruction results of CustomNeRF and the baseline method in the reference image and text-driven 3D scene editing tasks , CustomNeRF has achieved good editing results, not only achieving good alignment with the editing prompts, but also keeping the background area consistent with the original scene. In addition, Table 1 and Table 2 show the quantitative comparison of CustomNeRF with the baseline method when driven by images and text. The results show that CustomNeRF surpasses the baseline method in text alignment metrics, image alignment metrics, and human evaluation.
## ## with with text-driven editing # Table 1 Image driver Edit and Edit of the baseline 2 2 Text Drive Edit Quantitative comparison with baseline This paper innovatively proposes the CustomNeRF model and supports editing prompts for text descriptions or reference images. , and solves two key challenges - precise foreground-only editing and consistency across multiple views when using a single-view reference image. The scheme includes the local-global iterative editing (LGIE) training scheme, which enables editing operations to focus on the foreground while keeping the background unchanged; and class-guided regularization, which alleviates view inconsistencies in image-driven editing, and has been verified through extensive experiments CustomNeRF enables accurate editing of 3D scenes prompted by textual descriptions and reference images in a variety of real-world scenarios. Summary
The above is the detailed content of CVPR 2024 | New framework CustomNeRF accurately edits 3D scenes with only text or image prompts. For more information, please follow other related articles on the PHP Chinese website!

Hot AI Tools

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Undress AI Tool
Undress images for free

Clothoff.io
AI clothes remover

AI Hentai Generator
Generate AI Hentai for free.

Hot Article

Hot Tools

Notepad++7.3.1
Easy-to-use and free code editor

SublimeText3 Chinese version
Chinese version, very easy to use

Zend Studio 13.0.1
Powerful PHP integrated development environment

Dreamweaver CS6
Visual web development tools

SublimeText3 Mac version
God-level code editing software (SublimeText3)

Hot Topics

There are many ways to install DeepSeek, including: compile from source (for experienced developers) using precompiled packages (for Windows users) using Docker containers (for most convenient, no need to worry about compatibility) No matter which method you choose, Please read the official documents carefully and prepare them fully to avoid unnecessary trouble.

DeepSeekAI Tool User Guide and FAQ DeepSeek is a powerful AI intelligent tool. This article will answer some common usage questions to help you get started quickly. FAQ: The difference between different access methods: There is no difference in function between web version, App version and API calls, and App is just a wrapper for web version. The local deployment uses a distillation model, which is slightly inferior to the full version of DeepSeek-R1, but the 32-bit model theoretically has 90% full version capability. What is a tavern? SillyTavern is a front-end interface that requires calling the AI model through API or Ollama. What is breaking limit

Grayscale Investment: The channel for institutional investors to enter the cryptocurrency market. Grayscale Investment Company provides digital currency investment services to institutions and investors. It allows investors to indirectly participate in cryptocurrency investment through the form of trust funds. The company has launched several crypto trusts, which has attracted widespread market attention, but the impact of these funds on token prices varies significantly. This article will introduce in detail some of Grayscale's major crypto trust funds. Grayscale Major Crypto Trust Funds Available at a glance Grayscale Investment (founded by DigitalCurrencyGroup in 2013) manages a variety of crypto asset trust funds, providing institutional investors and high-net-worth individuals with compliant investment channels. Its main funds include: Zcash (ZEC), SOL,

ElizaOSv2: Empowering AI and leading the new economy of Web3. AI is evolving from auxiliary tools to independent entities. ElizaOSv2 plays a key role in it, which gives AI the ability to manage funds and operate Web3 businesses. This article will dive into the key innovations of ElizaOSv2 and how it shapes an AI-driven future economy. AI Automation: Going to independently operate ElizaOS was originally an AI framework focusing on Web3 automation. v1 version allows AI to interact with smart contracts and blockchain data, while v2 version achieves significant performance improvements. Instead of just executing simple instructions, AI can independently manage workflows, operate business and develop financial strategies. Architecture upgrade: Enhanced A

The entry of top market maker Castle Securities into Bitcoin market maker is a symbol of the maturity of the Bitcoin market and a key step for traditional financial forces to compete for future asset pricing power. At the same time, for retail investors, it may mean the gradual weakening of their voice. On February 25, according to Bloomberg, Citadel Securities is seeking to become a liquidity provider for cryptocurrencies. The company aims to join the list of market makers on various exchanges, including exchanges operated by CoinbaseGlobal, BinanceHoldings and Crypto.com, people familiar with the matter said. Once approved by the exchange, the company initially planned to set up a market maker team outside the United States. This move is not only a sign

Researchers from Shanghai Jiaotong University, Shanghai AILab and the Chinese University of Hong Kong have launched the Visual-RFT (Visual Enhancement Fine Tuning) open source project, which requires only a small amount of data to significantly improve the performance of visual language big model (LVLM). Visual-RFT cleverly combines DeepSeek-R1's rule-based reinforcement learning approach with OpenAI's reinforcement fine-tuning (RFT) paradigm, successfully extending this approach from the text field to the visual field. By designing corresponding rule rewards for tasks such as visual subcategorization and object detection, Visual-RFT overcomes the limitations of the DeepSeek-R1 method being limited to text, mathematical reasoning and other fields, providing a new way for LVLM training. Vis

Weekly Observation: Businesses Hoarding Bitcoin – A Brewing Change I often point out some overlooked market trends in weekly memos. MicroStrategy's move is a stark example. Many people may say, "MicroStrategy and MichaelSaylor are already well-known, what are you going to pay attention to?" This is true, but many investors regard it as a special case and ignore the deeper market forces behind it. This view is one-sided. In-depth research on the adoption of Bitcoin as a reserve asset in recent months shows that this is not an isolated case, but a major trend that is emerging. I predict that in the next 12-18 months, hundreds of companies will follow suit and buy large quantities of Bitcoin

This guide will provide a comprehensive overview of how to install the latest installation packages from EV Exchange on iOS devices. Ouyi Exchange is a leading cryptocurrency trading platform that provides a wide range of cryptocurrency trading, asset management and investment services. By following the step-by-step instructions provided in this guide, users can easily and easily install the Euyi Exchange app on their iPhone or iPad. This guide is suitable for all iOS devices, from older models to the latest generation, and includes clear screenshots and detailed instructions to ensure a seamless installation process.
