In two sentences, let AI generate VR scenes! Or the kind of 3D or HDR panorama?

​Produced by Big Data Digest

Author: Caleb

Recently, ChatGPT can be said to be extremely popular.

On November 30, OpenAI released the chat robot ChatGPT and opened it to the public for free for testing. Since then, it has been popular in China.

To talk to a robot is to ask the robot to execute a certain instruction, such as entering a key The words allow AI to generate corresponding pictures.

This seems to be nothing unusual. Didn’t OpenAI also update a new version of DALL-E in April?

OpenAI, how old are you? (Why is it always you?)

What if the digest bacteria said that the generated images are 3D images, HDR panoramas, or VR-based image content?

Recently, a research team from Nanyang Technological University in Singapore proposed such an AI. As long as the user inputs a clearly described scene in text, the system can generate a realistic 3D scene.

Let’s first take a look at the effect. For example, when entering “a brown wooden pier on the lake during the day surrounded by green trees”, the system will give such an answer. The light and detail effects are directly Laman.

The research has been titled Text2Light: Zero-Shot Text-Driven HDR Panorama Generation Published.

## Paper link: https://arxiv.org/abs/2209.09898

Without training, 3D HDRIs can be generated

High-quality HDRI (high dynamic range images), also known as HDR panoramas, are currently used to create realistic 360-degree 3D scenes popular methods.

Considering the difficulty of capturing HDRIs, although there are many technologies that can use AI to generate 3D scenes, they basically require a series of parameter settings or a large amount of data. Deep learning.

So, the researchers proposed a zero-shot text-driven framework, namely Text2Light, to generate 4K resolution HDRIs, and the entire process does not require corresponding training data.

The process of generating HDRIs can be divided into two steps.

In the first step, the input text is translated into an LDR panorama based on the discrete representation of the dual codebook. The input text is first mapped to text embeddings by a pre-trained CLIP model; secondly, a text-conditional global sampler learns to sample the overall semantics from the global codebook according to the input text; then, a structure-aware local sampler synthesizes local patches and Make a synthesis.

The second step is to upgrade the LDR results of the first stage according to the structured latent encoding as a continuous representation. The super-resolution inverse tone mapping operator (SR-iTMO) proposed by the researchers can simultaneously improve the spatial resolution and dynamic range of the panorama.

In this way, 4K resolution can be generated without training The HDRIs, which are also the most advanced image generation models to date, clean up the instability of the conversion from LDR to HDR and create a pair of panoramas and text for learning.

However, this technology is still in the early research stage and can only produce low-resolution 360-degree panoramic image content. However, the research team plans to use the current technology in the future. Generate panoramic images for upgrade, and add HDR image enhancement effects to make the generated 3D images or VR scenes more smooth and attractive to watch.

Use text driver to generate HDRI

Next, let’s take a look at some operation processes.

Download the checkpoints first, and note that the team has released models for outdoor (local sampler outdoor) and indoor (local sampler indoor) scenes respectively.

Generate HDR panorama from a sentence:

python text2light.py -rg logs/global_sampler_clip -rl logs/local_sampler_outdoor --outdir ./generated_panorama --text "YOUR SCENE DESCRIPTION" --clip clip_emb.npy --sritmo ./logs/sritmo.pth --sr_factor 4

From the series text description Generate HDR panorama:

# assume your texts is stored in alt.txtpython text2light.py -rg logs/global_sampler_clip -rl logs/local_sampler_outdoor --outdir ./generated_panorama --text ./alt.txt --clip clip_emb.npy --sritmo ./logs/sritmo.pth --sr_factor 4

Generate low Resolution (512x1024) LDR Panorama:

##

# assume your texts is stored in alt.txtpython text2light.py -rg logs/global_sampler_clip -rl logs/local_sampler_outdoor --outdir ./generated_panorama --text ./alt.txt --clip clip_emb.npy

The HDR panorama generated in this way can be used directly in any modern graphics. Take the rendering of the San Francisco landscape in the three-dimensional computer graphics software Blender as an example. When inputting landscape photography of mountain ranges under purple and pink skies, we will get an image like this:

为了便于批处理，例如使用多个hdri进行渲染，在命令行中也可以提供渲染3D的脚本。

解包，检查检查Blender的使用情况：

# assume your downloaded version is 3.1.2tar -xzvf blender-3.1.2-linux-x64.tar.xzcd blender-3.1.2-linux-x64./blender --help

添加别名：

# PATH_TO_DOWNLOADED_BLENDER indicates the parent directory where you save the downloaded blenderalias blender="/PATH_TO_DOWNLOADED_BLENDER/blender-3.1.2-linux-x64/blender"

然后回到Text2Light代码库，为不同的呈现设置运行以下命令：

blender --background --python rendering_shader_ball.py -- ./rendered_balls 100 1000 PATH_TO_HDRI

就能得到这样的结果：

该项目也在GitHub上开源了：

GitHub链接：https://github.com/FrozenBurning/Text2Light

这个项目也得到了不少网友的好评。有网友就感叹到，“人类的想象力是没有边界的”，照这个趋势来看我们距离“输入文字就能3D打印出一个真实物体”的时代也不远了。

也有网友表示，当试图输入“一个四层半的榻榻米房间，房间内有推拉门、拉门、餐桌、14寸黑白电视、黑色电话机”，仍然会担心AI能否比较准确地再现这种场景。毕竟在想象中，“这应该是一个有异国情调的房间”。

对这个速成HDR全景图的AI，大家有什么看法呢？也欢迎小伙伴们在评论区分享自己的使用心得~

相关报道：https://www.itmedia.co.jp/news/articles/2210/11/news036.html

The above is the detailed content of In two sentences, let AI generate VR scenes! Or the kind of 3D or HDR panorama?. For more information, please follow other related articles on the PHP Chinese website!