0.5 seconds, no GPU required, Stability AI and Chinese team VAST open source single image to generate 3D model TripoSR

Recently, Vincent’s video model Sora has set off a new wave of generative AI models, and the model’s multi-modal capabilities have attracted widespread attention.

Now, AI models have made new breakthroughs in 3D content generation.

After successfully launching image generation and video generation projects, Stability AI, which specializes in visual content generation, is now committed to expanding into the 3D field. The latest news shows that the company has cooperated with the Chinese team VAST to jointly open source a single-image generated 3D model project called TripoSR. This move further consolidates Stability AI’s leading position in the field of digital content generation and brings users a more diverse and innovative product experience.

TripoSR is able to generate high-quality 3D models from a single image in 0.5s and doesn’t even require a GPU to run.

• TripoSR model code: https://github.com/VAST-AI-Research/TripoSR

• TripoSR Model Weights: https://huggingface.co/stabilityai/TripoSR

• TripoSR Demo: https://huggingface.co/spaces/stabilityai/TripoSR

When TripoSR was tested on an NVIDIA A100, it was able to generate a high-quality sketch 3D mesh model with textures in about 0.5 seconds, a performance that exceeded other open source image-to-3D model tools. Such as OpenLRM. In addition to speed, TripoSR is fully usable by users with or without a GPU.

TripoSR is inspired by LRM launched by Adobe in November 2023. This is an innovative technology for image generation 3D Large Reconstruction Model (LRM). LRM is unique in that it can generate a corresponding 3D model from any single input image in just a few seconds. This technology is groundbreaking in its efficiency and accuracy, allowing users to easily convert flat images into realistic three-dimensional models. TripoSR has been further developed and optimized based on this technology, allowing users to create stunning three-dimensional scenes more quickly and accurately. By combining the LRM The 3D task can be understood as the process of translating image language into 3D model language. The "words" in the image language (analogous to the tokens of the language model and the patches of the video model) are small pieces that the user input image is divided into; while in the LRM method, the "words" of the 3D model language are a type of information called For each small block in the three-dimensional representation of "triplane", what LRM does is to translate the "words" in the image language into "words" in the 3D model language, and realize the input image and output 3D model.

With the support of the transformer architecture, LRM was trained on more than one million public 3D data and demonstrated phenomenal graph-generating 3D effects and efficiency. Therefore, it has attracted great attention in academia and industry. sensation. However, the relevant codes and models are not open source, and the huge training cost (128 A100s run for a week) also prohibits small research organizations. These factors have greatly hindered the civilian development of this technology.

This time Tripo AI and Stability AI jointly launched the first high-quality open source implementation of LRM - TripoSR, which can generate high-quality three-dimensional models based on images provided by users in almost real-time, greatly filling the gap of 3D generative models. A critical gap in the field of artificial intelligence.

According to Stability’s blog and technical report, the model is based on LRM’s original algorithm, through a finely filtered and rendered subset of the Objaverse dataset, and a series of model and training improvements. , significantly improves the ability to generalize from limited training data, while also enhancing the fidelity of 3D reconstruction. Until the emergence of TripoSR, the academic and open source communities have been lacking an open, fast, and powerful 3D generation basic model and framework with strong generalization capabilities. Although there have been widely watched open source projects such as threestudio, generating a 3D model is slow and resource-intensive due to the technologies it relies on (such as distillation score sampling) that require long optimization and calculation times. Stability AI's previously released Stable Zero123 project along this line and its integration attempts in threestudio have made some progress but still failed to fully address these issues.

TripoSR open source enables researchers, developers and creative workers around the world to access the most advanced 3D generative AI models, enabling companies of all types to use 3D content to create more complex products and services and explore new creations in the 3D industry possibilities and promote a more active and competitive market.

# This chart shows the relationship between 3D performance F-SCORE (the higher the better) and the reasoning time (the lower the better).

3D content generation technology has experienced steady development in the fields of computer graphics and computer vision in recent years. In the past year or so, especially with the emergence of large-scale public 3D datasets and the advancement of powerful generative models in the field of 2D images and videos, 3D generative technology has achieved tremendous and rapid progress, attracting widespread attention from the industry. In this context, although technologies based on score distillation sampling (SDS) such as DreamFusion (proposed by the Google Research team) have made breakthroughs in generating 3D models from multiple perspectives, they still face long generation times and difficulty in practical applications. Fine-grained control over constraints such as generated models.

In contrast, generation technology solutions based on large-scale 3D data sets and large-scale scalable model architectures, such as the TripoSR released this time, demonstrate the ability to efficiently train on different 3D data sets, and their generation The 3D model process requires only fast forward inference and allows for easy and fine control over the 3D model results during generation. The emergence of this type of technology not only opens up a new path for the rapid development of 3D generation technology, but also provides new possibilities for wider applications in the industry.

^{#                           Image and data source: TripoSR: Fast 3D Object Reconstruction from a Single Image}

It is worth noting that Stability AI is jointly open sourced with Tripo AI this time. As a cutting-edge research team in the field of 3D content generation, VAST AI Research, the research institution behind Tripo AI, has been committed to open source community contributions since its inception, and has successively open sourced the codes and weights of high-quality research work such as Wonder3D, CSD, and TGS.

Tripo is a universal 3D generative model launched by VAST since December 2023 (www.tripo3d.ai). It can generate a 3D mesh model from text or pictures in 8 seconds, and refine it in 5 minutes. The quality of the generated model is close to the manual level in terms of geometry and material.

According to the blog of VAST AI Research, the rapid development of AI in the field of 3D generation requires a "universal approach" that breaks away from reliance on human experience and uses larger data, more scalable models and sufficient Harness powerful computing power to “learn.” This “universal approach” should include the unification of training data for multiple modalities, the unification of control conditions for multiple modalities, and a generative model infrastructure common to multiple modalities.

To achieve this goal, VAST believes that work needs to be done from three directions: representation, model and data. Among them, the choice of "representation" is crucial. It is necessary to find a 3D representation that is both flexible and computationally friendly, while ensuring compatibility with existing graphics pipelines. In addition, exploring “3D tokenizers” is also a promising direction, converting 3D representations into forms similar to language tokens, which can help apply existing understanding and generation models to the 3D field.

At the “model” level, VAST’s research aims to make full use of the prior knowledge, design guidelines and training experience of large models in other modalities to enhance the model’s learning ability for 3D data. The challenges at the “data” level cannot be ignored. The scarcity of high-quality, native, and diverse 3D data set resources limits the final performance and generalization capabilities of the model.

TripoSR allows us to see the potential of generative AI models in the 3D direction, and we look forward to more new explorations in the field of 3D generation in 2024.

^{Reference link:}

^{https://stability.ai/news/triposr-3d-generation?utm_source =x&utm_medium=website&utm_campaign=blog}

The above is the detailed content of 0.5 seconds, no GPU required, Stability AI and Chinese team VAST open source single image to generate 3D model TripoSR. For more information, please follow other related articles on the PHP Chinese website!