GTA6 trailer has been played more than 1 billion times, and the AI ​​giant can quickly enter the role of GTA gangster

Have you seen the new GTA game trailer? It is said that this trailer has broken three Guinness World Records and has been viewed over 100 million times.

If I told you that the three AI giants could also play roles in the "Grand Theft Auto" game, would you still be able to tell them apart?

AI Big Three: Yann LeCun, Geoffrey Hinton and Yoshua Bengio.

This group photo was synthesized using Tencent’s FaceStudio AI model, showing a GTA-style effect. The uniqueness of this AI model lies in its excellent character recognition, which pushes the widely used "AI photo" technology to a higher level

In the rapid development of artificial intelligence technology Today, AI photo-taking has become a popular direction for the application of AI technology. In the field of AI image applications, AI photo products such as Miaoya Camera have demonstrated great potential and popularity. The Miaoya Camera has attracted a lot of attention on social media just a few weeks after its launch, and its rapid growth highlights the huge potential of this market. Despite this, many AI photo-taking products still have certain technical limitations. For example, users need to upload multiple photos with large differences and need to wait for a long time to obtain the composite effect, which undoubtedly affects the user experience.

In this wave of image innovation led by artificial intelligence, Tencent’s latest research result FaceStudio shows a further technological breakthrough. This research not only focuses on quickly synthesizing portraits, but also focuses more on retaining the identity information of the portrait to meet aesthetic needs while maintaining the uniqueness and recognition of the character. It not only inherits the core advantages of the open source algorithm StableDiffusion, but also makes innovative improvements in multiple key functions. The most eye-catching one is its ability to use hybrid guidance for image generation, especially in processing multi-person photos and stylized images.

The core technology of FaceStudio lies in its Stylized character image synthesis can be achieved without sacrificing personal identity characteristics. Traditional AI image synthesis technology often sacrifices the uniqueness and recognition of characters while pursuing visual beauty. However, through an advanced hybrid guidance mechanism, FaceStudio is able to simultaneously consider text cues, style images, and identity images when generating images, thereby achieving diverse style transfer while maintaining individual characteristics. This is not only a major breakthrough in technology, but also provides users with richer and more personalized image synthesis options.

In addition, FaceStudio’s unique multi-identity cross-attention mechanism makes it particularly good at processing images containing multiple people. Traditional methods often encounter problems in accurately distinguishing and maintaining the characteristics of each person when processing such images. But this mechanism of FaceStudio can accurately map the characteristic information of different identities to the corresponding parts of the image, which is excellent in maintaining the uniqueness of each character and the coordination of the overall style.

FaceStudio supports a variety of interesting face-related applications

• Paper address: https://arxiv.org/abs/2312.02663
• Homepage address: https://icoz69.github .io/facestudio/

Method Overview

Hybrid Bootstrap Design

One of FaceStudio’s core features is its hybrid boot design. The team used a unique approach that allows the model to receive both image and text cues simultaneously, thereby generating images with specific identity characteristics. The image prompt-based boot module contains two sub-modules:

• Image Guided Module: In this part, FaceStudio uses the CLIP visual encoder to process human images. These images are often stylized and contain rich visual information such as color, texture, and composition. The CLIP encoder is able to extract complex style features from these images.
• Identity recognition module: Parallel to the image guidance module, the Tencent team also designed an identity recognition module that uses the Arcface model to process individual facial images. Its main purpose is to extract key identity features such as facial structure, expression and other unique biometric information from facial images.

After extracting the visual features of the stylized image and the identity features of the facial image, the two sets of features are fused together. This step is accomplished through a linear layer that combines both features to create a comprehensive guidance feature. The advantage of this method is that it not only retains the identity of the character, but also incorporates specific style and content into the image generation process

FaceStudio not only has the image guidance function; Integrated text guidance function. This feature is achieved by using a pre-trained PriorTransformer model. The model is able to map CLIP textual features to corresponding CLIP visual features. Then, similar to the image prompt guidance module, these visual features are combined with the features of the identity recognition module to form a comprehensive guidance feature that can respond to text prompts. Finally, the two prompt features are weighted and fused to achieve hybrid guidance

The content that needs to be rewritten is: Facebook Studio architecture diagram

Multi-person image synthesis

In the FaceStudio framework developed by the Tencent team, there is a key The innovation is the "processing multi-person images" part. This section focuses on compositing portraits of multiple people in a single image to ensure that each person maintains their unique identity in the final image. Faced with an image containing multiple people, FaceStudio uses a special attention mechanism. This mechanism ensures that during the image synthesis process, the features of each character area only access the corresponding identity information. This means the model can precisely control each character’s identity, ensuring they appear correctly in the final image. In order to achieve this precise control, the Tencent team used a character instance segmentation model. The model is able to identify different people in the image and associate each person's region with its corresponding identity features. In this way, the model can ensure that the identity information of each character is correctly guided when synthesizing images

Comparison between FaceStudio and the baseline algorithm Effects on multi-person image generation

Training strategy

##The Tencent team designed a method for FaceStudio based on human images Reconstruction-targeted training strategy. With this approach, they use the original image with masked facial areas as input to a stylized human image, and simultaneously use the cropped face from the same image as input to the identity. In this way, the model can more accurately preserve the identity of the person when generating the guidance image. Different from existing generative model training methods, this method only relies on portraits as training data and does not require text annotations, which greatly reduces the dependence on annotated data. It can better adapt to various styles of portraits

Result display

FaceStudio is shown by evaluating face similarity and portrait generation time its unique advantages. Experimental results show that FaceStudio takes less than 4 seconds to generate a single portrait, while the popular algorithm DreamBooth based on optimization takes up to 6 minutes. At the same time, FaceStudio better retains portrait features and has better facial similarity. The experimental results are compared as follows:

The researchers compared FaceStudio with the current best portrait generation model algorithm. For comparison, the same images were used as samples. The comparison results show that FaceStudio achieves better or the same level of results on almost all samples. This further proves that FaceStudio has strong robustness and generalization performance. The specific comparison results are as follows:

In addition, a variety of unique face image generation applications were demonstrated in FaceStudo experiments, including identity mixing and text-image mixing guided generation

Identity Mixed Image Generation Experiment

Text and image mixed guided image generation experiment

The portrait samples generated by FaceStudio have a variety of styles

Summary

To sum up, the emergence of FaceStudio marks a major progress in the field of personalized image generation. It offers rich stylization and text-driven image generation options while maintaining character identity. This capability is not only of great value to the artistic creation and entertainment industries, but may also play an important role in areas such as advertising, digital media production, and personalized content creation. By precisely controlling identity and style in images, FaceStudio opens up a new path for the future development of image generation technology, heralding innovation and change in this field

The above is the detailed content of GTA6 trailer has been played more than 1 billion times, and the AI ​​giant can quickly enter the role of GTA gangster. For more information, please follow other related articles on the PHP Chinese website!