The once popular InstantID has a new way to play: stylized image generation, which has been open source

InstantID The original team launched InstantStyle, a new method of style migration.

Stylized image generation is often called style transfer, and its goal is to generate images that are consistent in style with the reference image and consistent with the original image content. This technology is based on diffusion methods (such as LoRA) for batch training of the same style data and cannot be migrated to new styles. Or based on inversion operations (such as StyleAlign), by restoring the style image to latent noise, the K and V obtained by forward propagation are used to replace the style image in generation. This method often degrades the generation style due to the inversion operation.

Recently, the original team of InstantID launched InstantStyle, a new method of style migration. Unlike Face ID, it is a general image style injection framework that uses two simple but very effective techniques to achieve effective separation of style and content from reference images. This method provides a simple but very effective technique to achieve effective separation of style and content from the reference image, thus achieving an effective fusion of style and content.

• Thesis: InstantStyle: Free Lunch towards Style-Preserving in Text-to-Image Generation

• Paper address: https://huggingface.co/papers/2404.02733

• Project homepage: https://instantstyle.github.io/

• Code link: https://github.com/InstantStyle/InstantStyle

• Demo address: https://huggingface.co/spaces/InstantX/InstantStyle

As a motivation for automatic introduction, (1) First of all, style is undefined and there is no clear standard to define it. It includes many elements, such as color, atmosphere, material , layout, etc., even with manual evaluation, it is difficult to reach a unified conclusion; (2) The previous inversion-based method will produce obvious style degradation, which is unacceptable for some styles, such as texture; (3) Image style Injection, the most critical issue is how to balance the intensity of style injection and the content leakage of the reference image.

In this regard, the author's team conducted a series of experimental analyses. They found that the problems of IP-Adapter were obviously exaggerated by many other methods. The author only manually adjusted the image The weight of feature injection solves the content leakage problem claimed in most papers. Nonetheless, in some cases it is still difficult for the IP-Adapter to find a suitable threshold to balance. Since the current Adapter-based methods generally use CLIP to extract image features, the author confirmed through the example of image retrieval that in the CLIP feature space, the features of images and text can be added and subtracted. The answer is obvious, why not display it before injecting it into the network? How to decouple content and style from image features by subtracting content information that may be leaked?

Finally, inspired by the B-LoRA method, the author carefully analyzed the effect of IP-Adapter injection at each layer, and was surprised to find that there are two independent layers that respond to style and spatial layout information respectively. To this end, the authors introduce the proposed method.

Method introduction

Based on the above observations and experiments, the author proposed the InstantStyle method, as shown in the figure. The core of this method contains two modules:

(1) Feature subtraction: Use the idle characteristics of CLIP to explicitly perform feature subtraction, remove the content information in the image features, and reduce the impact of the reference image content on the generated image. Compared with the underdetermination of style, content information is often easy to describe simply through text, so CLIP's text encoder can be used to extract content features for decoupling.

(2) Style layer injection only: Feature injection is only completed in a specific style layer, implicitly achieving decoupling of style and content. Near the mid block of UNet, the author discovered two specific layers that control style and spatial layout respectively, and found that in some styles, spatial layout may also be a type of style.

Overall, the idea of ​​InstantStyle is quite simple and easy to understand. With just a few lines of code, it alleviates the most troublesome content leakage problem in style migration.

Experimental results

The author shows the generation results of two strategies in the article. These two strategies are not limited to specific models and can be used separately and independently. Both are implemented. Excellent results.

Result of feature subtraction:

Only style layer injection:

Comparison with the current leading method:

Stylization based on the original image:

Community gameplay

InstantStyle Rich code implementations have been provided, and developers can find them directly through GitHub, including Wenshengtu, Tushengtu and Inpainting. Recently, it has also been used as a recommended stylization tool by the video generation project AnyV2V. For community users, InstantStyle also natively supports ComfyUI (the author of this node is also a co-author of InstantStyle), and users can quickly try it by updating the IP-Adapter node.

As the author of InstantID, how come we are not co-branded with InstantID? Compared to InstantID which only uses text to control the generated style, InstantStyle can undoubtedly make the style more diverse. The author team will officially support the face stylization function after the GitHub star reaches 1,000.

#The author also officially supports Huggingface Demo and can be tried online.

The above is the detailed content of The once popular InstantID has a new way to play: stylized image generation, which has been open source. For more information, please follow other related articles on the PHP Chinese website!