AI dimensionality reduction attacks human painters, Vincentian graphs are introduced into ControlNet, and depth and edge information are fully reusable

With the emergence of large text-image models, generating an attractive image has become very simple. All the user needs to do is to enter a simple prompt with the movement of their fingers. After obtaining the image through a series of operations, we will inevitably have several questions: Can the image generated based on prompt meet our requirements? What kind of architecture should we build to handle the various requirements raised by users? Can large models still maintain the advantages and capabilities gained from billions of images in specific tasks?

In order to answer these questions, researchers from Stanford conducted a large number of investigations on various image processing applications and came to the following three findings:

First of all, the available data in a specific field is actually less than the data for training general models. This is mainly reflected in the fact that for example, the largest data set on a specific problem (such as gesture understanding, etc.) is usually less than 100k, which is smaller than large-scale, The multimodal text image dataset LAION 5B is 5 × 10^4 orders of magnitude smaller. This requires the neural network to be robust to avoid model overfitting and to have good generalization when targeting specific problems.

Secondly, when using data-driven processing of image tasks, large computing clusters are not always available. This is where fast training methods become important, methods that can optimize large models for specific tasks within acceptable time and memory space. Furthermore, fine-tuning, transfer learning and other operations may be required in subsequent processing.

Finally, various problems encountered in the image processing process will have different forms of definition. When solving these problems, although the image diffusion algorithm can be adjusted in a "procedural" way, for example, constraining the denoising process, editing multi-head attention activation, etc., these hand-crafted rules are basically dictated by human instructions ,Considering some specific tasks such as depth-image, pose-person, etc., these problems essentially require the interpretation of raw inputs into object-level or scene-level understanding, which makes hand-crafted procedural approaches less feasible. Therefore, to provide solutions in multiple tasks, end-to-end learning is essential.

Based on the above findings, this paper proposes an end-to-end neural network architecture ControlNet, which can control the diffusion model (such as Stable Diffusion) by adding additional conditions, thereby improving the graph. Picture effect, and can generate full-color pictures from line drawings, generate pictures with the same depth structure, and optimize the generation of hands through key points of the hands.

Paper address: https://arxiv.org/pdf/2302.05543.pdf

Project Address: https://github.com/lllyasviel/ControlNet

Effect display

So what is the effect of ControlNet?

Canny Edge Detection: By extracting line drawing from the original image, an image of the same composition can be generated.

Depth detection: By extracting the depth information in the original image, a map with the same depth structure can be generated.

ControlNet with semantic segmentation:

Using learning-based The deep Hough transform detects straight lines from Places2 and then uses BLIP to generate subtitles.

HED edge detection illustration.

Illustration of human posture recognition.

Method Introduction

ControlNet is a neural network architecture that enhances pre-trained image diffusion models with task-specific conditions. Let's first look at the basic structure of ControlNet.

ControlNet manipulates the input conditions of neural network blocks, thereby further controlling the overall behavior of the entire neural network. Here "network block" refers to a group of neural layers that are put together as a common unit for building neural networks, such as resnet block, multi-head attention block, and Transformer block.

Taking 2D features as an example, given a feature map x ϵ R^h×w×c, where {h, w, c} are the height, width and number of channels respectively. A neural network block F (・; Θ) with a set of parameters Θ transforms x into another feature map y as shown in equation (1) below.

This process is shown in Figure 2-(a) below.

Neural network blocks are connected by a unique convolution layer called "zero convolution", which is the weight 1×1 convolutional layer with zero initialization and bias. The researcher represents the zero convolution operation as Z (・;・) and uses two parameter instances {Θ_z1, Θ_z2} to form the ControlNet structure, as shown in the following formula (2).

where y_c becomes the output of the neural network block, as shown in Figure 2-(b) below.

ControlNet in image diffusion model

##The researcher took Stable Diffusion as an example to introduce how to use ControlNet control Large-scale diffusion models with task-specific conditions. Stable Diffusion is a large-scale text-to-image diffusion model trained on billions of images, essentially a U-net consisting of an encoder, intermediate blocks, and a residual-connected decoder.

As shown in Figure 3 below, the researcher uses ControlNet to control each layer of U-net. Note that the way ControlNet is connected here is computationally efficient: since the original weights are locked, the gradient calculation on the original encoder does not require training. And because half of the gradient calculations on the original model are reduced, training can be accelerated and GPU memory can be saved. Training a Stable Diffusion model using ControlNet only requires approximately 23% more GPU memory and 34% more time per training iteration (tested on a single Nvidia A100 PCIE 40G).

#Specifically, the researchers used ControlNet to create 12 encoding blocks and 1 Stable Diffusion intermediate block that are trainable copy. The 12 coding blocks come in 4 resolutions, 64×64, 32×32, 16×16, and 8×8, with 3 blocks in each resolution. The output is added to the U-net with 12 residual connections and 1 intermediate block. Since Stable Diffusion is a typical U-net structure, it is likely that this ControlNet architecture can be used in other diffusion models.

Training and Boosting Training

Given an image z_0, the diffusion algorithm progressively adds noise to the image and generates noise Image z_t, t is the number of times noise is added. When t is large enough, the image approximates pure noise. Given a set of conditions including time step t, text prompts c_t, and task-specific conditions c_f, the image diffusion algorithm learns a network ϵ_θ to predict the noise added to a noisy image z_t, as shown in Equation (10) below.

During the training process, the researchers randomly replaced 50% of the text prompts c_t with empty strings, which is beneficial to ControlNet's ability to identify semantic content from the input condition map.

In addition, the researchers also discussed several strategies to improve the training of ControlNets, especially when computing devices are very limited (such as laptops) or very powerful (such as with large-scale GPUs available). computing cluster).

Please refer to the original paper for more technical details.

The above is the detailed content of AI dimensionality reduction attacks human painters, Vincentian graphs are introduced into ControlNet, and depth and edge information are fully reusable. For more information, please follow other related articles on the PHP Chinese website!