The first ultra-large-scale GAN model! The generation speed is 20+ times faster than Diffusion, and the image is produced in 0.13 seconds. It supports up to 16 million pixels.

With the release of DALL-E 2 by OpenAI, autoregressive and diffusion models have become the new standard for large-scale generative models overnight. Before that, Generative Adversarial Networks (GAN) had always been the mainstream choice and derived from StyleGAN. and other technologies.

Behind the explosion of AIGC, from a technical perspective, the architecture of the image generation model has undergone tremendous changes.

With the release of DALL-E 2 by OpenAI, autoregressive and diffusion models have become the new standard for large-scale generative models overnight. Before that, generative adversarial networks (GAN) had been They are all mainstream choices and have derived technologies such as StyleGAN.

The architectural shift from GAN to diffusion model also raises the question: Can the scale of GAN model be scaled up? , for example, can we further improve performance in large data sets like LAION?

Recently, researchers from Pohang University of Science and Technology (South Korea), Carnegie Mellon University, and Adobe Research proposed that increasing the capacity of the StyleGAN architecture will lead to instability. A new generative adversarial network architecture, GigaGAN, breaks the size limit of the model and shows that GAN can still be competent as a text-to-image synthesis model.

Paper link: https://arxiv.org/abs/2303.05511

Project link: https://mingukkang. github.io/GigaGAN/

GigaGAN has three major advantages.

1. It is faster during inference. Compared with Stable Diffusion-v1.5 with the same level of parameters, the generation speed at 512 resolution is shortened from 2.9 seconds to 0.13 seconds. .

2. High-resolution images can be synthesized, for example, a 16-megapixel image can be synthesized in 3.66 seconds.

3. Supports various latent space editing applications, such as latent interpolation, style mixing and vector arithmetic operations.

Has GAN reached its limit?

The recently released series of models, such as DALL-E 2, Imagen, Parti and Stable Diffusion, have ushered in a new era of image generation, reaching unprecedented levels of image quality and model flexibility.

The currently dominant paradigms "diffusion model" and "autoregressive model" both rely on the double-edged sword of iterative reasoning, because iterative methods can perform stable training with simple goals, but in reasoning Higher computational costs will be incurred in the process.

In contrast, the Generative Adversarial Network (GAN) only requires one forward pass to generate an image, so it is inherently more efficient.

Although GAN models dominated the "previous era" of generative modeling, due to instability in the training process, extending GAN requires careful adjustment of network structure and training considerations. , so although GANs are excellent at modeling single or multiple object categories, scaling to complex datasets (let alone open-world object generation) is still challenging.

So currently very large models, data and computing resources are mainly focused on diffusion and autoregressive models.

In this work, researchers mainly address the following questions:

Can GAN continue to scale and potentially benefit from these resources? Or is GAN already reaching its limit? What hinders further expansion of GANs? Can these obstacles be overcome?

Stable Training GAN

The researchers first conducted experiments with StyleGAN2 and observed that simply expanding the backbone network would lead to unstable training. After identifying several key issues Finally, techniques for stabilizing training while increasing model capacity are proposed.

First, effectively expand the capacity of the generator by retaining a bank of filters and taking a linear combination of specific samples.

Adapted several techniques commonly used in the context of diffusion models and confirmed that they can bring similar performance improvements to GANs, such as incorporating self-attention mechanisms (images only) and Cross-attention (image-text) intertwined with convolutional layers improves performance.

In addition, the researchers reintroduced multi-scale training and found a new scheme that can improve image-text alignment and generate low-frequency details in the output.

Multi-scale training allows GAN-based generators to use parameters in low-resolution blocks more efficiently, resulting in better image-text alignment and image quality.

Builder

The generator of GigaGAN consists of a text encoding branch, a style mapping network, and a multi-scale synthesis network. network), supplemented by stable attention and adaptive kernel selection.

In the text encoding branch, a pre-trained CLIP model and a learned attention layer T are first used to extract text embeddings, and then the embedding process is passed to the style mapping network M, Generate style vectors similar to StyleGAN w

##The synthesis network uses style encoding as modulation and text embedding as attention. Generate an image pyramid, and on this basis, introduce a sample adaptive kernel selection algorithm to achieve adaptive selection of convolution kernels based on input text conditions.

Discriminator

Similar to the generator, GigaGAN’s discriminator consists of two branches, which are used to process image and text conditions respectively.

The text branch handles something similar to the text branch of the generator; the image branch receives an image pyramid as input and makes independent predictions for each image scale.

Multiple additional loss functions are introduced into the formula to promote fast convergence.

Experimental Results Systematic, controlled evaluation on large-scale text-image synthesis tasks is difficult because most existing models are not publicly available is available, even if the training code is available, the cost of training a new model from scratch would be prohibitive.

The researchers chose to compare with Imagen, Latent Diffusion Models (LDM), Stable Diffusion, and Parti in their experiments, while acknowledging that there are significant differences in the training data set, number of iterations, batch size, and model size. There are considerable differences.

For quantitative evaluation indicators, Frechet Inception Distance (FID) is mainly used to measure the authenticity of the output distribution, and CLIP score is used to evaluate image-text alignment.

Five different experiments were conducted in this paper:

#1. Demonstrate the effectiveness of the proposed method by gradually incorporating each technical component;

2. Text-image synthesis results show that GigaGAN performs comparably to stable diffusion (SD-v1.5) FID while generating results hundreds of times faster than diffusion or autoregressive models;

3. Compare GigaGAN with A comparison of distillation-based diffusion models shows that GigaGAN can synthesize higher-quality images faster than distillation-based diffusion models;

##4. Verified the advantages of GigaGAN's upsampler over other upsamplers in conditional and unconditional super-resolution tasks;

5. The results show that large-scale GANs still enjoy the continuous and decomposed latent space operations of GANs, enabling new image editing modes.

After adjusting parameters, researchers achieved stability and reliability on large-scale data sets such as LAION2B-en. Training of extended billion-parameter GAN (GigaGAN).

And this method adopts a multi-stage approach, first generating at 64×64, and then upsampling to 512 ×512, both networks are modular and powerful enough to be used in a plug-and-play manner.

The results show that although the images of the diffusion model are never seen during training, the GAN upsampling network based on text conditions can be used as an efficient and high-quality upsampler for the basic diffusion model (such as DALL-E 2).

These results combined make GigaGAN far beyond previous GAN models, 36 times larger than StyleGAN2. 6 times larger than StyleGAN-XL and XMC-GAN.

Although GiGAN’s 1 billion parameters are still lower than the largest synthetic models released recently, such as Imagen (3B), DALL-E 2 (5.5B) and Parti (20B), but no mass saturation with respect to model size has yet been observed.

GigaGAN achieved a zero-shot FID of 9.09 on the COCO2014 data set, which is lower than the FID of DALL-E 2, Parti-750M and Stable Diffusion

Application scenario

Prompt interpolation

GigaGAN can smoothly interpolate between prompts. The four corners in the figure below are formed by the same Latent code generation, but with different text prompts.

Disentangled prompt mixing

GigaGAN Reserved A separated latent space enables combining the coarse style of one sample with the fine style of another sample, and GigaGAN can directly control the style through text hints.

##Coarse-to-fine sytle swapping

The GAN-based model architecture preserves a separate latent space, enabling the coarse style of one sample to be mixed with the fine style of another sample.

Reference:​

https://mingukkang.github.io/GigaGAN/

The above is the detailed content of The first ultra-large-scale GAN model! The generation speed is 20+ times faster than Diffusion, and the image is produced in 0.13 seconds. It supports up to 16 million pixels.. For more information, please follow other related articles on the PHP Chinese website!