Using small tricks to excavate the diffusion model, the generated images are almost replicas of the original training data, and privacy is about to be exposed.

Denoising diffusion models are an emerging class of generative neural networks that generate images from a training distribution through an iterative denoising process. This type of diffusion model produces higher quality samples and is easier to scale and control than previous methods such as GANs and VAEs. As a result, after rapid development, they can already produce high-resolution images, and there is great public interest in large models such as the DALL-E 2.

The beauty of generative diffusion models lies in their ability to synthesize new images that are, superficially, unlike anything in the training set. In fact, large-scale training efforts in the past have not found that overfitting will be a problem. Researchers in privacy-sensitive fields have even suggested that diffusion models can be used to protect privacy by generating synthetic examples to generate real images. This series of work is conducted under the assumption that the diffusion model does not memorize and regenerate training data. Doing so would violate all privacy guarantees and breed many problems with model generalization and digital forgery.

In this article, researchers from Google, DeepMind and other institutions have proven that the SOTA diffusion model can indeed remember and regenerate a single training example.

## Paper address: https://arxiv.org/pdf/2301.13188v1.pdf

First, the research proposes and implements a new definition of memory in image models. Then, the study designed a two-stage data extraction attack, using standard methods to generate images and label some images. The study applied the method to Stable Diffusion and Imagen, resulting in the extraction of more than 100 nearly identical copies of training images, which ranged from personally identifiable photos to trademarked logos (Figure 1).

In order to better understand how memory works and why, researchers trained hundreds of people on CIFAR10 Diffusion model to analyze the impact of model accuracy, hyperparameters, augmentation, and deduplication on privacy. Diffusion models are the least private form of image models evaluated in the study, leaking twice as much training data as GANs. Worse, research also finds that existing privacy-enhancing technologies fail to provide acceptable privacy-utility trade-offs. Overall, this paper highlights the tension that exists between increasingly powerful generative models and data privacy, and raises questions about how diffusion models work and how they can be deployed appropriately.

Why do you want to do this research?

There are two motivations behind understanding how diffusion models remember and regenerate training data.

The first is to understand the privacy risks. Regenerating diffusion models that scrape data from the Internet may pose similar privacy and copyright risks as language models. For example, it has been pointed out that memorizing and regenerating copyrighted text and source code present potential indicators of infringement. Likewise, copying an image created by a professional artist is called digital forgery, and there is a debate in the art world.

The second is to understand generalization. In addition to data privacy, understanding how and why diffusion models remember training data helps understand their ability to generalize. For example, a common question with large-scale generative models is whether their impressive results result from true generation or are the result of directly copying and remixing the training data. By studying memory, it is possible to provide a concrete empirical description of the rate at which generative models perform such data replication.

Extract data from SOTA diffusion model

Extract data from Stable Diffusion

Now Extract training data from Stable Diffusion, the largest and most popular open source diffusion model.

This extraction applies the method of previous work to the image, including two steps:

1. Diffusion model using standard sampling method And generate multiple examples using the known prompts from the previous section.

2. Perform inference to separate the new generation model from the memorized training model.

To evaluate the effectiveness of the intrusion, the study selected 350,000 most repeated examples from the training dataset and generated 500 candidate images for each prompt (generating 175 million in total image).

First, the study sorts all these generated images to determine which ones are generated by memorizing the training data. Each of these generated images is then compared to the training images under Definition 1 in the paper, and each image is annotated as extracted or not extracted. The study found that 94 images were extracted. To ensure that these images did not just fit some arbitrary definition, the study also manually annotated the first 1,000 generated images through visual analysis, either with or without memory. And another 13 (total 109 images) were found to be almost copies of the training examples, even though they did not meet the study L_2 norm definition. Figure 3 shows a subset of the extracted images, which are reproduced with near-perfect pixel accuracy.

The experiment also gives the calculation curve given an annotated ordered image set, Evaluate the number of extracted images versus the false positive rate of the intrusion. The intrusion is exceptionally accurate: out of 175 million generated images, 50 memory images with 0 false positives could be identified, and all memory images could be extracted with over 50% accuracy. Figure 4 contains precision-recall curves for both definitions of memory.

Extract data from image

Although Stable Diffusion is currently the best choice among publicly available diffusion models, some non-public models have achieved stronger performance using larger models and datasets. Previous research has found that larger models are more likely to remember training data, so this study looked at Imagen, a 2 billion parameter text-to-image diffusion model.

Surprisingly, research has found that hacking non-distributed images in Imagen is more effective than in Stable Diffusion. On Imagen, the study tried to extract 500 images with the highest out-of-distribution (OOD) score. Imagen memorizes and copies 3 of the images (which are unique in the training data set). In contrast, when the study applied the same method to Stable Diffusion, it failed to identify any memories even after trying to extract the 10,000 most outlier samples. Therefore, Imagen is less private than Stable Diffusion on both copied and non-copied images. This may be due to the fact that Imagen uses a larger model than Stable Diffusion and therefore remembers more images. Additionally, Imagen trains with more iterations on smaller datasets, which can also help improve memory levels.

The above is the detailed content of Using small tricks to excavate the diffusion model, the generated images are almost replicas of the original training data, and privacy is about to be exposed.. For more information, please follow other related articles on the PHP Chinese website!