Prompt offensive and defensive battle! Columbia University proposed BPE word-making method, which can bypass the review mechanism. DALL-E 2 has been tricked

​What is the most valuable thing in 2022? prompt!

Text-guided image generation (text-guided image generation) model, such as DALL-E 2, has been popular among netizens since it became popular.

But if you want the model to generate clear and usable target images, you must master the correct "spell", that is, the prompt must be carefully designed before it can be used. Some people even set up a website to sell prompts

If the prompt is an evil spell, the generated pictures may be "suspected of violations."

Although DALL-E 2 has set up various mechanisms to prevent the model from being abused when it was released, such as deleting violent, hateful or inappropriate images from the training data; using technical means to prevent super-generated faces. Realistic photos, especially of public figures.

During the generation phase, DALL-E 2 also sets a prompt filter that does not allow user-entered prompt words to contain violent, adult or political content.

But recently, researchers at Columbia University discovered that some seemingly gibberish words can be added to the prompt, making it impossible for the filter to recognize the meaning of the word, but the AI ​​system can eventually return meaningful generated images.

Paper link: https://arxiv.org/pdf/2208.04135.pdf

The author proposes two methods of constructing prompts. The first one is called It is called macaronic prompting, where the original meaning of the word macaronic refers to the mixing of words from multiple languages ​​to generate new vocabulary. For example, in Pakistan, mixed words of Urdu and English are very common.

The training corpus of DALL-E 2 is usually data collected from the Internet. The process of establishing conceptual connections between text and images will more or less involve multi-language learning, so that the trained model has The ability to recognize concepts in multiple languages ​​simultaneously.

So you can use multi-language combinations to form new words, bypass the prompt filter designed by humans, and achieve the purpose of fighting against attacks.

For example, the word "birds" is Vögel in German, uccelli in Italian, oiseaux in French, and pájaros in Spanish. The CLIP model uses the byte pair encoding (BPE) algorithm to input prompt sentences After word segmentation, it can be split into multiple subwords.

After rearranging the subwords into new words, such as uccoisegeljaros, DALL-E 2 can still generate images of birds, but humans cannot understand the word at all. meaning.

Even if the boundaries of subword are not strictly adhered to, for example, if replaced with voiscellpajaraux and oisvogajaro, the model can still generate bird images.

In addition to birds, researchers found that the method of combining multiple languages ​​can achieve good results in different image domains, and the image generation results show a very high consistency.

The generation of relevant images from the animal kingdom to landscapes, vehicles, scenes, and emotions is a breeze.

Although different text-guided image generation models have different architectures, training data, and word segmentation methods, in principle, macaronic hints can be applied to any multilingual data The same effect can also be found in trained models, such as the DALL-E mini model.

It is worth noting that despite the similar names, DALL-E 2 and DALL-E mini are quite different. They have different architectures (DALL-E mini does not use a diffusion model), are trained on different data sets, and use different tokenizers (DALL-E mini uses the BART tokenizer, which may behave differently than the CLIP tokenizer split words).

Despite these differences, macaronic hints can still work on both models, and the principles behind them need to be further studied.

But not all macaronic cues transfer appropriately between different models. For example, while farpapmaripterling produced a butterfly image for DALL-E 2 as expected, it produced a mushroom in DALL-E mini. image.

The researchers speculate that perhaps larger models trained on larger datasets are more susceptible to macaronic cues because they Stronger associative relationships are learned between subword units and visual concepts in different languages.

This may explain why some macaronic tips that produce the expected results in DALL-E 2 don't work in DALL-E mini, but there are few examples to the contrary.

This trend may not be good news, as large-scale models may be more vulnerable to adversarial attacks using macaronic hints.

In addition to using a single compound word as a prompt, compound words can also be embedded into English syntax to form sentences, and the effect of generating images is similar to the original words.

# And another advantage of compound words is that they can be combined to produce more specific and complex scenes. While complex macaronic cues need to conform to the syntactic structure of English, making the generated results easier to interpret than cues using synthetic strings, the information conveyed to the model is still relatively vague.

For most people, without prior exposure to macaronic cues and knowledge of the language used for hybridization, it can be difficult to guess what kind of scenario would result from the prompt An eidelucertlagarzard eating a maripofarterling .

Furthermore, such complex prompts will not trigger blacklist-based content filters, despite the fact that they use ordinary English words, as long as the censored concepts are sufficiently "encrypted" using macaronic methods .

macaronic tip It is not necessary to combine subwords in multiple languages. Combining them within a single language can also produce a valid visual concept, but people familiar with English may guess the intended effect of the string, such as It is easy to guess that the word happy is a compound word of happy and cheerful.

The second method is called Evocative Prompting. Unlike macaronic, evocative does not need to trigger visual association from existing word combinations, but from specific fields. The statistical significance of certain letter combinations is "evoked" to create a new word.

Referring to the Binomial Nomenclature in biological classification, you can create a new "pseudo-Latin word" based on the "genus name" and "species epithet", and DALL-E can create a new "pseudo-Latin word" based on the corresponding Topics generate corresponding species.

# New drug pictures can also be generated according to the naming rules of drugs.

Evocative cues can also be applied to associations between specific features of a language and visual features related to the place and culture of the corresponding language. For example, based on the name of the building, the model can infer which country's style it is. For example, the scene generated by Woldenbüchel looks like a German or Austrian village; Valtorigiano looks like an ancient Italian town; Beaussoncour looks like a historical town in France.

However, they are not necessarily all buildings. For example, the last image generated with DALL-E mini is a 17th-century French portrait, not a location in France, but The connection with French culture has been preserved.

Evocative hints can also be combined with lexical hybridization to gain more control over the specific features of the output.

Introducing chunks of English words into the pseudo-Latin nomenclature will cause DALL-E 2 to generate images of animals with specific attributes. For example, the prompt word scariosus ferocianensis combines scary (scary) and ferocious (ferocious) with the pseudo-Latin terms. Combined, they can produce images of traditionally fearsome "reptiles" such as scorpions.

cutiosus adorablensis combines cute and adorable with pseudo-Latin terms to generate images of cute mammals in the traditional sense; watosus swimensis combines water and swimming (swimming) combined with pseudo-Latin affixes can produce images of aquatic animals; flyosus wingensis combines fly (fly) and winged (winged) with pseudo-Latin affixes to produce images of flying insects.

In principle, the vocabulary generated by the macaronic method can provide a simple and seemingly reliable method to bypass the prompt filter. People with ulterior motives can use it to generate harmful, offensive, and illegal words. or other sensitive content, including violent, hateful, racist, sexist or pornographic images, as well as images that may infringe intellectual property rights or depict real individuals.

While companies that provide image generation services have made extensive efforts to prevent the generation of this type of output in accordance with their content policies, macaronic prompts can still pose a significant threat to the security protocols of commercial image generation systems. .

The threat posed by evocative cues is less obvious, because it does not provide a very effective and reliable way to trigger specific visual associations for strings, and it is mostly limited to broad morphological features of words or languages. Vague associations with related concepts.

In general, macaronic tips are more operable than evocative tips, and keyword-based blacklist content filtering in this type of model is not enough to resist attacks.

Is DALL-E 2 going to go dark? ​

The above is the detailed content of Prompt offensive and defensive battle! Columbia University proposed BPE word-making method, which can bypass the review mechanism. DALL-E 2 has been tricked. For more information, please follow other related articles on the PHP Chinese website!