ControlNet star count exceeds 10,000! In 2023, will AI painting go crazy?

From a horse-riding astronaut to a three-dimensional young lady, AI painting seems to have made revolutionary progress in less than a year.

This "horse-riding astronaut" is drawn by the Vincentian graph model DALL・E 2 launched by OpenAI in April 2022 . Its predecessor, DALL・E, demonstrated to people in 2021 the ability to generate images directly from text, breaking the dimensional wall between natural language and vision. On this basis, DALL・2 goes a step further and allows people to edit the original image, such as adding a corgi to the picture. This seemingly simple operation actually reflects the improvement in the controllability of the AI ​​painting model.

However, in terms of influence, the most popular Vincent diagram model in 2022 is not DALL・E 2, but another model with similar functions to it-Stable Diffusion. Like DALL・E 2, Stable Diffusion also allows creators to edit the generated images, but the advantage is that the model is open source and can run on consumer-grade GPUs. Therefore, after its release in August 2022, Stable Diffusion quickly became popular and became the most popular Vincent diagram model in just a few months.

During this period, people are also further exploring various methods of controlling this type of model, such as Runway, one of the teams behind Stable Diffusion, released An image Erase and Replace tool that can modify any part of the image. All the user needs to do is erase the area and write a natural language description, and the program does the rest.

Researchers from Google and Boston University have proposed a "personalized" text-to-image diffusion model ​​DreamBooth​, users only need to provide 3 to 5 sample sentences, and AI can customize photo-realistic images.

In addition, a research team from UC Berkeley also proposed a new method to edit images based on human instructions ​​InstructPix2Pix​​​, this model combines GPT-3 and Stable Diffusion. Given an input image and a text description that tells the model what to do, the model can follow the description instructions to edit the image. For example, to replace the sunflowers in the painting with roses, you only need to directly say "replace sunflowers with roses" directly to the model.

Entering 2023, a model called ControlNet has pushed the flexibility of this type of control to its peak.

The core idea of ​​ControlNet is to add some additional conditions to the text description to control the diffusion model (such as Stable Diffusion), thereby better controlling the character pose, depth, and screen of the generated image. structure and other information.

The additional conditions here are input in the form of an image. The model can perform Canny edge detection, depth detection, semantic segmentation, Hough transform line detection, and overall nesting based on this input image. edge detection (HED), human pose recognition, etc., and then retain this information in the generated image. Using this model, we can directly convert line drawings or graffiti into full-color images, generate images with the same depth structure, etc., and optimize the generation of character hands through hand key points.

This model has set off a huge wave in the field of AI painting, and the number of GitHub stars of related projects has exceeded 10,000.

## Project link: https://github.com/lllyasviel/ControlNet

Although many people currently only use it to generate two-dimensional and three-dimensional ladies, its wider uses have also been gradually discovered, such as house design, photography, film and television production, and advertising design. wait. In these scenarios, ControlNet is used together with some previous tools, such as LoRA to handle large model fine-tuning problems, video-to-animation conversion tool EbSynth, etc. The combined application of these tools accelerates the integration of AI painting models into the production process.

# of Image source: https://creativetechnologydigest.substack.com/p/controlling-artistic-chaos-with-controlnet (complete tutorial included)

​Use ControlNet and Houdini tools to generate 3D models. Image source: https://www.reddit.com/r/StableDiffusion/comments/115eax6/im_working_on_api_for_the_a1111_controlnet/​

​

Use Dreambooth and ControlNet to change the lighting of 2D images, which can be used for post-production of photos and videos. Image source: https://www.reddit.com/r/StableDiffusion/comments/1175id9/when_i_say_mindblowing_i_mean_it_new_experiments/​

​

Use ControlNet and EbSynth to convert animation into real person. Although the results are not great yet, it has shown the potential of adapting anime into live-action without the need for actors to appear. Image source https://www.reddit.com/r/StableDiffusion/comments/117ewr9/anime_to_live_action_with_controlnet_ebsynth_not/​

#################someone The designer used ControlNet to generate the famous brand's "new logo". Image source: https://twitter.com/fofrAI/status/1628882166900744194################# In addition to surprises, the progress of these technologies has also surprised practitioners in painting and other fields. Falling into anxiety and anger. The worry is that AI may take away your job. Angryly, many of the images generated by AI are plagiarisms and imitations of current painters, and the intellectual property rights of painters have been infringed. ####################### Source: https://www.zhihu.com/question/583294094########## #####With these issues unresolved, AI painting has become a sharp problem in the circle of painters. Many people believe that everyone should boycott AI painting and jointly defend their rights. Therefore, when news spread that a well-known artist was suspected of using AI painting to contribute to a game studio, other artists were completely outraged. ############ At the same time, gamers were also angered. Because there are still some limitations in AI painting at present, such as not being able to handle hand details well (you can see this if you look carefully at the girl in the first picture of this article), it cannot meet the requirements of players for exquisite visual effects, characters with personality and creativity, etc. , many players felt "fooled". Therefore, the above-mentioned game studio can only issue an emergency statement saying that it "will not use AI painting in its products." ######

But how long will this situation last? When the level of AI painting reaches a level that is difficult to distinguish with the naked eye, how do you know whether the game you are playing is from the artist or the AI, or a "team" composed of the two?

## Source: https://m .weibo.cn/2268335814/4870844515358190

Perhaps in a few months, AI painting tools will become as essential to painters’ daily work as Copilot used by programmers Few tools. Of course, this has also virtually raised the bar for this industry, just like other industries that have been "invaded" by AI. How to maintain one's competitiveness in such a wave may be a question that everyone should think about.

The above is the detailed content of ControlNet star count exceeds 10,000! In 2023, will AI painting go crazy?. For more information, please follow other related articles on the PHP Chinese website!