A new breakthrough in video generation: PixelDance, easily presenting complex movements and cool special effects

Recently, in addition to the widely watched large-scale language models that continue to occupy the headlines, video generation technology has also continued to make major breakthroughs, and many companies have successively released new models

First of all, Runway, as one of the earliest leaders to explore the field of video generation, has upgraded its Gen-2 model, bringing film-level high definition that is eye-catching. At the same time, the consistency of video generation has also been significantly improved

However, this improvement in consistency seems to come at the expense of video dynamics. It can be seen from the official promotional video of Gen-2 that although multiple short clips are assembled, the dynamics of each clip are relatively weak, making it difficult to capture clear actions and movements of characters, animals or objects.

Recently, Meta also released the video generation model Emu Video. As can be seen from Emu Video's official examples, the dynamics of its videos are significantly improved compared to Gen-2, but they are still limited to simpler actions.

Stability.ai, the company that developed the classic Vincent graph model Stable Diffusion, also recently released the open source video generation model Stable Video Diffusion ( SVD), which has attracted a lot of attention and discussion in the open source community. The effect of SVD is comparable to that of Gen-2. From the test sample, it can be seen that the video generated by SVD is relatively lacking in dynamics.

In the SVD paper, it was pointed out that the current video generated by SVD has insufficient dynamics

The above examples show that generating videos with high consistency and rich dynamics, so that the video content can truly move, is currently the biggest challenge in the field of video generation.

In this regard, the latest research results PixelDance have taken a critical step, and the dynamics of the generated results are significantly better than other existing models. , attracted the attention of the industry.

The PixelDance news forwarded by @_akhaliq, a well-known AI blogger on Twitter, has received nearly 80,000 views

On the official website (https://makepixelsdance.github.io), PixelDance gives Two different video generation modes.

There are two modes to choose from, the first is Basic Mode. In this mode, users only need to provide a guiding image and text description, and PixelDance can generate a highly consistent and dynamic video. The guidance picture can be a real photo, or it can be generated by the existing text generation model

From the results displayed, real style, animation style, two-dimensional style, PixelDance can handle all magical styles, including character movements, facial expressions, camera perspective control, and special effects movements. PixelDance can also complete them well. All I can say is tql!

The second is the advanced Magic Mode, which gives users more room to use their imagination and creativity. In this mode, users need to provide two guidance pictures and text descriptions, which can better generate more difficult video content. The website shows various cool special effects shots made with magic mode.

In addition, the official website also shows a 3-minute short story film entirely produced using PixelDance

What is very shocking is that using PixelDance can make the music as the user expected. A story, crafting every scene and corresponding action. Whether it is a real scene (such as Egypt, the Great Wall, etc.) or an imaginary scene (such as an alien planet), PixelDance can generate videos with rich details and action, even various special effects shots.

The protagonist Mr. Polar Bear’s black top hat and red bow tie are well maintained in various scenes. Now generating long videos is no longer a matter of simply piecing together irrelevant short video clips!

To achieve such outstanding video generation effects, it does not rely on complex data sets and large-scale model training. PixelDance only uses 1.5B on the public WebVid-10M data set. Small and large models achieve the above effects.

Paper address: https://arxiv.org/abs/2311.10982

Rewrite the content , without changing the original meaning, rewritten into Chinese: Please visit the following website to get the demo: https://makepixelsdance.github.io

In the corresponding paper "Make Pixels Dance: High-Dynamic Video Generation", the author pointed out the reason why video generation is difficult to achieve good results: compared with image generation, video generation has the characteristics of a significantly larger feature space and significantly greater action diversity. This makes it difficult for existing video generation methods to learn effective time-domain action information. Although the generated videos have high picture quality, their dynamics are very limited.

In response to the above problems, PixelDance proposes a video generation method based on text guidance and first and last frame picture guidance, so that the model can more fully pay attention to and learn the dynamic information of the video.

In video generation, the first frame of the picture provides the framework and material for the entire video content. At the same time, longer videos can be generated by using the last frame of the previous video clip as a guide for the first frame of the next clip. The video text description describes the specific content of the video action. The last frame picture guidance provides end status information for the video generation process. The author proposes an adaptation method that enables the model to accept relatively rough images as guidance, which allows users to use basic image editing tools to obtain end frame image guidance

official website The information shows that it is still actively iterating the model effect, and a model that everyone can try will be released in the next 2-3 months. At present, the author also provides a way to support everyone in sending samples they want to test. Currently, some user test samples have been released on the official website:

It seems that with PixelDance, everyone can become a "million-dollar special effects master" as long as they have a wild imagination!

The above is the detailed content of A new breakthrough in video generation: PixelDance, easily presenting complex movements and cool special effects. For more information, please follow other related articles on the PHP Chinese website!