Open source AI video tool, you only need to be a director, built by HuggingFace engineers

To make videos with Clapper, you only need to be the director.

As soon as Sora came out, the video field seems to have entered the era of generative AI. But until today, we still have not used OpenAI’s official video generation tool, and people who can’t wait have begun to look for other methods.

In recent weeks, Clapper, an open source video editing tool, has attracted people’s attention.

Unlike the video generators provided by many technology companies, Clapper is an open source AI story visualization tool that launched as a prototype a year ago. It's not designed to replace traditional video editors, or modern AI editors that use 3D scenes as input.

Clapper’s philosophy is to bring together various generative AI technologies to allow anyone to create videos using AI through an interactive, iterative and intuitive process. No external tools, filmmaking or AI engineering skills required. In Clapper, you don’t need to directly edit sequences of video and audio files, but instead iterate on your story based on AI Agents by adjusting high-level, abstract concepts such as characters, locations, weather, time periods, styles, and more.

Clapper author Julian Bilcke is an AI front-end engineer at HuggingFace. He said that in order to continue working in this direction, he is also developing a director mode: the goal is to allow users to play videos in full screen, sit comfortably in the director's chair (or sofa), shout commands to the Agent, and let the AI ​​make movies.

In recent days, Julian Bilcke has launched new features such as using large models to convert arbitrary text into timelines. The popularity of Clapper has also increased, and it already has more than 1,100 stars on GitHub.

• GitHub link: https://github.com/jbilcke-hf/clapper
• HuggingFace link: https://huggingface.co/spaces/jbilcke-hf/clapper/tree/main
• Trial URL: https://clapper.app/

How to use

Since it is an open source tool, the main thing we look at is of course whether it is easy to use.

Do you still remember the experience of AI master Karpathy creating AI short videos? In order to turn the first three sentences of "Pride and Prejudice" into an animated version, it took this top expert a full hour. Although there are only three sentences and three scenes, this workflow is far more complicated than three sentences. He first used Claude to generate a series of image prompt words based on the original text, then input these prompt words into the Vincent graph model to generate the corresponding images, and then handed it over to the video model to make animations. The dubbing task was assigned to Elevenlabs, and finally he put it in Veed Studio. Put all the pieces together.

So, after Karpathy finished, he tweeted to complain, saying: "Entrepreneurs, the opportunity has come! The market is in urgent need of an AI tool that can integrate and simplify these processes."

Clapper is exactly one A one-stop platform integrating all these features.

Usually if you want to make a short video, you need to go through the following steps. First, you need a story and script, then draw storyboards based on the script, then shoot or find materials based on the storyboards, put them together in editing software, add animation effects and special effects, and then selectively add spoken word and background music Or sound effects. Therefore, the division of labor in the film and television production industry such as choreography, directing, photography, editing, post-production, and dubbing came into being.

At Clapper, video production follows another logic. Each track of it does not correspond to video or picture material like Premier, Cutting and other editing software, but corresponds to a specific type of work.

                                                                                                                                                                                               ^{的 Clapper's track}

^{In the matter of using AI for video, we are Party A. Clapper is like a crew made up of the best AI in the industry. Clapper has built-in a series of "top-notch" large models such as GPT-4o, Claude 3.5 (Sonnet), etc. It is like Party B's executive director, responsible for connecting your needs to the corresponding "AI director."}

As you can see from the picture above, the first track represents the storyboard and talks to Clapper’s built-in large model. It will call the Vincent diagram model through the API and let the AI ​​storyboard teacher generate the corresponding Pictures serve as the basis for video images. Er Through CLAPPER, you can access the above Wensheng map model. Take the samples given by Clapper as an example. The next track corresponds to the scene, narration, camera perspective, background music, and sound effects. You can ask ElevenLabs or Fal.ai to generate some wind sounds from ruins or explosions from gunfights for this Western wasteland story.

And Clapper also has a feature that may really take a big step towards the dream of "making movies by talking". We can directly import the script into Clapper and carefully create a character for your protagonist in the "Story" column.

Taking "The Wizard of Oz" as an example, we can not only add more personalized character descriptions to the characters, but also upload pictures to set the visual image of the heroine Dorothy. That means we can ask any actor in the world to play this role, even if you want to see an 18-year-old DiCaprio playing Dorothy, you can do it. The functions of Clapper are so detailed that you can adjust the age and timbre of the characters, the furnishings of each scene, what furniture is in Dorothy's room, and what the house in their adventure destination "Emerald City" looks like, all can be adjusted in Clapper. Adjustment.

Of course, you can also use AI to draw some atmosphere pictures first, which may further stimulate your inspiration and creativity.

However, although the function of Clapper has fully considered the needs of making videos, its effect is somewhat unsatisfactory. Not only are the movements of the characters in the picture a bit "ghostly", they do not conform to the laws of physical movement. The overall effect of the video is more like a moving PPT, lacking transitions and continuity between shots, and the soundtrack is also full of AI, sounds without melody, and has some noise.

It may take a long time for generative AI to change the video production process, but the emergence of Clapper may be able to provide major manufacturers that are still expanding AI functions for traditional video editing software. Developed a new implementation idea.

Reference content:

https://news.ycombinator.com/item?id=41221399

https://x.com/aigclink/status/1818 111874531205216

The above is the detailed content of Open source AI video tool, you only need to be a director, built by HuggingFace engineers. For more information, please follow other related articles on the PHP Chinese website!