Home Technology peripherals AI Alibaba's 'trajectory controllable version of Sora” bids farewell to 'drawing cards” and makes video generation more consistent with physical laws

Alibaba's 'trajectory controllable version of Sora” bids farewell to 'drawing cards” and makes video generation more consistent with physical laws

Aug 05, 2024 pm 03:10 PM
industry Alibaba Tora

You specify the route, and Tora will generate a video of the corresponding trajectory.


Currently, diffusion models are capable of generating diverse and high-quality images or videos. Previously, video diffusion models used the U-Net architecture, which mainly focused on synthesizing videos of limited duration (usually around two seconds), with fixed constraints on resolution and aspect ratio.

The emergence of Sora breaks this limitation. It uses the Diffusion Transformer (DiT) architecture, which is not only good at producing high-quality videos of 10 to 60 seconds, but also because it generates different resolutions, various aspect ratios, and stand out by their ability to obey actual physical laws.

It can be said that Sora is the most favorable demonstration of DiT architecture, however, Transformer-based diffusion model has not been fully explored in effectively generating controllable action videos.

In response to this problem, researchers from Alibaba proposed Tora, the first trajectory-oriented DiT architecture, which integrates text, visual and trajectory conditions simultaneously to generate videos.

Alibabas trajectory controllable version of Sora” bids farewell to drawing cards” and makes video generation more consistent with physical laws

Tora is designed to work seamlessly with DiT’s scalability, allowing precise control of video content with different durations, aspect ratios and resolutions. Extensive experiments have proven that Tora excels at achieving high motion fidelity while also simulating the motion of the physical world in detail.

Alibabas trajectory controllable version of Sora” bids farewell to drawing cards” and makes video generation more consistent with physical laws

  • Paper address: https://arxiv.org/pdf/2407.21705
  • Paper homepage: https://ali-videoai.github.io/tora_video/
  • Paper title : Tora: Trajectory-oriented Diffusion Transformer for Video Generation

An old-fashioned wooden sailing boat glides smoothly on the mist-shrouded river along the prescribed route, surrounded by dense green forests.

Alibabas trajectory controllable version of Sora” bids farewell to drawing cards” and makes video generation more consistent with physical laws

A crucian fish swims gracefully across the red rock surface of Mars, the fish's trajectory is to the left, and the Mars' trajectory is to the right.

Alibabas trajectory controllable version of Sora” bids farewell to drawing cards” and makes video generation more consistent with physical laws

The hot air balloon rises into the night sky along different trajectories, one along the prescribed oblique line and the other along the curved trajectory.

Alibabas trajectory controllable version of Sora” bids farewell to drawing cards” and makes video generation more consistent with physical laws

Two cute kittens walking side by side on the quiet golden beach.

Alibabas trajectory controllable version of Sora” bids farewell to drawing cards” and makes video generation more consistent with physical laws

Bubbles float gently along the tracks among the blooming wildflowers.

Alibabas trajectory controllable version of Sora” bids farewell to drawing cards” and makes video generation more consistent with physical laws

The maple leaves tremble on the clear lake, reflecting the autumn forest.

Alibabas trajectory controllable version of Sora” bids farewell to drawing cards” and makes video generation more consistent with physical laws

The waterfalls in the mountains pour down, and the themes and backgrounds can be moved along different routes.

Alibabas trajectory controllable version of Sora” bids farewell to drawing cards” and makes video generation more consistent with physical laws

In the comparison between Tora and other methods, it can be seen that the video generated by Tora is smoother, follows the trajectory more, and the object does not have deformation problems and has better fidelity.

Alibabas trajectory controllable version of Sora” bids farewell to drawing cards” and makes video generation more consistent with physical laws

Alibabas trajectory controllable version of Sora” bids farewell to drawing cards” and makes video generation more consistent with physical laws

方法の紹介

Tora は、軌道抽出器 (TE、Trajectory Extractor)、時空間 DiT (Spatial-Temporal DiT)、およびモーション ガイダンス フュージョン ( MGF)、モーションガイドフューザー)。 TE は 3D ビデオ圧縮ネットワークを使用して、任意の軌跡を階層的な時空間モーション パッチにエンコードします。 MGF は、モーション パッチを DiT ブロックに統合して、軌跡をたどる一貫したビデオを生成します。図 3 は Tora のワークフローの概要を示しています。
Alibabas trajectory controllable version of Sora” bids farewell to drawing cards” and makes video generation more consistent with physical laws
時空 DiT (ST-DiT)

ST-DiT アーキテクチャには、空間 DiT ブロック (S-DiT-B) と時間 DiT ブロック (T-DiT-) の 2 つの異なるブロック タイプが含まれています。 B) 交互に配置されます。 S-DiT-B は 2 つのアテンション レイヤーで構成され、それぞれが空間セルフ アテンション (SSA) とクロス アテンションを順番に実行し、その後に隣接する T-DiT-B ピースを接続するポイントワイズ フィードフォワード レイヤーが続きます。 T-DiT-B は、SSA を一時的セルフアテンション (TSA) に置き換えることによってのみこのアーキテクチャを変更し、アーキテクチャの一貫性を維持します。各ブロックでは、正規化後の入力はスキップ接続を介してブロックの出力に接続されます。可変長シーケンスを処理する機能を活用することで、ST-DiT のノイズ除去は可変長のビデオを処理できます。

Trajectory Extractor

Trajectory は、生成されたビデオのモーションを制御するためのよりユーザーフレンドリーな方法であることが証明されています。ただし、DiT モデルでは、ビデオ オートエンコーダーと、ビデオをビデオ パッチに変換するパッチ処理プロセスが採用されています。ここでは、各パッチは複数のフレームにわたってエクスポートされるため、フレーム間オフセットを直接取得することは不適切です。この問題を解決するために、本論文で提案する TE は、軌跡をビデオ パッチと同じ潜在空間に位置するモーション パッチに変換します。

モーション ガイド付きフュージョン

DiT ベースのビデオ生成と軌跡を組み合わせるために、このペーパーでは、モーション パッチを各 ST-DiT ブロックに注入する 3 つのフュージョン アーキテクチャのバリアントを検討します。これらの設計を図 4 に示します。

Alibabas trajectory controllable version of Sora” bids farewell to drawing cards” and makes video generation more consistent with physical laws

実験結果

実装の詳細に関して、研究者は OpenSora v1.2 の重みに基づいて Tora をトレーニングしました。トレーニング ビデオの解像度は 144p ~ 720p です。異なる解像度と反復ごとのフレーム数で FLOP をトレーニングするために必要なメモリのバランスをとるために、バッチ サイズを 1 から 25 まで調整しました。

トレーニング インフラストラクチャに関しては、研究者は 4 台の NVIDIA A100 と学習率 2 × 10^−5 の Adam オプティマイザーを使用しました。

研究者らは、Tora を一般的な運動指導ビデオ生成方法と比較しました。評価では、16、64、および 128 フレームの 3 つの設定が使用され、すべて解像度 512 × 512 でした。

結果を以下の表 1 に示します。U-Net メソッドで一般的に使用される 16 フレーム設定では、MotionCtrl と DragNUWA は提供された軌道とよりよく位置合わせできますが、それでも Tora よりは弱いです。フレーム数が増加すると、一部のフレームで U-Net メソッドが大幅に偏り、位置ずれエラーの伝播により、後続のシーケンスで変形、モーション ブラー、またはオブジェクトの消失が発生する可能性があります。

Alibabas trajectory controllable version of Sora” bids farewell to drawing cards” and makes video generation more consistent with physical laws

対照的に、Tora は、Transformer の統合されたスケーリング機能のおかげで、フレーム レートの変更に対して高い堅牢性を示します。 Tora によって生成される動きはよりスムーズで、物理世界とより一致しています。 128 フレームのテスト設定での評価では、Tora の軌道精度は他の方法の 3 ~ 5 倍であり、優れたモーション制御能力を示しています。

以下の図 5 では、研究者がさまざまな解像度と期間で軌道エラーを分析しています。結果は、時間の経過とともに明らかな軌道誤差が現れる U-Net とは異なり、Tora の軌道誤差は時間の経過とともに徐々に増加することを示しています。これは、DiT モデルで時間が増加するにつれてビデオ品質が低下することと一致しています。 Tora は効果的な軌道制御を長期間維持します。
Alibabas trajectory controllable version of Sora” bids farewell to drawing cards” and makes video generation more consistent with physical laws
下の図 6 は、Tora と主流のモーション制御手法の比較分析を示しています。2 人が一緒に移動するシーンでは、どの手法も比較的正確なモーション軌跡を生成できます。ただし、Tora の視覚的な品質は、より長いシーケンス フレームを使用することで優れており、よりスムーズなモーション軌跡とよりリアルな背景レンダリングを実現できます。

Tora によって生成された自転車のシーンでは、人間の脚がリアルなペダリングの動きを示しているのに対し、DragNUWA の脚はほぼ水平に浮遊し、物理的現実を侵害していることがわかります。さらに、DragNUWA と MotionCtrl の両方で、ビデオの最後に深刻なモーション ブラーが発生します。

ランタンが生成される別のシーンでは、DragNUWA は提供された軌道が上昇と下降を続けるため、激しい変形を示します。 MotionCtrl の軌道は比較的正確でしたが、結果のビデオは 2 つのランタンの説明と一致しませんでした。 Tora は軌跡を厳密に追従するだけでなく、オブジェクトの変形を最小限に抑え、より忠実なアクション表現を保証します。

Alibabas trajectory controllable version of Sora” bids farewell to drawing cards” and makes video generation more consistent with physical laws

技術的な詳細と実験結果については、元の論文を参照してください。

The above is the detailed content of Alibaba's 'trajectory controllable version of Sora” bids farewell to 'drawing cards” and makes video generation more consistent with physical laws. For more information, please follow other related articles on the PHP Chinese website!

Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

Video Face Swap

Video Face Swap

Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Tools

Notepad++7.3.1

Notepad++7.3.1

Easy-to-use and free code editor

SublimeText3 Chinese version

SublimeText3 Chinese version

Chinese version, very easy to use

Zend Studio 13.0.1

Zend Studio 13.0.1

Powerful PHP integrated development environment

Dreamweaver CS6

Dreamweaver CS6

Visual web development tools

SublimeText3 Mac version

SublimeText3 Mac version

God-level code editing software (SublimeText3)

Hot Topics

Java Tutorial
1664
14
PHP Tutorial
1266
29
C# Tutorial
1239
24
DeepMind robot plays table tennis, and its forehand and backhand slip into the air, completely defeating human beginners DeepMind robot plays table tennis, and its forehand and backhand slip into the air, completely defeating human beginners Aug 09, 2024 pm 04:01 PM

But maybe he can’t defeat the old man in the park? The Paris Olympic Games are in full swing, and table tennis has attracted much attention. At the same time, robots have also made new breakthroughs in playing table tennis. Just now, DeepMind proposed the first learning robot agent that can reach the level of human amateur players in competitive table tennis. Paper address: https://arxiv.org/pdf/2408.03906 How good is the DeepMind robot at playing table tennis? Probably on par with human amateur players: both forehand and backhand: the opponent uses a variety of playing styles, and the robot can also withstand: receiving serves with different spins: However, the intensity of the game does not seem to be as intense as the old man in the park. For robots, table tennis

The first mechanical claw! Yuanluobao appeared at the 2024 World Robot Conference and released the first chess robot that can enter the home The first mechanical claw! Yuanluobao appeared at the 2024 World Robot Conference and released the first chess robot that can enter the home Aug 21, 2024 pm 07:33 PM

On August 21, the 2024 World Robot Conference was grandly held in Beijing. SenseTime's home robot brand "Yuanluobot SenseRobot" has unveiled its entire family of products, and recently released the Yuanluobot AI chess-playing robot - Chess Professional Edition (hereinafter referred to as "Yuanluobot SenseRobot"), becoming the world's first A chess robot for the home. As the third chess-playing robot product of Yuanluobo, the new Guoxiang robot has undergone a large number of special technical upgrades and innovations in AI and engineering machinery. For the first time, it has realized the ability to pick up three-dimensional chess pieces through mechanical claws on a home robot, and perform human-machine Functions such as chess playing, everyone playing chess, notation review, etc.

Claude has become lazy too! Netizen: Learn to give yourself a holiday Claude has become lazy too! Netizen: Learn to give yourself a holiday Sep 02, 2024 pm 01:56 PM

The start of school is about to begin, and it’s not just the students who are about to start the new semester who should take care of themselves, but also the large AI models. Some time ago, Reddit was filled with netizens complaining that Claude was getting lazy. "Its level has dropped a lot, it often pauses, and even the output becomes very short. In the first week of release, it could translate a full 4-page document at once, but now it can't even output half a page!" https:// www.reddit.com/r/ClaudeAI/comments/1by8rw8/something_just_feels_wrong_with_claude_in_the/ in a post titled "Totally disappointed with Claude", full of

At the World Robot Conference, this domestic robot carrying 'the hope of future elderly care' was surrounded At the World Robot Conference, this domestic robot carrying 'the hope of future elderly care' was surrounded Aug 22, 2024 pm 10:35 PM

At the World Robot Conference being held in Beijing, the display of humanoid robots has become the absolute focus of the scene. At the Stardust Intelligent booth, the AI ​​robot assistant S1 performed three major performances of dulcimer, martial arts, and calligraphy in one exhibition area, capable of both literary and martial arts. , attracted a large number of professional audiences and media. The elegant playing on the elastic strings allows the S1 to demonstrate fine operation and absolute control with speed, strength and precision. CCTV News conducted a special report on the imitation learning and intelligent control behind "Calligraphy". Company founder Lai Jie explained that behind the silky movements, the hardware side pursues the best force control and the most human-like body indicators (speed, load) etc.), but on the AI ​​side, the real movement data of people is collected, allowing the robot to become stronger when it encounters a strong situation and learn to evolve quickly. And agile

ACL 2024 Awards Announced: One of the Best Papers on Oracle Deciphering by HuaTech, GloVe Time Test Award ACL 2024 Awards Announced: One of the Best Papers on Oracle Deciphering by HuaTech, GloVe Time Test Award Aug 15, 2024 pm 04:37 PM

At this ACL conference, contributors have gained a lot. The six-day ACL2024 is being held in Bangkok, Thailand. ACL is the top international conference in the field of computational linguistics and natural language processing. It is organized by the International Association for Computational Linguistics and is held annually. ACL has always ranked first in academic influence in the field of NLP, and it is also a CCF-A recommended conference. This year's ACL conference is the 62nd and has received more than 400 cutting-edge works in the field of NLP. Yesterday afternoon, the conference announced the best paper and other awards. This time, there are 7 Best Paper Awards (two unpublished), 1 Best Theme Paper Award, and 35 Outstanding Paper Awards. The conference also awarded 3 Resource Paper Awards (ResourceAward) and Social Impact Award (

Li Feifei's team proposed ReKep to give robots spatial intelligence and integrate GPT-4o Li Feifei's team proposed ReKep to give robots spatial intelligence and integrate GPT-4o Sep 03, 2024 pm 05:18 PM

Deep integration of vision and robot learning. When two robot hands work together smoothly to fold clothes, pour tea, and pack shoes, coupled with the 1X humanoid robot NEO that has been making headlines recently, you may have a feeling: we seem to be entering the age of robots. In fact, these silky movements are the product of advanced robotic technology + exquisite frame design + multi-modal large models. We know that useful robots often require complex and exquisite interactions with the environment, and the environment can be represented as constraints in the spatial and temporal domains. For example, if you want a robot to pour tea, the robot first needs to grasp the handle of the teapot and keep it upright without spilling the tea, then move it smoothly until the mouth of the pot is aligned with the mouth of the cup, and then tilt the teapot at a certain angle. . this

Hongmeng Smart Travel S9 and full-scenario new product launch conference, a number of blockbuster new products were released together Hongmeng Smart Travel S9 and full-scenario new product launch conference, a number of blockbuster new products were released together Aug 08, 2024 am 07:02 AM

This afternoon, Hongmeng Zhixing officially welcomed new brands and new cars. On August 6, Huawei held the Hongmeng Smart Xingxing S9 and Huawei full-scenario new product launch conference, bringing the panoramic smart flagship sedan Xiangjie S9, the new M7Pro and Huawei novaFlip, MatePad Pro 12.2 inches, the new MatePad Air, Huawei Bisheng With many new all-scenario smart products including the laser printer X1 series, FreeBuds6i, WATCHFIT3 and smart screen S5Pro, from smart travel, smart office to smart wear, Huawei continues to build a full-scenario smart ecosystem to bring consumers a smart experience of the Internet of Everything. Hongmeng Zhixing: In-depth empowerment to promote the upgrading of the smart car industry Huawei joins hands with Chinese automotive industry partners to provide

Distributed Artificial Intelligence Conference DAI 2024 Call for Papers: Agent Day, Richard Sutton, the father of reinforcement learning, will attend! Yan Shuicheng, Sergey Levine and DeepMind scientists will give keynote speeches Distributed Artificial Intelligence Conference DAI 2024 Call for Papers: Agent Day, Richard Sutton, the father of reinforcement learning, will attend! Yan Shuicheng, Sergey Levine and DeepMind scientists will give keynote speeches Aug 22, 2024 pm 08:02 PM

Conference Introduction With the rapid development of science and technology, artificial intelligence has become an important force in promoting social progress. In this era, we are fortunate to witness and participate in the innovation and application of Distributed Artificial Intelligence (DAI). Distributed artificial intelligence is an important branch of the field of artificial intelligence, which has attracted more and more attention in recent years. Agents based on large language models (LLM) have suddenly emerged. By combining the powerful language understanding and generation capabilities of large models, they have shown great potential in natural language interaction, knowledge reasoning, task planning, etc. AIAgent is taking over the big language model and has become a hot topic in the current AI circle. Au

See all articles