Home Technology peripherals AI Whether it's real or AI video, 'Mosca” can reconstruct and restore 4D dynamic renderable scenes.

Whether it's real or AI video, 'Mosca” can reconstruct and restore 4D dynamic renderable scenes.

Jun 22, 2024 am 07:09 AM
industry neural information processing system

Whether its real or AI video, Mosca” can reconstruct and restore 4D dynamic renderable scenes.
The AIxiv column is a column where academic and technical content is published on this site. In the past few years, the AIxiv column of this site has received more than 2,000 reports, covering top laboratories from major universities and companies around the world, effectively promoting academic exchanges and dissemination. If you have excellent work that you want to share, please feel free to contribute or contact us for reporting. Submission email: liyazhou@jiqizhixin.com; zhaoyunfeng@jiqizhixin.com

Lei Jiahui, PhD student in the Department of Computer Science, University of Pennsylvania (2020 - present), his supervisor is Professor Kostas Daniilidis, his current main research direction is four-dimensional dynamic scenes Geometric modeling representations and algorithms with applications. He has published 7 articles as the first or co-author in top computer vision and machine learning conferences (CVPR, NeurIPS, ICML, ECCV). His previous undergraduate degree (2016-2020) graduated from the Control Department of Zhejiang University and the mixed class of Zhu Kezhen College with the first place in his major.

Reconstructing renderable dynamic scenes from arbitrary monocular video is a holy grail in computer vision research. In this paper, a team of researchers from the University of Pennsylvania and Stanford University attempts to take a small step toward this goal.

There are massive monocular videos on the Internet, which contain a large amount of information about the physical world. However, 3D vision still lacks effective means to extract 3D dynamic information from these videos to support future 3D large model modeling and Understanding the dynamic physical world. Although important, this inverse problem is extremely challenging.

  • First, real-shot 2D videos often lack multi-view information, so multi-view geometry cannot be used for 3D reconstruction. In many cases, it is even impossible to solve the camera pose and internal parameters through existing software (such as COLMAP).

  • Second, the degree of freedom of dynamic scenes is extremely high, and the four-dimensional representation of its deformation and long-term information fusion is still immature, making this difficult inverse problem more complicated.

This article proposes a novel neural information processing system - MoSca, which only needs to provide a series of video frame pictures without any additional information, and can generate videos, movie and TV series clips from SORA , reconstruct renderable dynamic scenes from monocular in-the-wild videos from , Internet videos and public datasets.

Whether its real or AI video, Mosca” can reconstruct and restore 4D dynamic renderable scenes.

  • ArXiv address: https://arxiv.org/pdf/2405.17421
  • Code (will be open source in the near future): www.github.com/JiahuiLei/MoSca
  • Project website: www. cis.upenn.edu/~leijh/projects/mosca/
  • Video (bilibili): www.bilibili.com/video/BV1uU411o75P/?vd_source=177d8c87be5e898a43e8937dbef9bed4

The following are two video demos. Whether its real or AI video, Mosca” can reconstruct and restore 4D dynamic renderable scenes.Whether its real or AI video, Mosca” can reconstruct and restore 4D dynamic renderable scenes.

Method Overview

In order to overcome the above difficulties, Mosca first utilized the strong prior knowledge stored in computer vision foundation models to reduce the problem solution space.

Whether its real or AI video, Mosca” can reconstruct and restore 4D dynamic renderable scenes.

Specifically, Mosca uses the monocular metric-depth estimation (mono metric-depth) model UniDepth, video any point long-term tracking (track any point) model CoTracker, and optical flow estimation (optical flow) ) The epipolar geometric error (epipolar error) calculated by the model RAFT, and the semantic features provided by the pre-trained semantic model DINO-v2. See Chapter 3.1 of the paper for details. We observe that most real-world dynamic deformations are compact and sparse in nature, and their complexity is often much lower than that of real geometric structures. For example, the motion of a hard object can be represented by rotation and translation, and the motion of a person can be roughly approximated by the rotation and translation of multiple joints.

Based on this observation, this article proposes a

novel compact dynamic scene representation - 4D Motion Scaffold

, which upgrades the above cornerstone model output from two dimensions to four dimensions and fuses it, while also integrating physics Inspired deformation regularization (ARAP).

The four-dimensional motion scaffold is a graph. Each node of the graph is a string of rigid body motion (SE (3)) trajectories. The topology of the graph is the nearest neighbor edge constructed by considering the distance between the rigid body motion trajectory curves globally. Deformations at any point in space can be represented by smoothing the rigid body trajectories of nodes on the interpolated graph in space-time using dual-quaternions. This representation greatly simplifies the motion parameters that need to be solved. (See Chapter 3.2 of the paper for details). Whether its real or AI video, Mosca” can reconstruct and restore 4D dynamic renderable scenes.

Another huge advantage of the four-dimensional motion scaffold is that it can be directly initialized by monocular depth and video two-dimensional point tracking, and then the unknown occlusion point position and local coordinate system direction can be solved through efficient physical regular term optimization. Please refer to Chapter 3.3 of the paper for details.

With the four-dimensional motion scaffolding, any point at any time can be deformed to any target time, which makes it possible to globally fuse observation information. Specifically, each frame of the video can be back-projected into three-dimensional space using the estimated depth map and initialized with a three-dimensional Gaussian (3DGS). These Gaussians are "bound" to the four-dimensional motion scaffolding and can travel freely at any time. If you want to render the scene at a certain moment, you only need to transfer the Gaussians of all other global moments to the current moment through the four-dimensional scaffolding for fusion. This dynamic scene representation based on four-dimensional motion scaffolding and Gaussian can be efficiently optimized by the Gaussian renderer (see Chapter 3.4 of the paper for details).

Finally, it is worth mentioning that Mosca is a system that does not require internal and external parameters of the camera. By using the epipolar geometry error output by the above-mentioned cornerstone model to determine the static background mask, and using the depth and point tracking output by the cornerstone model, Mosca can efficiently optimize the reprojection error and solve the global bundle adjustment to directly output Camera internal parameters and pose, and continue to optimize the camera through subsequent rendering (see Chapter 3.5 of the paper for details).

Experimental results

Mosca can reconstruct dynamic scenes in DAVIS dataset videos. It is worth noting that Mosca flexibly supports multiple Gaussian-based renderers. In addition to the native 3DGS renderer, this article also tested the recent Gaussian surface reconstruction renderer GOF (Gaussian Opacity Field). As shown in the rightmost train in the picture, GOF can render higher quality normal and depth.

Whether its real or AI video, Mosca” can reconstruct and restore 4D dynamic renderable scenes.

Moska achieves significant improvements on the challenging IPhone DyCheck dataset, while also comparing other methods on the widely comparable Nvidia dataset.

Whether its real or AI video, Mosca” can reconstruct and restore 4D dynamic renderable scenes.

Whether its real or AI video, Mosca” can reconstruct and restore 4D dynamic renderable scenes.

Whether its real or AI video, Mosca” can reconstruct and restore 4D dynamic renderable scenes.

Whether its real or AI video, Mosca” can reconstruct and restore 4D dynamic renderable scenes.

The above is the detailed content of Whether it's real or AI video, 'Mosca” can reconstruct and restore 4D dynamic renderable scenes.. For more information, please follow other related articles on the PHP Chinese website!

Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

Video Face Swap

Video Face Swap

Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Tools

Notepad++7.3.1

Notepad++7.3.1

Easy-to-use and free code editor

SublimeText3 Chinese version

SublimeText3 Chinese version

Chinese version, very easy to use

Zend Studio 13.0.1

Zend Studio 13.0.1

Powerful PHP integrated development environment

Dreamweaver CS6

Dreamweaver CS6

Visual web development tools

SublimeText3 Mac version

SublimeText3 Mac version

God-level code editing software (SublimeText3)

Hot Topics

Java Tutorial
1664
14
PHP Tutorial
1266
29
C# Tutorial
1239
24
DeepMind robot plays table tennis, and its forehand and backhand slip into the air, completely defeating human beginners DeepMind robot plays table tennis, and its forehand and backhand slip into the air, completely defeating human beginners Aug 09, 2024 pm 04:01 PM

But maybe he can’t defeat the old man in the park? The Paris Olympic Games are in full swing, and table tennis has attracted much attention. At the same time, robots have also made new breakthroughs in playing table tennis. Just now, DeepMind proposed the first learning robot agent that can reach the level of human amateur players in competitive table tennis. Paper address: https://arxiv.org/pdf/2408.03906 How good is the DeepMind robot at playing table tennis? Probably on par with human amateur players: both forehand and backhand: the opponent uses a variety of playing styles, and the robot can also withstand: receiving serves with different spins: However, the intensity of the game does not seem to be as intense as the old man in the park. For robots, table tennis

The first mechanical claw! Yuanluobao appeared at the 2024 World Robot Conference and released the first chess robot that can enter the home The first mechanical claw! Yuanluobao appeared at the 2024 World Robot Conference and released the first chess robot that can enter the home Aug 21, 2024 pm 07:33 PM

On August 21, the 2024 World Robot Conference was grandly held in Beijing. SenseTime's home robot brand "Yuanluobot SenseRobot" has unveiled its entire family of products, and recently released the Yuanluobot AI chess-playing robot - Chess Professional Edition (hereinafter referred to as "Yuanluobot SenseRobot"), becoming the world's first A chess robot for the home. As the third chess-playing robot product of Yuanluobo, the new Guoxiang robot has undergone a large number of special technical upgrades and innovations in AI and engineering machinery. For the first time, it has realized the ability to pick up three-dimensional chess pieces through mechanical claws on a home robot, and perform human-machine Functions such as chess playing, everyone playing chess, notation review, etc.

Claude has become lazy too! Netizen: Learn to give yourself a holiday Claude has become lazy too! Netizen: Learn to give yourself a holiday Sep 02, 2024 pm 01:56 PM

The start of school is about to begin, and it’s not just the students who are about to start the new semester who should take care of themselves, but also the large AI models. Some time ago, Reddit was filled with netizens complaining that Claude was getting lazy. "Its level has dropped a lot, it often pauses, and even the output becomes very short. In the first week of release, it could translate a full 4-page document at once, but now it can't even output half a page!" https:// www.reddit.com/r/ClaudeAI/comments/1by8rw8/something_just_feels_wrong_with_claude_in_the/ in a post titled "Totally disappointed with Claude", full of

At the World Robot Conference, this domestic robot carrying 'the hope of future elderly care' was surrounded At the World Robot Conference, this domestic robot carrying 'the hope of future elderly care' was surrounded Aug 22, 2024 pm 10:35 PM

At the World Robot Conference being held in Beijing, the display of humanoid robots has become the absolute focus of the scene. At the Stardust Intelligent booth, the AI ​​robot assistant S1 performed three major performances of dulcimer, martial arts, and calligraphy in one exhibition area, capable of both literary and martial arts. , attracted a large number of professional audiences and media. The elegant playing on the elastic strings allows the S1 to demonstrate fine operation and absolute control with speed, strength and precision. CCTV News conducted a special report on the imitation learning and intelligent control behind "Calligraphy". Company founder Lai Jie explained that behind the silky movements, the hardware side pursues the best force control and the most human-like body indicators (speed, load) etc.), but on the AI ​​side, the real movement data of people is collected, allowing the robot to become stronger when it encounters a strong situation and learn to evolve quickly. And agile

ACL 2024 Awards Announced: One of the Best Papers on Oracle Deciphering by HuaTech, GloVe Time Test Award ACL 2024 Awards Announced: One of the Best Papers on Oracle Deciphering by HuaTech, GloVe Time Test Award Aug 15, 2024 pm 04:37 PM

At this ACL conference, contributors have gained a lot. The six-day ACL2024 is being held in Bangkok, Thailand. ACL is the top international conference in the field of computational linguistics and natural language processing. It is organized by the International Association for Computational Linguistics and is held annually. ACL has always ranked first in academic influence in the field of NLP, and it is also a CCF-A recommended conference. This year's ACL conference is the 62nd and has received more than 400 cutting-edge works in the field of NLP. Yesterday afternoon, the conference announced the best paper and other awards. This time, there are 7 Best Paper Awards (two unpublished), 1 Best Theme Paper Award, and 35 Outstanding Paper Awards. The conference also awarded 3 Resource Paper Awards (ResourceAward) and Social Impact Award (

Li Feifei's team proposed ReKep to give robots spatial intelligence and integrate GPT-4o Li Feifei's team proposed ReKep to give robots spatial intelligence and integrate GPT-4o Sep 03, 2024 pm 05:18 PM

Deep integration of vision and robot learning. When two robot hands work together smoothly to fold clothes, pour tea, and pack shoes, coupled with the 1X humanoid robot NEO that has been making headlines recently, you may have a feeling: we seem to be entering the age of robots. In fact, these silky movements are the product of advanced robotic technology + exquisite frame design + multi-modal large models. We know that useful robots often require complex and exquisite interactions with the environment, and the environment can be represented as constraints in the spatial and temporal domains. For example, if you want a robot to pour tea, the robot first needs to grasp the handle of the teapot and keep it upright without spilling the tea, then move it smoothly until the mouth of the pot is aligned with the mouth of the cup, and then tilt the teapot at a certain angle. . this

Hongmeng Smart Travel S9 and full-scenario new product launch conference, a number of blockbuster new products were released together Hongmeng Smart Travel S9 and full-scenario new product launch conference, a number of blockbuster new products were released together Aug 08, 2024 am 07:02 AM

This afternoon, Hongmeng Zhixing officially welcomed new brands and new cars. On August 6, Huawei held the Hongmeng Smart Xingxing S9 and Huawei full-scenario new product launch conference, bringing the panoramic smart flagship sedan Xiangjie S9, the new M7Pro and Huawei novaFlip, MatePad Pro 12.2 inches, the new MatePad Air, Huawei Bisheng With many new all-scenario smart products including the laser printer X1 series, FreeBuds6i, WATCHFIT3 and smart screen S5Pro, from smart travel, smart office to smart wear, Huawei continues to build a full-scenario smart ecosystem to bring consumers a smart experience of the Internet of Everything. Hongmeng Zhixing: In-depth empowerment to promote the upgrading of the smart car industry Huawei joins hands with Chinese automotive industry partners to provide

Distributed Artificial Intelligence Conference DAI 2024 Call for Papers: Agent Day, Richard Sutton, the father of reinforcement learning, will attend! Yan Shuicheng, Sergey Levine and DeepMind scientists will give keynote speeches Distributed Artificial Intelligence Conference DAI 2024 Call for Papers: Agent Day, Richard Sutton, the father of reinforcement learning, will attend! Yan Shuicheng, Sergey Levine and DeepMind scientists will give keynote speeches Aug 22, 2024 pm 08:02 PM

Conference Introduction With the rapid development of science and technology, artificial intelligence has become an important force in promoting social progress. In this era, we are fortunate to witness and participate in the innovation and application of Distributed Artificial Intelligence (DAI). Distributed artificial intelligence is an important branch of the field of artificial intelligence, which has attracted more and more attention in recent years. Agents based on large language models (LLM) have suddenly emerged. By combining the powerful language understanding and generation capabilities of large models, they have shown great potential in natural language interaction, knowledge reasoning, task planning, etc. AIAgent is taking over the big language model and has become a hot topic in the current AI circle. Au

See all articles