Give me a picture and generate a 30-second video!
Has AI advanced again?
#And it is the kind that generates a continuous 30-second video from one picture.
emm....Isn’t the quality a bit too blurry
Be aware that this is only generated from a single image (the first frame) and does not have any geometry information displayed.
This is a general framework for image modeling and vision tasks based on probabilistic frame prediction recently proposed by DeepMind - Transframer.
#To put it simply, Transframer is used to predict the probability of any frame.
These frames can be conditioned on one or more annotated context frames, which can be previous video frames, time stamps, or camera-tagged view scenes. .
Transframer architecture
Let’s first take a look at how this magical Transframer architecture works.
The paper address is posted below. Interested children can take a look ~ https://arxiv.org /abs/2203.09494
To estimate the predicted distribution on a target image, we need an expression generation model that can produce diverse, high-quality output.
Although the results of DC Transformer on a single image domain can meet the needs, it is not in the multi-image text set {(In,an)}n we need as condition.
# Therefore, we extended DC Transformer to enable image and annotation conditional prediction.
We replace DC Transformer with a Vision-Transformer style encoder that operates on a single DCT image using a multi-frame U-Net architecture, using For processing a set of annotated frames and partially hidden target DCT images.
#Let’s take a look at how the Transframer architecture works.
(a) Transframer takes as input the DCT images (a1 and a2) as well as the partially hidden target DCT image (aT) and additional annotations, which are processed by the multi-frame U-Net encoder. Next, the U-Net output is passed to the DC-Transformer decoder through cross attention, which automatically regresses to generate a DCT Token sequence (green letters) corresponding to the hidden part of the target image. (b) The multi-frame U-Net block consists of NF-Net convolution blocks and multi-frame self-attention blocks, which exchange information between input frames and Transformer-style residual MLP.
Let’s take a look at Multi-Frame U-Net that processes image input.
The input to U-Net is a sequence consisting of N DCT frames and partially hidden target DCT frames, annotated information is associated with each input frame Provided in vector form.
The core component of U-Net is a computational block that first applies a shared NF-ResNet convolution block to each input frame and then applies A Transformer-style self-attention block to aggregate information across frames. (Figure 2 b)
NF-ResNet block consists of grouped convolutions and squeeze and excitation layers, aiming to improve the performance of TPU.
Below, Figure (a) compares the sparsity of absolute and residual DCT representations of RoboNet (128x128) and KITTI videos.
Since RoboNet consists of static videos with only a few moving elements, the sparsity of the residual frame representation increases significantly.
And KITTI videos often have a moving camera, resulting in differences almost everywhere in consecutive frames.
#But in this case, the benefits of small sparsity are also weakened.
Multi-vision task powerhouse
Through a series of tests on data sets and tasks, the results show Transframer can be applied to a wide range of tasks.
This includes video modeling, new view synthesis, semantic segmentation, object recognition, depth estimation, optical flow prediction, etc.
Video Modeling
By Transframer Predict the next frame given a sequence of input video frames.
The researchers trained Transframer's performance in video generation on the KITTI and RoboNet data sets respectively.
For KITTI, given 5 context frames and 25 sample frames, the results show that the Transframer model performs better on all metrics The performance of LPIPS and FVD has been improved, among which the improvements of LPIPS and FVD are the most obvious.
On RoboNet, the researchers were given 2 context frames and 10 sampling frames, 64x64 and 128x128 respectively. The training was carried out at a certain resolution and finally achieved very good results.
View composition
In terms of view synthesis, we work by providing camera views as context and target annotations as described in Table 1 (line 3), and uniformly sampling multiple context views up to a specified maximum.
By providing 1-2 contextual views, the model Transframer is evaluated on the ShapeNet benchmark, significantly outperforming PixelNeRF and SRN.
Additionally after evaluation on the dataset Objectron, it can be seen that when given a single input view, the model produces Coherent output, but missing some features like crossed chair legs.
When 1 context view is given, the view synthesized at 128×128 resolution is as follows:
When two more context views are given, the view synthesized at 128×128 resolution is as follows:
Multiple Vision Tasks
Different computer vision tasks often use complex architecture and loss function to handle.
Here, the researchers jointly trained the Transframer model on 8 different tasks and datasets using the same loss function.
The 8 tasks are: optical flow prediction of a single image, object classification, detection and segmentation, semantic segmentation (on 2 data sets), Future frame prediction and depth estimation.
The results show that Transframer learns to generate different samples in completely different tasks. In some tasks, such as Cityscapes, the model produces output of good quality.
# However, the quality of model outputs on tasks such as future frame prediction and bounding box detection is variable, suggesting that modeling in this setting is more challenging. sex.
The above is the detailed content of Give me a picture and generate a 30-second video!. For more information, please follow other related articles on the PHP Chinese website!

Hot AI Tools

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Undress AI Tool
Undress images for free

Clothoff.io
AI clothes remover

AI Hentai Generator
Generate AI Hentai for free.

Hot Article

Hot Tools

Notepad++7.3.1
Easy-to-use and free code editor

SublimeText3 Chinese version
Chinese version, very easy to use

Zend Studio 13.0.1
Powerful PHP integrated development environment

Dreamweaver CS6
Visual web development tools

SublimeText3 Mac version
God-level code editing software (SublimeText3)

Hot Topics

Evaluating the cost/performance of commercial support for a Java framework involves the following steps: Determine the required level of assurance and service level agreement (SLA) guarantees. The experience and expertise of the research support team. Consider additional services such as upgrades, troubleshooting, and performance optimization. Weigh business support costs against risk mitigation and increased efficiency.

The lightweight PHP framework improves application performance through small size and low resource consumption. Its features include: small size, fast startup, low memory usage, improved response speed and throughput, and reduced resource consumption. Practical case: SlimFramework creates REST API, only 500KB, high responsiveness and high throughput

Writing clear and comprehensive documentation is crucial for the Golang framework. Best practices include following an established documentation style, such as Google's Go Coding Style Guide. Use a clear organizational structure, including headings, subheadings, and lists, and provide navigation. Provides comprehensive and accurate information, including getting started guides, API references, and concepts. Use code examples to illustrate concepts and usage. Keep documentation updated, track changes and document new features. Provide support and community resources such as GitHub issues and forums. Create practical examples, such as API documentation.

Choose the best Go framework based on application scenarios: consider application type, language features, performance requirements, and ecosystem. Common Go frameworks: Gin (Web application), Echo (Web service), Fiber (high throughput), gorm (ORM), fasthttp (speed). Practical case: building REST API (Fiber) and interacting with the database (gorm). Choose a framework: choose fasthttp for key performance, Gin/Echo for flexible web applications, and gorm for database interaction.

The learning curve of a PHP framework depends on language proficiency, framework complexity, documentation quality, and community support. The learning curve of PHP frameworks is higher when compared to Python frameworks and lower when compared to Ruby frameworks. Compared to Java frameworks, PHP frameworks have a moderate learning curve but a shorter time to get started.

Foreword This model combines the SigLIP visual model and the Gemma language model. Both models are open components, making PaliGemma excellent at processing tasks that combine vision and language. The usage scenarios of PaliGemma include image subtitles, image tags and visual question answering. These application scenarios take advantage of PaliGemma's ability to understand image content and extract key features, and then convert this information into language output to enable interaction with users or automated content generation. This flexibility makes PaliGemma suitable not only for research and development environments, but also for commercial applications such as customer service, content recommendation systems, etc. Pictures What can PaliGemma do? Pictures can be used when prompted.

According to benchmarks, for small, high-performance applications, Quarkus (fast startup, low memory) or Micronaut (TechEmpower excellent) are ideal choices. SpringBoot is suitable for large, full-stack applications, but has slightly slower startup times and memory usage.

In Go framework development, common challenges and their solutions are: Error handling: Use the errors package for management, and use middleware to centrally handle errors. Authentication and authorization: Integrate third-party libraries and create custom middleware to check credentials. Concurrency processing: Use goroutines, mutexes, and channels to control resource access. Unit testing: Use gotest packages, mocks, and stubs for isolation, and code coverage tools to ensure sufficiency. Deployment and monitoring: Use Docker containers to package deployments, set up data backups, and track performance and errors with logging and monitoring tools.
