Google and MIT's 'Iterative Joint Certification” video question and answer model: SOTA performance, using 80% less computing power

Video is a ubiquitous source of media content that touches many aspects of people’s daily lives. An increasing number of real-world video applications, such as video subtitling, content analysis, and video question answering (VideoQA), rely on models that can connect video content to text or natural language.

Among them, the video question and answer model is particularly challenging because it requires simultaneous grasp of semantic information, such as targets in the scene, and temporal information, such as how things move and interact. Both types of information must be placed in the context of a natural language question with a specific intent. Additionally, since videos have many frames, processing all of them to learn spatiotemporal information may be computationally prohibitive.

Paper link: https://arxiv.org/pdf/2208.00934.pdf For To solve this problem, in the article "Video Question Answering with Iterative Video-Text Co-Tokenization", researchers from Google and MIT introduced a new method of video-text learning, called "Iterative Co-Tokenization", which can effectively Fusion of spatial, temporal and linguistic information for information processing in video question answering.

This method is multi-stream, using independent backbone models to handle different scales Video, produces video representations that capture different features, such as high spatial resolution or long duration videos. The model applies the "co-authentication" module to learn effective representations from the fusion of video streams and text. The model is very computationally efficient, requiring only 67 GFLOPs, which is at least 50% lower than the previous method, and has better performance than other SOTA models.

Video-Text Iteration

The main goal of this model is to generate features from video and text (i.e. user questions) that together allow their corresponding inputs to interact. The second goal is to do this in an efficient way, which is very important for videos since they contain tens to hundreds of frames of input.

The model learns to label joint video-language input into smaller sets of labels to jointly and efficiently represent both modalities. When tokenizing, researchers use both modes to produce a joint compact representation, which is fed into a transformation layer to produce the next-level representation.

A challenge here, which is also a typical problem in cross-modal learning, is that video frames often do not directly correspond to related text. The researchers solved this problem by adding two learnable linear layers to unify the visual and textual feature dimensions before tokenization. This allowed the researchers to have both video and text condition how video tags were learned.

Furthermore, a single tokenization step does not allow further interaction between the two modes. To do this, the researchers use this new feature representation to interact with the video input features and produce another set of tokenized features, which are then fed into the next transformer layer. This iterative process creates new features or markers that represent the continuous improvement of the joint representation of the two modes. Finally, these features are fed into a decoder that generates text output.

As is common practice in video quality assessment, the researchers fine-tuned the model before fine-tuning it on individual video quality assessment datasets. Do pre-training. In this work, the researchers automatically annotated videos with text based on speech recognition, using the HowTo100M dataset instead of pre-training on the large VideoQA dataset. This weaker pre-training data still enabled the researchers' model to learn video-text features.

Implementation of efficient video question answering

The researchers applied the video language iterative co-authentication algorithm to three major VideoQA benchmarks, MSRVTT-QA, MSVD-QA and IVQA , and demonstrate that this approach achieves better results than other state-of-the-art models without making the model too large. In addition, iterative co-label learning also requires lower computing power on video-text learning tasks.

This model only uses 67GFLOPS computing power, which is the computing power required for 3D-ResNet video model and text (360GFLOPS ), which is more than twice as efficient as the X3D model. and generates highly accurate results, exceeding state-of-the-art methods.

Multi-stream video input

For VideoQA or some other tasks involving video input, researchers found that multi-stream input is more accurate for answering questions about spatial and temporal relationships The question is very important.

The researchers utilized three video streams of different resolutions and frame rates: a low-resolution, high-frame-rate input video stream (32 frames per second, spatial resolution 64x64, denoted as 32x64x64); a high-resolution, low-frame-rate video (8x224x224); and one in between (16x112x112).

Although there is obviously more information to process with three data streams, a very efficient model is obtained due to the iterative co-labeling method. At the same time, these additional data streams allow the most relevant information to be extracted.

For example, as shown below, questions related to a specific activity will produce higher activations in a video input with a lower resolution but a higher frame rate than with a general activity Related questions can be answered from high-resolution inputs with few frames.

Another benefit of this algorithm is that the tokenization will be based on the question asked. Different and changed.

Conclusion

The researchers proposed a new video language learning method that focuses on joint learning across video-text modalities. Researchers tackle the important and challenging task of video question answering. The researchers' approach is efficient and accurate, outperforming current state-of-the-art models despite being more efficient.

The Google researchers' approach has a modest model size and could gain further performance improvements with larger models and data. The researchers hope this work will spark more research in visual language learning to enable more seamless interactions with visual-based media.

The above is the detailed content of Google and MIT's 'Iterative Joint Certification” video question and answer model: SOTA performance, using 80% less computing power. For more information, please follow other related articles on the PHP Chinese website!