UC Berkeley successfully developed a large general visual reasoning model, and three senior scholars joined forces to participate in the research

How far can we go with visual (pixel) models alone? A new paper from UC Berkeley and Johns Hopkins University explores this problem and demonstrates the potential of large vision models (LVM) on a variety of CV tasks.

In recent times, large language models (LLM) such as GPT and LLaMA have become popular around the world.

Building large-scale visual models (LVM) is a problem of great concern. What do we need to achieve it?

The ideas provided by visual language models such as LLaVA are interesting and worth exploring, but according to the laws of the animal kingdom, we already know that visual ability and language ability are not related. For example, many experiments have shown that the visual world of non-human primates is very similar to that of humans, even though their language systems are "identical" to humans.

A recent paper discusses the answer to another question: how far can we go with pixels alone. The paper was written by researchers from the University of California, Berkeley, and Johns Hopkins University

• Paper link: https://arxiv.org/ abs/2312.00785

• Project homepage: https://yutongbai.com/lvm.html

The LLM that researchers try to emulate in LVM Key features: 1) Growth according to the scale of data In order to expand the business, we need to find new market opportunities. We plan to further expand our product line to meet growing demand. At the same time, we will strengthen marketing strategies and increase brand awareness. By actively participating in industry exhibitions and promotion activities, we will strive to develop more customer groups. We believe that through these efforts we can achieve greater success and achieve continued growth, 2) Flexibly specify tasks through prompts (contextual learning).

They specify three main components, namely data, architecture and loss function.

In terms of data, researchers want to take advantage of the significant diversity in visual data. Starting with just unannotated raw images and videos, and then leveraging various annotated visual data sources produced over the past few decades (including semantic segmentation, depth reconstruction, keypoints, multi-view 3D objects, etc.). They defined a common format - a "visual sentence" - to represent these different annotations without requiring any meta-knowledge beyond pixels. The total size of the training set is 1.64 billion images/frame.

In terms of architecture, the researchers used a large transformer architecture (3 billion parameters) to train on visual data represented as token sequences, and used the learned tokenizer to map each image to 256 vectorsQuantification token string.

Regarding the loss function, researchers draw inspiration from the natural language community, that is, mask token modeling has "given way" to the sequence autoregressive prediction method. Once images, videos, and annotated images can all be represented as sequences, the trained model can minimize the cross-entropy loss when predicting the next token.

Through this extremely simple design, the researchers demonstrated the following noteworthy behaviors:

• As the model size and data size increase, the model automatically Demonstrate Appropriate In order to expand our business, we need to look for new market opportunities. We plan to further expand our product line to meet growing demand. At the same time, we will strengthen marketing strategies and increase brand awareness. By actively participating in industry exhibitions and promotion activities, we will strive to develop more customer groups. We believe that through these efforts we can achieve greater success and achieve continued growth behavior

• Many different visual tasks can now be solved by designing appropriate prompts at test time. While not as high-performance as a custom, specially trained model, the fact that a single vision model can solve so many tasks is very encouraging;

• Supervised data significantly contributes to performance on a variety of vision tasks

• There are already signs of general visual reasoning capabilities when processing out-of-distribution data and performing new tasks, but Further research is still needed

The co-author of the paper, Yutong Bai, a fourth-year CS doctoral student at Johns Hopkins University and a visiting doctoral student at Berkeley, tweeted to promote their work.

## The original image source comes from the Twitter account: https://twitter.com/YutongBAI1002/status/1731512110247473608

Among the authors of the paper, the last three are senior scholars at UC Berkeley in the field of CV. Professor Trevor Darrell is the founding co-director of BAIR, the Berkeley Artificial Intelligence Research Laboratory, Professor Jitendra Malik won the 2019 IEEE Computer Pioneer Award, and Professor Alexei A. Efros is especially famous for nearest neighbor research.

^{From left to right are Trevor Darrell, Jitendra Malik, Alexei A. Efros.}

Method introduction

The article uses a two-stage method: 1) train a large visual tokenizer (operating on a single image) to be able to combine each Convert an image into a series of visual tokens; 2) Train an autoregressive transformer model on visual sentences, and each sentence is represented as a series of tokens. The method is shown in Figure 2

Image Tokenization

In order to apply the Transformer model to the image, typical operations include: Divide images into patches and treat them as sequences; or use a pretrained image tokenizer, such as VQVAE or VQGAN, to aggregate image features into a grid of discrete tokens. This article adopts the latter method, using the VQGAN model to generate semantic tokens.

The LVM framework includes encoding and decoding mechanisms and also has quantization layers, where the encoder and decoder are built with convolutional layers. The encoder is equipped with multiple downsampling modules to shrink the spatial dimensions of the input, while the decoder is equipped with a series of equivalent upsampling modules to restore the image to its original size. For a given image, the VQGAN tokenizer generates 256 discrete tokens.

The VQGAN architecture in this paper adopts the implementation details proposed by Chang et al. and follows their setup. Specifically, the downsampling factor is f=16 and the codebook size is 8192. This means that for an image of size 256×256, the VQGAN tokenizer will generate 16×16=256 tokens, and each token can take on 8192 different values. In addition, this article trained tokenizer on a 1.5B subset of the LAION 5B data set

Visual sentence sequence modeling

Use VQGAN to convert images into discrete tokens Finally, this paper concatenates discrete tokens in multiple images into a one-dimensional sequence and treats visual sentences as a unified sequence. Importantly, none of the visual sentences were specially processed - that is, no special tokens were used to indicate a specific task or format.

^{The function of visual sentences is to format different visual data into a unified image sequence structure}

Implementation details. After tokenizing each image in the visual sentence into 256 tokens, this paper concatenates them to form a 1D token sequence. On the visual token sequence, the Transformer model in this article is actually the same as the autoregressive language model, so they adopt LLaMA’s Transformer architecture.

This content uses a context length of 4096 tokens, which is similar to the language model. Add a [BOS] (beginning of sentence) token at the beginning of each visual sentence and an [EOS] (end of sentence) token at the end, and use sequence splicing during training to improve efficiency

This article is used throughout UVDv1 The model was trained on the data set (420 billion tokens), and a total of 4 models with different numbers of parameters were trained: 300 million, 600 million, 1 billion and 3 billion.

Experimental results need to be rewritten

The study conducted experiments to evaluate the model. In order to expand the business, we need to find new market opportunities. We plan to further expand our product line to meet growing demand. At the same time, we will strengthen marketing strategies and increase brand awareness. By actively participating in industry exhibitions and promotion activities, we will strive to develop more customer groups. We believe that through these efforts we can achieve greater success and achieve continued growth in our capabilities and ability to understand and answer a variety of tasks.

In order to expand our business, we need to find new market opportunities. We plan to further expand our product line to meet growing demand. At the same time, we will strengthen marketing strategies and increase brand awareness. By actively participating in industry exhibitions and promotion activities, we will strive to develop more customer groups. We believe that through these efforts, we can achieve greater achievements and achieve sustained growth

As shown in Figure 3, this study first examined the training loss of LVMs of different sizes

As shown in Figure 4 below, the larger model has lower complexity in all tasks, indicating that the overall performance of the model can be transferred to a series of downstream tasks.

As shown in Figure 5, each data component has an important impact on downstream tasks. LVM not only benefits from larger data, but also improves with the diversity of the data set

Rewrite content without changing the original meaning, The language needs to be rewritten to Chinese. The original sentence should appear

In order to test LVM’s ability to understand various prompts, this study first conducted an evaluation experiment on LVM on a sequence reasoning task. Among them, prompt is very simple: provide the model with a sequence of 7 images and ask it to predict the next image. The experimental results need to be rewritten as shown in Figure 6 below:

The study also treats the list of items of a given category as a sequence to let LVM predict images of the same category. The experimental results need to be rewritten as shown in Figure 15 below:

So, how much context is needed to accurately predict subsequent frames?

In this study, we evaluate the frame generation perplexity of our model by giving contextual prompts of varying lengths (1 to 15 frames). The results show that the perplexity gradually improves as the number of frames increases. The specific data is shown in Figure 7 below. The confusion improved significantly from frame 1 to frame 11, and then stabilized (62.1 → 48.4)

Analogy Prompt

This study also tested the advanced interpretation capabilities of LVM by evaluating more complex prompt structures such as analogy prompts

Figure 8 below shows the results of Analogy Prompt for a number of tasks Qualitative results:

Based on comparison with visual prompting, it can be seen that sequence LVM is better than previous methods on almost all tasks

Synthetic tasks. Figure 9 shows the results of combining multiple tasks using a single prompt

Other prompts

Researchers have tried to The model provides various prompts that it has never seen before. To observe the model, in order to expand the business, we need to find new market opportunities. We plan to further expand our product line to meet growing demand. At the same time, we will strengthen marketing strategies and increase brand awareness. By actively participating in industry exhibitions and promotion activities, we will strive to develop more customer groups. We believe that through these efforts, we can achieve greater success and achieve continued growth. Figure 10 below shows some such prompts working well.

Figure 11 below shows some prompts that are difficult to describe in words. LVM may eventually outperform LLM on these tasks.

In the non-verbal human IQ test, Figure 13 shows preliminary qualitative results for a typical visual reasoning question

Read the original article for more details.

The above is the detailed content of UC Berkeley successfully developed a large general visual reasoning model, and three senior scholars joined forces to participate in the research. For more information, please follow other related articles on the PHP Chinese website!