PixelLM, a byte multi-modal large model that efficiently implements pixel-level reasoning without SA dependency

Multi-modal large models have exploded, are you ready to enter practical applications in fine-grained tasks such as image editing, autonomous driving and robotics?

The capabilities of most current models are still limited to generating text descriptions of the overall image or specific areas, and their capabilities in pixel-level understanding (such as object segmentation) are relatively limited.

In response to this problem, some work has begun to explore the use of multi-modal large models to process user segmentation instructions (for example, "Please segment the fruits rich in vitamin C in the picture").

However, the methods on the market all suffer from two major shortcomings:

1) cannot handle tasks involving multiple target objects, which is indispensable in real-world scenarios;

2) Relies on pre-trained image segmentation models like SAM, and the amount of calculation required for one forward propagation of SAM is enough for Llama-7B to generate more than 500 tokens.

In order to solve this problem, the Bytedance Intelligent Creation Team teamed up with researchers from Beijing Jiaotong University and University of Science and Technology Beijing to propose PixelLM, the first large-scale efficient pixel-level inference model that does not rely on SAM.

Before introducing it in detail, let’s experience the actual segmentation effect of several groups of PixelLM:

Compared with previous work, the advantages of PixelLM are:

• It can Proficiently handle any number of open domain targets and diverse complex inference segmentation tasks.
• Avoids additional and costly segmentation models, improving efficiency and migration capabilities to different applications.

Further, in order to support model training and evaluation in this research field, the research team built a data set for multi-objective reasoning segmentation scenarios based on the LVIS data set with the help of GPT-4V. MUSE, which contains more than 200,000 question-answer pairs involving more than 900,000 instance segmentation masks.

In order to achieve the above effects, how was this research conducted?

Principle behind

Picture

As shown in the framework diagram in the paper, the PixelLM architecture is very simple and consists of four main parts. The latter two are the core of PixelLM:

1. Pre-trained CLIP-ViT visual encoder
2. Large language model
3. Lightweight pixel decoder
4. Seg Codebook

The Seg codebook contains learnable tokens, which are used to encode target information at different scales of CLIP-ViT. Then, the pixel decoder generates object segmentation results based on these tokens and the image features of CLIP-ViT. Thanks to this design, PixelLM can generate high-quality segmentation results without an external segmentation model, significantly improving model efficiency.

According to the researcher's description, the tokens in the Seg codebook can be divided into L groups, each group contains N tokens, and each group corresponds to a scale from CLIP-ViT visual features.

For the input image, PixelLM extracts L scale features from the image features produced by the CLIP-ViT visual encoder. The last layer covers the global image information and will be used by LLM to understand the image content. .

The tokens of the Seg codebook will be input into the LLM together with the text instructions and the last layer of image features to produce output in the form of autoregression. The output will also include Seg codebook tokens processed by LLM, which will be input into the pixel decoder together with L scale CLIP-ViT features to produce the final segmentation result.

Picture

Picture

So why do we need to set each group to contain N tokens? ? The researchers explained in conjunction with the following figure:

In scenarios involving multiple targets or targets with very complex semantics, although LLM can provide detailed text responses, it may not be fully captured using only a single token. The entire content of the target semantics.

In order to enhance the model's ability in complex reasoning scenarios, the researchers introduced multiple tokens within each scale group and performed a linear fusion operation of one token. Before the token is passed to the decoder, a linear projection layer is used to merge the tokens within each group.

The following figure shows the effect when there are multiple tokens in each group. The attention map is what each token looks like after being processed by the decoder. This visualization shows that multiple tokens provide unique and complementary information, resulting in more effective segmentation output.

Picture

In addition, in order to enhance the model's ability to distinguish multiple targets, PixelLM also designed an additional Target Refinement Loss.

MUSE Dataset

Although the above solutions have been proposed, in order to fully utilize the capabilities of the model, the model still requires appropriate training data. Reviewing the currently available public data sets, we found that the existing data have the following major limitations:

1) Insufficient description of object details;
2) Lack of problems with complex reasoning and multiple target numbers -The answer is correct.

In order to solve these problems, the research team used GPT-4V to build an automated data annotation pipeline, and thus generated the MUSE data set. The figure below shows an example of the prompts used when generating MUSE and the data generated.

Picture

In MUSE, all instance masks are from the LVIS dataset, and additionally detailed text descriptions generated based on the image content are added. MUSE contains 246,000 question-answer pairs, and each question-answer pair involves an average of 3.7 target objects. In addition, the research team conducted an exhaustive statistical analysis of the dataset:

Category statistics: There are more than 1,000 categories in MUSE from the original LVIS dataset, and 900,000 instances with unique descriptions based on The question-answer pairs vary depending on the context. Figure (a) shows the number of instances of each category across all question-answer pairs.

Token number statistics: Figure (b) shows the distribution of the number of tokens described in the instance. Some instance descriptions contain more than 100 tokens. These descriptions are not limited to simple category names; instead, they are enriched with detailed information about each instance, including appearance, properties, and relationships to other objects, through a GPT-4V-based data generation process. The depth and diversity of information in the dataset enhances the generalization ability of the trained model, allowing it to effectively solve open domain problems.

Statistics of the number of targets: Figure (c) shows the statistics of the number of targets in each question-answer pair. The average number of targets is 3.7, and the maximum number of targets can reach 34. This number can cover most target inference scenarios for a single image.

Algorithm Evaluation

The research team evaluated the performance of PixelLM on three benchmarks, including MUSE benchmark, referring segmentation benchmark, and multi-referring segmentation benchmark. In the multi-referring segmentation benchmark, The research team requires the model to continuously segment multiple targets contained in each image in the referring segmentation benchmark in one problem.

At the same time, since PixelLM is the first model to handle complex pixel reasoning tasks involving multiple targets, the research team established four baselines to conduct comparative analysis of the models.

Three of the baselines are based on LISA, the most relevant work on PixelLM, including:

1) Original LISA;

2) LISA_rec: First enter the question into LLAVA-13B to Get the target's text reply, and then use LISA to split the text;

3) LISA_aug: directly add MUSE to the LISA training data.

4) The other is SEEM, a general segmentation model that does not use LLM.

Picture

In most indicators of the three benchmarks, PixelLM's performance is better than other methods, and because PixelLM does not rely on SAM, Its TFLOPs are far lower than models of the same size.

Interested friends can follow the wave first and wait for the code to be open source~

Reference link:
[1]https://www.php.cn/ link/9271858951e6fe9504d1f05ae8576001
[2]https://www.php.cn/link/f1686b4badcf28d33ed632036c7ab0b8

The above is the detailed content of PixelLM, a byte multi-modal large model that efficiently implements pixel-level reasoning without SA dependency. For more information, please follow other related articles on the PHP Chinese website!