University of Wisconsin-Madison and others jointly issued a post! The latest multi-modal large model LLaVA is released, and its level is close to GPT-4

Visual command fine-tuning is popular.

This paper is called Visual Instruction Tuning and is jointly produced by the University of Wisconsin-Madison, Microsoft Research and Columbia University.

The authors include Haotian Liu, Chunyuan Li, Qingyang Wu and Yong Jae Lee.

At the same time, the researchers also open sourced their code, models and data sets on GitHub. The paper is also posted on Arxiv.

Explanation

So what exactly is the function of LLaVA? If you dive into the paper, it may be a bit strange to those who are not familiar with it.

Let’s first see what LLaVA itself says.

## The above answer is generated from LLaVA’s Chatbot. The question I entered is, what is the function of LLaVA and what should users do? Take advantage of it?

The answer is as follows:

LLaVA is a large language and visual assistant trained in the UW Madison WAIV laboratory. LLaVA is designed to help users handle various tasks by leveraging its natural language understanding capabilities and visual content analysis capabilities.

LLaVA’s main features include:

· Text-based tasks: LLaVA can process and analyze text , allowing users to ask questions, LLaVA can chat with users, or complete tasks entered by users, such as refining document summaries, sentiment analysis, entity recognition, etc.

· Image-based tasks: LLaVA can analyze images, describe images, perform object recognition, and analyze and understand scenes.

If users want to better apply LLaVA, they can directly talk to LLaVA, just like talking to humans. Provide as much detail as possible and LLaVA will generate a more complete answer.

At the same time, you should remember that after all, LLaVA is a machine learning model and an AI tool. As long as it is a tool, it has its limitations. LLaVA may not perfectly answer the questions entered by the user every time.

Of course, LLaVA is constantly learning and improving, and its functions will become more and more powerful.

In general, it can be seen from LLaVA’s answer that the functions of Chatbot are not much different from GPT. But LLaVA can handle some tasks related to images.

Exhibiting Color

The researchers detailed the technical details of LLaVA in their Arxiv paper.

It is important to know that using machine-generated instructions to follow data to fine-tune instructions for large language models (LLMs) improves the zero-point capability of new tasks, but this idea is not explored in the multi-modal field. less.

In the paper, the researchers attempted for the first time to use language-only GPT-4 to generate instruction-following data for multi-modal language images.

By conditioning instructions on this generated data, the researchers introduce LLaVA: a large-scale language and vision assistant that is an end-to-end trained large-scale multi-modal Stateful model, which connects a visual encoder and LLM for general vision and language understanding.

Early experiments show that LLaVA demonstrates impressive multi-modal chat capabilities, sometimes outputting multi-modal GPT-4 performance on unseen images/commands, and on synthetic multi-modal chats. Compared with GPT-4 on the modal instruction following data set, it achieved a relative score of 85.1%.

When fine-tuned for Science Magazine, the synergy of LLaVA and GPT-4 achieved a new state-of-the-art accuracy of 92.53%.

Researchers have disclosed the data, models and code base for visual command adjustments generated by GPT-4.

Multimodal model

First clarify the definition.

Large-scale multimodal model refers to a model based on machine learning technology that can process and analyze multiple input types, such as text and images.

These models are designed to handle a wider range of tasks and are able to understand different forms of data. By taking text and images as input, these models improve their ability to understand and compile explanations to generate more accurate and relevant answers.

Humans interact with the world through multiple channels, including vision and language, because each individual channel has unique advantages in representing and conveying certain concepts of the world, thereby facilitating greater Understand the world well.

One of the core aspirations of artificial intelligence is to develop a universal assistant that can effectively follow multi-modal visual and language instructions, be consistent with human intentions, and complete various real-life tasks. World Mission.

As a result, the developer community has witnessed a renewed interest in developing language-enhanced basic vision models with powerful capabilities in open-world visual understanding such as classification, detection, segmentation, description, and visual generation and editing.

In these features, each task is independently solved by a single large visual model, with task instructions implicitly considered in the model design.

Furthermore, language is only used to describe image content. While this allows language to play an important role in mapping visual signals to linguistic semantics - a common channel for human communication. However, this results in models that often have fixed interfaces with limited interactivity and adaptability to user instructions.

And large language models (LLM) show that language can play a broader role: a common interface for a general assistant, various task instructions can be explicitly expressed in language, and guide the end to The end-trained neural assistant switches to the task of interest to solve it.

For example, the recent success of ChatGPT and GPT-4 have demonstrated the ability of this LLM to follow human instructions and stimulated huge interest in developing open source LLM.

LLaMA is an open source LLM whose performance is equivalent to GPT-3. Ongoing work leverages various machine-generated high-quality instruction following samples to improve LLM's alignment capabilities, reporting impressive performance compared to proprietary LLMs. Importantly, this line of work is text-only.

In this paper, researchers propose visual command tuning, which is the first attempt to extend command tuning into a multimodal space and paves the way for building a universal visual assistant. the way. Specifically, the main contents of the paper include:

Multimodal instruction following data. A key challenge is the lack of visual language instructions to follow the data. We present a data reform perspective and pipeline that uses ChatGPT/GPT-4 to convert image-text pairs into appropriate command-following formats.

Large multi-modal model. The researchers developed a large multimodal model (LMM) by connecting CLIP's open-set visual encoder and language decoder LaMA, and fine-tuned them end-to-end on the generated instructional visual-verbal data. Empirical studies verify the effectiveness of LMM instruction tuning using generated data and provide practical suggestions for building a general instruction-following visual agent. With GPT 4, the research team achieved state-of-the-art performance on the Science QA multi-modal inference dataset.

Open source. The research team released the following to the public: the generated multimodal instruction data, a code library for data generation and model training, model checkpoints, and a visual chat demonstration.

Result Display

It can be seen that LLaVA can handle all kinds of problems, and the answers generated are both comprehensive and logical.

LLaVA shows some multi-modal capabilities close to the level of GPT-4, with a GPT-4 relative score of 85% in terms of visual chat.

In terms of reasoning question and answer, LLaVA even reached the new SoTA-92.53%, defeating the multi-modal thinking chain.

The above is the detailed content of University of Wisconsin-Madison and others jointly issued a post! The latest multi-modal large model LLaVA is released, and its level is close to GPT-4. For more information, please follow other related articles on the PHP Chinese website!