Microsoft launches LLaVA-Med AI model to analyze medical pathology cases

According to news on June 14, Microsoft researchers recently demonstrated the LLaVA-Med model, which is mainly used for biomedical research and can infer the pathological conditions of patients based on CT, X-ray pictures, etc.

It is reported that Microsoft researchers have cooperated with a group of hospitals and obtained a large data set corresponding to biomedical image text to train a multi-modal AI model. The data set includes chest X-ray, MRI, histology, pathology and CT images, etc., with relatively comprehensive coverage.

▲ Picture source Microsoft

Microsoft uses GPT-4, based on Vision Transformer and Vicuna language model, to run LLaVA-Med on eight NVIDIA A100 GPUs It is trained to include "all pre-analytical information for each image" and used to generate questions and answers about images, meeting the vision of an assistant that can "answer questions about biomedical images in natural language."

In the learning process, the model mainly focuses on "describing the content of such images" and "elaborating on biomedical concepts (IT House Note: Judge what it looks like from the picture)". According to Microsoft, the model ultimately has “excellent multi-modal dialogue capabilities” and “On three standard biomedical datasets used to answer visual questions, LLaVA-Med leads other advanced models in the industry in some indicators. ".

▲ Picture source Microsoft

The research team stated: “While we believe that the LLaVA-Med model represents a step towards building useful biomedical visual assistants is an important step, but the current LLaVA-Med model still has certain shortcomings, namely the common problems of false examples and poor accuracy in large models. The research team will focus on improving the quality and reliability of the model in the future to make the model One day it can be applied in commercial biomedicine."

IT House noticed that the model is now open source, and you canfind relevant information on GitHub.

The above is the detailed content of Microsoft launches LLaVA-Med AI model to analyze medical pathology cases. For more information, please follow other related articles on the PHP Chinese website!