


Covering text, positioning and segmentation tasks, Zhiyuan and Hong Kong Chinese jointly proposed the first multi-functional 3D medical multi-modal large model
作者 | 香港中文大学白帆
编辑 | ScienceAI
近日,香港中文大学和智源联合提出的 M3D 系列工作,包括 M3D-Data, M3D-LaMed 和 M3D-Bench,从数据集、模型和测评全方面推动 3D 医学图像分析的发展。
(1)M3D-Data 是目前最大的 3D 医学图像数据集,包括 M3D-Cap (120K 3D 图文对), M3D-VQA (510K 问答对),M3D-Seg(150K 3D Mask),M3D-RefSeg (3K 推理分割)共四个子数据集。
(2)M3D-LaMed 是目前最多功能的 3D 医学多模态大模型,能够解决文本(疾病诊断、图像检索、视觉问答、报告生成等),定位(目标检测、视觉定位等)和分割(语义分割、指代分割、推理分割等)三类医学分析任务。
(3)M3D-Bench 能够全面和自动评估 8 种任务,涵盖文本、定位和分割三个方面,并提供人工校验后的测试数据。
我们最早于 2024 年 4 月发布了数据集、模型和代码。
近期,我们提供了更小和更强的 M3D-LaMed-Phi-3-4B 模型,并增加了线上 demo 供大家体验!
最新进展请关注 GitHub 库的更新 ,如果有任何疑问和建议可以及时联系,欢迎大家讨论和支持我们的工作。
- 论文链接:https://arxiv.org/abs/2404.00578
- 代码:https://github.com/BAAI-DCAI/M3D
- 模型:https://huggingface.co/GoodBaiBai88/M3D-LaMed-Phi-3-4B
- 数据集:https://github.com/BAAI-DCAI/M3D?tab=readme-ov-file#data
- 线上 Demo:https://baai.rpailab.xyz/
我们能为医学图像相关研究者提供什么?
- M3D-Data, 最大的 3D 医学多模态数据集;
- M3D-Seg,整合了几乎所有开源 3D 医学分割数据集,共计 25 个;
- M3D-LaMed, 支持文本、定位和分割的最多功能的 3D 医学多模态大模型,提供了简洁清晰的代码框架,研究者可以轻易魔改每个模块的设置;
- M3D-CLIP,基于 M3D-Cap 3D 图文对,我们训练了一个图文对比学习的 M3D-CLIP 模型,共提供其中的视觉预训练权重 3DViT;
- M3D-Bench,全面和清晰的测评方案和代码。
- 本文涉及的所有资源全部开放,希望能帮助研究者共同推进 3D 医学图像分析的发展。
线上Demo视频。
医学图像分析对临床诊断和治疗至关重要,多模态大语言模型 (MLLM) 对此的支持日益增多。然而,先前的研究主要集中在 2D 医学图像上,尽管 3D 图像具有更丰富的空间信息,但对其的研究和探索还不够。
本文旨在利用 MLLM 推进 3D 医学图像分析。为此,我们提出了一个大规模 3D 多模态医学数据集 M3D-Data,其中包含 120K 个图像-文本对和 662K 个指令-响应对,专门针对各种 3D 医学任务量身定制,例如图文检索、报告生成、视觉问答、定位和分割。
此外,我们提出了 M3D-LaMed,这是一种用于 3D 医学图像分析的多功能多模态大语言模型。
我们还引入了一个新的 3D 多模态医学基准 M3D-Bench,它有助于在八个任务中进行自动评估。通过综合评估,我们的方法被证明是一种稳健的 3D 医学图像分析模型,其表现优于现有解决方案。所有代码、数据和模型均可在以下网址公开获取。
数据集
M3D-Data 共包括4个子数据集,分别为M3D-Cap(图文对), M3D-VQA(视觉问答对), M3D-RefSeg(推理分割)和 M3D-Seg(整合 25 个 3D 分割数据集)。
数据集统计情况。
M3D-VQA 数据集分布。其中问题类型主要包括平面、期相、器官、异常和定位五类常见的3D图像问题。
我们整合了几乎所有开源的 3D 医学分割数据集,组成了 M3D-Seg,共计 25 个。数据集可以被用来做语义分割、推理分割、指代分割和相应的检测定位任务。
M3D-Seg。
模型
M3D-LaMed 模型结构如下图所示。(a)3D 图像编码器通过跨模态对比学习损失由图文数据进行预训练,可直接应用于图文检索任务。(b)在 M3D-LaMed 模型中,3D 医学图像被输入到预先训练的 3D 图像编码器和高效的 3D 空间池化感知器中,并将视觉 token 插入 LLM,输出的 [SEG] 作为 prompt 驱动分割模块。
M3D-LaMed 模型结构。
实验
图文检索
在 3D 图文检索中,模型旨在根据相似性从数据集中匹配图像和文本,通常涉及两个任务:文本到图像检索 (TR) 和图像到文本检索 (IR)。
由于缺乏合适的方法,我们将 2D 医学的代表模型 PMC-CLIP 应用于 3D 图文检索中,我们发现由于缺乏空间信息,几乎无法和 3D 图文检索模型对比。
报告生成
在报告生成中,该模型根据从 3D 医学图像中提取的信息生成文本报告。
封闭式视觉问答
在封闭式视觉问答中,需要为模型提供封闭的答案候选,例如 A,B,C,D,要求模型从候选中选出正确答案。
我们发现在医学领域 M3D-LaMed 超过通用的 GPT-4V。
开放式视觉问答
在开放式视觉问答中,模型生成开放式的答案,不存在任何答案提示和候选。
我们发现在医学领域 M3D-LaMed 超过通用的 GPT-4V。不过需注意目前GPT-4V 限制了医疗相关问题的回答。
定位
Positioning is crucial in visual language tasks, especially those involving input and output boxes. Tasks in the output box, such as referent expression understanding (REC), aim to locate a target object in an image based on a referent representation. In contrast, input box tasks, such as referent expression generation (REG), require the model to generate a description of a specific region based on an image and a location box.
Segmentation
The segmentation task is crucial in 3D medical image analysis because of its recognition and localization capabilities. To address various textual cues, segmentation is divided into semantic segmentation and referential expression segmentation. For semantic segmentation, the model generates segmentation masks based on semantic labels. Referential expression segmentation requires target segmentation based on natural language expression description, which requires the model to have certain understanding and reasoning capabilities.
Case Study of Out-of-Distribution (OOD) Problems
We tested the M3D-LaMed model on an OOD conversation, which means that all problems are not relevant to our training data. We found that M3D-LaMed has strong generalization capabilities and can produce reasonable answers to OOD problems rather than gibberish. In each set of conversations, the avatar and questions on the left come from the user, and the avatar and answers on the right come from M3D-LaMed.
The model has strong reasoning and generalization capabilities.
Our latest trained smaller M3D-LaMed-Phi-3-4B model has better performance, everyone is welcome to use it! GoodBaiBai88/M3D-LaMed-Phi-3-4B · Hugging Face
Report generation test results
Closed VQA test results
Reviewed on TotalSegmentator Semantic Segmentation Dice Results
Summary
Our M3D series of studies promotes the use of MLLM for 3D medical image analysis. Specifically, we build a large-scale 3D multimodal medical dataset M3D-Data, which contains 120K 3D image-text pairs and 662K instruction-response pairs, tailored for 3D medical tasks. Furthermore, we propose M3D-LaMed, a general model that handles image text retrieval, report generation, visual question answering, localization, and segmentation. Furthermore, we introduce a comprehensive benchmark, M3D-Bench, which is carefully designed for eight tasks.
Our approach lays a solid foundation for MLLM to understand the vision and language of 3D medical scenes. Our data, code, and models will facilitate further exploration and application of 3D medical MLLM in future research. We hope that our work can be helpful to researchers in the field, and everyone is welcome to use and discuss it.
The above is the detailed content of Covering text, positioning and segmentation tasks, Zhiyuan and Hong Kong Chinese jointly proposed the first multi-functional 3D medical multi-modal large model. For more information, please follow other related articles on the PHP Chinese website!

Hot AI Tools

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Undress AI Tool
Undress images for free

Clothoff.io
AI clothes remover

Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Article

Hot Tools

Notepad++7.3.1
Easy-to-use and free code editor

SublimeText3 Chinese version
Chinese version, very easy to use

Zend Studio 13.0.1
Powerful PHP integrated development environment

Dreamweaver CS6
Visual web development tools

SublimeText3 Mac version
God-level code editing software (SublimeText3)

Hot Topics











In modern manufacturing, accurate defect detection is not only the key to ensuring product quality, but also the core of improving production efficiency. However, existing defect detection datasets often lack the accuracy and semantic richness required for practical applications, resulting in models unable to identify specific defect categories or locations. In order to solve this problem, a top research team composed of Hong Kong University of Science and Technology Guangzhou and Simou Technology innovatively developed the "DefectSpectrum" data set, which provides detailed and semantically rich large-scale annotation of industrial defects. As shown in Table 1, compared with other industrial data sets, the "DefectSpectrum" data set provides the most defect annotations (5438 defect samples) and the most detailed defect classification (125 defect categories

The open LLM community is an era when a hundred flowers bloom and compete. You can see Llama-3-70B-Instruct, QWen2-72B-Instruct, Nemotron-4-340B-Instruct, Mixtral-8x22BInstruct-v0.1 and many other excellent performers. Model. However, compared with proprietary large models represented by GPT-4-Turbo, open models still have significant gaps in many fields. In addition to general models, some open models that specialize in key areas have been developed, such as DeepSeek-Coder-V2 for programming and mathematics, and InternVL for visual-language tasks.

Editor |KX To this day, the structural detail and precision determined by crystallography, from simple metals to large membrane proteins, are unmatched by any other method. However, the biggest challenge, the so-called phase problem, remains retrieving phase information from experimentally determined amplitudes. Researchers at the University of Copenhagen in Denmark have developed a deep learning method called PhAI to solve crystal phase problems. A deep learning neural network trained using millions of artificial crystal structures and their corresponding synthetic diffraction data can generate accurate electron density maps. The study shows that this deep learning-based ab initio structural solution method can solve the phase problem at a resolution of only 2 Angstroms, which is equivalent to only 10% to 20% of the data available at atomic resolution, while traditional ab initio Calculation

For AI, Mathematical Olympiad is no longer a problem. On Thursday, Google DeepMind's artificial intelligence completed a feat: using AI to solve the real question of this year's International Mathematical Olympiad IMO, and it was just one step away from winning the gold medal. The IMO competition that just ended last week had six questions involving algebra, combinatorics, geometry and number theory. The hybrid AI system proposed by Google got four questions right and scored 28 points, reaching the silver medal level. Earlier this month, UCLA tenured professor Terence Tao had just promoted the AI Mathematical Olympiad (AIMO Progress Award) with a million-dollar prize. Unexpectedly, the level of AI problem solving had improved to this level before July. Do the questions simultaneously on IMO. The most difficult thing to do correctly is IMO, which has the longest history, the largest scale, and the most negative

In 2023, almost every field of AI is evolving at an unprecedented speed. At the same time, AI is constantly pushing the technological boundaries of key tracks such as embodied intelligence and autonomous driving. Under the multi-modal trend, will the situation of Transformer as the mainstream architecture of AI large models be shaken? Why has exploring large models based on MoE (Mixed of Experts) architecture become a new trend in the industry? Can Large Vision Models (LVM) become a new breakthrough in general vision? ...From the 2023 PRO member newsletter of this site released in the past six months, we have selected 10 special interpretations that provide in-depth analysis of technological trends and industrial changes in the above fields to help you achieve your goals in the new year. be prepared. This interpretation comes from Week50 2023

Editor |ScienceAI Question Answering (QA) data set plays a vital role in promoting natural language processing (NLP) research. High-quality QA data sets can not only be used to fine-tune models, but also effectively evaluate the capabilities of large language models (LLM), especially the ability to understand and reason about scientific knowledge. Although there are currently many scientific QA data sets covering medicine, chemistry, biology and other fields, these data sets still have some shortcomings. First, the data form is relatively simple, most of which are multiple-choice questions. They are easy to evaluate, but limit the model's answer selection range and cannot fully test the model's ability to answer scientific questions. In contrast, open-ended Q&A

Editor | KX Retrosynthesis is a critical task in drug discovery and organic synthesis, and AI is increasingly used to speed up the process. Existing AI methods have unsatisfactory performance and limited diversity. In practice, chemical reactions often cause local molecular changes, with considerable overlap between reactants and products. Inspired by this, Hou Tingjun's team at Zhejiang University proposed to redefine single-step retrosynthetic prediction as a molecular string editing task, iteratively refining the target molecular string to generate precursor compounds. And an editing-based retrosynthetic model EditRetro is proposed, which can achieve high-quality and diverse predictions. Extensive experiments show that the model achieves excellent performance on the standard benchmark data set USPTO-50 K, with a top-1 accuracy of 60.8%.

Editor | ScienceAI Based on limited clinical data, hundreds of medical algorithms have been approved. Scientists are debating who should test the tools and how best to do so. Devin Singh witnessed a pediatric patient in the emergency room suffer cardiac arrest while waiting for treatment for a long time, which prompted him to explore the application of AI to shorten wait times. Using triage data from SickKids emergency rooms, Singh and colleagues built a series of AI models that provide potential diagnoses and recommend tests. One study showed that these models can speed up doctor visits by 22.3%, speeding up the processing of results by nearly 3 hours per patient requiring a medical test. However, the success of artificial intelligence algorithms in research only verifies this
