Home Technology peripherals AI Microsoft Research Asia launches TinyMIM: improving the performance of small ViT through knowledge distillation

Microsoft Research Asia launches TinyMIM: improving the performance of small ViT through knowledge distillation

Sep 09, 2023 pm 10:37 PM
theory Microsoft Research Asia knowledge distillation

Re-expressed: Research motivation


Mask modeling (MIM, MAE) has proven to be a very effective self-supervised training method. However, as shown in Figure 1, MIM works relatively better for larger models. When the model is very small (such as ViT-T 5M parameters, such a model is very important for the real world), MIM may even reduce the effect of the model to a certain extent. For example, the classification effect of ViT-L trained with MAE on ImageNet is 3.3% higher than that of the model trained with ordinary supervision, but the classification effect of ViT-T trained with MAE on ImageNet is 0.6% lower than that of the model trained with ordinary supervision.

In this work we proposed TinyMIM, which uses Distillation methods transfer knowledge from large models to small models.

Microsoft Research Asia launches TinyMIM: improving the performance of small ViT through knowledge distillation



  • ##Paper address :https://arxiv.org/pdf/2301.01296.pdf
  • Code address: https://github.com/OliverRensu/TinyMIM

We systematically studied the impact of distillation objectives, data enhancement, regularization, auxiliary loss functions, etc. on distillation. In the case of strictly using only ImageNet-1K as training data (including the Teacher model also only using ImageNet-1K training) and ViT-B as the model, our method achieves the best performance currently. As shown in the picture:

Microsoft Research Asia launches TinyMIM: improving the performance of small ViT through knowledge distillation



#put our The method (TinyMIM) is compared with the mask reconstruction-based method MAE, and the supervised learning method DeiT trained from scratch. MAE has significant performance improvements when the model is relatively large, but when the model is relatively small, the improvement is limited and may even harm the final effect of the model. Our method, TinyMIM, achieves substantial improvements across different model sizes.

Our contributions are as follows:

1. Distillation targets: 1) Distillation token The relationship between them is more effective than distilling class tokens or feature maps alone; 2) It is more effective to use the middle layer as the target of distillation.
2. Data enhancement and model regularization (Data and network regularization): 1) The effect of using masked images is worse; 2) The student model requires a drop path, but the teacher model does not.
3. Auxiliary losses: MIM is meaningless as an auxiliary loss function.
4. Macro distillation strategy: We found that serialized distillation (ViT-B -> ViT-S -> ViT-T) works best.

2. Method

# #


We systematically investigated the distillation goals, input images, and distillation target modules.

2.1 Factors affecting the distillation effect

1) Features:

a. Intermediate block features and output features

Microsoft Research Asia launches TinyMIM: improving the performance of small ViT through knowledge distillation



##When i=L, it refers to the characteristics of the Transformer output layer. When i

b. Attention (Attention) features and feed-forward layer (FFN) layer features

Microsoft Research Asia launches TinyMIM: improving the performance of small ViT through knowledge distillation



Transformer Each block has an Attention layer and a FFN layer. Different distillation layers will have different effects.

c.QKV Features

Microsoft Research Asia launches TinyMIM: improving the performance of small ViT through knowledge distillation



##There will be Q, K, and V features in the Attention layer. These features are used to calculate the attention mechanism. We have also investigated direct distillation of these features.

2) Relationship

Microsoft Research Asia launches TinyMIM: improving the performance of small ViT through knowledge distillation


##Q, K, V are used to calculate the attention map, and the relationship between these features can also be used as the target of knowledge distillation.
#3) Input: whether to mask
Traditional knowledge distillation is to directly input the complete image. Our method is to explore the distillation mask modeling model, so we also explore whether masked images are suitable as input for knowledge distillation.

2.2 Comparison of knowledge distillation methods
1) Class Token distillation:
The simplest method is to directly distill the class token of the MAE pre-trained model similar to DeiT:

Microsoft Research Asia launches TinyMIM: improving the performance of small ViT through knowledge distillation


where
refers to the class token of the student model, and
refers to the class token of the teacher model.
Microsoft Research Asia launches TinyMIM: improving the performance of small ViT through knowledge distillationMicrosoft Research Asia launches TinyMIM: improving the performance of small ViT through knowledge distillation
2) Feature distillation: We directly refer to feature distillation [1] as a comparison

Microsoft Research Asia launches TinyMIM: improving the performance of small ViT through knowledge distillation



Microsoft Research Asia launches TinyMIM: improving the performance of small ViT through knowledge distillation

##3) Relational Distillation: We proposed also The default distillation strategy of this article


Microsoft Research Asia launches TinyMIM: improving the performance of small ViT through knowledge distillation


## 3. Experiment

3.1 Main experimental results

Our method is Pre-trained on ImageNet-1K, and the teacher model is also pre-trained on ImageNet-1K. We then fine-tuned our pre-trained model on downstream tasks (classification, semantic segmentation). The model performance is as shown in the figure:



Our method significantly outperforms previous MAE-based methods, especially for small models. Specifically, for the ultra-small model ViT-T, our method achieves a classification accuracy of 75.8%, an improvement of 4.2 compared to the MAE baseline model. For the small model ViT-S, we achieve 83.0% classification accuracy, an improvement of 1.4 over the previous best method. For Base-sized models, our method outperforms the MAE baseline model and the previous best model by CAE 4.1 and 2.0, respectively.

At the same time, we also tested the robustness of the model, as shown in the figure:

Microsoft Research Asia launches TinyMIM: improving the performance of small ViT through knowledge distillation



Compared with MAE-B, TinyMIM-B improved by 6.4 and 4.6 in ImageNet-A and ImageNet-R respectively.

3.2 Ablation experiment

1) Distillation of different relationships

Microsoft Research Asia launches TinyMIM: improving the performance of small ViT through knowledge distillation



## Distills the QK, VV relationship at the same time and has Softmax when calculating the relationship Achieved the best results.

2) Different distillation strategies

Microsoft Research Asia launches TinyMIM: improving the performance of small ViT through knowledge distillation



TinyMIM This method of distilling relationships achieves better results than the MAE baseline model, class token distillation, and feature map distillation, and this is true for models of various sizes. .

3) Distillation middle layer

Microsoft Research Asia launches TinyMIM: improving the performance of small ViT through knowledge distillation



We found that the eighteenth layer of distillation achieved the best results.

4. Conclusion

In this article, we proposed TinyMIM, which is The first model to successfully enable small models to benefit from mask reconstruction modeling (MIM) pre-training. Instead of adopting mask reconstruction as the task, we pre-train the small model by training the small model to simulate the relationships of the large model in a knowledge distillation manner. TinyMIM's success can be attributed to a comprehensive study of various factors that may affect TinyMIM pre-training, including distillation targets, distillation inputs, and intermediate layers. Through extensive experiments, we conclude that relation distillation is superior to feature distillation and class label distillation, etc. With its simplicity and powerful performance, we hope that our method will provide a solid foundation for future research.

[1] Wei, Y., Hu, H., Xie, Z., Zhang, Z., Cao, Y., Bao, J. , ... & Guo, B. (2022). Contrastive learning rivals masked image modeling in fine-tuning via feature distillation. arXiv preprint arXiv:2205.14141.

The above is the detailed content of Microsoft Research Asia launches TinyMIM: improving the performance of small ViT through knowledge distillation. For more information, please follow other related articles on the PHP Chinese website!

Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

Video Face Swap

Video Face Swap

Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Tools

Notepad++7.3.1

Notepad++7.3.1

Easy-to-use and free code editor

SublimeText3 Chinese version

SublimeText3 Chinese version

Chinese version, very easy to use

Zend Studio 13.0.1

Zend Studio 13.0.1

Powerful PHP integrated development environment

Dreamweaver CS6

Dreamweaver CS6

Visual web development tools

SublimeText3 Mac version

SublimeText3 Mac version

God-level code editing software (SublimeText3)

Breaking through the boundaries of traditional defect detection, 'Defect Spectrum' achieves ultra-high-precision and rich semantic industrial defect detection for the first time. Breaking through the boundaries of traditional defect detection, 'Defect Spectrum' achieves ultra-high-precision and rich semantic industrial defect detection for the first time. Jul 26, 2024 pm 05:38 PM

In modern manufacturing, accurate defect detection is not only the key to ensuring product quality, but also the core of improving production efficiency. However, existing defect detection datasets often lack the accuracy and semantic richness required for practical applications, resulting in models unable to identify specific defect categories or locations. In order to solve this problem, a top research team composed of Hong Kong University of Science and Technology Guangzhou and Simou Technology innovatively developed the "DefectSpectrum" data set, which provides detailed and semantically rich large-scale annotation of industrial defects. As shown in Table 1, compared with other industrial data sets, the "DefectSpectrum" data set provides the most defect annotations (5438 defect samples) and the most detailed defect classification (125 defect categories

NVIDIA dialogue model ChatQA has evolved to version 2.0, with the context length mentioned at 128K NVIDIA dialogue model ChatQA has evolved to version 2.0, with the context length mentioned at 128K Jul 26, 2024 am 08:40 AM

The open LLM community is an era when a hundred flowers bloom and compete. You can see Llama-3-70B-Instruct, QWen2-72B-Instruct, Nemotron-4-340B-Instruct, Mixtral-8x22BInstruct-v0.1 and many other excellent performers. Model. However, compared with proprietary large models represented by GPT-4-Turbo, open models still have significant gaps in many fields. In addition to general models, some open models that specialize in key areas have been developed, such as DeepSeek-Coder-V2 for programming and mathematics, and InternVL for visual-language tasks.

Training with millions of crystal data to solve the crystallographic phase problem, the deep learning method PhAI is published in Science Training with millions of crystal data to solve the crystallographic phase problem, the deep learning method PhAI is published in Science Aug 08, 2024 pm 09:22 PM

Editor |KX To this day, the structural detail and precision determined by crystallography, from simple metals to large membrane proteins, are unmatched by any other method. However, the biggest challenge, the so-called phase problem, remains retrieving phase information from experimentally determined amplitudes. Researchers at the University of Copenhagen in Denmark have developed a deep learning method called PhAI to solve crystal phase problems. A deep learning neural network trained using millions of artificial crystal structures and their corresponding synthetic diffraction data can generate accurate electron density maps. The study shows that this deep learning-based ab initio structural solution method can solve the phase problem at a resolution of only 2 Angstroms, which is equivalent to only 10% to 20% of the data available at atomic resolution, while traditional ab initio Calculation

Google AI won the IMO Mathematical Olympiad silver medal, the mathematical reasoning model AlphaProof was launched, and reinforcement learning is so back Google AI won the IMO Mathematical Olympiad silver medal, the mathematical reasoning model AlphaProof was launched, and reinforcement learning is so back Jul 26, 2024 pm 02:40 PM

For AI, Mathematical Olympiad is no longer a problem. On Thursday, Google DeepMind's artificial intelligence completed a feat: using AI to solve the real question of this year's International Mathematical Olympiad IMO, and it was just one step away from winning the gold medal. The IMO competition that just ended last week had six questions involving algebra, combinatorics, geometry and number theory. The hybrid AI system proposed by Google got four questions right and scored 28 points, reaching the silver medal level. Earlier this month, UCLA tenured professor Terence Tao had just promoted the AI ​​Mathematical Olympiad (AIMO Progress Award) with a million-dollar prize. Unexpectedly, the level of AI problem solving had improved to this level before July. Do the questions simultaneously on IMO. The most difficult thing to do correctly is IMO, which has the longest history, the largest scale, and the most negative

Nature's point of view: The testing of artificial intelligence in medicine is in chaos. What should be done? Nature's point of view: The testing of artificial intelligence in medicine is in chaos. What should be done? Aug 22, 2024 pm 04:37 PM

Editor | ScienceAI Based on limited clinical data, hundreds of medical algorithms have been approved. Scientists are debating who should test the tools and how best to do so. Devin Singh witnessed a pediatric patient in the emergency room suffer cardiac arrest while waiting for treatment for a long time, which prompted him to explore the application of AI to shorten wait times. Using triage data from SickKids emergency rooms, Singh and colleagues built a series of AI models that provide potential diagnoses and recommend tests. One study showed that these models can speed up doctor visits by 22.3%, speeding up the processing of results by nearly 3 hours per patient requiring a medical test. However, the success of artificial intelligence algorithms in research only verifies this

To provide a new scientific and complex question answering benchmark and evaluation system for large models, UNSW, Argonne, University of Chicago and other institutions jointly launched the SciQAG framework To provide a new scientific and complex question answering benchmark and evaluation system for large models, UNSW, Argonne, University of Chicago and other institutions jointly launched the SciQAG framework Jul 25, 2024 am 06:42 AM

Editor |ScienceAI Question Answering (QA) data set plays a vital role in promoting natural language processing (NLP) research. High-quality QA data sets can not only be used to fine-tune models, but also effectively evaluate the capabilities of large language models (LLM), especially the ability to understand and reason about scientific knowledge. Although there are currently many scientific QA data sets covering medicine, chemistry, biology and other fields, these data sets still have some shortcomings. First, the data form is relatively simple, most of which are multiple-choice questions. They are easy to evaluate, but limit the model's answer selection range and cannot fully test the model's ability to answer scientific questions. In contrast, open-ended Q&A

PRO | Why are large models based on MoE more worthy of attention? PRO | Why are large models based on MoE more worthy of attention? Aug 07, 2024 pm 07:08 PM

In 2023, almost every field of AI is evolving at an unprecedented speed. At the same time, AI is constantly pushing the technological boundaries of key tracks such as embodied intelligence and autonomous driving. Under the multi-modal trend, will the situation of Transformer as the mainstream architecture of AI large models be shaken? Why has exploring large models based on MoE (Mixed of Experts) architecture become a new trend in the industry? Can Large Vision Models (LVM) become a new breakthrough in general vision? ...From the 2023 PRO member newsletter of this site released in the past six months, we have selected 10 special interpretations that provide in-depth analysis of technological trends and industrial changes in the above fields to help you achieve your goals in the new year. be prepared. This interpretation comes from Week50 2023

The accuracy rate reaches 60.8%. Zhejiang University's chemical retrosynthesis prediction model based on Transformer was published in the Nature sub-journal The accuracy rate reaches 60.8%. Zhejiang University's chemical retrosynthesis prediction model based on Transformer was published in the Nature sub-journal Aug 06, 2024 pm 07:34 PM

Editor | KX Retrosynthesis is a critical task in drug discovery and organic synthesis, and AI is increasingly used to speed up the process. Existing AI methods have unsatisfactory performance and limited diversity. In practice, chemical reactions often cause local molecular changes, with considerable overlap between reactants and products. Inspired by this, Hou Tingjun's team at Zhejiang University proposed to redefine single-step retrosynthetic prediction as a molecular string editing task, iteratively refining the target molecular string to generate precursor compounds. And an editing-based retrosynthetic model EditRetro is proposed, which can achieve high-quality and diverse predictions. Extensive experiments show that the model achieves excellent performance on the standard benchmark data set USPTO-50 K, with a top-1 accuracy of 60.8%.

See all articles