Can BERT also be used on CNN? ByteDance's research results selected for ICLR 2023 Spotlight

​

How to run BERT on a convolutional neural network?

You can directly use SparK - Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling proposed by the ByteDance technical team. Recently, it has been included as a Spotlight focus paper by the top artificial intelligence conference : ​

##Paper link: ​

​

​https://www.php.cn/link/e38e37a99f7de1f45d169efcdb288dd1​​​

Open source code: ​##​

​

https://www.php.cn/link/9dfcf16f0adbc5e2a55ef02db36bac7f​​​

This is also BERT’s first success in convolutional neural networks (CNN)

##. Let’s first feel the performance of SparK in pre-training. ​Enter an incomplete picture:

​

Restore a puppy:

​

Another A mutilated picture:

​

It turns out to be a bagel sandwich:

​

Other scenes can also achieve picture restoration:

​

The perfect match between BERT and Transformer

​“

Any great actions and thoughts, They all have a humble beginning.”​Behind the BERT pre-training algorithm is a simple and profound design . BERT uses "cloze": randomly delete several words in a sentence and let the model learn to recover.

​

BERT relies heavily on the

core model in the NLP field - Transformer. ​Transformer is naturally suitable for processing variable-length sequence data (such as an English sentence), so it can easily cope with BERT's "random deletion" of cloze ".

CNN in the visual field also wants to enjoy BERT: What are the two challenges?

​

Looking back at the development history of computer vision, Convolutional neural network model condenses the essence of many classic models such as translational equivariance, multi-scale structure, etc. , can be described as the mainstay of the CV world. But what is very different from Transformer is that CNN is inherently unable to adapt to data that is "hollowed out" by cloze and full of "random holes", so it cannot enjoy the dividends of BERT pre-training at first glance. ​

## Above picture a. What is shown is the work of MAE (Masked Autoencoders are Scalable Visual Learners). Since it uses the Transformer model instead of the CNN model, it can flexibly cope with inputs with holes, which is a "natural match" with BERT.

​

The picture on the right b. shows a rough way to fuse the BERT and CNN models - that is, "blacken" all the empty areas, and When this "black mosaic" image is input into CNN, the result can be imagined, which will cause serious pixel intensity distribution shift problem and lead to poor performance (verified later). This is the

challenge that hinders the successful application of BERT on CNN. #In addition, the author team also pointed out that the BERT algorithm originating from the field of NLP naturally does not have the characteristics of "multi-scale", and the multi-scale pyramid structure It can be called the "gold standard" in the long history of computer vision. The conflict between single-scale BERT and natural multi-scale CNN is

Challenge 2. ​Solution SparK: Sparse and Hierarchical Mask Modeling

​

##The author team proposed SparK (Sparse and hierarchical masKed modeling) to solve the previous two problems a challenge.

​

First, inspired by three-dimensional point cloud data processing, the author team proposed to treat the fragmented images after masking operation (hollowing operation) as Sparse point clouds are encoded using Submanifold Sparse Convolution. This allows the convolutional network to handle randomly deleted images easily.

Secondly, inspired by the elegant design of UNet, the author team naturally designed an encoder-decoder model with horizontal connections, allowing Multi-scale features flow between multiple levels of the model, allowing BERT to fully embrace the multi-scale gold standard of computer vision. ​

At this point, SparK, a sparse, multi-scale mask modeling algorithm tailored for convolutional networks (CNN), was born. ​

SparK is

general:

It can Can be directly used on any convolutional network without any modification to their structure or the introduction of any additional components - whether it is the familiar classic ResNet or the recent advanced model ConvNeXt, you can directly benefit from SparK . ​From ResNet to ConvNeXt: Performance improvements in three major visual tasks

​

The author team selected two representative convolution model families, ResNet and ConvNeXt, and conducted performance tests on image classification, target detection, and instance segmentation tasks. ​

On the classic ResNet-50 model, SparK serves as the only generative pre-training,

has achieved State-of-the-art level:

On the ConvNeXt model, SparK still leads . Before pre-training, ConvNeXt and Swin-Transformer were evenly matched; after pre-training, ConvNeXt overwhelmingly surpassed Swin-Transformer in three tasks: ​

When verifying SparK on the complete model family from small to large, you can observe: ​

##No matter Models big or small, new or old, can all benefit from SparK, and as the model size/training overhead increases, the increase is even higher, reflecting the scaling capability of the SparK algorithm: ​

##Finally, the author team also designed a confirmatory ablation experiment, from which we can see

Sparse MaskandHierarchical Structure Lines 3 and 4 Lines) are very critical designs. Once missing, it will cause serious performance degradation: ​

The above is the detailed content of Can BERT also be used on CNN? ByteDance's research results selected for ICLR 2023 Spotlight. For more information, please follow other related articles on the PHP Chinese website!