How to improve model efficiency with limited resources? An article summarizing efficient NLP methods

Training increasingly larger deep learning models has become an emerging trend in the past decade. As shown in the figure below, the continuous increase in the number of model parameters makes the performance of neural networks better and better, and also generates some new research directions, but there are also more and more problems with the model.

#First of all, this type of model often has limited access and is not open source, or even if it is open source, it still requires a lot of computing resources to run. Second, the parameters of these network models are not universal, so a large amount of resources are required for training and derivation. Third, the model cannot be expanded indefinitely because the size of the parameters is limited by the hardware. To address these issues, a new research trend focusing on improving efficiency is emerging.

Recently, more than a dozen researchers from Hebrew University, University of Washington and other institutions jointly wrote a review summarizing efficient methods in the field of natural language processing (NLP).

Paper address: https://arxiv.org/pdf/2209.00099.pdf

Efficiency usually refers to the input system The relationship between resources and system output. An efficient system can produce output without wasting resources. In the field of NLP, we think of efficiency as the relationship between the cost of a model and the results it produces.

Equation (1) describes the training cost (Cost) of an artificial intelligence model to produce a certain result (R) and three (incomplete) factors Proportional to:

(1) The cost of executing the model on a single sample (E);

(2) The size of the training data set (D);

(3) Number of training runs (H) required for model selection or parameter adjustment.

Cost(·) can then be measured along multiple dimensions, as each of computational, time or environmental costs can be further quantified in a variety of ways. For example, the computational cost can include the total number of floating point operations (FLOPs) or the number of model parameters. Since using a single cost metric can be misleading, this study collects and organizes work on multiple aspects of efficient NLP and discusses which aspects are beneficial for which use cases.

This study aims to give a basic introduction to a wide range of methods to improve NLP efficiency. Therefore, this study organizes this survey according to the typical NLP model pipeline (Figure 2 below) and introduces how to make each stage More efficient existing methods.

This work provides a practical efficiency guide for NLP researchers, mainly for two types of readers:

(1 ) researchers from various fields of NLP to help them work in resource-limited environments: depending on resource bottlenecks, readers can jump directly to an aspect covered by the NLP pipeline. For example, if the main limitation is inference time, Chapter 6 of the paper describes related efficiency improvements.

(2) Researchers interested in improving the current efficiency of NLP methods. This paper can serve as an entry point to identify opportunities for new research directions.

Figure 3 below outlines the efficient NLP method summarized in this study.

In addition, although the choice of hardware has a large impact on the efficiency of the model, most NLP researchers do not directly control decisions about hardware, and Most hardware optimizations are useful for all stages in the NLP pipeline. Therefore, this study focuses the work on algorithms but provides a brief introduction to hardware optimization in Chapter 7. Finally, the paper further discusses how to quantify efficiency, what factors should be considered during the evaluation process, and how to decide on the most appropriate model.

Interested readers can read the original text of the paper to learn more research details.

The above is the detailed content of How to improve model efficiency with limited resources? An article summarizing efficient NLP methods. For more information, please follow other related articles on the PHP Chinese website!