NUS and Byte collaborated cross-industry to achieve 72 times faster training through model optimization, and won the AAAI2023 Outstanding Paper.

Recently, the top international artificial intelligence conference AAAI 2023 announced the selection results. The CowClip technical paper collaborated by the National University of Singapore (NUS) and ByteDance Machine Learning Team (AML) was shortlisted for Distinguished Papers. CowClip is a model training optimization strategy that can increase model training speed on a single GPU by 72 times while ensuring model accuracy. The relevant code is now open source. ​

Paper address: https://arxiv.org/abs/ 2204.06240​

Open source address: https://github.com/bytedance/LargeBatchCTR​

AAAI is an annual conference hosted by the International Association for the Advancement of Artificial Intelligence. It is one of the oldest top academic conferences in the field of artificial intelligence. AAAI 2023 received a total of 8777 paper submissions, of which 1721 papers were accepted, with an acceptance rate of 19.6%. The Department of Computer Science of Oxford University won the highest award of the conference (Outstanding Paper Award), and the collaborative paper from Peking University and other institutions won the Outstanding Student Paper Award. In addition, the conference also selected 12 Distinguished Papers, covering multiple directions such as model training strategy optimization, graph neural network optimization, and neural architecture search.

#How to improve model training speed is an eternal topic in the field of machine learning. Since Google proposed the first pre-trained large model BERT in 2018, large model training has gradually become a trend and trend in the field of deep learning. However, the increasing size of the model also means that a complete training will take a lot of time and computational cost. According to information previously released by Google, when training the 11 billion parameter T5 (the pre-trained model launched by Google in 2019) variant, the single running cost exceeds US$1.3 million. ​

The CowClip model training optimization strategy selected as an outstanding paper can achieve more sufficient GPU performance by ensuring the model accuracy of a larger batch size (batch size) Excavate to achieve the purpose of increasing training speed. Experiments show that the model trained using CowClip not only has higher accuracy than other methods, but also greatly improves the training speed. Training the Deep FM model on a single GPU can reduce the training time from 12 hours to 10 minutes based on the data of the public data set. The model training speed is increased by 72 times at one time. ​

With efficient computing and more accurate analysis and decision-making capabilities, artificial intelligence technology is increasingly being used in medical care, finance, manufacturing, and education and e-commerce and other fields, and the accuracy and efficiency of model training will continue to become key factors affecting the development of the artificial intelligence industry.

#According to reports, Bytedance Machine Learning Team (AML) has implemented CowClip’s core technology in some of the company’s businesses. The team provides machine learning middle-end services for the company, including large-scale training systems and inference systems for business scenarios such as recommendation, advertising, and search for products such as Toutiao, Douyin, and Xigua Video, and provides simple and easy-to-use services to enterprise customers through the Volcano Engine. Easy-to-use, stable and reliable machine learning platform. ​

The above is the detailed content of NUS and Byte collaborated cross-industry to achieve 72 times faster training through model optimization, and won the AAAI2023 Outstanding Paper.. For more information, please follow other related articles on the PHP Chinese website!