


The cost of training large models has been reduced by nearly half! National University of Singapore's latest optimizer has been put into use
The optimizer occupies a large amount of memory resources in the training of large language models.
Now there is a new optimization method that reduces memory consumption by half while maintaining performance.
This result was created by the National University of Singapore. It won the Outstanding Paper Award at the ACL conference and has been put into practical application.
Picture
As the number of parameters of large language models continues to increase, the problem of memory consumption during training becomes more severe.
The research team proposed the CAME optimizer, which has the same performance as Adam while reducing memory consumption.
Picture
The CAME optimizer has achieved the same or even better training performance than the Adam optimizer in the pre-training of multiple commonly used large-scale language models. And it shows stronger robustness to large batch pre-training scenarios.
Furthermore, training large language models through the CAME optimizer can significantly reduce the cost of large model training.
Implementation method
The CAME optimizer is improved based on the Adafactor optimizer, which often causes a loss of training performance in the pre-training tasks of large-scale language models.
The non-negative matrix decomposition operation in Adafactor will inevitably produce errors in the training of deep neural networks, and the correction of these errors is the source of performance loss.
Through comparison, it is found that when the difference between the starting value mt and the current value t is small, the confidence level of mt is higher .
Picture
Inspired by this, the team proposed a new optimization algorithm.
The blue part in the picture below is the increased part of CAME compared to Adafactor.
Picture
The CAME optimizer performs update amount correction based on the confidence of the model update, and at the same time performs a non-negative matrix decomposition operation on the introduced confidence matrix.
In the end, CAME successfully obtained the effect of Adam with the consumption of Adafactor.
The same effect only consumes half the resources
The team used CAME to train BERT, GPT-2 and T5 models respectively.
The previously commonly used Adam (better effect) and Adafactor (lower consumption) are the references for measuring CAME performance.
Among them, in the process of training BERT, CAME achieved the same accuracy as Adafaactor in only half the number of steps.
△The left side is 8K scale, the right side is 32K scale
For GPT-2, from the perspective of loss and confusion, CAME’s The performance is very close to Adam.
In the training of the T5 model, CAME also showed similar results.
As for the fine-tuning of the model, CAME’s performance in accuracy is not inferior to the benchmark.
In terms of resource consumption, when using PyTorch to train BERT with 4B data volume, the memory resources consumed by CAME were reduced by nearly half compared with the baseline.
Team Profile
The HPC-AI Laboratory of the National University of Singapore is a high-performance computing and artificial intelligence laboratory led by Professor You Yang.
The laboratory is committed to the research and innovation of high-performance computing, machine learning systems and distributed parallel computing, and promotes applications in fields such as large-scale language models.
The head of the laboratory, You Yang, is the President Young Professor(Presidential Young Professor) of the Department of Computer Science at the National University of Singapore.
You Yang was selected into the Forbes Under 30 Elite List (Asia) in 2021 and won the IEEE-CS Supercomputing Outstanding Newcomer Award. His current research focus is on the distributed optimization of large-scale deep learning training algorithms.
Luo Yang, the first author of this article, is a master's student in the laboratory. His current research focus is on the stability and efficient training of large model training.
Paper address: https://arxiv.org/abs/2307.02047
GitHub项目页:https://github.com/huawei-noah/Pretrained-Language-Model/tree/master/CAME
The above is the detailed content of The cost of training large models has been reduced by nearly half! National University of Singapore's latest optimizer has been put into use. For more information, please follow other related articles on the PHP Chinese website!

Hot AI Tools

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Undress AI Tool
Undress images for free

Clothoff.io
AI clothes remover

AI Hentai Generator
Generate AI Hentai for free.

Hot Article

Hot Tools

Notepad++7.3.1
Easy-to-use and free code editor

SublimeText3 Chinese version
Chinese version, very easy to use

Zend Studio 13.0.1
Powerful PHP integrated development environment

Dreamweaver CS6
Visual web development tools

SublimeText3 Mac version
God-level code editing software (SublimeText3)

Hot Topics

0.What does this article do? We propose DepthFM: a versatile and fast state-of-the-art generative monocular depth estimation model. In addition to traditional depth estimation tasks, DepthFM also demonstrates state-of-the-art capabilities in downstream tasks such as depth inpainting. DepthFM is efficient and can synthesize depth maps within a few inference steps. Let’s read about this work together ~ 1. Paper information title: DepthFM: FastMonocularDepthEstimationwithFlowMatching Author: MingGui, JohannesS.Fischer, UlrichPrestel, PingchuanMa, Dmytr

Imagine an artificial intelligence model that not only has the ability to surpass traditional computing, but also achieves more efficient performance at a lower cost. This is not science fiction, DeepSeek-V2[1], the world’s most powerful open source MoE model is here. DeepSeek-V2 is a powerful mixture of experts (MoE) language model with the characteristics of economical training and efficient inference. It consists of 236B parameters, 21B of which are used to activate each marker. Compared with DeepSeek67B, DeepSeek-V2 has stronger performance, while saving 42.5% of training costs, reducing KV cache by 93.3%, and increasing the maximum generation throughput to 5.76 times. DeepSeek is a company exploring general artificial intelligence

Earlier this month, researchers from MIT and other institutions proposed a very promising alternative to MLP - KAN. KAN outperforms MLP in terms of accuracy and interpretability. And it can outperform MLP running with a larger number of parameters with a very small number of parameters. For example, the authors stated that they used KAN to reproduce DeepMind's results with a smaller network and a higher degree of automation. Specifically, DeepMind's MLP has about 300,000 parameters, while KAN only has about 200 parameters. KAN has a strong mathematical foundation like MLP. MLP is based on the universal approximation theorem, while KAN is based on the Kolmogorov-Arnold representation theorem. As shown in the figure below, KAN has

Boston Dynamics Atlas officially enters the era of electric robots! Yesterday, the hydraulic Atlas just "tearfully" withdrew from the stage of history. Today, Boston Dynamics announced that the electric Atlas is on the job. It seems that in the field of commercial humanoid robots, Boston Dynamics is determined to compete with Tesla. After the new video was released, it had already been viewed by more than one million people in just ten hours. The old people leave and new roles appear. This is a historical necessity. There is no doubt that this year is the explosive year of humanoid robots. Netizens commented: The advancement of robots has made this year's opening ceremony look like a human, and the degree of freedom is far greater than that of humans. But is this really not a horror movie? At the beginning of the video, Atlas is lying calmly on the ground, seemingly on his back. What follows is jaw-dropping

I cry to death. The world is madly building big models. The data on the Internet is not enough. It is not enough at all. The training model looks like "The Hunger Games", and AI researchers around the world are worrying about how to feed these data voracious eaters. This problem is particularly prominent in multi-modal tasks. At a time when nothing could be done, a start-up team from the Department of Renmin University of China used its own new model to become the first in China to make "model-generated data feed itself" a reality. Moreover, it is a two-pronged approach on the understanding side and the generation side. Both sides can generate high-quality, multi-modal new data and provide data feedback to the model itself. What is a model? Awaker 1.0, a large multi-modal model that just appeared on the Zhongguancun Forum. Who is the team? Sophon engine. Founded by Gao Yizhao, a doctoral student at Renmin University’s Hillhouse School of Artificial Intelligence.

AI is indeed changing mathematics. Recently, Tao Zhexuan, who has been paying close attention to this issue, forwarded the latest issue of "Bulletin of the American Mathematical Society" (Bulletin of the American Mathematical Society). Focusing on the topic "Will machines change mathematics?", many mathematicians expressed their opinions. The whole process was full of sparks, hardcore and exciting. The author has a strong lineup, including Fields Medal winner Akshay Venkatesh, Chinese mathematician Zheng Lejun, NYU computer scientist Ernest Davis and many other well-known scholars in the industry. The world of AI has changed dramatically. You know, many of these articles were submitted a year ago.

What? Is Zootopia brought into reality by domestic AI? Exposed together with the video is a new large-scale domestic video generation model called "Keling". Sora uses a similar technical route and combines a number of self-developed technological innovations to produce videos that not only have large and reasonable movements, but also simulate the characteristics of the physical world and have strong conceptual combination capabilities and imagination. According to the data, Keling supports the generation of ultra-long videos of up to 2 minutes at 30fps, with resolutions up to 1080p, and supports multiple aspect ratios. Another important point is that Keling is not a demo or video result demonstration released by the laboratory, but a product-level application launched by Kuaishou, a leading player in the short video field. Moreover, the main focus is to be pragmatic, not to write blank checks, and to go online as soon as it is released. The large model of Ke Ling is already available in Kuaiying.

Recently, the military circle has been overwhelmed by the news: US military fighter jets can now complete fully automatic air combat using AI. Yes, just recently, the US military’s AI fighter jet was made public for the first time and the mystery was unveiled. The full name of this fighter is the Variable Stability Simulator Test Aircraft (VISTA). It was personally flown by the Secretary of the US Air Force to simulate a one-on-one air battle. On May 2, U.S. Air Force Secretary Frank Kendall took off in an X-62AVISTA at Edwards Air Force Base. Note that during the one-hour flight, all flight actions were completed autonomously by AI! Kendall said - "For the past few decades, we have been thinking about the unlimited potential of autonomous air-to-air combat, but it has always seemed out of reach." However now,
