Google internal documents leaked: Open source large models are too scary, even OpenAI can't stand it!

I saw an article today, saying that Google leaked a document "We have no moat, and neither does OpenAI." It describes the views of a certain Google employee (non-Google company) on open source AI. The view is very interesting, roughly What it means is this:

After ChatGPT became popular, all major manufacturers are flocking to LLM and investing crazily.

Google is also working hard, hoping to make a comeback, but no one can win this arms race because third parties are quietly eating this big cake.

This third party is a large open source model.

Open source large models have already done this:

1. Run the basic model on Pixel 6 at a speed of 5 tokens per second.

2. You can fine-tune personalized AI on your PC in one night:

Although OpenAI and Google’s models have advantages in quality, The gap is closing at an alarming rate:

Open source models are faster, customizable, more private, and more powerful.

Open source large models use $100 and 13B parameters to do things and get it done within a few weeks; while Google struggles with $10 million and 540B parameters within a few months.

When free, unrestricted alternatives are as good as the closed model, people will definitely abandon the closed model.

It all started when Facebook open sourced LLaMA. In early March, the open source community got this truly capable basic model. Although there was no instruction, conversation tuning, or RLHF, the community immediately realized the importance of this thing. .

The subsequent innovation is simply crazy, even measured in days:

2-24: Facebook launches LLaMA, which at this time is only licensed to research institutions and used by government organizations

3-03: LLaMA was leaked on the Internet. Although it was not allowed for commercial use, suddenly anyone could play it.

3-12: Running LLaMA on Raspberry Pi is very slow and impractical

3-13: Stanford released Alpaca and added instruction tuning for LLaMA, which is even more "scary" Yes, Eric J. Wang of Stanford used an RTX 4090 graphics card to train a model equivalent to Alpaca in just 5 hours, reducing the computing power requirements of such models to consumer levels.

3-18: 5 days later, Georgi Gerganov uses 4-bit quantization technology to run LLaMA on MacBook CPU, which is the first "GPUless" solution.

3-19: Just one day later, researchers from the University of California, Berkeley, CMU, Stanford University, and the University of California, San Diego jointly launched Vicuna, which claims to have achieved more than 90% of the quality of OpenAI ChatGPT and Google Bard. It also outperforms other models such as LLaMA and Stanford Alpaca in more than 90% of cases.

3-25: Nomic created GPT4all, which is both a model and an ecosystem. For the first time we see multiple models gathered in one place

…

In just one month, instruction tuning, quantization, quality improvements, human evals, multimodality, RLHF, etc. all appeared.

More importantly, the open source community has solved the scalability problem, and the threshold for training has been lowered from a large company to one person, one night and a powerful personal computer.

So the author said at the end: OpenAI also made mistakes like us, and it cannot withstand the impact of open source. We need to build an ecosystem to make open source work for Google.

Google has applied this paradigm on Android and Chrome with great success. You should establish yourself as a leader in large model open source and continue to solidify your position as a thought leader and leader.

To be honest, the development of large language models in the past month or so has been really dazzling and overwhelming, and I am bombarded every day.

This reminds me of the early years when the Internet was just starting out. One exciting website pops up today, and another pops up tomorrow. And when the mobile Internet broke out, one app became popular today, and another app became popular tomorrow...

Personally, I don’t want these big language models to be controlled by giants. We can only “parasitize” these giant models, call their APIs, and develop some applications. This is very unpleasant. of. It is best to let a hundred flowers bloom and be accessible to the masses, so that everyone can build their own private model.

Now the cost of training should be affordable for small companies. If programmers have the ability to train, it may be a good opportunity combined with specific industries and fields.

If programmers want to be proficient in large-scale privatization models, in addition to the principles, they still have to practice by themselves. There are also dozens of people on our planet practicing in teams. Although the open source community has greatly reduced the cost, it still requires I want to train a useful model, but the requirements for the hardware environment are still too high. Graphics cards are very expensive. RTX4090 costs tens of thousands, which is painful. The price of renting a GPU for training in the cloud is even more uncontrollable. If the training fails, money will be lost. Throw it away in vain. It's not like learning a language or framework and downloading a few installation packages. It's almost zero cost.

I hope the threshold will be lowered further!

The above is the detailed content of Google internal documents leaked: Open source large models are too scary, even OpenAI can't stand it!. For more information, please follow other related articles on the PHP Chinese website!