Beating GPT-4o in seconds, beating Llama 3 70B in 22B, Mistral AI opens its first code model

Mistral AI, the French AI unicorn benchmarking against OpenAI, has made a new move: Codestral, the first large code model, was born.

As an open generative AI model designed specifically for code generation tasks, Codestral helps developers write and Interact with the code. Codestral's proficiency in coding and English allows software developers to design advanced AI applications.

Codestral’s parameter size is 22B and follows the new Mistral AI Non-Production License. It can be used for research and testing purposes, but commercial use is prohibited.

Currently, the model is available for download on HuggingFace.

• ##Download address: https://huggingface .co/mistralai/Codestral-22B-v0.1
• Free trial address: https://t.co/LsgC84GCYw

Guillaume Lample, co-founder and chief scientist of Mistral AI, said that Codestral can be easily integrated into the VS Code plug-in.

Some users compared Codestral with GPT-4o, and Codestral was directly faster than GPT-4o.

Proficient in 80 Programming Languages

Codestral in a diverse dataset of 80 programming languages Online training, including Python, Java, C, C, JavaScript, Bash and other popular programming languages. It also performs well on programming languages ​​such as Swift and Fortran.

Thus, a broad language base ensures that Codestral can help developers in a variety of coding environments and projects.

Codestral can competently write code, write tests and use the fill-in-the-middle mechanism to complete any code part, saving developers time and energy. Using Codestral at the same time can also help improve developers' coding skills and reduce the risk of errors and bugs.

New standard for code generation performance

As a 22B parameter model, Codestral has better code generation performance than previous large code models. and latency headroom set new standards.

As you can see from Figure 1 below, the context window length of Codestral is 32k, the competing product CodeLlama 70B is 4k, DeepSeek Coder 33B is 16k, and Llama 3 70B is 8k. Results show that Codestral outperforms other models on the code generation remote evaluation benchmark RepoBench.

Mistral AI also compared Codestral to existing code-specific models, which require higher hardware requirements.

Performance on Python. The researchers used the HumanEval pass@1 and MBPP sanitised pass@1 benchmarks to evaluate Codestral's Python code generation capabilities; in addition, the researchers also used CruxEval and RepoBench EM benchmark evaluations.

Performance on SQL. To evaluate the performance of Codestral in SQL, the researchers used the Spider benchmark.

Performance on other programming languages. The researchers also evaluated Codestral in six other programming languages, including C, bash, Java, PHP, Typescript, and C#, and calculated the average of these evaluations.

FIM Benchmark. The researchers also evaluated Codestral's ability to complete code when there are gaps in the code fragments, mainly conducting experiments on Python, JavaScript and Java. The results showed that users can run the code completed by Codestral immediately.

Blog address: https://mistral.ai/news/codestral/

The above is the detailed content of Beating GPT-4o in seconds, beating Llama 3 70B in 22B, Mistral AI opens its first code model. For more information, please follow other related articles on the PHP Chinese website!