The strongest API calling model is here! Based on LLaMA fine-tuning, the performance exceeds GPT-4

After the alpaca, there is another model named after an animal, this time it is the gorilla.

Although LLM is currently booming, making a lot of progress, and its performance in various tasks is also remarkable, the potential of these models to effectively use tools through API calls is still urgently needed. Digging.

Even for today’s most advanced LLMs, such as GPT-4, API calls are a challenging task, mainly due to their inability to generate accurate input parameters and the ease with which LLMs Hallucinations caused by incorrect use of API calls.

No, the researchers have developed Gorilla, a fine-tuned LLaMA-based model, whose performance even exceeds GPT-4 in writing API calls.

When combined with a document retriever, Gorilla also demonstrates powerful performance, making user updates or version changes more flexible.

In addition, Gorilla also greatly alleviates the hallucination problem that LLM often encounters.

To evaluate the model’s capabilities, the researchers also introduced the API Benchmark, a comprehensive dataset composed of HuggingFace, TorchHub and TensorHub APIs

Gorilla

Needless to introduce the powerful abilities of LLMs, including natural conversation ability, mathematical reasoning ability, and program synthesis ability.

However, despite its powerful performance, LLM still suffers from some limitations. Moreover, LLM also needs to be retrained to update their knowledge base and reasoning capabilities in a timely manner.

By authorizing the tools available to LLM, researchers can allow LLM to access a vast and ever-changing knowledge base to complete complex computing tasks.

By providing access to search technologies and databases, researchers can enhance LLM's ability to address larger and more dynamic knowledge spaces.

Similarly, by providing the use of calculation tools, LLM can also complete complex calculation tasks.

Therefore, technology giants have begun to try to integrate various plug-ins to enable LLM to call external tools through APIs.

The transition from a smaller, hand-coded tool to one capable of calling a vast, ever-changing cloud API space can transform LLM into computing infrastructure, and The main interface required by the network.

Tasks from booking an entire vacation to hosting a conference can become as simple as talking to an LLM with access to web APIs for flights, car rentals, hotels, dining and entertainment.

However, much prior work integrating tools into LLM considers a small set of well-documented APIs that can be easily injected into prompts.

Supporting a web-scale collection of potentially millions of changing APIs requires a rethinking of how researchers integrate tools.

It is no longer possible to describe all APIs in a single environment. Many APIs will have overlapping functionality, with subtle limitations and constraints. Simply evaluating LLM in this new environment requires new benchmarks.

In this paper, researchers explore methods for using self-structure fine-tuning and retrieval to enable LLM to accurately derive data from large, overlapping, and varying data expressed using its API and API documentation. Make selections in the toolset.

Researchers built API Bench by scraping ML APIs (models) from public model centers, a large corpus of APIs with complex and often overlapping functionality.

The researchers chose three main model hubs to build the dataset: TorchHub, TensorHub, and HuggingFace.

The researchers exhaustively included every API call in TorchHub (94 API calls) and TensorHub (696 API calls).

For HuggingFace, due to the large number of models, the researchers selected the 20 most downloaded models in each task category (925 in total).

The researchers also used Self-Instruct to generate prompts for 10 user questions for each API.

Therefore, each entry in the data set becomes an instruction reference API pair. The researchers employed common AST subtree matching techniques to evaluate the functional correctness of the generated APIs.

The researcher first parses the generated code into an AST tree, then finds a subtree whose root node is the API call the researcher cares about, and then uses this to index the researcher's data set.

Researchers check the functional correctness and hallucination issues of LLMs and provide feedback on the corresponding accuracy. The researchers then fine-tuned Gorilla, a model based on LLaMA-7B, to perform document retrieval using the researchers' data set.

The researchers found that Gorilla significantly outperformed GPT-4 in terms of API functionality accuracy and reduced illusory errors.

The researchers show an example in Figure 1.

In addition, the researchers’ retrieval-aware training of Gorilla enabled the model to adapt to changes in API documentation.

Finally, the researchers also demonstrated Gorilla’s ability to understand and reason about constraints.

In addition, Gorilla also performed well in terms of illusion.

The following figure is a comparison of accuracy and hallucination in four cases, zero samples (i.e., without any retriever) and using retrievers of BM25, GPT and Oracle.

BM25 and GPT are commonly used search engines, while the Oracle search engine will return relevant documents with 100% relevance, indicating an upper limit.

The one with higher accuracy and fewer illusions in the picture has better effect.

Across the entire dataset, Gorilla improves accuracy while reducing hallucinations.

#In order to collect the data set, the researchers carefully recorded HuggingFace’s The Model Hub, PyTorch Hub and TensorFlow Hub models All online models.

The HuggingFace platform hosts and serves a total of 203,681 models.

However, the documentation for many of these models is poor.

To filter out these low-quality models, the researchers finally selected the top 20 models from each domain.

The researchers considered 7 domains for multimodal data, 8 domains for CV, 12 domains for NLP, 5 domains for audio, 2 domains for tabular data, and 2 areas of reinforcement learning.

After filtering, the researchers obtained a total of 925 models from HuggingFace. The versions of TensorFlow Hub are divided into v1 and v2.

The latest version (v2) has a total of 801 models, and the researchers processed all models. After filtering out models with little information, 626 models remained.

Similar to TensorFlow Hub, the researchers obtained 95 models from Torch Hub.

Under the guidance of the self-instruct paradigm, the researchers adopted GPT-4 to generate synthetic instruction data.

The researchers provided three in-context examples, as well as a reference API document, and tasked the model with generating real use cases for calling the API.

The researchers specifically instructed the model not to use any API names or hints when creating instructions. The researchers built six examples (instruction-API pairs) for each of the three model hubs.

These 18 points are the only manually generated or modified data.

And Gorilla is a retrieval-aware LLaMA-7B model, specifically used for API calls.

As shown in Figure 3, the researchers used self-construction to generate {instruction, API} pairs.

To fine-tune LLaMA, the researchers converted it into a user-agent chat-style conversation, where each data point is a conversation and the user and agent take turns talking.

The researchers then performed standard instruction fine-tuning on the basic LLaMA-7B model. In experiments, the researchers trained Gorilla with and without a retriever.

In the study, the researchers focused on techniques designed to improve LLM's ability to accurately identify appropriate APIs for specific tasks - something that is critical in the development of this technology. But an aspect that is often overlooked.

Because the API functions as a universal language that enables effective communication between different systems, proper use of the API can improve LLM's ability to interact with a wider range of tools.

Gorilla outperformed state-of-the-art LLM (GPT-4) on three large-scale datasets collected by the researchers. Gorilla produces reliable ML models of API calls without hallucinations and satisfies constraints when selecting APIs.

Desiring to find a challenging dataset, the researchers chose ML APIs because of their similar functionality. A potential drawback of ML-focused APIs is that if trained on biased data, they have the potential to produce biased predictions that may disadvantage certain subgroups.

To address this concern and promote deeper understanding of these APIs, researchers are releasing a more extensive dataset that includes more than 11,000 instruction-API pairs.

In the example below, researchers used abstract syntax tree (AST) subtree matching to evaluate the correctness of API calls.

Abstract syntax tree is a tree representation of the source code structure, which helps to better analyze and understand the code.

First, the researchers built the relevant API tree from the API calls returned by Gorilla (left). This is then compared to the dataset to see if the API dataset has a subtree match.

In the above example, the matching subtree is highlighted in brown, indicating that the API call is indeed correct. Where Pretrained=True is an optional parameter.

This resource will serve the broader community as a valuable tool for studying and measuring existing APIs, contributing to more equitable and optimal use of machine learning.

The above is the detailed content of The strongest API calling model is here! Based on LLaMA fine-tuning, the performance exceeds GPT-4. For more information, please follow other related articles on the PHP Chinese website!