Tsinghua Microsoft open sourced a new prompt word compression tool, the length dropped by 80%! GitHub gets 3.1K stars

In natural language processing, a lot of information is actually repeated.

If the prompt words can be effectively compressed, it is equivalent to expanding the length of the context supported by the model to some extent.

Existing information entropy methods reduce this redundancy by removing certain words or phrases.

However, the calculation based on information entropy only covers the one-way context of the text and may ignore key information required for compression; moreover, the calculation method of information entropy is not fully consistent with the compression tips the actual purpose of the word.

To meet these challenges, researchers from Tsinghua University and Microsoft jointly proposed a new data processing process called LLMLingua-2. It aims to extract knowledge from large language models (LLM) and achieve information refinement by compressing prompt words while ensuring that key information is not lost.

The project has gained 3.1k stars on GitHub

The results show that LLMLingua-2 can The text length is significantly reduced to the original 20%, effectively reducing processing time and costs.

In addition, LLMLingua 2’s processing speed is increased by 3 to 6 times compared to the previous version of LLMLingua and other similar technologies.

Paper address: https://arxiv.org/abs/2403.12968

In this process , raw text is first fed into the model.

The model will evaluate the importance of each word and decide whether to retain or delete it, while also taking into account the relationship between words.

Finally, the model will select those words with the highest scores to form a shorter prompt word.

The team tested the LLMLingua-2 model on multiple datasets including MeetingBank, LongBench, ZeroScrolls, GSM8K and BBH.

Although this model is small, it achieves significant performance improvements in benchmark tests and demonstrates its performance on different large language models (from GPT-3.5 to Mistral- 7B) Excellent generalization ability across languages ​​(from English to Chinese).

System prompt:

As an outstanding linguist, you are good at converting long Condensing paragraphs of text into brief expressions by removing unimportant words while retaining as much information as possible.

User Tips:

Please compress the given text into A short expression that allows you (GPT-4) to restore the original text as accurately as possible. Different from regular text compression, I need you to follow the following five conditions:

1. Only remove unimportant words.

2. Keep the order of the original words unchanged.

3. Keep the original vocabulary unchanged.

4. Do not use any abbreviations or emoticons.

5. Do not add any new words or symbols.

Please compress the original text as much as possible while retaining as much information as possible. If you understand, please compress the following text: {Text to be compressed}

The compressed text is: [...]

The results show that LLMLingua-2 significantly outperforms the original LLMLingua model and other selective context strategies in multiple language tasks such as question and answer, summary writing, and logical reasoning.

It is worth mentioning that this compression method is equally effective for different large language models (from GPT-3.5 to Mistral-7B) and different languages ​​(from English to Chinese) .

Moreover, the deployment of LLMLingua-2 can be achieved with just two lines of code.

Currently, the model has been integrated into the widely used RAG frameworks LangChain and LlamaIndex.

Implementation method

In order to overcome the problems faced by existing information entropy-based text compression methods, LLMLingua-2 adopts an innovation data extraction strategy.

This strategy extracts essential information from large language models such as GPT-4, achieving efficient text editing without losing key content and avoiding adding erroneous information. compression.

Prompt Design

To fully utilize the text compression potential of GPT-4, the key lies in how to set the precise compression instructions.

That is, when compressing text, instruct GPT-4 to only remove those words that are not so important in the original text, while avoiding the introduction of any new words in the process.

The purpose of this is to ensure that the compressed text maintains the authenticity and integrity of the original text as much as possible.

##Annotation and filtering

The researchers used the GPT-4 Using the knowledge extracted from large language models, a novel data annotation algorithm was developed.

This algorithm can mark each word in the original text and clearly indicate which words must be retained during the compression process.

In order to ensure the high quality of the constructed data set, they also designed two quality monitoring mechanisms specifically to identify and exclude those data samples with poor quality.

Compressor

Finally, the researchers solved the problem of text compression It is transformed into a task of classifying each word (Token), and a powerful Transformer is used as the feature extractor.

This tool can understand the context of text to accurately capture the information critical for text compression.

By training on a carefully constructed data set, the researchers' model is able to calculate a probability value to decide whether a word should be retained based on its importance. In the final compressed text, it should still be discarded.

Performance Evaluation

The researchers tested the performance of LLMLingua-2 on a range of tasks, These tasks include context learning, text summarization, dialogue generation, multi- and single-document question answering, code generation, and synthesis tasks, including both in-domain and out-of-domain datasets.

Test results show that the researchers’ method reduces minimal performance loss while maintaining high performance, and performs outstandingly among task-unspecific text compression methods.

- In-domain test (MeetingBank)

The researchers compared the performance of LLMLingua-2 on the MeetingBank test set with Other powerful baseline methods are compared.

Although their model size is much smaller than the LLaMa-2-7B used in the baseline, the researchers' method not only significantly improved performance on question answering and text summarization tasks, but also was consistent with The original text prompts performed similarly.

##-Out-of-domain testing (LongBench, GSM8K and BBH)

Considering that the researchers’ model was only trained on MeetingBank’s meeting record data, the researchers further explored its generalization capabilities in different scenarios such as long text, logical reasoning, and contextual learning.

It is worth mentioning that although LLMLingua-2 was only trained on one dataset, in out-of-domain testing, its performance was not only comparable to the current state-of-the-art task-independent compression The methods are comparable, and in some cases even better.

Even the researchers’ smaller model (BERT-base size) was able to achieve comparable performance to the original hint, in some cases Down or even slightly above the original tip.

While the researchers’ approach achieved promising results, it still has shortcomings when compared with other task-aware compression methods, such as LongLLMlingua on Longbench.

The researchers attribute this performance gap to the extra information they get from the questions. However, the researchers' model is task-agnostic, making it an efficient option with good generalizability when deployed in different scenarios.

Table 4 above lists the results of different methods using Mistral-7Bv0.1 4 as the target LLM.

Compared with other baseline methods, the researchers' method has a significant improvement in performance, demonstrating its good generalization ability on the target LLM.

It is worth noting that LLMLingua-2 performs even better than the original prompt.

Researchers speculate that Mistral-7B may not be as good at managing long contexts as GPT-3.5-Turbo.

The researchers’ approach effectively improves Mistral7B’s final inference performance by providing short hints with higher information density.

Table 5 above shows the latency of different systems on the V100-32G GPU with different compression ratios.

The results show that compared with other compression methods, LLMLlingua2 has much less computational overhead and can achieve an end-to-end speed improvement of 1.6 times to 2.9 times.

In addition, the researchers' method can reduce GPU memory costs by 8 times, thus reducing the demand for hardware resources.

Context-Aware Observations The researchers observed that as the compression ratio increases, LLMLingua-2 can effectively maintain the most informative words with complete context.

This is thanks to the adoption of a bidirectional context-aware feature extractor and a strategy that is explicitly optimized towards the goal of timely compression.

The researchers observed that as the compression ratio increases, LLMLingua-2 can effectively maintain the most informative words related to the complete context. .

This is thanks to the adoption of a bidirectional context-aware feature extractor and a strategy that is explicitly optimized towards the goal of timely compression.

Finally the researchers had GPT-4 reconstruct the original tones from the LLMLlingua-2 compression prompts.

The results show that GPT-4 can effectively reconstruct the original tip, indicating that no essential information is lost during LLMLingua-2 compression.

The above is the detailed content of Tsinghua Microsoft open sourced a new prompt word compression tool, the length dropped by 80%! GitHub gets 3.1K stars. For more information, please follow other related articles on the PHP Chinese website!