Rankgpt en tant qu'agent de réensemble de RAG (tutoriel)-IA-php.cn

La génération augmentée (RAG) de la récupération est une technique qui rend les modèles de langage de grands (LLM) plus intelligents et plus précis en leur permettant d'utiliser des informations extérieures lors de la génération de texte.

Le grand défi, cependant, est de choisir les bons documents ou les bons passages d'une énorme collection de données.

RankGPT aborde ce problème en améliorant l'étape de reconstitution dans les pipelines de chiffon. Il utilise les capacités de compréhension approfondie des LLM pour mieux évaluer et (re) classer quelles informations sont les plus pertinentes.

Dans cet article, nous présenterons RANKGPT et démontrerons comment vous pouvez l'intégrer dans vos applications RAG AI.

Comprendre la génération augmentée de récupération (RAG)

La génération augmentée (RAG) de la récupération est une méthode qui combine les LLM avec des systèmes de récupération d'informations. Cela signifie que lorsqu'un LLM est invité à générer du texte, il peut extraire des informations pertinentes provenant de sources externes, ce qui rend ses réponses plus précises et informées.

Rag se compose de deux composants principaux - le retriever et le générateur - et un composant facultatif, le Reranker:

Retriever - Le travail du retriever consiste à trouver des documents ou des segments de texte pertinents à partir d'un large ensemble de documents basés sur la requête de l'utilisateur. Il utilise des algorithmes comme BM25 pour classer les documents par leur pertinence.
RERANKER (Facultatif) - Le Reranker prend l'ensemble initial de documents récupérés et les réordit pour s'assurer que les plus pertinents sont en haut. Cela aide à filtrer les informations moins utiles et à se concentrer sur ce qui est important.
Générateur - Le générateur est le LLM qui utilise les documents récupérés pour générer la sortie finale. L'accès aux données externes pertinentes peut produire des réponses plus précises.

le rôle et les avantages du rankgpt dans Rag

RankGPT utilise des LLM pour évaluer la pertinence des documents récupérés ou des segments de texte, en veillant à ce que les plus importants soient en haut. Avec RankGPT, le générateur du pipeline de chiffon obtient les entrées de meilleure qualité, ce qui entraîne des réponses plus précises.

Amélioration de la pertinence et des performances

RankGpt va au-delà de la simple correspondance des mots clés en comprenant le sens et le contexte plus profond des requêtes et des documents. Cela lui permet de fournir des informations plus précises pour les LLM, identifiant le contenu le plus pertinent en fonction de sa signification réelle, pas seulement des mots clés.

Lors de l'utilisation de GPT-4 avec une génération de permutation pédagogique à tirs zéro, RankGPT surpasse les systèmes supervisés sur diverses références comme TREC, Beir et Mr.TyDi.

Distillation efficace et rentable

RankGPT utilise la distillation de permutation pour transférer les capacités de classement des grands modèles comme GPT-4 en modèles plus petits et spécialisés.

Ces modèles plus petits maintiennent des performances élevées tout en étant beaucoup plus efficaces. Par exemple, un modèle distillé de 440m a surpassé un modèle supervisé 3B sur la référence Beir, réduisant considérablement les coûts de calcul tout en obtenant de meilleurs résultats.

Gestion des informations nouvelles et inconnues

RANKGPT comprend le NOUVELLE EST TEST pour garantir la robustesse et les préoccupations de contamination des données. Cet ensemble évalue la capacité du modèle à classer les passages en fonction des informations récentes et inconnues.

GPT-4 a atteint des performances de pointe sur ce test, démontrant sa capacité à gérer efficacement les requêtes nouvelles et invisibles.

RANKGPT Benchmark Performance

RANKGPT (GPT-4) surpasse tous les autres modèles à travers TREC et BEIR, avec un score NDCG @ 10 de 53,68 moyen comme indiqué dans le tableau ci-dessous. Il a obtenu les résultats les plus élevés dans les ensembles de données Beir, battant de solides modèles supervisés comme Monot5 (3B) et Cohere Rerank-V2. Même avec GPT-3.5-turbo, RankGPT score de manière compétitive, prouvant que c'est un RERANKER très efficace.

Rankgpt en tant qu'agent de réensemble de RAG (tutoriel)

Source: Weiwei Sun et al., 2023

RANKGPT (GPT-4) fonctionne également fortement sur les ensembles de données Mr.TyDi, menant avec un score NDCG @ 10 de 62,93 moyen, battant à la fois BM25 et MMARCOCE. Il surpasse constamment BM25 et dépasse même Mmarcoce dans de nombreuses langues, en particulier en indonésien et swahili.

Dans l'ensemble, Rankgpt a obtenu le score le plus élevé dans de nombreuses langues, comme le bengali, l'Indonésien et le Japon, avec seulement quelques cas où il est légèrement à la traîne de Mmarcoce.

Rankgpt en tant qu'agent de réensemble de RAG (tutoriel)

Source: Weiwei Sun et al., 2023

Enfin, RankGPT a été testé sur le nouveau jeu de données Reeval, qui mesure la façon dont un modèle peut classer les passages en fonction des informations récentes et inconnues. RankGPT (GPT-4) a marqué le plus haut métrique de l'évaluation (NDCG @ 1, NDCG @ 5 et NDCG @ 10), en particulier avec le score NDCG @ 10 de 90,45. Il a surperformé d'autres modèles forts comme Monot5 (3B) et Monobert (340m), ce qui met en évidence ses fortes performances en tant que re-Reranker.

Rankgpt en tant qu'agent de réensemble de RAG (tutoriel)

Source: Weiwei Sun et al., 2023

À travers tous les résultats de référence, RankGPT (GPT-4) surpasse constamment d'autres méthodes, qu'elle soit supervisée ou non supervisée, démontrant sa capacité supérieure à redirige.

Implémentation de RankGPT dans les pipelines de chiffon

Voici comment nous pouvons intégrer RankGPT dans un pipeline de chiffons.

Étape 1: cloner le référentiel Rankgpt

Tout d'abord, vous devrez cloner le référentiel RANKGPT. Exécutez la commande suivante dans votre terminal:

git clone https://github.com/sunnweiwei/RankGPT

Copier après la connexion

Étape 2: Configurez votre environnement

Accédez au répertoire RANKGPT et installez les packages requis. Vous souhaiterez peut-être créer un environnement virtuel et installer des packages à l'aide des exigences fournies.txt:

pip install -r requirements.txt

Copier après la connexion

Étape 3: Implémentation de RankGPT

Ici, nous utilisons l'exemple simpliste de requête et des documents récupérés fournis par le référentiel RANGPT d'origine.

item = {
    'query': 'How much impact do masks have on preventing the spread of the COVID-19?',
    'hits': [
        {'content': 'Title: Universal Masking is Urgent in the COVID-19 Pandemic: SEIR and Agent Based Models, Empirical Validation, Policy Recommendations Content: We present two models for the COVID-19 pandemic predicting the impact of universal face mask wearing upon the spread of the SARS-CoV-2 virus--one employing a stochastic dynamic network based compartmental SEIR (susceptible-exposed-infectious-recovered) approach, and the other employing individual ABM (agent-based modelling) Monte Carlo simulation--indicating (1) significant impact under (near) universal masking when at least 80% of a population is wearing masks, versus minimal impact when only 50% or less of the population is wearing masks, and (2) significant impact when universal masking is adopted early, by Day 50 of a regional outbreak, versus minimal impact when universal masking is adopted late. These effects hold even at the lower filtering rates of homemade masks. To validate these theoretical models, we compare their predictions against a new empirical data set we have collected'},
        {'content': 'Title: Masking the general population might attenuate COVID-19 outbreaks Content: The effect of masking the general population on a COVID-19 epidemic is estimated by computer simulation using two separate state-of-the-art web-based softwares, one of them calibrated for the SARS-CoV-2 virus. The questions addressed are these: 1. Can mask use by the general population limit the spread of SARS-CoV-2 in a country? 2. What types of masks exist, and how elaborate must a mask be to be effective against COVID-19? 3. Does the mask have to be applied early in an epidemic? 4. A brief general discussion of masks and some possible future research questions regarding masks and SARS-CoV-2. Results are as follows: (1) The results indicate that any type of mask, even simple home-made ones, may be effective. Masks use seems to have an effect in lowering new patients even the protective effect of each mask (here dubbed"one-mask protection") is'},
        {'content': 'Title: To mask or not to mask: Modeling the potential for face mask use by the general public to curtail the COVID-19 pandemic Content: Face mask use by the general public for limiting the spread of the COVID-19 pandemic is controversial, though increasingly recommended, and the potential of this intervention is not well understood. We develop a compartmental model for assessing the community-wide impact of mask use by the general, asymptomatic public, a portion of which may be asymptomatically infectious. Model simulations, using data relevant to COVID-19 dynamics in the US states of New York and Washington, suggest that broad adoption of even relatively ineffective face masks may meaningfully reduce community transmission of COVID-19 and decrease peak hospitalizations and deaths. Moreover, mask use decreases the effective transmission rate in nearly linear proportion to the product of mask effectiveness (as a fraction of potentially infectious contacts blocked) and coverage rate (as'}
    ]
}

Copier après la connexion

Vous pouvez utiliser le pipeline de permutation fourni pour remanier facilement les documents récupérés avec RANKGPT.

from rank_gpt import permutation_pipeline
new_item = permutation_pipeline(
    item,
    rank_start=0,
    rank_end=3,
    model_name='gpt-3.5-turbo',
    api_key='Your OPENAI Key!'
)
print(new_item)

Copier après la connexion

Cela se traduira par le nouvel ordre des documents suivant:

{
    'query': 'How much impact do masks have on preventing the spread of the COVID-19?',
    'hits': [
        {'content': 'Title: Universal Masking is Urgent in the COVID-19 Pandemic: SEIR and Agent Based Models, Empirical Validation, Policy Recommendations Content: We present two models for the COVID-19 pandemic predicting the impact of universal face mask wearing upon the spread of the SARS-CoV-2 virus--one employing a stochastic dynamic network based compartmental SEIR (susceptible-exposed-infectious-recovered) approach, and the other employing individual ABM (agent-based modelling) Monte Carlo simulation--indicating (1) significant impact under (near) universal masking when at least 80% of a population is wearing masks, versus minimal impact when only 50% or less of the population is wearing masks, and (2) significant impact when universal masking is adopted early, by Day 50 of a regional outbreak, versus minimal impact when universal masking is adopted late. These effects hold even at the lower filtering rates of homemade masks. To validate these theoretical models, we compare their predictions against a new empirical data set we have collected'},
        {'content': 'Title: To mask or not to mask: Modeling the potential for face mask use by the general public to curtail the COVID-19 pandemic Content: Face mask use by the general public for limiting the spread of the COVID-19 pandemic is controversial, though increasingly recommended, and the potential of this intervention is not well understood. We develop a compartmental model for assessing the community-wide impact of mask use by the general, asymptomatic public, a portion of which may be asymptomatically infectious. Model simulations, using data relevant to COVID-19 dynamics in the US states of New York and Washington, suggest that broad adoption of even relatively ineffective face masks may meaningfully reduce community transmission of COVID-19 and decrease peak hospitalizations and deaths. Moreover, mask use decreases the effective transmission rate in nearly linear proportion to the product of mask effectiveness (as a fraction of potentially infectious contacts blocked) and coverage rate (as'},
        {'content': 'Title: Masking the general population might attenuate COVID-19 outbreaks Content: The effect of masking the general population on a COVID-19 epidemic is estimated by computer simulation using two separate state-of-the-art web-based softwares, one of them calibrated for the SARS-CoV-2 virus. The questions addressed are these: 1. Can mask use by the general population limit the spread of SARS-CoV-2 in a country? 2. What types of masks exist, and how elaborate must a mask be to be effective against COVID-19? 3. Does the mask have to be applied early in an epidemic? 4. A brief general discussion of masks and some possible future research questions regarding masks and SARS-CoV-2. Results are as follows: (1) The results indicate that any type of mask, even simple home-made ones, may be effective. Masks use seems to have an effect in lowering new patients even the protective effect of each mask (here dubbed"one-mask protection") is'}
    ]
}

Copier après la connexion

Génération de permutation pédagogique étape par étape

Pour une implémentation plus étape par étape du pipeline de permutation, vous pouvez directement interagir avec RankGPT pour créer et traiter les instructions de permutation comme suit:

from rank_gpt import (
    create_permutation_instruction,
    run_llm,
    receive_permutation
)
# Create permutation generation instruction
messages = create_permutation_instruction(
    item=item,
    rank_start=0,
    rank_end=3,
    model_name='gpt-3.5-turbo'
)

Copier après la connexion

[{'role': 'system',
  'content': 'You are RankGPT, an intelligent assistant that can rank passages based on their relevancy to the query.'},
 {'role': 'user',
  'content': 'I will provide you with 3 passages, each indicated by number identifier []. \nRank the passages based on their relevance to query: How much impact do masks have on preventing the spread of the COVID-19?.'},
 {'role': 'assistant', 'content': 'Okay, please provide the passages.'},
 {'role': 'user',
  'content': '[1] Title: Universal Masking is Urgent in the COVID-19 Pandemic: SEIR and Agent Based Models, Empirical Validation, Policy Recommendations Content: We present two models for the COVID-19 pandemic predicting the impact of universal face mask wearing upon the spread of the SARS-CoV-2 virus--one employing a stochastic dynamic network based compartmental SEIR (susceptible-exposed-infectious-recovered) approach, and the other employing individual ABM (agent-based modelling) Monte Carlo simulation--indicating (1) significant impact under (near) universal masking when at least 80% of a population is wearing masks, versus minimal impact when only 50% or less of the population is wearing masks, and (2) significant impact when universal masking is adopted early, by Day 50 of a regional outbreak, versus minimal impact when universal masking is adopted late. These effects hold even at the lower filtering rates of homemade masks. To validate these theoretical models, we compare their predictions against a new empirical data set we have collected'},
 {'role': 'assistant', 'content': 'Received passage [1].'},
 {'role': 'user',
  'content': '[2] Title: Masking the general population might attenuate COVID-19 outbreaks Content: The effect of masking the general population on a COVID-19 epidemic is estimated by computer simulation using two separate state-of-the-art web-based softwares, one of them calibrated for the SARS-CoV-2 virus. The questions addressed are these: 1. Can mask use by the general population limit the spread of SARS-CoV-2 in a country? 2. What types of masks exist, and how elaborate must a mask be to be effective against COVID-19? 3. Does the mask have to be applied early in an epidemic? 4. A brief general discussion of masks and some possible future research questions regarding masks and SARS-CoV-2. Results are as follows: (1) The results indicate that any type of mask, even simple home-made ones, may be effective. Masks use seems to have an effect in lowering new patients even the protective effect of each mask (here dubbed"one-mask protection") is'},
 {'role': 'assistant', 'content': 'Received passage [2].'},
 {'role': 'user',
  'content': '[3] Title: To mask or not to mask: Modeling the potential for face mask use by the general public to curtail the COVID-19 pandemic Content: Face mask use by the general public for limiting the spread of the COVID-19 pandemic is controversial, though increasingly recommended, and the potential of this intervention is not well understood. We develop a compartmental model for assessing the community-wide impact of mask use by the general, asymptomatic public, a portion of which may be asymptomatically infectious. Model simulations, using data relevant to COVID-19 dynamics in the US states of New York and Washington, suggest that broad adoption of even relatively ineffective face masks may meaningfully reduce community transmission of COVID-19 and decrease peak hospitalizations and deaths. Moreover, mask use decreases the effective transmission rate in nearly linear proportion to the product of mask effectiveness (as a fraction of potentially infectious contacts blocked) and coverage rate (as'},
 {'role': 'assistant', 'content': 'Received passage [3].'},
 {'role': 'user',
  'content': 'Search Query: How much impact do masks have on preventing the spread of the COVID-19?. \nRank the 3 passages above based on their relevance to the search query. The passages should be listed in descending order using identifiers. The most relevant passages should be listed first. The output format should be [] > [], e.g., [1] > [2]. Only response the ranking results, do not say any word or explain.'}]

Copier après la connexion

# Get ChatGPT predicted permutation
permutation = run_llm(
    messages,
    api_key='Your OPENAI Key!',
    model_name='gpt-3.5-turbo'
)

Copier après la connexion

'[1] > [3] > [2]'

Copier après la connexion

# Use permutation to re-rank the passage
item = receive_permutation(
    item,
    permutation,
    rank_start=0,
    rank_end=3
)

Copier après la connexion

{'query': 'How much impact do masks have on preventing the spread of the COVID-19?',
 'hits': [{'content': 'Title: Universal Masking is Urgent in the COVID-19 Pandemic: SEIR and Agent Based Models, Empirical Validation, Policy Recommendations Content: We present two models for the COVID-19 pandemic predicting the impact of universal face mask wearing upon the spread of the SARS-CoV-2 virus--one employing a stochastic dynamic network based compartmental SEIR (susceptible-exposed-infectious-recovered) approach, and the other employing individual ABM (agent-based modelling) Monte Carlo simulation--indicating (1) significant impact under (near) universal masking when at least 80% of a population is wearing masks, versus minimal impact when only 50% or less of the population is wearing masks, and (2) significant impact when universal masking is adopted early, by Day 50 of a regional outbreak, versus minimal impact when universal masking is adopted late. These effects hold even at the lower filtering rates of homemade masks. To validate these theoretical models, we compare their predictions against a new empirical data set we have collected'},
  {'content': 'Title: To mask or not to mask: Modeling the potential for face mask use by the general public to curtail the COVID-19 pandemic Content: Face mask use by the general public for limiting the spread of the COVID-19 pandemic is controversial, though increasingly recommended, and the potential of this intervention is not well understood. We develop a compartmental model for assessing the community-wide impact of mask use by the general, asymptomatic public, a portion of which may be asymptomatically infectious. Model simulations, using data relevant to COVID-19 dynamics in the US states of New York and Washington, suggest that broad adoption of even relatively ineffective face masks may meaningfully reduce community transmission of COVID-19 and decrease peak hospitalizations and deaths. Moreover, mask use decreases the effective transmission rate in nearly linear proportion to the product of mask effectiveness (as a fraction of potentially infectious contacts blocked) and coverage rate (as'},
  {'content': 'Title: Masking the general population might attenuate COVID-19 outbreaks Content: The effect of masking the general population on a COVID-19 epidemic is estimated by computer simulation using two separate state-of-the-art web-based softwares, one of them calibrated for the SARS-CoV-2 virus. The questions addressed are these: 1. Can mask use by the general population limit the spread of SARS-CoV-2 in a country? 2. What types of masks exist, and how elaborate must a mask be to be effective against COVID-19? 3. Does the mask have to be applied early in an epidemic? 4. A brief general discussion of masks and some possible future research questions regarding masks and SARS-CoV-2. Results are as follows: (1) The results indicate that any type of mask, even simple home-made ones, may be effective. Masks use seems to have an effect in lowering new patients even the protective effect of each mask (here dubbed"one-mask protection") is'}]}

Copier après la connexion

Stratégie de fenêtre coulissante (SWA) pour RANKGPT

Si vous avez besoin de classer plus de documents que le modèle ne peut gérer immédiatement, utilisez une stratégie de fenêtre coulissante. Voici comment appliquer une stratégie de fenêtre coulissante aux documents de ré-classement:

from rank_gpt import sliding_windows
api_key = "Your OPENAI Key"
new_item = sliding_windows(
    item,
    rank_start=0,
    rank_end=3,
    window_size=2,
    step=1,
    model_name='gpt-3.5-turbo',
    api_key=api_key
)
print(new_item)

Copier après la connexion

Dans cet exemple, la fenêtre coulissante a une taille de 2 et une taille de pas de 1, ce qui signifie qu'il traite deux documents à la fois, déplaçant un document pour la prochaine passe de classement.

Conclusion

En utilisant les LLM pour mieux évaluer la pertinence des informations, RankGPT améliore la précision du tri et du contenu de nouveau-ranking.

Cela traite des problèmes communs tels que la garantie du contenu est sur le point, l'amélioration de l'efficacité et la réduction de la probabilité de générer des informations trompeuses.

Dans l'ensemble, RankGPT contribue à créer des applications de chiffon plus fiables et plus précises.