retrieval-augmented generation

RAG looks and acts like a standard seq2seq model, meaning it takes in one sequence and outputs a corresponding sequence. There is an intermediary step though, which differentiates and elevates RAG above the usual seq2seq methods. Rather than passing the input directly to the generator, RAG instead uses the input to retrieve a set of relevant documents, in our case from Wikipedia. Given the prompt “When did the first mammal appear on Earth?” for instance, RAG might surface documents for “Mammal,” “History of Earth,” and “Evolution of Mammals.” These supporting documents are then concatenated as context with the original input and fed to the seq2seq model that produces the actual output. RAG thus has two sources of knowledge: the knowledge that seq2seq models store in their parameters (parametric memory) and the knowledge stored in the corpus from which RAG retrieves passages (nonparametric memory). These two sources complement each other. We found that RAG uses its nonparametric memory to “cue” the seq2seq model into generating correct responses, essentially combining the flexibility of the “closed-book” or parametric-only approach with the performance of “open-book” or retrieval-based methods. RAG employs a form of late fusion to integrate knowledge from all retrieved documents, meaning it makes individual answer predictions for document-question pairs and then aggregates the final prediction scores. Critically, using late fusion allows us to back-propagate error signals in the output to the retrieval mechanism, which can substantially improve the performance of the end-to-end system. Leider kann dieses Video nicht richtig abgespielt werden.Bitte installiere zum Abspielen des Videos die neueste Version deines Internetbrowsers. Combining a retrieval-based component with a generative component has advantages even in purely extractive tasks, such as the open-domain NaturalQuestions task. Performance improves when RAG has access to documents that contain clues to the correct answer but where the answer is never stated verbatim, and RAG even generates correct answers in certain situations where the correct answer is nowhere to be found in any of the retrieved documents. We obtained very strong results on NaturalQuestions, CuratedTrec, and WebQuestions with RAG, demonstrating that state-of-the-art machine reading performance can be achieved with a generative, rather than extractive, reader.. RAG’s true strength lies in its flexibility. Changing what a pretrained language model knows entails retraining the entire model with new documents. With RAG, we control what it knows simply by swapping out the documents it uses for knowledge retrieval. We tested this behavior by replacing our original Wikipedia dataset with an older one and then asking questions like “Who is the prime minister of Iceland?” Results showed RAG leveraged the knowledge in the swapped-in corpus to adjust its answers, even though the parametric knowledge remained static. This adaptive approach is invaluable in situations where facts (or our understanding of the facts) evolve over time. We see broad potential for RAG, which is why we released it today as a component of the Hugging Face transformer library. Hugging Face’s Transformers has become a de facto standard in open source NLP, thanks to its low barrier to entry and coverage of state-of-the-art models, and it integrates with the new Datasets library to provide the indexed knowledge source that RAG relies on. Now, with RAG’s inclusion, we believe the community will be able to apply retrieval-based generation to both the knowledge-intensive tasks we already explored and some we haven’t even imagined yet. We will be presenting RAG at NeurIPS 2020 in December, and look forward in the meantime to seeing how the community responds to it. We also plan to continue pursuing these ideas by refining RAG’s retrieval component, scaling the retrieval corpus and model, and improving RAG’s overall performance on knowledg