Rag Performance Prediction for Question Answering
Or Dado, David Carmel. Oren Kurland
TLDR
This paper introduces a novel supervised predictor that effectively predicts the performance gain of RAG for question answering by modeling semantic relationships.
Key contributions
- Addresses the task of predicting RAG performance gain for question answering.
- Evaluates existing pre-retrieval, post-retrieval, and post-generation predictors.
- Introduces a novel supervised post-generation predictor for RAG gain.
- The novel predictor models semantic relationships between question, passages, and answer, achieving best quality.
Why it matters
Understanding when RAG improves QA performance is crucial for its effective deployment and resource optimization. This paper offers a novel method to predict RAG's gain, helping practitioners make informed decisions.
Original Abstract
We address the task of predicting the gain of using RAG (retrieval augmented generation) for question answering with respect to not using it. We study the performance of a few pre-retrieval and post-retrieval predictors originally devised for ad hoc retrieval. We also study a few post-generation predictors, one of which is novel to this study and posts the best prediction quality. Our results show that the most effective prediction approach is a novel supervised predictor that explicitly models the semantic relationships among the question, retrieved passages, and the generated answer.
📬 Weekly AI Paper Digest
Get the top 10 AI/ML arXiv papers from the week — summarized, scored, and delivered to your inbox every Monday.