Beyond Relevance: Utility-Centric Retrieval in the LLM Era
Hengran Zhang, Minghao Tang, Keping Bi, Jiafeng Guo
TLDR
Retrieval in the LLM era must move beyond relevance to utility, focusing on how retrieved documents enhance LLM generation quality for user tasks.
Key contributions
- Traditional IR optimizes for relevance; RAG requires utility-centric retrieval for LLMs.
- Proposes evaluating retrieval by its contribution to LLM generation quality, not just relevance metrics.
- Introduces a unified framework for LLM-agnostic/specific and context-independent/dependent utility.
- Connects utility to LLM information needs and the emerging concept of agentic RAG.
Why it matters
This paper is crucial for evolving information retrieval in the LLM era. It proposes a foundational shift from simple relevance to the practical utility of retrieved information for enhancing LLM outputs. This guidance is essential for developers building next-generation RAG systems.
Original Abstract
Information retrieval systems have traditionally optimized for topical relevance-the degree to which retrieved documents match a query. However, relevance only approximates a deeper goal: utility, namely, whether retrieved information helps accomplish a user's underlying task. The emergence of retrieval-augmented generation (RAG) fundamentally changes this paradigm. Retrieved documents are no longer consumed directly by users but instead serve as evidence for large language models (LLMs) that produce answers. As a result, retrieval effectiveness must be evaluated by its contribution to generation quality rather than by relevance-based ranking metrics alone. This tutorial argues that retrieval objectives are evolving from relevance-centric optimization toward LLM-centric utility. We present a unified framework covering LLM-agnostic versus LLM-specific utility, context-independent versus context-dependent utility, and the connection with LLM information needs and agentic RAG. By synthesizing recent advances, the tutorial provides conceptual foundations and practical guidance for designing retrieval systems aligned with the requirements of LLM-based information access.
📬 Weekly AI Paper Digest
Get the top 10 AI/ML arXiv papers from the week — summarized, scored, and delivered to your inbox every Monday.