BracketRank: Large Language Model Document Ranking via Reasoning-based Competitive Elimination
Abdelrahman Abdallah, Mohammed Ali, Bhawna Piryani, Adam Jatowt
TLDR
BracketRank improves LLM document ranking by using a reasoning-based competitive elimination framework, significantly outperforming SOTA.
Key contributions
- Introduces adaptive grouping to manage LLM context limits effectively.
- Uses reasoning-enhanced prompts requiring step-by-step relevance explanations.
- Employs a bracket-style elimination structure with winner and loser tracks.
- Achieves state-of-the-art performance on BRIGHT and TREC benchmarks.
Why it matters
Current LLM rerankers struggle with deep semantic inference due to context constraints. BracketRank addresses this by introducing a novel competitive elimination framework with explicit reasoning. This significantly advances LLM-based retrieval for complex, multi-step tasks.
Original Abstract
Reasoning-intensive retrieval requires deep semantic inference beyond surface-level keyword matching, posing a challenge for current LLM-based rerankers limited by context constraints and order sensitivity. We propose \textbf{\BracketRank}, a framework that treats document reranking as a reasoning-driven competitive tournament. Our approach introduces three key innovations: (1) adaptive grouping based on model context limits, (2) reasoning-enhanced prompts that mandate step-by-step relevance explanations, and (3) a bracket-style elimination structure with winner and loser tracks. This design ensures robust document advancement while enabling parallel processing across competition stages. Evaluation on the BRIGHT reasoning benchmark shows that \BracketRank achieves \textbf{26.56 nDCG@10}, significantly outperforming state-of-the-art baselines including RankGPT-4 (17.0) and Rank-R1-14B (20.5). On TREC datasets, BracketRank achieves 77.90 nDCG@5 on DL 19 and 75.85 nDCG@5 on DL 20, exceeding all baselines, establishing that explicit reasoning within competitive elimination is a powerful paradigm for complex, multi-step retrieval tasks. https://github.com/DataScienceUIBK/BracketRank
📬 Weekly AI Paper Digest
Get the top 10 AI/ML arXiv papers from the week — summarized, scored, and delivered to your inbox every Monday.