![[2602.18613] Diagnosing LLM Reranker Behavior Under Fixed Evidence Pools](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.18613] Diagnosing LLM Reranker Behavior Under Fixed Evidence Pools
Summary
This paper presents a diagnostic method for evaluating LLM reranker behavior using fixed evidence pools, isolating ranking policies from retrieval quality.
Why It Matters
Understanding LLM reranker behavior is crucial for improving information retrieval systems. This study provides insights into how different models handle redundancy and lexical coverage, which can inform the development of more effective ranking algorithms.
Key Takeaways
- Introduces a controlled diagnostic to evaluate LLM rerankers.
- Findings reveal varied redundancy patterns across different models.
- LLMs show underperformance in lexical coverage at smaller selection budgets.
- The method is model-agnostic, applicable to various rankers.
- Eliminating retrieval variance allows for direct attribution of differences to ranking policies.
Computer Science > Machine Learning arXiv:2602.18613 (cs) [Submitted on 20 Feb 2026] Title:Diagnosing LLM Reranker Behavior Under Fixed Evidence Pools Authors:Baris Arat, Emre Sefer View a PDF of the paper titled Diagnosing LLM Reranker Behavior Under Fixed Evidence Pools, by Baris Arat and 1 other authors View PDF HTML (experimental) Abstract:Standard reranking evaluations study how a reranker orders candidates returned by an upstream retriever. This setup couples ranking behavior with retrieval quality, so differences in output cannot be attributed to the ranking policy alone. We introduce a controlled diagnostic that isolates reranking by using Multi-News clusters as fixed evidence pools. We limit each pool to exactly eight documents and pass identical inputs to all rankers. Within this setup, BM25 and MMR serve as interpretable reference points for lexical matching and diversity optimization. Across 345 clusters, we find that redundancy patterns vary by model: one LLM implicitly diversifies at larger selection budgets, while another increases redundancy. In contrast, LLMs underperform on lexical coverage at small selection budgets. As a result, LLM rankings diverge substantially from both baselines rather than consistently approximating either strategy. By eliminating retrieval variance, we can attribute these differences directly to the ranking policy. This diagnostic is model-agnostic and applicable to any ranker, including open source systems and proprietary APIs. S...
