![[2602.19969] ReAttn: Improving Attention-based Re-ranking via Attention Re-weighting](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.19969] ReAttn: Improving Attention-based Re-ranking via Attention Re-weighting
Summary
The paper presents ReAttn, a novel strategy to enhance attention-based re-ranking in large language models by reducing lexical bias and improving attention distribution.
Why It Matters
As large language models become increasingly integral to information retrieval, improving their ranking accuracy is crucial. ReAttn addresses key limitations in current attention mechanisms, potentially leading to more relevant search results and better user experiences in applications relying on these models.
Key Takeaways
- ReAttn proposes a post-hoc re-weighting strategy for attention-based re-ranking.
- The method reduces bias by down-weighting frequently overlapping query tokens.
- Entropy-based regularization encourages a more balanced attention distribution.
- ReAttn operates without additional training, making it efficient to implement.
- Extensive experiments validate the effectiveness of the proposed method.
Computer Science > Computation and Language arXiv:2602.19969 (cs) [Submitted on 23 Feb 2026] Title:ReAttn: Improving Attention-based Re-ranking via Attention Re-weighting Authors:Yuxing Tian, Fengran Mo, Weixu Zhang, Yiyan Qi, Jian-Yun Nie View a PDF of the paper titled ReAttn: Improving Attention-based Re-ranking via Attention Re-weighting, by Yuxing Tian and 4 other authors View PDF HTML (experimental) Abstract:The strong capabilities of recent Large Language Models (LLMs) have made them highly effective for zero-shot re-ranking task. Attention-based re-ranking methods, which derive relevance scores directly from attention weights, offer an efficient and interpretable alternative to generation-based re-ranking methods. However, they still face two major limitations. First, attention signals are highly concentrated a small subset of tokens within a few documents, making others indistinguishable. Second, attention often overemphasizes phrases lexically similar to the query, yielding biased rankings that irrelevant documents with mere lexical resemblance are regarded as relevant. In this paper, we propose \textbf{ReAttn}, a post-hoc re-weighting strategy for attention-based re-ranking methods. It first compute the cross-document IDF weighting to down-weight attention on query-overlapping tokens that frequently appear across the candidate documents, reducing lexical bias and emphasizing distinctive terms. It then employs entropy-based regularization to mitigate over-concentr...