[2604.09494] RecaLLM: Addressing the Lost-in-Thought Phenomenon with Explicit In-Context Retrieval

Computer Science > Computation and Language arXiv:2604.09494 (cs) [Submitted on 10 Apr 2026] Title:RecaLLM: Addressing the Lost-in-Thought Phenomenon with Explicit In-Context Retrieval Authors:Kyle Whitecross, Negin Rahimi View a PDF of the paper titled RecaLLM: Addressing the Lost-in-Thought Phenomenon with Explicit In-Context Retrieval, by Kyle Whitecross and 1 other authors View PDF HTML (experimental) Abstract:We propose RecaLLM, a set of reasoning language models post-trained to make effective use of long-context information. In-context retrieval, which identifies relevant evidence from context, and reasoning are deeply intertwined: retrieval supports reasoning, while reasoning often determines what must be retrieved. However, their interaction remains largely underexplored. In preliminary experiments on several open-source LLMs, we observe that in-context retrieval performance substantially degrades even after a short reasoning span, revealing a key bottleneck for test-time scaling that we refer to as lost-in-thought: reasoning steps that improve performance also make subsequent in-context retrieval more challenging. To address this limitation, RecaLLM interleaves reasoning with explicit in-context retrieval, alternating between reasoning and retrieving context information needed to solve intermediate subproblems. We introduce a negligible-overhead constrained decoding mechanism that enables verbatim copying of evidence spans, improving the grounding of subsequent ge...