[2603.04759] Stacked from One: Multi-Scale Self-Injection for Context Window Extension

Computer Science > Computation and Language arXiv:2603.04759 (cs) [Submitted on 5 Mar 2026] Title:Stacked from One: Multi-Scale Self-Injection for Context Window Extension Authors:Wei Han, Pan Zhou, Shuicheng Yan View a PDF of the paper titled Stacked from One: Multi-Scale Self-Injection for Context Window Extension, by Wei Han and Pan Zhou and Shuicheng Yan View PDF HTML (experimental) Abstract:The limited context window of contemporary large language models (LLMs) remains a primary bottleneck for their broader application across diverse domains. Although continual pre-training on long-context data offers a straightforward solution, it incurs prohibitive data acquisition and computational costs. To address this challenge, we propose~\modelname, a novel framework based on multi-grained context compression and query-aware information acquisition. SharedLLM comprises two stacked short-context LLMs: a lower model serving as a compressor and an upper model acting as a decoder. The lower model compresses long inputs into compact, multi-grained representations, which are then forwarded to the upper model for context-aware processing. To maximize efficiency, this information transfer occurs exclusively at the lowest layers, bypassing lengthy forward passes and redundant cross-attention operations. This entire process, wherein the upper and lower models are derived from the same underlying LLM layers, is termed~\textit{self-injection}. To support this architecture, a specialized t...