[2604.04281] Preservation Is Not Enough for Width Growth: Regime-Sensitive Selection of Dense LM Warm Starts

Computer Science > Artificial Intelligence arXiv:2604.04281 (cs) [Submitted on 5 Apr 2026] Title:Preservation Is Not Enough for Width Growth: Regime-Sensitive Selection of Dense LM Warm Starts Authors:Eren Unlu View a PDF of the paper titled Preservation Is Not Enough for Width Growth: Regime-Sensitive Selection of Dense LM Warm Starts, by Eren Unlu View PDF HTML (experimental) Abstract:Width expansion offers a practical route to reuse smaller causal-language-model checkpoints, but selecting a widened warm start is not solved by zero-step preservation alone. We study dense width growth as a candidate-selection problem over full training states, including copied weights, optimizer moments, and scheduler state. In a small-scale TinyStories proxy, we compare exact-copy, perturbative, asymmetric-reset, and structured non-clone warm starts under matched continuation budgets. We evaluate zero-step preservation, short-lag probe metrics, and downstream continuation utility in deterministic and stochastic regimes. The picture is mixed and partially replicated through a reduced-pool seed-1 check. Exact-copy symmetric warm starts rank first in every completed 16-step probe and in the completed stochastic 128-step continuations at seed-0 steps 1000 and 2000 plus reduced seed-1 step 2000. By contrast, the structured non-clone challenger wins deterministic 128-step continuation. Early escape from the inherited cloned subspace is therefore not a universal selector: it helps in long deter...