![[2509.17196] Evolution of Concepts in Language Model Pre-Training](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2509.17196] Evolution of Concepts in Language Model Pre-Training
Summary
This article examines the evolution of concepts in language model pre-training, revealing how feature development influences performance during training stages.
Why It Matters
Understanding the dynamics of feature evolution in language models can enhance the development of more effective AI systems. This research sheds light on the pre-training process, which is crucial for improving model performance and interpretability in natural language processing.
Key Takeaways
- Feature evolution in language models occurs in distinct phases.
- Causal connections exist between feature development and downstream performance.
- The study introduces a method to track representation progress during training.
Computer Science > Computation and Language arXiv:2509.17196 (cs) [Submitted on 21 Sep 2025 (v1), last revised 14 Feb 2026 (this version, v2)] Title:Evolution of Concepts in Language Model Pre-Training Authors:Xuyang Ge, Wentao Shu, Jiaxing Wu, Yunhua Zhou, Zhengfu He, Xipeng Qiu View a PDF of the paper titled Evolution of Concepts in Language Model Pre-Training, by Xuyang Ge and 5 other authors View PDF HTML (experimental) Abstract:Language models obtain extensive capabilities through pre-training. However, the pre-training process remains a black box. In this work, we track linear interpretable feature evolution across pre-training snapshots using a sparse dictionary learning method called crosscoders. We find that most features begin to form around a specific point, while more complex patterns emerge in later training stages. Feature attribution analyses reveal causal connections between feature evolution and downstream performance. Our feature-level observations are highly consistent with previous findings on Transformer's two-stage learning process, which we term a statistical learning phase and a feature learning phase. Our work opens up the possibility to track fine-grained representation progress during language model learning dynamics. Comments: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI) Cite as: arXiv:2509.17196 [cs.CL] (or arXiv:2509.17196v2 [cs.CL] for this version) https://doi.org/10.48550/arXiv.2509.17196 Focus to learn mo...