![[2602.15029] Symmetry in language statistics shapes the geometry of model representations](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.15029] Symmetry in language statistics shapes the geometry of model representations
Summary
This article explores how symmetry in language statistics influences the geometric representation of models in machine learning, particularly in large language models (LLMs).
Why It Matters
Understanding the geometric structures that emerge from language statistics can enhance our grasp of how neural networks process and represent language. This insight is crucial for improving model performance and robustness in natural language processing tasks.
Key Takeaways
- Language statistics exhibit translation symmetry affecting model geometry.
- Geometric structures like circular representations of time persist despite perturbations.
- The underlying continuous latent variable governs co-occurrence statistics in language models.
Computer Science > Machine Learning arXiv:2602.15029 (cs) [Submitted on 16 Feb 2026] Title:Symmetry in language statistics shapes the geometry of model representations Authors:Dhruva Karkada, Daniel J. Korchinski, Andres Nava, Matthieu Wyart, Yasaman Bahri View a PDF of the paper titled Symmetry in language statistics shapes the geometry of model representations, by Dhruva Karkada and 4 other authors View PDF HTML (experimental) Abstract:Although learned representations underlie neural networks' success, their fundamental properties remain poorly understood. A striking example is the emergence of simple geometric structures in LLM representations: for example, calendar months organize into a circle, years form a smooth one-dimensional manifold, and cities' latitudes and longitudes can be decoded by a linear probe. We show that the statistics of language exhibit a translation symmetry -- e.g., the co-occurrence probability of two months depends only on the time interval between them -- and we prove that the latter governs the aforementioned geometric structures in high-dimensional word embedding models. Moreover, we find that these structures persist even when the co-occurrence statistics are strongly perturbed (for example, by removing all sentences in which two months appear together) and at moderate embedding dimension. We show that this robustness naturally emerges if the co-occurrence statistics are collectively controlled by an underlying continuous latent variable. W...
