[2602.24264] Compositional Generalization Requires Linear, Orthogonal Representations in Vision Embedding Models

Computer Science > Computer Vision and Pattern Recognition arXiv:2602.24264 (cs) [Submitted on 27 Feb 2026] Title:Compositional Generalization Requires Linear, Orthogonal Representations in Vision Embedding Models Authors:Arnas Uselis, Andrea Dittadi, Seong Joon Oh View a PDF of the paper titled Compositional Generalization Requires Linear, Orthogonal Representations in Vision Embedding Models, by Arnas Uselis and 2 other authors View PDF Abstract:Compositional generalization, the ability to recognize familiar parts in novel contexts, is a defining property of intelligent systems. Although modern models are trained on massive datasets, they still cover only a tiny fraction of the combinatorial space of possible inputs, raising the question of what structure representations must have to support generalization to unseen combinations. We formalize three desiderata for compositional generalization under standard training (divisibility, transferability, stability) and show they impose necessary geometric constraints: representations must decompose linearly into per-concept components, and these components must be orthogonal across concepts. This provides theoretical grounding for the Linear Representation Hypothesis: the linear structure widely observed in neural representations is a necessary consequence of compositional generalization. We further derive dimension bounds linking the number of composable concepts to the embedding geometry. Empirically, we evaluate these predict...