[2601.18858] Representational Homomorphism Predicts and Improves Compositional Generalization In Transformer Language Model

Computer Science > Machine Learning arXiv:2601.18858 (cs) [Submitted on 26 Jan 2026 (v1), last revised 24 Mar 2026 (this version, v2)] Title:Representational Homomorphism Predicts and Improves Compositional Generalization In Transformer Language Model Authors:Zhiyu An, Wan Du View a PDF of the paper titled Representational Homomorphism Predicts and Improves Compositional Generalization In Transformer Language Model, by Zhiyu An and 1 other authors View PDF HTML (experimental) Abstract:Compositional generalization-the ability to interpret novel combinations of familiar components-remains a persistent challenge for neural networks. Behavioral evaluations reveal \emph{when} models fail but offer limited insight into \emph{why} failures arise at the representational level. We introduce \textit{Homomorphism Error} (HE), a structural metric that measures the inconsistency between a set of established rules for which words combine to form new meaning (linguistic syntax) and model's learned rules for which hidden states combine to form new states (semantic syntax). We formulate this inconsistency as deviations from approximate homomorphisms between the linguistic expression algebra and a model's hidden-state space. We designed experiments to test if i) HE predicts compositional generalization performance, and ii) will regularizing for low HE during training improve such performance. To avoid the effect of data spoilage, we train small decoder-only Transformers from scratch using a...