[2603.05143] Feature Resemblance: On the Theoretical Understanding of Analogical Reasoning in Transformers

Computer Science > Computation and Language arXiv:2603.05143 (cs) [Submitted on 5 Mar 2026] Title:Feature Resemblance: On the Theoretical Understanding of Analogical Reasoning in Transformers Authors:Ruichen Xu, Wenjing Yan, Ying-Jun Angela Zhang View a PDF of the paper titled Feature Resemblance: On the Theoretical Understanding of Analogical Reasoning in Transformers, by Ruichen Xu and 2 other authors View PDF HTML (experimental) Abstract:Understanding reasoning in large language models is complicated by evaluations that conflate multiple reasoning types. We isolate analogical reasoning (inferring shared properties between entities based on known similarities) and analyze its emergence in transformers. We theoretically prove three key results: (1) Joint training on similarity and attribution premises enables analogical reasoning through aligned representations; (2) Sequential training succeeds only when similarity structure is learned before specific attributes, revealing a necessary curriculum; (3) Two-hop reasoning ($a \to b, b \to c \implies a \to c$) reduces to analogical reasoning with identity bridges ($b = b$), which must appear explicitly in training data. These results reveal a unified mechanism: transformers encode entities with similar properties into similar representations, enabling property transfer through feature alignment. Experiments with architectures up to 1.5B parameters validate our theory and demonstrate how representational geometry shapes inducti...