[2509.21993] Bilinear representation mitigates reversal curse and enables consistent model editing

Computer Science > Artificial Intelligence arXiv:2509.21993 (cs) [Submitted on 26 Sep 2025 (v1), last revised 2 Mar 2026 (this version, v3)] Title:Bilinear representation mitigates reversal curse and enables consistent model editing Authors:Dong-Kyum Kim, Minsung Kim, Jea Kwon, Nakyeong Yang, Meeyoung Cha View a PDF of the paper titled Bilinear representation mitigates reversal curse and enables consistent model editing, by Dong-Kyum Kim and 4 other authors View PDF HTML (experimental) Abstract:The reversal curse--a language model's inability to infer an unseen fact "B is A" from a learned fact "A is B"--is widely considered a fundamental limitation. We show that this is not an inherent failure but an artifact of how models encode knowledge. Our results demonstrate that training from scratch on synthetic relational knowledge graphs leads to the emergence of a bilinear relational structure within the models' hidden representations. This structure alleviates the reversal curse and facilitates inference of unseen reverse facts. Crucially, this bilinear geometry is foundational for consistent model editing: updates to a single fact propagate correctly to its reverse and logically dependent relations. In contrast, models lacking this representation suffer from the reversal curse and fail to generalize model edits, leading to logical inconsistencies. Our results establish that training on a relational knowledge dataset induces the emergence of bilinear internal representations, ...