![[2602.23164] MetaOthello: A Controlled Study of Multiple World Models in Transformers](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.23164] MetaOthello: A Controlled Study of Multiple World Models in Transformers
Summary
The paper presents MetaOthello, a study exploring how transformers manage multiple world models through a controlled suite of Othello variants, revealing insights into shared representation and model organization.
Why It Matters
Understanding how transformers organize multiple world models is crucial for advancing machine learning interpretability and improving model design. This research provides a framework for analyzing complex model behaviors in generative tasks, which can enhance applications in AI safety and robustness.
Key Takeaways
- MetaOthello introduces a controlled environment for studying transformers' handling of multiple game rules.
- Transformers trained on mixed data do not isolate their capacities but develop a shared representation across variants.
- The findings suggest that early layers maintain general representations while later layers specialize based on game identity.
Computer Science > Machine Learning arXiv:2602.23164 (cs) [Submitted on 26 Feb 2026] Title:MetaOthello: A Controlled Study of Multiple World Models in Transformers Authors:Aviral Chawla, Galen Hall, Juniper Lovato View a PDF of the paper titled MetaOthello: A Controlled Study of Multiple World Models in Transformers, by Aviral Chawla and 2 other authors View PDF HTML (experimental) Abstract:Foundation models must handle multiple generative processes, yet mechanistic interpretability largely studies capabilities in isolation; it remains unclear how a single transformer organizes multiple, potentially conflicting "world models". Previous experiments on Othello playing neural-networks test world-model learning but focus on a single game with a single set of rules. We introduce MetaOthello, a controlled suite of Othello variants with shared syntax but different rules or tokenizations, and train small GPTs on mixed-variant data to study how multiple world models are organized in a shared representation space. We find that transformers trained on mixed-game data do not partition their capacity into isolated sub-models; instead, they converge on a mostly shared board-state representation that transfers causally across variants. Linear probes trained on one variant can intervene on another's internal state with effectiveness approaching that of matched probes. For isomorphic games with token remapping, representations are equivalent up to a single orthogonal rotation that generali...
