![[2504.01445] Compositional-ARC: Assessing Systematic Generalization in Abstract Spatial Reasoning](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2504.01445] Compositional-ARC: Assessing Systematic Generalization in Abstract Spatial Reasoning
Summary
The paper introduces Compositional-ARC, a dataset for evaluating systematic generalization in abstract spatial reasoning, demonstrating the effectiveness of meta-learning in enhancing model performance beyond linguistic tasks.
Why It Matters
This research addresses the limitations of large language models in systematic generalization, particularly in abstract spatial reasoning. By extending meta-learning approaches, it opens new avenues for developing more robust AI models capable of handling complex compositional tasks, which is crucial for advancing AI capabilities in various applications.
Key Takeaways
- Compositional-ARC dataset evaluates systematic generalization in spatial reasoning.
- Meta-learning significantly enhances model performance in compositional tasks.
- A small transformer model outperforms larger state-of-the-art LLMs in systematic behavior.
- The findings suggest broader applicability of meta-learning beyond linguistic tasks.
- This research contributes to the ongoing discourse on AI's capacity for systematic generalization.
Computer Science > Artificial Intelligence arXiv:2504.01445 (cs) [Submitted on 2 Apr 2025 (v1), last revised 26 Feb 2026 (this version, v3)] Title:Compositional-ARC: Assessing Systematic Generalization in Abstract Spatial Reasoning Authors:Philipp Mondorf, Shijia Zhou, Monica Riedler, Barbara Plank View a PDF of the paper titled Compositional-ARC: Assessing Systematic Generalization in Abstract Spatial Reasoning, by Philipp Mondorf and 3 other authors View PDF HTML (experimental) Abstract:Systematic generalization refers to the capacity to understand and generate novel combinations from known components. Despite recent progress by large language models (LLMs) across various domains, these models often fail to extend their knowledge to novel compositional scenarios, revealing notable limitations in systematic generalization. There has been an ongoing debate about whether neural networks possess the capacity for systematic generalization, with recent studies suggesting that meta-learning approaches designed for compositionality can significantly enhance this ability. However, these insights have largely been confined to linguistic problems, leaving their applicability to other tasks an open question. In this study, we extend meta-learning for compositionality to the domain of abstract spatial reasoning. To this end, we introduce $\textit{Compositional-ARC}\unicode{x2014}$a dataset designed to evaluate the capacity of models to systematically generalize from known geometric t...
