[2603.23517] Beyond Accuracy: Introducing a Symbolic-Mechanistic Approach to Interpretable Evaluation

Computer Science > Machine Learning arXiv:2603.23517 (cs) [Submitted on 6 Mar 2026] Title:Beyond Accuracy: Introducing a Symbolic-Mechanistic Approach to Interpretable Evaluation Authors:Reza Habibi, Darian Lee, Magy Seif El-Nasr View a PDF of the paper titled Beyond Accuracy: Introducing a Symbolic-Mechanistic Approach to Interpretable Evaluation, by Reza Habibi and 2 other authors View PDF HTML (experimental) Abstract:Accuracy-based evaluation cannot reliably distinguish genuine generalization from shortcuts like memorization, leakage, or brittle heuristics, especially in small-data regimes. In this position paper, we argue for mechanism-aware evaluation that combines task-relevant symbolic rules with mechanistic interpretability, yielding algorithmic pass/fail scores that show exactly where models generalize versus exploit patterns. We demonstrate this on NL-to-SQL by training two identical architectures under different conditions: one without schema information (forcing memorization), one with schema (enabling grounding). Standard evaluation shows the memorization model achieves 94% field-name accuracy on unseen data, falsely suggesting competence. Our symbolic-mechanistic evaluation reveals this model violates core schema generalization rules, a failure invisible to accuracy metrics. Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Symbolic Computation (cs.SC) Cite as: arXiv:2603.23517 [cs.LG]   (or arXiv:2603.2351...