[2603.25579] The Rules-and-Facts Model for Simultaneous Generalization and Memorization in Neural Networks

Statistics > Machine Learning arXiv:2603.25579 (stat) [Submitted on 26 Mar 2026] Title:The Rules-and-Facts Model for Simultaneous Generalization and Memorization in Neural Networks Authors:Gabriele Farné, Fabrizio Boncoraglio, Lenka Zdeborová View a PDF of the paper titled The Rules-and-Facts Model for Simultaneous Generalization and Memorization in Neural Networks, by Gabriele Farn\'e and 2 other authors View PDF HTML (experimental) Abstract:A key capability of modern neural networks is their capacity to simultaneously learn underlying rules and memorize specific facts or exceptions. Yet, theoretical understanding of this dual capability remains limited. We introduce the Rules-and-Facts (RAF) model, a minimal solvable setting that enables precise characterization of this phenomenon by bridging two classical lines of work in the statistical physics of learning: the teacher-student framework for generalization and Gardner-style capacity analysis for memorization. In the RAF model, a fraction $1 - \varepsilon$ of training labels is generated by a structured teacher rule, while a fraction $\varepsilon$ consists of unstructured facts with random labels. We characterize when the learner can simultaneously recover the underlying rule - allowing generalization to new data - and memorize the unstructured examples. Our results quantify how overparameterization enables the simultaneous realization of these two objectives: sufficient excess capacity supports memorization, while regul...