[2506.05639] FictionalQA: A Dataset for Studying Memorization and Knowledge Acquisition

Computer Science > Computation and Language arXiv:2506.05639 (cs) [Submitted on 5 Jun 2025 (v1), last revised 2 Mar 2026 (this version, v2)] Title:FictionalQA: A Dataset for Studying Memorization and Knowledge Acquisition Authors:John Kirchenbauer, Janny Mongkolsupawan, Yuxin Wen, Tom Goldstein, Daphne Ippolito View a PDF of the paper titled FictionalQA: A Dataset for Studying Memorization and Knowledge Acquisition, by John Kirchenbauer and 4 other authors View PDF HTML (experimental) Abstract:When language models are trained on textual data, they acquire both knowledge about the structure of language as well as knowledge of facts about the world. At inference time, their knowledge of facts can be leveraged to solve interesting problems and perform useful knowledge work for users. It is well known that language models can verbatim memorize long sequences from their training data. However, it is much less well understood how language models memorize facts seen during training. In this work, we propose a new dataset to specifically empower researchers to study the dual processes of fact memorization and verbatim sequence memorization. The dataset consists of synthetically-generated, webtext-like documents about fictional events, as well as question-answer pairs about the events. We conduct training experiments showing how synthetic data about fictional events can be useful for studying different forms of memorization. We also document some challenges in effectively building ...