[2509.22641] Death of the Novel(ty): Beyond n-Gram Novelty as a Metric for Textual Creativity

Computer Science > Computation and Language arXiv:2509.22641 (cs) [Submitted on 26 Sep 2025 (v1), last revised 3 Mar 2026 (this version, v2)] Title:Death of the Novel(ty): Beyond n-Gram Novelty as a Metric for Textual Creativity Authors:Arkadiy Saakyan, Najoung Kim, Smaranda Muresan, Tuhin Chakrabarty View a PDF of the paper titled Death of the Novel(ty): Beyond n-Gram Novelty as a Metric for Textual Creativity, by Arkadiy Saakyan and 3 other authors View PDF HTML (experimental) Abstract:N-gram novelty is widely used to evaluate language models' ability to generate text outside of their training data. More recently, it has also been adopted as a metric for measuring textual creativity. However, theoretical work on creativity suggests that this approach may be inadequate, as it does not account for creativity's dual nature: novelty (how original the text is) and appropriateness (how sensical and pragmatic it is). We investigate the relationship between this notion of creativity and n-gram novelty through 8,618 expert writer annotations of novelty, pragmaticality, and sensicality via close reading of human- and AI-generated text. We find that while n-gram novelty is positively associated with expert writer-judged creativity, approximately 91% of top-quartile n-gram novel expressions are not judged as creative, cautioning against relying on n-gram novelty alone. Furthermore, unlike in human-written text, higher n-gram novelty in open-source LLMs correlates with lower pragmati...