[2510.10902] Auditing Information Disclosure During LLM-Scale Gradient Descent Using Gradient Uniqueness

Computer Science > Machine Learning arXiv:2510.10902 (cs) [Submitted on 13 Oct 2025 (v1), last revised 3 Mar 2026 (this version, v2)] Title:Auditing Information Disclosure During LLM-Scale Gradient Descent Using Gradient Uniqueness Authors:Sleem Abdelghafar, Maryam Aliakbarpour, Chris Jermaine View a PDF of the paper titled Auditing Information Disclosure During LLM-Scale Gradient Descent Using Gradient Uniqueness, by Sleem Abdelghafar and 2 other authors View PDF HTML (experimental) Abstract:Disclosing information via the publication of a machine learning model poses significant privacy risks. However, auditing this disclosure across every datapoint during the training of Large Language Models (LLMs) is computationally prohibitive. In this paper, we present Gradient Uniqueness (GNQ), a principled, attack-agnostic metric derived from an information-theoretic upper bound on the amount of information embedded in a model about individual training points via gradient descent. While naively computing GNQ requires forming and inverting an $P \times P$ matrix for every datapoint (for a model with $P$ parameters), we introduce Batch-Space Ghost GNQ (BS-Ghost GNQ). This efficient algorithm performs all computations in a much smaller batch-space and leverages ghost kernels to compute GNQ ``in-run'' with minimal computational overhead. We empirically validate that GNQ successfully accounts for prior/common knowledge. Our evaluation demonstrates that GNQ strongly predicts sequence ext...