[2603.29357] BenchScope: How Many Independent Signals Does Your Benchmark Provide?

Computer Science > Artificial Intelligence arXiv:2603.29357 (cs) [Submitted on 31 Mar 2026] Title:BenchScope: How Many Independent Signals Does Your Benchmark Provide? Authors:Tommy Sha, Stella Zhao View a PDF of the paper titled BenchScope: How Many Independent Signals Does Your Benchmark Provide?, by Tommy Sha and 1 other authors View PDF HTML (experimental) Abstract:AI evaluation suites often report many scores without checking whether those scores carry independent information. We introduce Effective Dimensionality (ED), the participation ratio of a centered benchmark-score spectrum, as a fast, population-conditional upper-bound diagnostic of measurement breadth. Applied at per-instance granularity to 22 benchmarks across 8 domains and more than 8,400 model evaluations, ED reveals substantial redundancy: the six-score Open LLM Leaderboard behaves like roughly two effective measurement axes (ED = 1.7), BBH and MMLU-Pro are near-interchangeable (rho = 0.96, stable across seven subpopulations), and measurement breadth varies more than 20x across current benchmarks. We show that relative ED rankings are stable under matched-dimension controls and that ED can flag redundant suite components, monitor performance-conditional compression, and guide benchmark maintenance. Because binary spectra overestimate absolute latent dimensionality, we interpret ED as a screening statistic rather than a literal factor count and complement it with null, reliability, and saturation analyses...