[2603.05293] Knowledge Divergence and the Value of Debate for Scalable Oversight

Computer Science > Machine Learning arXiv:2603.05293 (cs) [Submitted on 5 Mar 2026] Title:Knowledge Divergence and the Value of Debate for Scalable Oversight Authors:Robin Young View a PDF of the paper titled Knowledge Divergence and the Value of Debate for Scalable Oversight, by Robin Young View PDF HTML (experimental) Abstract:AI safety via debate and reinforcement learning from AI feedback (RLAIF) are both proposed methods for scalable oversight of advanced AI systems, yet no formal framework relates them or characterizes when debate offers an advantage. We analyze this by parameterizing debate's value through the geometry of knowledge divergence between debating models. Using principal angles between models' representation subspaces, we prove that the debate advantage admits an exact closed form. When models share identical training corpora, debate reduces to RLAIF-like where a single-agent method recovers the same optimum. When models possess divergent knowledge, debate advantage scales with a phase transition from quadratic regime (debate offers negligible benefit) to linear regime (debate is essential). We classify three regimes of knowledge divergence (shared, one-sided, and compositional) and provide existence results showing that debate can achieve outcomes inaccessible to either model alone, alongside a negative result showing that sufficiently strong adversarial incentives cause coordination failure in the compositional regime, with a sharp threshold separating...