[2602.09924] LLMs Encode Their Failures: Predicting Success from Pre-Generation Activations

Computer Science > Computation and Language arXiv:2602.09924 (cs) [Submitted on 10 Feb 2026 (v1), last revised 6 Apr 2026 (this version, v3)] Title:LLMs Encode Their Failures: Predicting Success from Pre-Generation Activations Authors:William Lugoloobi, Thomas Foster, William Bankes, Chris Russell View a PDF of the paper titled LLMs Encode Their Failures: Predicting Success from Pre-Generation Activations, by William Lugoloobi and 3 other authors View PDF HTML (experimental) Abstract:Running LLMs with extended reasoning on every problem is expensive, but determining which inputs actually require additional compute remains challenging. We investigate whether their own likelihood of success is recoverable from their internal representations before generation, and if this signal can guide more efficient inference. We train linear probes on pre-generation activations to predict policy-specific success on math and coding tasks, substantially outperforming surface features such as question length and TF-IDF. Using E2H-AMC, which provides both human and model performance on identical problems, we show that models encode a model-specific notion of difficulty that is distinct from human difficulty, and that this distinction increases with extended reasoning. Leveraging these probes, we demonstrate that routing queries across a pool of models can exceed the best-performing model whilst reducing inference cost by up to 70\% on MATH, showing that internal representations enable practi...