[2603.27343] Beyond Completion: Probing Cumulative State Tracking to Predict LLM Agent Performance

Computer Science > Artificial Intelligence arXiv:2603.27343 (cs) [Submitted on 28 Mar 2026] Title:Beyond Completion: Probing Cumulative State Tracking to Predict LLM Agent Performance Authors:Dengzhe Hou, Lingyu Jiang, Deng Li, Zirui Li, Fangzhou Lin, Kazunori D Yamada View a PDF of the paper titled Beyond Completion: Probing Cumulative State Tracking to Predict LLM Agent Performance, by Dengzhe Hou and 5 other authors View PDF HTML (experimental) Abstract:Task-completion rate is the standard proxy for LLM agent capability, but models with identical completion scores can differ substantially in their ability to track intermediate state. We introduce Working Memory Fidelity-Active Manipulation (WMF-AM), a calibrated no-scratchpad probe of cumulative arithmetic state tracking, and evaluate it on 20 open-weight models (0.5B-35B, 13 families) against a released deterministic 10-task agent battery. In a pre-specified, Bonferroni-corrected analysis, WMF-AM predicts agent performance with Kendall's tau = 0.612 (p < 0.001, 95% CI [0.360, 0.814]); exploratory partial-tau analyses suggest this signal persists after controlling for completion score and model scale. Three construct-isolation ablations (K = 1 control, non-arithmetic ceiling, yoked cancellation) support the interpretation that cumulative state tracking under load, rather than single-step arithmetic or entity tracking alone, is the primary difficulty source. K-calibration keeps the probe in a discriminative range where p...