[2604.01025] Fast and Accurate Probing of In-Training LLMs' Downstream Performances

Computer Science > Machine Learning arXiv:2604.01025 (cs) [Submitted on 1 Apr 2026] Title:Fast and Accurate Probing of In-Training LLMs' Downstream Performances Authors:Zhichen Liu, Tianle Lun, Zhibin Wen, Hao An, Yulin Ou, Jianhui Xu, Hao Zhang, Wenyi Fang, Yang Zheng, Yang Xu View a PDF of the paper titled Fast and Accurate Probing of In-Training LLMs' Downstream Performances, by Zhichen Liu and 9 other authors View PDF HTML (experimental) Abstract:The paradigm of scaling Large Language Models (LLMs) in both parameter size and test time has pushed the boundaries of AI capabilities, but at the cost of making the traditional generative evaluation paradigm prohibitively expensive, therefore making the latency of LLM's in-training downstream performance evaluation unbearable. However, simple metrics like training loss (perplexity) are not always correlated with downstream performance, as sometimes their trends diverge from the actual task outcomes. This dilemma calls for a method that is computationally efficient and sufficiently accurate in measuring model capabilities. To address this challenge, we introduce a new in-training evaluation paradigm that uses a lightweight probe for monitoring downstream performance. The probes take the internal representations of LLM checkpoints (during training) as input and directly predict the checkpoint's performance on downstream tasks measured by success probability (i.e., pass@1). We design several probe architectures, validating their...