[2603.21840] Select, Label, Evaluate: Active Testing in NLP

Computer Science > Computation and Language arXiv:2603.21840 (cs) [Submitted on 23 Mar 2026] Title:Select, Label, Evaluate: Active Testing in NLP Authors:Antonio Purificato, Maria Sofia Bucarelli, Andrea Bacciu, Amin Mantrach, Fabrizio Silvestri View a PDF of the paper titled Select, Label, Evaluate: Active Testing in NLP, by Antonio Purificato and 4 other authors View PDF HTML (experimental) Abstract:Human annotation cost and time remain significant bottlenecks in Natural Language Processing (NLP), with test data annotation being particularly expensive due to the stringent requirement for low-error and high-quality labels necessary for reliable model evaluation. Traditional approaches require annotating entire test sets, leading to substantial resource requirements. Active Testing is a framework that selects the most informative test samples for annotation. Given a labeling budget, it aims to choose the subset that best estimates model performance while minimizing cost and human effort. In this work, we formalize Active Testing in NLP and we conduct an extensive benchmarking of existing approaches across 18 datasets and 4 embedding strategies spanning 4 different NLP tasks. The experiments show annotation reductions of up to 95%, with performance estimation accuracy difference from the full test set within 1%. Our analysis reveals variations in method effectiveness across different data characteristics and task types, with no single approach emerging as universally superi...