![[2602.14696] A Critical Look at Targeted Instruction Selection: Disentangling What Matters (and What Doesn't)](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.14696] A Critical Look at Targeted Instruction Selection: Disentangling What Matters (and What Doesn't)
Summary
This paper critically examines targeted instruction selection for fine-tuning large language models, analyzing data representation and selection algorithms to improve performance predictability.
Why It Matters
As the use of large language models grows, understanding how to effectively select training instructions is crucial for practitioners. This paper clarifies existing methods, providing actionable insights that can enhance model performance and guide future research in instruction selection.
Key Takeaways
- Gradient-based data representations consistently predict performance across datasets.
- Greedy round-robin selection performs best at low instruction budgets.
- No single method dominates; performance varies with selection algorithms and budgets.
- The paper unifies existing algorithms under a framework of distance minimization.
- Findings provide a foundation for principled data selection in LLM fine-tuning.
Computer Science > Machine Learning arXiv:2602.14696 (cs) [Submitted on 16 Feb 2026] Title:A Critical Look at Targeted Instruction Selection: Disentangling What Matters (and What Doesn't) Authors:Nihal V. Nayak, Paula Rodriguez-Diaz, Neha Hulkund, Sara Beery, David Alvarez-Melis View a PDF of the paper titled A Critical Look at Targeted Instruction Selection: Disentangling What Matters (and What Doesn't), by Nihal V. Nayak and 4 other authors View PDF HTML (experimental) Abstract:Instruction fine-tuning of large language models (LLMs) often involves selecting a subset of instruction training data from a large candidate pool, using a small query set from the target task. Despite growing interest, the literature on targeted instruction selection remains fragmented and opaque: methods vary widely in selection budgets, often omit zero-shot baselines, and frequently entangle the contributions of key components. As a result, practitioners lack actionable guidance on selecting instructions for their target tasks. In this work, we aim to bring clarity to this landscape by disentangling and systematically analyzing the two core ingredients: data representation and selection algorithms. Our framework enables controlled comparisons across models, tasks, and budgets. We find that only gradient-based data representations choose subsets whose similarity to the query consistently predicts performance across datasets and models. While no single method dominates, gradient-based representat...
