![[2602.18584] GIST: Targeted Data Selection for Instruction Tuning via Coupled Optimization Geometry](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.18584] GIST: Targeted Data Selection for Instruction Tuning via Coupled Optimization Geometry
Summary
The paper presents GIST, a method for targeted data selection in instruction tuning, improving efficiency by aligning training gradients with task-specific subspaces.
Why It Matters
As machine learning models grow in complexity, efficient instruction tuning becomes crucial for optimizing performance. GIST addresses limitations in current methods by enhancing data selection processes, which can lead to significant improvements in computational efficiency and model effectiveness.
Key Takeaways
- GIST improves targeted data selection for instruction tuning.
- The method addresses limitations of existing approaches by considering cross-parameter coupling.
- GIST achieves superior performance with significantly reduced storage and computational costs.
- The approach utilizes spectral filtering to align training gradients with task-specific subspaces.
- This innovation can enhance the efficiency of fine-tuning in various AI applications.
Computer Science > Machine Learning arXiv:2602.18584 (cs) [Submitted on 20 Feb 2026] Title:GIST: Targeted Data Selection for Instruction Tuning via Coupled Optimization Geometry Authors:Guanghui Min, Tianhao Huang, Ke Wan, Chen Chen View a PDF of the paper titled GIST: Targeted Data Selection for Instruction Tuning via Coupled Optimization Geometry, by Guanghui Min and 3 other authors View PDF HTML (experimental) Abstract:Targeted data selection has emerged as a crucial paradigm for efficient instruction tuning, aiming to identify a small yet influential subset of training examples for a specific target task. In practice, influence is often measured through the effect of an example on parameter updates. To make selection scalable, many approaches leverage optimizer statistics (e.g., Adam states) as an axis-aligned surrogate for update geometry (i.e., diagonal precondition), implicitly treating parameters as coordinate-wise independent. We show that this assumption breaks down in parameter-efficient fine-tuning (PEFT) methods such as LoRA. In this setting, the induced optimization geometry exhibits strong cross-parameter coupling with non-trivial off-diagonal interactions, while the task-relevant update directions are confined to a low-dimensional subspace. Motivated by this mismatch, we propose GIST (Gradient Isometric Subspace Transformation), a simple yet principled alternative that replaces axis-aligned scaling with robust subspace alignment. GIST recovers a task-specif...
