![[2602.14536] Explainable Token-level Noise Filtering for LLM Fine-tuning Datasets](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.14536] Explainable Token-level Noise Filtering for LLM Fine-tuning Datasets
Summary
The paper presents XTF, an explainable token-level noise filtering framework designed to enhance the fine-tuning of Large Language Models (LLMs) by addressing token-level noise in datasets.
Why It Matters
As LLMs become increasingly integral to various applications, optimizing their fine-tuning datasets is crucial for improving performance. This research highlights a novel approach to mitigate token-level noise, which can significantly enhance the effectiveness of LLMs across different tasks, thereby advancing the field of AI and NLP.
Key Takeaways
- XTF framework improves LLM fine-tuning by filtering token-level noise.
- The framework assesses token contributions based on reasoning importance, knowledge novelty, and task relevance.
- Experiments show performance improvements of up to 13.7% in downstream tasks.
- Token-level optimization is essential for effective LLM training.
- The study emphasizes the need for explainable AI in training mechanisms.
Computer Science > Computation and Language arXiv:2602.14536 (cs) [Submitted on 16 Feb 2026] Title:Explainable Token-level Noise Filtering for LLM Fine-tuning Datasets Authors:Yuchen Yang, Wenze Lin, Enhao Huang, Zhixuan Chu, Hongbin Zhou, Lan Tao, Yiming Li, Zhan Qin, Kui Ren View a PDF of the paper titled Explainable Token-level Noise Filtering for LLM Fine-tuning Datasets, by Yuchen Yang and 8 other authors View PDF HTML (experimental) Abstract:Large Language Models (LLMs) have seen remarkable advancements, achieving state-of-the-art results in diverse applications. Fine-tuning, an important step for adapting LLMs to specific downstream tasks, typically involves further training on corresponding datasets. However, a fundamental discrepancy exists between current fine-tuning datasets and the token-level optimization mechanism of LLMs: most datasets are designed at the sentence-level, which introduces token-level noise, causing negative influence to final performance. In this paper, we propose XTF, an explainable token-level noise filtering framework. XTF decomposes the complex and subtle contributions of token-level data to the fine-tuning process into three distinct and explicit attributes (reasoning importance, knowledge novelty, and task relevance), which can be assessed using scoring methods, and then masks the gradients of selected noisy tokens accordingly to optimize the performance of fine-tuned LLMs. We conduct extensive experiments on three representative downst...
