![[2602.17664] Sink-Aware Pruning for Diffusion Language Models](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.17664] Sink-Aware Pruning for Diffusion Language Models
Summary
The paper presents Sink-Aware Pruning, a novel method for optimizing Diffusion Language Models (DLMs) by identifying and removing unstable attention sinks, improving efficiency without retraining.
Why It Matters
As Diffusion Language Models become more prevalent, optimizing their performance is crucial for practical applications. This research addresses the inefficiencies in existing pruning methods, providing a new approach that enhances quality and efficiency, which is vital for both academic research and industry applications in AI.
Key Takeaways
- Existing pruning methods for autoregressive models do not apply effectively to DLMs.
- Sink-Aware Pruning identifies and removes unstable attention sinks, enhancing model efficiency.
- The proposed method achieves better quality-efficiency trade-offs without the need for retraining.
- This approach challenges traditional assumptions about the importance of attention sinks in language models.
- Code for the proposed method is publicly available, promoting further research and application.
Computer Science > Computation and Language arXiv:2602.17664 (cs) [Submitted on 19 Feb 2026] Title:Sink-Aware Pruning for Diffusion Language Models Authors:Aidar Myrzakhan, Tianyi Li, Bowei Guo, Shengkun Tang, Zhiqiang Shen View a PDF of the paper titled Sink-Aware Pruning for Diffusion Language Models, by Aidar Myrzakhan and Tianyi Li and Bowei Guo and Shengkun Tang and Zhiqiang Shen View PDF HTML (experimental) Abstract:Diffusion Language Models (DLMs) incur high inference cost due to iterative denoising, motivating efficient pruning. Existing pruning heuristics largely inherited from autoregressive (AR) LLMs, typically preserve attention sink tokens because AR sinks serve as stable global anchors. We show that this assumption does not hold for DLMs: the attention-sink position exhibits substantially higher variance over the full generation trajectory (measured by how the dominant sink locations shift across timesteps), indicating that sinks are often transient and less structurally essential than in AR models. Based on this observation, we propose ${\bf \texttt{Sink-Aware Pruning}}$, which automatically identifies and prunes unstable sinks in DLMs (prior studies usually keep sinks for AR LLMs). Without retraining, our method achieves a better quality-efficiency trade-off and outperforms strong prior pruning baselines under matched compute. Our code is available at this https URL. Comments: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Mach...