![[2602.21224] Make Every Draft Count: Hidden State based Speculative Decoding](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.21224] Make Every Draft Count: Hidden State based Speculative Decoding
Summary
The paper presents a novel approach to speculative decoding in large language models (LLMs), focusing on reusing discarded draft tokens to enhance computational efficiency and speed up inference.
Why It Matters
As LLMs become increasingly integral to various applications, optimizing their inference processes is crucial. This research addresses inefficiencies in speculative decoding, potentially leading to significant performance improvements in real-world applications.
Key Takeaways
- Introduces a system that reuses discarded draft tokens to improve efficiency.
- Proposes a draft model architecture based on auto-regressive hidden states.
- Demonstrates up to a 3.3x speedup compared to standard speculative decoding.
- Highlights a token information injection mechanism for high-quality draft token trees.
- Addresses hardware utilization to maximize computational resources.
Computer Science > Computation and Language arXiv:2602.21224 (cs) [Submitted on 2 Feb 2026] Title:Make Every Draft Count: Hidden State based Speculative Decoding Authors:Yuetao Chen, Xuliang Wang, Xinzhou Zheng, Ming Li, Peng Wang, Hong Xu View a PDF of the paper titled Make Every Draft Count: Hidden State based Speculative Decoding, by Yuetao Chen and 5 other authors View PDF HTML (experimental) Abstract:Speculative decoding has emerged as a pivotal technique to accelerate LLM inference by employing a lightweight draft model to generate candidate tokens that are subsequently verified by the target model in parallel. However, while this paradigm successfully increases the arithmetic intensity of memory-bound inference, it causes significant compute inefficiency: the majority of draft tokens fail verification and are discarded, resulting in waste of computation. Motivated by the goal of recollecting this wasted computation, we propose a novel system that transforms discarded drafts into reusable tokens. Our key insight is to perform auto-regressive prediction at the hidden states level and postpone the integrating token information after the hidden states generation, so the draft hidden states are not contaminated by incorrect tokens, enabling hidden state reuse. To implement such a system, first we introduce a draft model architecture based on auto-regressive hidden states, which preserves richer semantics than token-based drafters to facilitate draft repurposing. Second, ...
