[2603.05210] Balancing Coverage and Draft Latency in Vocabulary Trimming for Faster Speculative Decoding

Computer Science > Computation and Language arXiv:2603.05210 (cs) [Submitted on 5 Mar 2026] Title:Balancing Coverage and Draft Latency in Vocabulary Trimming for Faster Speculative Decoding Authors:Ofir Ben Shoham View a PDF of the paper titled Balancing Coverage and Draft Latency in Vocabulary Trimming for Faster Speculative Decoding, by Ofir Ben Shoham View PDF HTML (experimental) Abstract:Speculative decoding accelerates inference for Large Language Models by using a lightweight draft model to propose candidate tokens that are verified in parallel by a larger target model. Prior work shows that the draft model often dominates speculative decoding latency, since it generates tokens sequentially and incurs high cost from its language modeling head as vocabulary size grows. This exposes a fundamental trade-off in draft model design: larger vocabularies improve token coverage and agreement with the target model, but incur higher draft latency, while smaller vocabularies reduce latency at the risk of missing tokens required for accurate draft generation. We address this trade-off through vocabulary trimming for draft models, motivated by the observation that domain-specific workloads use only a small fraction of the full vocabulary. We cast draft vocabulary selection as a constrained optimization problem that balances token coverage and draft latency. Coverage is computed over assistant responses in the training data, while latency is estimated using architecture-aware FLOPs...