![[2602.09725] Efficient Remote Prefix Fetching with GPU-native Media ASICs](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.09725] Efficient Remote Prefix Fetching with GPU-native Media ASICs
Summary
The paper presents KVFetcher, a novel solution for efficient remote key-value (KV) cache reuse using GPU-native video codecs, significantly reducing time-to-first-token in low-bandwidth scenarios.
Why It Matters
As large language models (LLMs) become increasingly prevalent, optimizing their inference speed is crucial. This research addresses bandwidth limitations in remote KV cache fetching, enhancing performance and efficiency, which is vital for real-time applications and resource-constrained environments.
Key Takeaways
- KVFetcher utilizes GPU-native video codecs for efficient KV cache transmission.
- The proposed system reduces time-to-first-token (TTFT) by up to 3.51 times.
- Maintains lossless accuracy while improving performance in bandwidth-limited scenarios.
- Introduces a codec-friendly tensor layout for compact KV cache storage.
- Demonstrates effectiveness across a range of GPU hardware.
Computer Science > Distributed, Parallel, and Cluster Computing arXiv:2602.09725 (cs) [Submitted on 10 Feb 2026 (v1), last revised 12 Feb 2026 (this version, v2)] Title:Efficient Remote Prefix Fetching with GPU-native Media ASICs Authors:Liang Mi, Weijun Wang, Jinghan Chen, Ting Cao, Haipeng Dai, Yunxin Liu View a PDF of the paper titled Efficient Remote Prefix Fetching with GPU-native Media ASICs, by Liang Mi and 5 other authors View PDF HTML (experimental) Abstract:Remote KV cache reuse fetches KV cache for identical contexts from remote storage, avoiding recomputation, accelerating LLM inference. While it excels in high-speed networks, its performance degrades significantly in bandwidth-limited scenarios. Recent studies address this by transmitting KV caches in compressed form, but the associated heavyweight decompression counteracts the KV reuse benefits. In this paper, we propose an efficient and widely deployable remote KV cache reuse solution that leverages GPU-native video codecs. Our system, KVFetcher, enables effective KV cache coding with two techniques. The codec-friendly tensor layout compresses the KV cache in a highly compact video format, enabling fast transmission. The efficient KV fetcher orchestrates the transmission, decoding, and restoration of compressed KV caches in an efficient pipelined manner, eliminating resource contention, masking network fluctuations, and achieving minimum time-to-first-token (TTFT). We prototype KVFetcher on diverse GPUs from...