[2603.03301] From Exact Hits to Close Enough: Semantic Caching for LLM Embeddings

Computer Science > Computation and Language arXiv:2603.03301 (cs) [Submitted on 7 Feb 2026] Title:From Exact Hits to Close Enough: Semantic Caching for LLM Embeddings Authors:Dvir David Biton, Roy Friedman View a PDF of the paper titled From Exact Hits to Close Enough: Semantic Caching for LLM Embeddings, by Dvir David Biton and Roy Friedman View PDF HTML (experimental) Abstract:The rapid adoption of large language models (LLMs) has created demand for faster responses and lower costs. Semantic caching, reusing semantically similar requests via their embeddings, addresses this need but breaks classic cache assumptions and raises new challenges. In this paper, we explore offline policies for semantic caching, proving that implementing an optimal offline policy is NP-hard, and propose several polynomial-time heuristics. We also present online semantic aware cache policies that combine recency, frequency, and locality. Evaluations on diverse datasets show that while frequency based policies are strong baselines, our novel variant improves semantic accuracy. Our findings reveal effective strategies for current systems and highlight substantial headroom for future innovation. All code is open source. Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) Cite as: arXiv:2603.03301 [cs.CL]   (or arXiv:2603.03301v1 [cs.CL] for this version)   https://doi.org/10.48550/arXiv.2603.03301 Focus to learn more arXiv-issued DOI via DataCite Su...