[2601.22002] Rate-Distortion Optimization for Transformer Inference

Computer Science > Machine Learning arXiv:2601.22002 (cs) [Submitted on 29 Jan 2026 (v1), last revised 1 Apr 2026 (this version, v2)] Title:Rate-Distortion Optimization for Transformer Inference Authors:Anderson de Andrade, Alon Harell, Ivan V. Bajić View a PDF of the paper titled Rate-Distortion Optimization for Transformer Inference, by Anderson de Andrade and 2 other authors View PDF HTML (experimental) Abstract:Transformers achieve superior performance on many tasks, but impose heavy compute and memory requirements during inference. This inference can be made more efficient by partitioning the process across multiple devices, which, in turn, requires compressing its intermediate representations. We introduce a principled rate-distortion-based framework for lossy compression that learns compact encodings that explicitly trade bitrate for accuracy. Experiments on language benchmarks show that the simplest of the proposed codecs achieves substantial rate savings, outperforming more complex methods. We characterize and analyze the rate-distortion behaviour of transformers, offering a unified lens for understanding performance in representation coding. This formulation extends information-theoretic concepts to define the gap between rate and entropy, and derive some of its bounds. We further develop probably approximately correct (PAC)-style bounds for estimating this gap. For different architectures and tasks, we empirically demonstrate that their rates are driven by these...