[2603.00356] Token Management in Multi-Tenant AI Inference Platforms

Computer Science > Distributed, Parallel, and Cluster Computing arXiv:2603.00356 (cs) [Submitted on 27 Feb 2026] Title:Token Management in Multi-Tenant AI Inference Platforms Authors:William J. Cunningham View a PDF of the paper titled Token Management in Multi-Tenant AI Inference Platforms, by William J. Cunningham View PDF HTML (experimental) Abstract:Multi-tenant AI inference platforms must balance resource utilization against service-level guarantees under variable demand. Conventional approaches fail to achieve this balance: dedicated endpoints strand capacity on idle models, while rate limits ignore the heterogeneous cost of inference requests. We introduce \emph{token pools}, a control-plane abstraction that represents inference capacity as explicit entitlements expressed in inference-native units (token throughput, KV cache, concurrency). Unlike rate limits, which govern request admission without regard to execution cost, token pools authorize both admission and autoscaling from the same capacity model, ensuring consistency between what is promised and what is provisioned. The abstraction captures burst modes across multiple dimensions invisible to conventional throttling. Dynamic per-entitlement limits on each burst dimension enable fine-grained control over resource consumption while permitting work-conserving backfill by low-priority traffic. The design supports priority-aware allocation, service tiers with differentiated guarantees, and debt-based fairness mech...