[2603.26498] Rocks, Pebbles and Sand: Modality-aware Scheduling for Multimodal Large Language Model Inference

Computer Science > Distributed, Parallel, and Cluster Computing arXiv:2603.26498 (cs) [Submitted on 27 Mar 2026] Title:Rocks, Pebbles and Sand: Modality-aware Scheduling for Multimodal Large Language Model Inference Authors:Konstantinos Papaioannou, Thaleia Dimitra Doudali View a PDF of the paper titled Rocks, Pebbles and Sand: Modality-aware Scheduling for Multimodal Large Language Model Inference, by Konstantinos Papaioannou and Thaleia Dimitra Doudali View PDF HTML (experimental) Abstract:Multimodal Large Language Models (MLLMs) power platforms like ChatGPT, Gemini, and Copilot, enabling richer interactions with text, images, and videos. These heterogeneous workloads introduce additional inference stages, such as vision preprocessing and encoding, that inflate latency and memory demand. Existing LLM serving systems, optimized for text-only workloads, fail under multimodality: large requests (e.g., videos) monopolize resources, causing severe head-of-line blocking and performance degradation. Our key insight is that multimodal requests differ by orders of magnitude in resource demands, which we capture through a simple abstraction: videos behave like rocks, images like pebbles, and text like sand. We design RPS-Serve, a modality-aware scheduler that lets sand flow quickly through pebbles and rocks, ensuring interactive responsiveness while avoiding starvation. RPS-Serve classifies requests, prioritizes them dynamically, and applies aging to avoid starvation. Evaluation a...