[2512.05525] Poodle: Seamlessly Scaling Down Large Language Models with Just-in-Time Model Replacement

Computer Science > Databases arXiv:2512.05525 (cs) [Submitted on 5 Dec 2025 (v1), last revised 4 May 2026 (this version, v2)] Title:Poodle: Seamlessly Scaling Down Large Language Models with Just-in-Time Model Replacement Authors:Nils Strassenburg, Boris Glavic, Tilmann Rabl View a PDF of the paper titled Poodle: Seamlessly Scaling Down Large Language Models with Just-in-Time Model Replacement, by Nils Strassenburg and 2 other authors View PDF HTML (experimental) Abstract:Businesses increasingly rely on large language models (LLMs) to automate simple repetitive tasks instead of developing custom machine learning models. LLMs require few, if any, training examples and can be utilized by users without expertise in model development. However, this comes at the cost of substantially higher resource and energy consumption compared to smaller models, which often achieve similar predictive performance for simple tasks. In this paper, we present our vision for just-in-time model replacement (JITR), where, upon identifying a recurring task in calls to an LLM, the model is replaced transparently with a cheaper alternative that performs well for this specific task. JITR retains the ease of use and low development effort of LLMs, while saving significant cost and energy. We discuss the main challenges in realizing our vision regarding the identification of recurring tasks and the creation of a custom model. Specifically, we argue that model search and transfer learning will play a cru...