[2603.24239] DVM: Real-Time Kernel Generation for Dynamic AI Models

Computer Science > Programming Languages arXiv:2603.24239 (cs) [Submitted on 25 Mar 2026] Title:DVM: Real-Time Kernel Generation for Dynamic AI Models Authors:Jingzhi Fang, Xiong Gao, Renwei Zhang, Zichun Ye, Lei Chen, Jie Zhao, Chengnuo Huang, Hui Xu, Xuefeng Jin View a PDF of the paper titled DVM: Real-Time Kernel Generation for Dynamic AI Models, by Jingzhi Fang and 8 other authors View PDF HTML (experimental) Abstract:Dynamism is common in AI computation, e.g., the dynamic tensor shapes and the dynamic control flows in models. Due to the long compilation time, existing runtime compilation damages the model efficiency, while the offline compilers either suffer from the long compilation time and device memory footprint to cover all the possible execution instances of a dynamic model, or sacrifice optimization opportunities for usability. In this paper, we rethink the feasibility of runtime compilation for dynamic models and identify that the key for it to work is to speed up the compilation or hide the compilation overhead. To do this, we propose a real-time compiler, DVM. In DVM, we design a runtime operator compiler based on a bytecode virtual machine to perform effective and efficient compilation for each dynamic operator instance given its input. Specifically, instead of compiling programs into machine code, we encode the operator program into bytecode on the CPU and decode the bytecode into virtual instructions for direct execution on the NPU. Based on the runtime o...