[2603.00196] Your Inference Request Will Become a Black Box: Confidential Inference for Cloud-based Large Language Models

Computer Science > Cryptography and Security arXiv:2603.00196 (cs) [Submitted on 27 Feb 2026] Title:Your Inference Request Will Become a Black Box: Confidential Inference for Cloud-based Large Language Models Authors:Chung-ju Huang, Huiqiang Zhao, Yuanpeng He, Lijian Li, Wenpin Jiao, Zhi Jin, Peixuan Chen, Leye Wang View a PDF of the paper titled Your Inference Request Will Become a Black Box: Confidential Inference for Cloud-based Large Language Models, by Chung-ju Huang and 7 other authors View PDF HTML (experimental) Abstract:The increasing reliance on cloud-hosted Large Language Models (LLMs) exposes sensitive client data, such as prompts and responses, to potential privacy breaches by service providers. Existing approaches fail to ensure privacy, maintain model performance, and preserve computational efficiency simultaneously. To address this challenge, we propose Talaria, a confidential inference framework that partitions the LLM pipeline to protect client data without compromising the cloud's model intellectual property or inference quality. Talaria executes sensitive, weight-independent operations within a client-controlled Confidential Virtual Machine (CVM) while offloading weight-dependent computations to the cloud GPUs. The interaction between these environments is secured by our Reversible Masked Outsourcing (ReMO) protocol, which uses a hybrid masking technique to reversibly obscure intermediate data before outsourcing computations. Extensive evaluations show ...