[2603.19611] Demonstrations, CoT, and Prompting: A Theoretical Analysis of ICL

Computer Science > Machine Learning arXiv:2603.19611 (cs) [Submitted on 20 Mar 2026] Title:Demonstrations, CoT, and Prompting: A Theoretical Analysis of ICL Authors:Xuhan Tong, Yuchen Zeng, Jiawei Zhang View a PDF of the paper titled Demonstrations, CoT, and Prompting: A Theoretical Analysis of ICL, by Xuhan Tong and 2 other authors View PDF HTML (experimental) Abstract:In-Context Learning (ICL) enables pretrained LLMs to adapt to downstream tasks by conditioning on a small set of input-output demonstrations, without any parameter updates. Although there have been many theoretical efforts to explain how ICL works, most either rely on strong architectural or data assumptions, or fail to capture the impact of key practical factors such as demonstration selection, Chain-of-Thought (CoT) prompting, the number of demonstrations, and prompt templates. We address this gap by establishing a theoretical analysis of ICL under mild assumptions that links these design choices to generalization behavior. We derive an upper bound on the ICL test loss, showing that performance is governed by (i) the quality of selected demonstrations, quantified by Lipschitz constants of the ICL loss along paths connecting test prompts to pretraining samples, (ii) an intrinsic ICL capability of the pretrained model, and (iii) the degree of distribution shift. Within the same framework, we analyze CoT prompting as inducing a task decomposition and show that it is beneficial when demonstrations are well ch...