[2603.04464] Understanding the Dynamics of Demonstration Conflict in In-Context Learning

Computer Science > Machine Learning arXiv:2603.04464 (cs) [Submitted on 3 Mar 2026] Title:Understanding the Dynamics of Demonstration Conflict in In-Context Learning Authors:Difan Jiao, Di Wang, Lijie Hu View a PDF of the paper titled Understanding the Dynamics of Demonstration Conflict in In-Context Learning, by Difan Jiao and 2 other authors View PDF HTML (experimental) Abstract:In-context learning enables large language models to perform novel tasks through few-shot demonstrations. However, demonstrations per se can naturally contain noise and conflicting examples, making this capability vulnerable. To understand how models process such conflicts, we study demonstration-dependent tasks requiring models to infer underlying patterns, a process we characterize as rule inference. We find that models suffer substantial performance degradation from a single demonstration with corrupted rule. This systematic misleading behavior motivates our investigation of how models process conflicting evidence internally. Using linear probes and logit lens analysis, we discover that under corruption models encode both correct and incorrect rules in intermediate layers but develop prediction confidence only in late layers, revealing a two-phase computational structure. We then identify attention heads for each phase underlying the reasoning failures: Vulnerability Heads in early-to-middle layers exhibit positional attention bias with high sensitivity to corruption, while Susceptible Heads i...