![[2602.15580] How Vision Becomes Language: A Layer-wise Information-Theoretic Analysis of Multimodal Reasoning](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.15580] How Vision Becomes Language: A Layer-wise Information-Theoretic Analysis of Multimodal Reasoning
Summary
This paper analyzes how multimodal Transformers integrate visual and linguistic information, revealing a layer-wise evolution of predictive information through an innovative framework.
Why It Matters
Understanding the interplay between visual and linguistic reasoning in AI models is crucial for improving multimodal systems. This research provides insights into how information is processed and can guide future architectural designs to enhance performance in tasks requiring both vision and language.
Key Takeaways
- Introduces a layer-wise framework to analyze multimodal Transformers.
- Finds that visual-unique information peaks early, while language-unique information increases in later layers.
- Demonstrates that cross-modal synergy remains low, suggesting distinct processing pathways.
- Establishes causal relationships through targeted attention knockouts.
- Offers quantitative guidance for identifying architectural bottlenecks in multimodal systems.
Computer Science > Artificial Intelligence arXiv:2602.15580 (cs) [Submitted on 17 Feb 2026] Title:How Vision Becomes Language: A Layer-wise Information-Theoretic Analysis of Multimodal Reasoning Authors:Hongxuan Wu, Yukun Zhang, Xueqing Zhou View a PDF of the paper titled How Vision Becomes Language: A Layer-wise Information-Theoretic Analysis of Multimodal Reasoning, by Hongxuan Wu and 2 other authors View PDF HTML (experimental) Abstract:When a multimodal Transformer answers a visual question, is the prediction driven by visual evidence, linguistic reasoning, or genuinely fused cross-modal computation -- and how does this structure evolve across layers? We address this question with a layer-wise framework based on Partial Information Decomposition (PID) that decomposes the predictive information at each Transformer layer into redundant, vision-unique, language-unique, and synergistic components. To make PID tractable for high-dimensional neural representations, we introduce \emph{PID Flow}, a pipeline combining dimensionality reduction, normalizing-flow Gaussianization, and closed-form Gaussian PID estimation. Applying this framework to LLaVA-1.5-7B and LLaVA-1.6-7B across six GQA reasoning tasks, we uncover a consistent \emph{modal transduction} pattern: visual-unique information peaks early and decays with depth, language-unique information surges in late layers to account for roughly 82\% of the final prediction, and cross-modal synergy remains below 2\%. This traject...
