[2604.04411] Responses Fall Short of Understanding: Revealing the Gap between Internal Representations and Responses in Visual Document Understanding

Computer Science > Computation and Language arXiv:2604.04411 (cs) [Submitted on 6 Apr 2026] Title:Responses Fall Short of Understanding: Revealing the Gap between Internal Representations and Responses in Visual Document Understanding Authors:Haruka Kawasaki, Ryota Tanaka, Kyosuke Nishida View a PDF of the paper titled Responses Fall Short of Understanding: Revealing the Gap between Internal Representations and Responses in Visual Document Understanding, by Haruka Kawasaki and 2 other authors View PDF HTML (experimental) Abstract:Visual document understanding (VDU) is a challenging task for large vision language models (LVLMs), requiring the integration of visual perception, text recognition, and reasoning over structured layouts. Although recent LVLMs have shown progress on VDU benchmarks, their performance is typically evaluated based on generated responses, which may not necessarily reflect whether the model has actually captured the required information internally. In this paper, we investigate how information required to solve VDU tasks is represented across different layers of LLMs within LVLMs using linear probing. Our study reveals that (1) there is a clear gap between internal representations and generated responses, and (2) information required to solve the task is often encoded more linearly from intermediate layers than from the final layer. Motivated by these findings, we explore fine-tuning strategies that target intermediate layers. Experiments show that fin...