[2603.20275] Understanding Pruning Regimes in Vision-Language Models Through Domain-Aware Layer Selection

Computer Science > Computer Vision and Pattern Recognition arXiv:2603.20275 (cs) [Submitted on 17 Mar 2026] Title:Understanding Pruning Regimes in Vision-Language Models Through Domain-Aware Layer Selection Authors:Saeed Khaki, Nima Safaei, Kamal Ginotra View a PDF of the paper titled Understanding Pruning Regimes in Vision-Language Models Through Domain-Aware Layer Selection, by Saeed Khaki and 2 other authors View PDF HTML (experimental) Abstract:Transformer-based vision-language models (VLMs) contain substantial depth redundancy, yet the effect of removing specific decoder layers remains poorly understood, especially for domains that require tight coupling between perception and multi-step reasoning. We study structured decoder layer pruning through the lens of domain-aware activation similarity, measuring how strongly each layer transforms representations for math versus non-math inputs. This yields simple math-aware, non-math-aware, and mixed ranking criteria that identify layers whose input-output activations change least within a target domain. Across two state-of-the-art VLMs and a broad suite of math and general multimodal benchmarks, we uncover a consistent three-regime structure: at low pruning budgets, performance is highly sensitive to which layers are removed; at moderate budgets, methods converge as structural damage accumulates; and at high budgets, structural continuity dominates, favoring spacing-aware strategies. Our domain-aware rankings achieve the str...