[2604.04231] Subspace Control: Turning Constrained Model Steering into Controllable Spectral Optimization

Computer Science > Machine Learning arXiv:2604.04231 (cs) [Submitted on 5 Apr 2026] Title:Subspace Control: Turning Constrained Model Steering into Controllable Spectral Optimization Authors:Yancheng Huang, Changsheng Wang, Chongyu Fan, Yicheng Lang, Bingqi Shang, Yang Zhang, Mingyi Hong, Qing Qu, Alvaro Velasquez, Sijia Liu View a PDF of the paper titled Subspace Control: Turning Constrained Model Steering into Controllable Spectral Optimization, by Yancheng Huang and 9 other authors View PDF HTML (experimental) Abstract:Foundation models, such as large language models (LLMs), are powerful but often require customization before deployment to satisfy practical constraints such as safety, privacy, and task-specific requirements, leading to "constrained" optimization problems for model steering and adaptation. However, solving such problems remains largely underexplored and is particularly challenging due to interference between the primary objective and constraint objectives during optimization. In this paper, we propose a subspace control framework for constrained model training. Specifically, (i) we first analyze, from a model merging perspective, how spectral cross-task interference arises and show that it can be resolved via a one-shot solution that orthogonalizes the merged subspace; (ii) we establish a connection between this solution and gradient orthogonalization in the spectral optimizer Muon; and (iii) building on these insights, we introduce SIFT (spectral interf...