![[2602.18308] JPmHC Dynamical Isometry via Orthogonal Hyper-Connections](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.18308] JPmHC Dynamical Isometry via Orthogonal Hyper-Connections
Summary
The paper presents JPmHC, a framework enhancing deep learning stability by replacing identity skips in residual connections with a trainable linear mixer, improving performance and reducing memory overhead.
Why It Matters
This research addresses critical challenges in deep learning, such as training instability and scalability issues. By introducing JPmHC, the authors provide a solution that not only enhances model performance but also offers insights into the design of more efficient neural architectures, which is vital for advancing AI technologies.
Key Takeaways
- JPmHC replaces identity skips with a trainable linear mixer to enhance stability.
- The framework improves convergence speed, accuracy, and reduces computational costs.
- It introduces a free-probability analysis for better mixer selection.
- Memory-efficient implicit differentiation is utilized to lower activation memory.
- JPmHC contributes to the evolution of topological architecture design in deep learning.
Computer Science > Machine Learning arXiv:2602.18308 (cs) [Submitted on 20 Feb 2026] Title:JPmHC Dynamical Isometry via Orthogonal Hyper-Connections Authors:Biswa Sengupta, Jinhua Wang, Leo Brunswic View a PDF of the paper titled JPmHC Dynamical Isometry via Orthogonal Hyper-Connections, by Biswa Sengupta and 2 other authors View PDF Abstract:Recent advances in deep learning, exemplified by Hyper-Connections (HC), have expanded the residual connection paradigm by introducing wider residual streams and diverse connectivity patterns. While these innovations yield significant performance gains, they compromise the identity mapping property of residual connections, leading to training instability, limited scalability, and increased memory overhead. To address these challenges, we propose JPmHC (Jacobian-spectrum Preserving manifold-constrained Hyper-Connections), a framework that replaces identity skips with a trainable linear mixer acting on n parallel streams while explicitly controlling gradient conditioning. By constraining the mixer M on operator-norm-bounded manifolds (e.g., bistochastic, Stiefel, Grassmann), JPmHC prevents gradient pathologies and enhances stability. JPmHC introduces three key contributions: (i) a free-probability analysis that predicts Jacobian spectra for structured skips, providing actionable design rules for mixer selection; (ii) memory-efficient implicit differentiation for fixed-point projections, reducing activation memory and synchronization ove...
