![[2602.17809] Calibrated Adaptation: Bayesian Stiefel Manifold Priors for Reliable Parameter-Efficient Fine-Tuning](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.17809] Calibrated Adaptation: Bayesian Stiefel Manifold Priors for Reliable Parameter-Efficient Fine-Tuning
Summary
This paper introduces Stiefel-Bayes Adapters (SBA), a Bayesian framework for parameter-efficient fine-tuning of large language models, enhancing predictive uncertainty and performance across various benchmarks.
Why It Matters
The research addresses the limitations of existing parameter-efficient fine-tuning methods by providing a principled approach to uncertainty estimation. This is crucial for improving the reliability of AI models in real-world applications, especially under domain shifts.
Key Takeaways
- SBA uses a Matrix Langevin prior on the Stiefel manifold for improved fine-tuning.
- It achieves significant reductions in Expected Calibration Error compared to traditional methods.
- The framework outperforms existing models in OOD detection while being more parameter-efficient.
- SBA maintains orthogonality constraints, enhancing the conditioning of adapter subspaces.
- The research demonstrates the importance of geometric structure in Bayesian treatments.
Computer Science > Machine Learning arXiv:2602.17809 (cs) [Submitted on 19 Feb 2026] Title:Calibrated Adaptation: Bayesian Stiefel Manifold Priors for Reliable Parameter-Efficient Fine-Tuning Authors:Ibne Farabi Shihab, Sanjeda Akter, Anuj Sharma View a PDF of the paper titled Calibrated Adaptation: Bayesian Stiefel Manifold Priors for Reliable Parameter-Efficient Fine-Tuning, by Ibne Farabi Shihab and 2 other authors View PDF HTML (experimental) Abstract:Parameter-efficient fine-tuning methods such as LoRA enable practical adaptation of large language models but provide no principled uncertainty estimates, leading to poorly calibrated predictions and unreliable behavior under domain shift. We introduce Stiefel-Bayes Adapters (SBA), a Bayesian PEFT framework that places a Matrix Langevin prior over orthonormal adapter factors on the Stiefel manifold $\St$ and performs approximate posterior inference via tangent space Laplace approximation with geodesic retraction. Unlike Gaussian priors in flat space projected onto orthogonality constraints, our prior on the manifold naturally encodes the inductive bias that adapter subspaces should be well conditioned and orthogonal, while the posterior provides calibrated predictive uncertainty without recalibration. We prove formally that the tangent space approximation strictly avoids the structural variance inflation inherent in projecting from ambient space, establishing a rigorous theoretical advantage for intrinsic manifold inferen...
