![[2602.19271] Taming Preconditioner Drift: Unlocking the Potential of Second-Order Optimizers for Federated Learning on Non-IID Data](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.19271] Taming Preconditioner Drift: Unlocking the Potential of Second-Order Optimizers for Federated Learning on Non-IID Data
Summary
This paper presents FedPAC, a framework to enhance the stability and accuracy of second-order optimizers in federated learning on non-IID data by addressing preconditioner drift.
Why It Matters
As federated learning becomes increasingly important in machine learning applications, ensuring the reliability of optimization methods on non-IID data is crucial. This research addresses a significant challenge in the field, potentially improving the performance of various AI models across different tasks.
Key Takeaways
- FedPAC framework improves stability and accuracy in federated learning.
- Addresses preconditioner drift caused by heterogeneous data.
- Achieves up to 5.8% accuracy gain on CIFAR-100 with ViTs.
- Decouples parameter aggregation from geometry synchronization.
- Provides convergence guarantees with linear speedup under partial participation.
Computer Science > Machine Learning arXiv:2602.19271 (cs) [Submitted on 22 Feb 2026] Title:Taming Preconditioner Drift: Unlocking the Potential of Second-Order Optimizers for Federated Learning on Non-IID Data Authors:Junkang Liu, Fanhua Shang, Hongying Liu, Jin Liu, Weixin An, Yuanyuan Liu View a PDF of the paper titled Taming Preconditioner Drift: Unlocking the Potential of Second-Order Optimizers for Federated Learning on Non-IID Data, by Junkang Liu and 5 other authors View PDF HTML (experimental) Abstract:Second-order optimizers can significantly accelerate large-scale training, yet their naive federated variants are often unstable or even diverge on non-IID data. We show that a key culprit is \emph{preconditioner drift}: client-side second-order training induces heterogeneous \emph{curvature-defined geometries} (i.e., preconditioner coordinate systems), and server-side model averaging updates computed under incompatible metrics, corrupting the global descent direction. To address this geometric mismatch, we propose \texttt{FedPAC}, a \emph{preconditioner alignment and correction} framework for reliable federated second-order optimization. \texttt{FedPAC} explicitly decouples parameter aggregation from geometry synchronization by: (i) \textbf{Alignment} (i.e.,aggregating local preconditioners into a global reference and warm-starting clients via global preconditioner); and (ii) \textbf{Correction} (i.e., steering local preconditioned updates using a global preconditio...