![[2602.19945] DP-FedAdamW: An Efficient Optimizer for Differentially Private Federated Large Models](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.19945] DP-FedAdamW: An Efficient Optimizer for Differentially Private Federated Large Models
Summary
The paper introduces DP-FedAdamW, a novel optimizer designed for differentially private federated learning, addressing key challenges in convergence efficiency and robustness.
Why It Matters
As federated learning becomes increasingly important for privacy-preserving machine learning, optimizing performance while maintaining data privacy is crucial. DP-FedAdamW offers a solution that enhances training efficiency and reduces client drift, making it significant for both researchers and practitioners in AI.
Key Takeaways
- DP-FedAdamW stabilizes second-moment variance and removes bias in differentially private federated learning.
- The optimizer achieves a linearly accelerated convergence rate without heterogeneity assumptions.
- Empirical results show DP-FedAdamW outperforms existing methods by a notable margin on benchmark datasets.
Computer Science > Machine Learning arXiv:2602.19945 (cs) [Submitted on 23 Feb 2026] Title:DP-FedAdamW: An Efficient Optimizer for Differentially Private Federated Large Models Authors:Jin Liu, Yinbin Miao, Ning Xi, Junkang Liu View a PDF of the paper titled DP-FedAdamW: An Efficient Optimizer for Differentially Private Federated Large Models, by Jin Liu and 3 other authors View PDF HTML (experimental) Abstract:Balancing convergence efficiency and robustness under Differential Privacy (DP) is a central challenge in Federated Learning (FL). While AdamW accelerates training and fine-tuning in large-scale models, we find that directly applying it to Differentially Private FL (DPFL) suffers from three major issues: (i) data heterogeneity and privacy noise jointly amplify the variance of second-moment estimator, (ii) DP perturbations bias the second-moment estimator, and (iii) DP amplify AdamW sensitivity to local overfitting, worsening client drift. We propose DP-FedAdamW, the first AdamW-based optimizer for DPFL. It restores AdamW under DP by stabilizing second-moment variance, removing DP-induced bias, and aligning local updates to the global descent to curb client drift. Theoretically, we establish an unbiased second-moment estimator and prove a linearly accelerated convergence rate without any heterogeneity assumption, while providing tighter $(\varepsilon,\delta)$-DP guarantees. Our empirical results demonstrate the effectiveness of DP-FedAdamW across language and vision ...
