[2603.04422] FedEMA-Distill: Exponential Moving Average Guided Knowledge Distillation for Robust Federated Learning

Computer Science > Machine Learning arXiv:2603.04422 (cs) [Submitted on 15 Feb 2026] Title:FedEMA-Distill: Exponential Moving Average Guided Knowledge Distillation for Robust Federated Learning Authors:Hamza Reguieg, Mohamed El Kamili, Essaid Sabir View a PDF of the paper titled FedEMA-Distill: Exponential Moving Average Guided Knowledge Distillation for Robust Federated Learning, by Hamza Reguieg and 2 other authors View PDF HTML (experimental) Abstract:Federated learning (FL) often degrades when clients hold heterogeneous non-Independent and Identically Distributed (non-IID) data and when some clients behave adversarially, leading to client drift, slow convergence, and high communication overhead. This paper proposes FedEMA-Distill, a server-side procedure that combines an exponential moving average (EMA) of the global model with ensemble knowledge distillation from client-uploaded prediction logits evaluated on a small public proxy dataset. Clients run standard local training, upload only compressed logits, and may use different model architectures, so no changes are required to client-side software while still supporting model heterogeneity across devices. Experiments on CIFAR-10, CIFAR-100, FEMNIST, and AG News under Dirichlet-0.1 label skew show that FedEMA-Distill improves top-1 accuracy by several percentage points (up to +5% on CIFAR-10 and +6% on CIFAR-100) over representative baselines, reaches a given target accuracy in 30-35% fewer communication rounds, and re...