Knowledge Distillation in Federated Learning: Where and How to Distill?

Federated learning (FL) advances the field of distributed machine learning for facilitating the privacy-preserving management of edge devices and central servers. However, the majority of data among edge devices is non-IID (not Independent and Identically Distributed), making it challenging to achieve edge intelligence. In this work, we attempt to mitigate the above issue through knowledge distillation (KD) by sharing knowledge between the central server and edge devices. Specifically, we first investigate where (i.e. which feature layer) to conduct the distillation for knowledge sharing. We find that setting the feature layer to that before the classification head yields superior performance. Moreover, we investigate how to conduct the KD in terms of loss choices. We test various types of losses for enhancing the knowledge sharing and find that Center Kernel Alignment (CKA) achieves the best performance among the investigated loss metrics. Overall, this work sheds new light on where and how to perform KD in FL. Experimental results on MNIST and Fashion-MNIST demonstrate that our finding yields a performance gain of at least 4%.