Early Exists Federated Learning (EEFL): Brining Training to the Edge

Abstract

With the proliferation of IoT devices, the core network is experiencing overwhelming congestion. Edge Com-puting emerges as a valuable and promising solution to relocate computing and intelligence from the cloud to the edge, where end-user devices are located. Consequently, reducing the increasing privacy concerns about hosting data in the cloud. However, this transition presents numerous challenges due to the heterogeneity and limited resources available on edge devices. To cope with those challenges, Edge Intelligence has risen as a research domain that investigates various optimization and distribution techniques that allow AI models to be well-hosted at the Edge. The majority of the current work focuses on implementing model inference at the Edge by deploying multiple methods such as Quantization, Pruning, Early-Exit models, Knowledge Distillation, etc. Moreover, very few works consider training AI models at the Edge. Consequently, there is always a constant need to upload data to the cloud or Fog for incremental learning. This not only causes constant privacy concern but also goes against the future wave of bringing intelligence to where the end-users reside. Following that, and via the verdict of this article, we present a novel federated learning technique that enables edge devices to collaborate in training deep learning models locally. By drawing inspiration from federated learning and early-exit neural networks, we have designed a horizontally distributed learning technique that allows all connected edge devices to contribute to the training based on their available resources. If an edge device has limited memory and computing power, it will train the model up to the first or second early exit. Conversely, if an edge device possesses enough computing resources, it can train the entire model. Our approach allows us to exploit all edge devices' local data without violating privacy concerns. More importantly, our EEFL training approach enhances accuracy from 3% to 30% compared to training exclusively with powerful edge computing devices alone. Our real world implementation also shows that training at the Edge with EEFL is possible and reduces the training time up to 1h 10 min on LeNet and 47h on AlexNet on average.