Communication-Efficient Model Fusion

Abstract

We consider the problem of learning a federated model where the number of communication rounds is severely limited. We discuss recent works on model fusion, a special case of Federated Learning where only a single communication round is allowed. This setting has a unique feature where it is sufficient for clients to have a pre-trained model, but not the data. Data storage regulations such as GDPR make this setting appealing as the data can be immediately deleted after updating the local model before FL starts. However, model fusion methods are limited to relatively shallow neural network architectures. We discuss extensions of model fusion applicable to deep learning models that require more than one communication round, but remain very efficient in terms of communication budget, i.e., number of communication rounds and size of the messages exchanged between the clients and the server. We consider both homogeneous and heterogeneous client data scenarios, including scenarios where training on the aggregated data is suboptimal due to biases in the data. In addition to deep learning methods, we cover unsupervised settings such as mixture models, topic models, and hidden Markov models.

We compare the statistical efficiency of model fusion to that of a hypothetical centralized approach in which a learner with unlimited compute and storage capacity simply aggregates data from all clients and trains a model in a non-federated way. As we shall see, although the model fusion approach generally matches the convergence rate of the (hypothetical) centralized approach, it may not have the same efficiency. Further, this discrepancy between the centralized and federated approaches is amplified when client data is heterogeneous.