Improving the Resilience of Gradient-Based Distributed Learning Algorithms with FABA

Abstract

Distributed learning allows the implementation of algorithms that require much more computing power or memory capacity than a single machine can provide. However, this poses the problem of the confidence that must be placed in the work of each of the computing nodes. In this paper we are interested in the ability of the distributed gradient descent algorithm to tolerate Byzantine nodes and successfully converge to the correct optimum. The FABA algorithm of Qi Xia et al. tolerates up to 30% of Byzantine workers. We propose three new variants of this algorithm to improve its resilience to Byzantine workers. We realise it by using the distance to the isobarycenter, the interquartile range and the Z-score to identify Byzantine workers. The conducted experiments show that the proposed variants enhance the resilience of FABA. While the FABA algorithm tolerates 30% of Byzantine workers, the proposed variants tolerate up to 45%. Additionally, the variant based on the Z-score achieves convergence in less than 100 epochs, whereas the original FABA algorithm requires more than 150 epochs to reach the same level of convergence. In summary, the new proposed variants improve the resilience of the distributed gradient descent algorithm FABA by tolerating a higher percentage of Byzantine workers and accelerating convergence towards the desired optimum.