Byzantine-Robust Loopless Stochastic Variance-Reduced Gradient

Abstract

Distributed optimization with open collaboration is a popular field since it provides an opportunity for small groups/companies/universities, and individuals to jointly solve huge-scale problems. However, standard optimization algorithms are fragile in such settings due to the possible presence of so-called Byzantine workers – participants that can send (intentionally or not) incorrect information instead of the one prescribed by the protocol (e.g., send anti-gradient instead of stochastic gradients). Thus, the problem of designing distributed methods with provable robustness to Byzantine workers has been receiving a lot of attention recently. In particular, several works consider a very promising way to achieve Byzantine tolerance via exploiting variance reduction and robust aggregation. The existing approaches use SAGA- and SARAH-type variance reduced estimators, while another popular estimator – SVRG – is not studied in the context of Byzantine-robustness. In this work, we close this gap in the literature and propose a new method – Byzantine-Robust Loopless Stochastic Variance Reduced Gradient (BR-LSVRG). We derive non-asymptotic convergence guarantees for the new method in the strongly convex case and compare its performance with existing approaches in numerical experiments.

The research was supported by the Ministry of Science and Higher Education of the Russian Federation (Goszadaniye) 075-00337-20-03, project No. 0714-2020-0005.

A Examples of Robust Aggregators

In [20], the authors propose the procedure called bucketing (see Algorithm 2) that robustifies certain aggregation rules such as: • geometric median (GM) $\hat{x} = \arg \min _{x\in \mathbb {R}^d}\sum _{i=1}^n \Vert x - x_i\Vert$; • coordinate-wise median (CM) $\hat{x} = \arg \min _{x\in \mathbb {R}^d}\sum _{i=1}^n \Vert x - x_i\Vert _1$; • Krum estimator [3] $\arg \min _{x_i \in \{x_1, \ldots , x_n\}} \sum _{j \in S_i} \Vert x_j - x_i\Vert ^2$, where $S_i \subseteq \{x_1, \ldots , x_n\}$ is the subset of $n - |\mathcal{B}| - 2$ closest (w.r.t. $\ell _2$-norm) vectors to $x_i$.

The following result establishes the robustness of the aforementioned aggregation rules in combination with Bucketing.

Theorem 3

(Theorem D.1 from [12]). Assume that $\{x_1,x_2,\ldots ,x_n\}$ is such that there exists a subset $\mathcal{G}\subseteq [n]$, $|\mathcal{G}| = G \ge (1-\delta )n$ and $\sigma \ge 0$ such that $\frac{1}{G(G-1)}\sum _{i,l \in \mathcal{G}}\mathbb {E}\Vert x_i - x_l\Vert ^2 \le \sigma ^2$. Assume that $\delta \le \delta _{\max }$. If Algorithm 2 is run with , then

• GM $\circ$ Bucketing satisfies Definition 1 with $c = \mathcal{O}(1)$ and ,

• CM $\circ$ Bucketing satisfies Definition 1 with $c = \mathcal{O}(d)$ and ,

• Krum $\circ$ Bucketing satisfies Definition 1 with $c = \mathcal{O}(1)$ and .