Hybrid ADMM: a unifying and fast approach to decentralized optimization
 616 Downloads
Abstract
The present work introduces the hybrid consensus alternating direction method of multipliers (HCADMM), a novel framework for optimization over networks which unifies existing distributed optimization approaches, including the centralized and the decentralized consensus ADMM. HCADMM provides a flexible tool that leverages the underlying graph topology in order to achieve a desirable sweet spot between nodetonode communication overhead and rate of convergence—thereby alleviating known limitations of both CCADMM and DCADMM. A rigorous analysis of the novel method establishes linear convergence rate and also guides the choice of parameters to optimize this rate. The novel hybrid update rules of HCADMM lend themselves to “innetwork acceleration” that is shown to effect considerable—and essentially “freeofcharge”—performance boost over the fully decentralized ADMM. Comprehensive numerical tests validate the analysis and showcase the potential of the method in tackling efficiently, widely useful learning tasks.
Keywords
ADMM Distributed optimization Decentralized learning Hybrid Consensus1 Introduction
Recent advances in machine learning, signal processing, and data mining have led to important problems that can be formulated as distributed optimization over networks. Such problems entail parallel processing of data acquired by interconnected nodes and arise frequently in several applications, including data fusion and processing using sensor networks [1, 2, 3, 4], vehicle coordination [5, 6], power state estimation [7], clustering [8], classification [9], regression [10], filtering [11], and demodulation [12, 13], to name a few. Among the candidate solvers for such problems, the alternating direction method of multipliers (ADMM) [14, 15] stands out as an efficient and easily implementable algorithm of choice that has attracted much interest in recent years [16, 17, 18, 19], thanks to its simplicity, fast convergence, and easily decomposable structure.
Many distributed optimization problems can be formulated in a consensus form and solved efficiently by ADMM [15, 20]. The solver involves two basic steps: (i) a communication step for exchanging information with a central processing unit, the socalled fusion center (FC), and (ii) an update step for updating the local variables at each node. By alternating between the two, local iterates eventually converge to the global solution. This approach is referred to as centralized consensus ADMM (CCADMM), and although it has been successfully applied in various settings, it may not always present the preferable solver. In largescale systems for instance, the cost of connecting each node to the FC may become prohibitive as the overhead of communicating data to the FC may be overwhelming and the related storage requirement could surpass the capacity of a single FC. Furthermore, having one dedicated FC can lead to a single point of failure. In addition, there might be privacyrelated issues that restrict access to private data.
Decentralized optimization, on the other hand, forgoes with the FC by exchanging information only among singlehop neighbors. As long as the network is connected, local iterates can consent to the globally optimal decision variable, thanks to the aforementioned information exchange. This method—referred to as decentralized consensus ADMM (DCADMM)—has attracted considerable interest; see e.g., [20] for a review of applications in communications and networking. In largescale networks, DCADMM’s convergence slows down as the pernode information experiences large delays to reach remote destinations through multiple neighbortoneighbor communications.
1.1 Our contributions
 (i)
HCADMM features hybrid updates accommodating communications with both the FCs and singlehop neighbors, thus bridging centralized with fully decentralized updates. This makes HCADMM appealing for largescale networks with multiple local FCs—a situation none of the existing approached is designed to handle.
 (ii)
A novel formulation of DCADMM without duplicate constraints (dual variables commonly adopted by decentralized learning [7, 20, 21]) emerges simply by specializing the hybrid constraints to coincide with those arising from the purely neighborhoodbased formulation.
 (iii)
Linear convergence is established, along with a rate bound and specializes to C CADMM and DCADMM. The parameter setting to achieve the optimal bound is also provided.
 (iv)
HCADMM is flexible to deploy FCs as needed to maximize performance gains, thus striking a desirable tradeoff between the number of FCs deployed and convergence gain sought.
 (v)
The capability of handling hybrid constraints not only deals with mixed updates but also effects “innetwork acceleration” in decentralized operation without incurring noticeable increase in the overall complexity.
1.2 Related work
Distributed optimization over networks has attracted much attention since the seminal works in [14, 22], where gradientbased parallel algorithms were developed. Since then, several alternatives have been advocated, including subgradient methods [23, 24, 25, 26], stochastic subgradients [27], dual averaging [28, 29], and gossip algorithms [30].
The decomposability of CADMM makes it particularly well suited for distributed optimization. Along with its many variants, including centralized [14], decentralized [20, 21], weighted [3, 31], and Nesterov accelerated [17], CADMM has gained wide popularity.
ADMM was introduced in the 1970s [32], and its convergence analysis can be found in, e.g., [33, 34]. Local linear convergence of ADMM for linear or quadratic programs is established in [35]; see also [18] where the cost is a sum of component costs. Global linear convergence of a more general form of ADMM is reported in [19], and linear convergence for a generalized formulation of consensus ADMM using the socalled “communication matrix” in [31].
Though DCADMM has been applied to various problems [3, 4, 10, 12, 13, 36], its linear convergence remained open until recently [21] (see [37] for the weighted counterpart). A successive orthogonal projection approach for distributed learning over networked nodes is introduced in [38], where nodes cannot communicate, but each node can access only limited amounts of data, and agreement is enforced across nodes sharing the same data. A distributed ADMM algorithm that deals with node clusters was proposed, for which linear convergence was also substantiated [39]. However, it relies on the gossip algorithm for communication within clusters when cluster head is absent and the explicit rate bound is difficult to obtains. Moreover, it admits no closedform representation for general networks. The present contribution is the first principled attempt to tackling the hybrid consensus problem.
1.3 Outline and notation
The rest of the paper is organized as follows. Section 2 states the problem and outlines two existing solvers, namely CCADMM and DCADMM; Section 3 develops HCADMM and shows its connections to both CCADMM and DCADMM; Section 4 establishes linear convergence of HCADMM and discusses parameter settings that can afford optimal performance; Section 5 introduces the notion of “innetwork acceleration;” Section 6 reports the results of numerical tests; and Section 7 concludes this work.
Notation. Vectors are represented by bold lower case, and matrices by bold upper case letters; I_{N} denotes the identity matrix of size N×N, and 1(0) the all ones (zeros) vector of appropriate size.
2 Preliminaries
where f_{i}(·) is the ith cost—only available to node i; and \( {\mathbf {x}}\in {\mathbb {{R}}}^{l} \) is the common decision variable.
where {x_{i}} is the local copy and equality is enforced to ensure equivalence of (1) with (2). As a result, the global decision variable in (1) is successfully decoupled to facilitate distributed processing.
Each node optimizes locally its component of cost, and the equality constraints are effected by exchanging information among nodes, subject to restrictions. Indeed, in the centralized case, nodes communicate with a single FC, while in the fully decentralized case, nodes can only communicate with their immediate neighbors.
We model communication constraints in the decentralized setting as an undirected graph \( \mathcal {G}:= ({\mathcal {V}}, {\mathcal {E}}) \), with each vertex {v_{i}} corresponding to one node, and the presence of edge \( (v_{i}, v_{j}) \in {\mathcal {E}} \) denoting that nodes i and j can communicate. With N(resp. M) denoting the number of nodes (edges), we will label nodes (edges) using the set {1,2,…,N} (resp. {1,2,…,M}). We will further define the neighborhood set of node i as \( {\mathcal {N}}_{i} := \{j(v_{i}, v_{j}) \in {\mathcal {E}}\} \).
The following assumptions will be adopted about the graph and the local cost functions.
Assumption 1
(Connectivity) Graph \( \mathcal {G}:= ({\mathcal {V}}, {\mathcal {E}}) \) is connected.
Assumption 2
Assumption 3
For brevity, we will henceforth focus on l=1, but Appendix A outlines the generalization to l≥2.
2.1 Centralized consensus ADMM
where z represents the GFC’s decision variable (state).
2.2 Decentralized consensus ADMM
In the decentralized setting, no GFC is present and nodes can only communicate with their onehop neighbors. If the underlying graph \({\mathcal {{G}}}\) is connected, consensus constraints effect agreement across nodes.
where z is the vector concatenating all {z_{ij}} in arbitrary order, \( {\mathbf {A}} := \left [{\mathbf {A}}_{1}^{\top }, {\mathbf {A}}_{2}^{\top }\right ]^{\top } \) with \( {\mathbf {A}}_{1}, {\mathbf {A}}_{2} \in {\mathbb {{R}}}^{N\times 2M} \) defined such that if the qth element of z is z_{ij}, then (A_{1})_{qi}=1, (A_{2})_{qj}=1, and all other elements are zeros; while \( {\mathbf {B}} := \left [{{\mathbf {{I}}}}_{2M}^{\top }, {{\mathbf {{I}}}}_{2M}^{\top }\right ]^{\top } \).
where y:=(A_{1}−A_{2})^{⊤}β, and the inverse in (7a) is a shorthand for \( {x}_{i}^{k+1} \) being the solution of \( \nabla f_{i}\left ({x}_{i}^{k+1}\right) + {\rho } {\mathcal {N}}{x}_{i}^{k+1} = ({\rho }/2) \sum _{j\in {\mathcal {N}}_{i}}\left ({x}_{i}^{k+1} + {x}_{j}^{k+1}\right) \).
The fact that the pernode updates in (7) involve only singlehop neighbors justifies the term decentralized consensus ADMM (DCADMM).
3 Hybrid consensus ADMM
Rather than a single GFC that is connected to all nodes, here, we consider optimization over networks with multiple LFCs. Such a setup can arise in largescale networks, where bandwidth, power, and computational limits or even security concerns may discourage deployment of a single GFC. These considerations prompt the deployment of multiple LFCs each of which communicates with a limited number of nodes. No prior ADMMbased solver can deal with this setup as none is capable of handling hybrid constraints that are present when nodes exchange information not only with LFCs but also with their singlehop neighbors. This section introduces HCADMM that is particularly designed to handle this situation.
3.1 Problem formulation
In contrast to the simple graph \({\mathcal {{G}}}\) used in the fully decentralized setting, we will employ hypergrahs to cope with hybrid constraints. A hypergraph is a tuple \( {\mathcal {{H}}}:=({\mathcal {{V}}}, {\mathcal {{E}}}) \), where \( {\mathcal {{V}}} \) is the vertex set and \( {\mathcal {{E}}}:=\{{\mathcal {{E}}}_{i}\}_{i=1}^{M} \) denotes the collection of hyperedges. Each \( {\mathcal {{E}}}_{i} \) comprises a set of vertices, \({\mathcal {{E}}}_{i}\subset {\mathcal {{V}}} \) with cardinality \( {\mathcal {{E}}}_{i} \geq 2, \:\:\forall i \). If \( {\mathcal {{E}}}_{i} = 2\), then it reduces to a simple edge. A vertex v_{i} and an edge \( {\mathcal {{E}}}_{j} \) are said to be incident if \( v_{i} \in {\mathcal {{E}}}_{j} \). Hypergraphs are particularly suitable for modeling hybrid constraints because each LFC can be modeled as one hyperedge consisting of all its connected nodes.
Let now \({\mathbf {C}} \in \mathbb {R}^{N\times M}\) denote the incidence matrix of the hypergraph, formed with entries C_{ij}=1 if node i and edge j are incident, and C_{ij}=0 otherwise; d_{i} is the degree of node i (the number of incident edges of node i), e_{j} is the degree of hyperedge j (the number of incident nodes of hyperedge j), diagonal matrix \({\mathbf {D}} \in \mathbb {R}^{N\times N}\) is the node degree matrix (formed with d_{i} as its ith diagonal element), and likewise, \({\mathbf {E}} \in \mathbb {R}^{M\times M} \) is the edge degree matrix (formed with \(e_{j} = {\mathcal {{E}}}_{j} \) as its jth diagonal element). With these notational conventions, we prove in the Appendix the following.
Lemma 1
One can easily verify that each row of Ax−Bz=0 represents one constraint and Lemma 1 indeed holds.
3.2 Algorithm
Recursions (13)–(15) summarize our HCADMM, and their pernode forms are listed in Algorithm 1.
Remark 1
 (i)
Regardless of the number of attached nodes, each hyperedge serves as an LFC.
 (ii)
Each LFC performs local averaging. Indeed, the entrywise update of (13) shows that each hyperedge satisfies \( {z}_{i} = (1/E_{ii}) \sum _{j\in {\mathcal {N}}_{i}} {x}_{j} \). Hence, all hyperedges are treated equally in the sense that they are updated by the average value of all incident nodes.
3.3 Key relations
Here, we unveil a relationship satisfied by the iterates {x^{k}} generated by Algorithm 1, which not only provides a different view of HCADMM, but also serves as the starting point for establishing the convergence results in Section 4. This relation shows that x^{k+1} depends solely on the gradient of the local cost function, as well as the past {x^{k},x^{k−1},…,x^{0}} that is also not dependent on the variables z^{k} and y^{k}.
Lemma 2
Proof
from which we can readily solve for x^{k+1}. □
Lemma 2 shows that \( {x}_{i}^{k+1} \) is determined by its past \( \{{x}_{i}^{t}\}_{t=0}^{k}\) and the local gradient, namely \( \nabla f_{i}\left ({x}_{i}^{k+1}\right) \). This suggests a new update scheme, where each node maintains not only its current \( {x}_{i}^{k} \) but also \( \sum _{t=0}^{k} {x}_{i}^{t} \).
Among the things worth stressing in Lemma 2 is the appearance of matrices CE^{−1}C^{⊤} and D−CE^{−1}C^{⊤}. Since both play key roles in studying the evolution of (19), it is important to understand their properties and impact on the performance of the algorithm.
Lemma 3
Proof
See Appendix. □
3.4 HCADMM links to CCADMM and DCADMM
Modeling hybrid communication constraints as a hypergraph not only affords the flexibility to accommodate multiple LFCs, but also provides a unified view of consensusbased ADMM. Indeed, it is not difficult to show that by specializing the hypergraph, our proposed approach subsumes both centralized and decentralized consensus ADMM.
Proposition 1
HCADMM reduces to CCADMM when there is only one hyperedge capturing all nodes.
Proof
where we have used that y^{k}=A^{⊤}λ^{k}=λ^{k}.
Comparing (22) with (4), it is not difficult to see that (4a) is just the entrywise form of (22a) and likewise for (22b) and (4b). Therefore, CCADMM can be viewed as a special case of HCADMM with one hyperedge connecting all nodes. □
Proposition 2
HCADMM reduces to DCADMM when every edge is a hyperedge.
Proof
To relate \(\sum _{j\in {\mathcal {N}}{}}({x}_{i} + {x}_{j})\) in (7b) to (24b), notice that \( D_{ii} = {\mathcal {N}}_{i} \) and \( ({\mathbf {C}}{\mathbf {C}}^{\top {\mathbf {x}}})_{i} = \sum _{j\in {\mathcal {N}}_{i}} ({x}_{i} + {x}_{j}) \). Hence, it is straightforward to see that (7a) and (7b) are just the entrywise versions of (24a) and (24b). Therefore, DCADMM is also a special case of HCADMM with each simple edge viewed as a hyperedge. □
Remark 2
In past works of fully decentralized consensus ADMM [10, 12, 20, 21], one edge is often associated with two variables z_{ij}, z_{ji} in order to decouple the equality constraint x_{i}=x_{j} and express it as x_{i}=z_{ij}, x_{j}=z_{ji}. Although eventually the duplicate variables are shown equal (and therefore discarded), their sheer presence leads to duplicate Lagrange multipliers, which can complicate the algorithm. Proposition 2 suggests a novel derivation of DCADMM with only one variable attached to each edge, a property that can simplify the whole process considerably.
Example 1
Comparison of ADMM update rules at node 4 using CCADMM, DCADMM, and HCADMM to solve the problem in Example 1
Method  Update rules 

CCADMM  \({x}_{4}^{k+1} = \underset {{x}_{4}}{\text {arg\,min}} \left [ f_{4}({x}_{4}) + {\lambda }_{4}^{k}\left ({x}_{4}  \bar {{x}}^{k}\right) + \frac {{\rho }}{2}\{x}_{4}  \bar {{x}}^{k}\_{2}^{2}\right ] \) 
\( {\lambda }_{4}^{k+1} = {\lambda }_{4}^{k} + {\rho }{}\left ({x}_{4}^{k+1}  \bar {{x}}^{k+1}\right)\)  
DCADMM  \({x}_{4}^{k+1} = (\nabla f_{4} + 2{\rho } I)^{1} \left ({\rho } {x}_{4}^{k} + \frac {{\rho }}{2}\left ({x}_{2}^{k} + {x}_{5}^{k}\right)  {y}_{4}^{k}\right) \) 
\({y}_{4}^{k+1} = {y}_{4}^{k} + \frac {{\rho }}{2}\left (2{x}_{4}^{k+1}  {x}_{2}^{k+1}  {x}_{5}^{k+1}\right) \)  
HCADMM  \({x}_{4}^{k+1} = \underset {{x}_{4}}{\text {arg\,min}} \Bigg [ f_{4}({x}_{4}) + {y}_{4}^{k} {x}_{4} + \frac {{\rho }}{2} \Big (\{x}_{4}  \frac {1}{4}\sum _{i=1}^{4} {x}_{i}^{k}\^{2} + \{x}_{4}  \frac {1}{2} \sum _{i=1}^{2} {x}_{i}^{k}\^{2} \Big)\Bigg ] \) 
\({y}_{4}^{k+1} = {y}_{4}^{k} + {\rho }\left (2{x}_{4}^{k+1}  \frac {1}{4}\sum _{i=1}^{4} {x}_{i}^{k+1}  \frac {1}{2} \sum _{i=4}^{5} {x}_{i}^{k+1}\right) \) 
As CCADMM relies on a GFC connecting all nodes, every node receives updates from the GFC. This is demonstrated in the first row of Table 1.
DCADMM on the other hand, allows communication only along edges (solid line). Thus, each node can only receive updates from its singlehop neighbors. Specifically, node 4 can only receive information from nodes 2 and 5, as can be seen in the second row of Table 1.
HCADMM lies somewhere in between. It allows deployment of multiple LFCs, each of which is connected to a subset of nodes. The hypergraph model is shown in Fig. 1 with hyperedges marked by ellipsoids. The union of 3 solid ellipsoids corresponds to the LFC, while dash and dotted ones represent the edges between nodes 4, 5, and 6. Speaking of node 4, the information needed to update its local variables comes from its neighbors, the LFC represented by \( \sum _{i=1}^{4} {x}_{i}^{k+1} / 4 \); and node 5 represented by \( \left (x_{4}^{k+1} + x_{5}^{k+1}\right)/2 \). That is exactly what we see in the third row of Table 1.
Remark 3
For all three consensus ADMM algorithms, the fusion centers—both global and local—act as averaging operators that compute, store, and broadcast the mean values of the local estimates from all connected nodes.
Remark 4
Remark 5
4 Convergence rate analysis
In this section, we analyze the convergence behavior of the novel HCADMM algorithm. In particular, our main theorem establishes linear convergence and provides a bound on the rate of convergence, which depends on properties of both the objective function, as well as the underlying graph topology.
Apart from the assumptions made in Section 2, here, we also need an additional one:
Assumption 4
This assumption is required for the development of Algorithm 1 as well as for the analysis of its convergence rate. If (as4) does not hold, either the original problem is unsolvable or it entails unbounded subproblems or a diverging sequence of λ^{k} [19].
Assumptions 1–4 guarantee the existence of at least one optimal solution. In fact, we can further prove that any saddle point is actually the unique solution of the KKT conditions (25) and hence of Algorithm 1.
Lemma 4
If λ is initialized so that B^{⊤}λ^{0}=0, and (as1)–(as4) hold, then (x^{⋆},z^{⋆},y^{⋆}) is the unique optimal solution of (25).
Proof
See Appendix. □
4.1 Linear rate of convergence
Alternating direction methods, including ADMM, have been thoroughly investigated in [19]. Similar to DCADMM [21], conditions for establishing linear convergence rate in [19] are not necessarily satisfied by the HCADMM setup^{1}. Therefore, we cannot establish linear convergence rate simply by reformulating it as a special case of existing ADMM approaches.
One way to overcome this obstacle is to adopt a technique similar to [21], as we did in [40], to obtain a relatively loose bound on convergence, in the sense that it could not capture significant accelerations observed in practice by varying the topology of the underlying graph. For strongly convex costs, a tighter bound has been reported recently [31]. However, HCADMM is not amenable to the analysis in [31] since the linear constraint coefficients A and B cannot be recovered from the communication matrix.
These two sequences play an important role in establishing linear convergence of the proposed algorithm. Before we establish such a convergence result, we first need to bound the gradient of F(·).
Lemma 5
Proof
See. □
Let Λ:=λ_{N}(CE^{−1}C^{⊤}) denote the largest eigenvalue of CE^{−1}C^{⊤} and λ:=λ_{2}(D−CE^{−1}C^{⊤}) the second smallest eigenvalue of D−CE^{−1}C^{⊤}.
Theorem 1
Proof
See. □
Theorem 1 asserts that x^{k} converges linearly to the optimal solution x^{⋆} at a rate bounded by 1/(1+δ). Larger δ implies faster convergence.
Note that while Theorem 1 is proved for l=1, it can be generalized to l≥2 (see (40) in Appendix A).
Remark 6
Assumptions 2 and 3 are sufficient conditions for establishing linear convergence rate of HCADMM.
4.2 Finetuning the parameters
Theorem 1 characterizes the convergence of iterates generated by HCADMM. Parameter δ is determined by the local costs, the underlying communication graph topology, and the scalar ρ. By tuning these parameters, one can maximize the convergence bound to speed up convergence in practice too. With local costs and the graph fixed, one can maximize δ by tuning ρ. When possible to choose the number and locations of LFCs, we can effectively alter the topology of the communication graph—hence modify Λ and λ—to improve convergence.
Theorem 2
Proof
□
Clearly, the bound in (38) is a decreasing function of κ_{F} and κ_{G}. Therefore, decreasing both will drive the bound larger, possibly resulting in a faster rate of convergence. On the one hand, smaller cost condition number makes the cost easier to optimize; on the other hand, smaller graph condition number implies improved connectivity. Indeed, when the communication hypergraph is just a simple graph—a case for which HCADMM reduces to DCADMM with D−CE^{−1}C^{⊤}=L/2, then λ is the smallest nonzero eigenvalue of the Laplacian, which is related to bottlenecks in the underlying graph [41].
Remark 7
Theorem 2 shows that the number of iterations it takes to achieve an εaccurate solution is \( {\mathcal {O}\left ({\sqrt {\kappa _{F}}\log \left (\frac {1}{\epsilon }\right)}\right)} \). The dependence on \( 1/\sqrt {\kappa _{F}} \) improves over [21], which had established a dependence of \( 1/\kappa _{F}^{2} \).
5 Graphaware acceleration
Distributed optimization over networks using a central GFCs is not feasible for several reasons including communication constraints and privacy concerns. At the other end of the spectrum, decentralized schemes relying on singlehop communications may suffer from slow convergence, especially when the network has a large diameter or bottlenecks. HCADMM fills the gap by compromising between the two aforementioned extremes. By carefully deploying multiple LFCs, it becomes possible to achieve significant performance gains while abiding by cost and privacy constraints.

Hardware. Relying on virtual LFCs, innetwork acceleration requires no modifications in the actual topology and hardware.

Interface. The other nodes “see” exactly the same number of neighbors, so there is no change in the communication interface. However, the information exchanged is indeed different.

Computational complexity. Except for the host nodes, the update rules for both primal and dual variables remain the same. Each host however, serves a dual role: as an FC as well as an ordinary node. We know from Algorithm 1 that the computations performed per LFC involve averaging information from all connected nodes, which is simple compared to updating the local variables. Thus, the computational complexity remains of the same order, while the total computational cost decreases as less iterations are necessary to reach a target level of accuracy.

Communication cost. Since there is no change of the communication interface, the communication cost remains invariant. Once again, the total communication cost can further drop, since HCADMM enjoys faster convergence.
Given that the interface does not change, nor extra communication/computation cost is incurred, one can think of innetwork acceleration as a sort of “free lunch” approach, with particularly attractive practical implications.
Remark 8
The communication cost per iteration refers to the total number of transmission needed for updating each variable once, hence the total communication cost. In the graph of Fig. 3, the nodes need to transmit 2×5 times to finish one iteration (each node must send and receive once to update its variables and there are 5 edges), so the communication cost per iteration is 10 transmissions.
5.1 Strategies for selecting the local FCs
A reasonable question to ask at this point is: “How should one select the nodes to host the virtual FCs?” Unfortunately, there is no simple answer. The question would have been easier if we could choose as many LFCs as necessary to achieve the maximum possible acceleration. In practice, however, we do not always have the luxury to place as many virtual LFCs as we want, for reasons that include lack of control over some nodes and difficulty to modify internal updating rules. And even if we could, picking the right nodes to host the LFCs under a general network architecture might not be straightforward.
A reasonable way forth would be to maximize the convergence rate bound, δ, subject to a maximum number of LFCs, hoping that the optimal solution would yield the best rate of convergence in practice. However, this turns out to involve optimizing the ratio of eigenvalues, which is typically difficult to solve. For this reason, we will resort to heuristic methods.
Intuitively, one may choose the nodes with highest degree so as to maximize the effect of virtual LFCs. However, one should be careful when applying this simple approach to clustered graphs. For example, consider the graph consisting of two cliques (connected by a short path), comprising n_{1} and n_{2} nodes, respectively. Each node in the larger clique has higher degree than every node in the smaller one. As a result, always assigning the role of LFC to the largest degree nodes would disregard the nodes of the smaller clique (when our budget is less than n_{1}), while one could apparently take care of both cliques with as few as two LFCs. Taking this into account, we advocate a greedy LFC selection (Algorithm 2), which prohibits placing virtual LFCs within the neighborhood of other FCs.
Remark 9
For simplicity, Algorithm 2 relies on degree information to select LFCs. More elaborate strategies would involve richer structural properties of the underlying topology to identify more promising nodes at the expense of possibly computationally heavier LFC selection. In general, LFC selection strategies offer the potential of substantially increasing the convergence rate over random assignment. Note however, that regardless of the choice of virtual LFCs, HCADMM remains operational.
5.2 On HCADMM’s “free lunch”
Innetwork acceleration has several advantages, allowing for faster convergence essentially “without paying any price.” At first glance, this appears to be a “free lunch” type of benefit and deservedly makes one skeptical. Actually, the benefit comes from leveraging information that is completely overlooked by fully decentralized methods. To see this, recall that in DCADMM, each node communicates with only one neighbor each time, without accounting for the entire neighborhood. Instead, HCADMM manages to exploit network topology by creating virtual LFCs that gather and share information with the whole neighborhood. It is this additional information that enables faster flow of data and hence faster convergence. Therefore, it is not a “free lunch” for HCADMM, but a lunch “not even tasted” by DCADMM.
6 Numerical tests
In this section, we test numerically the performance of HCADMM and also validate our analytical findings.
6.1 Experimental settings
Throughout this section, we consider several interconnected nodes trying to estimate one value, x_{0}, based on local observations o_{i}=x_{0}+ε_{i}, where \( \epsilon _{i} \sim {\mathcal {N}}(0, 0.1) \) is the measurement noise. This can be solved by minimizing the leastsquares (LS) error \( F({\mathbf {x}}) = \frac {1}{2} \sum _{i=1}^{N} \o_{i}  x_{i}\_{2}^{2} \). Different from centralized LS, here, the observation o_{i} is available only to node i, and all nodes collaborate to obtain the final solution. In DCADMM, each node can only talk to its neighbors, while in HCADMM nodes can potentially communicate with the LFC and their neighbors.
We test DCADMM and HCADMM solvers with various parameter settings. We assess convergence using the relative accuracy metric defined as ∥x^{k}−x^{⋆}∥_{2}/∥x^{⋆}∥_{2} and report the number of iterations as well as the communication cost involved in reaching a target level of performance. The communication cost measures how many times local, and global decision variables are transferred across the network. Originally, we set ρ according to Theorem 2, but this choice did not work well our tests. For this reason, we tuned it manually to reach the best possible performance.
6.2 Acceleration of dedicated FCs
 HCADMM with mixed updates works well in practice. Solutions are obtained in fewer iterations and at lower communication cost.

The tests verify the linear convergence properties of both DCADMM and HCADMM, as can be seen in all four graphs.

The performance gap between dashed lines and solid lines of the same marker confirms the acceleration ability of HCADMM. The gap is larger for “badly connected” graphs, such as the lollipop graph and relatively small for “wellconnected” graphs, such as the Erdős Rényi graphs. In fact, this observation holds even when comparing the Erdős Rényi graphs. Indeed, for the ER(r=0.05) graph which is not as well connected as the ER(r=0.1), the performance gap is smaller.

Figure 5 suggests that the more nodes are connected to the FC, the larger the acceleration gains for HCADMM, which is intuitively reasonable since extra connections pay off. In view of this connectionsversusacceleration tradeoff, HCADMM can reach desirable sweet spots between performance gains and deployment cost.
6.3 Innetwork acceleration
In this test, we demonstrate the performance gain effected by innetwork acceleration. By creating virtual FCs among nodes, this technique does not require dedicated FCs and new links. As detailed in Section 5, innetwork accelerated HCADMM exchanges information along existing edges, essentially leading to a communication cost that follows the same pattern with iteration complexity. Therefore, in this experiment, we report both metrics in one figure.
All the results illustrate that HCADMM with innetwork acceleration offers a significant boost in convergence rate over DCADMM, especially for graphs with relatively large diameters (or graphs that are not well connected), such as the line graph, the cycle graph, or the lollipop graph. On the other hand, the performance gain is minimal for the star graph, whose diameter is 2 regardless of the number of nodes, as well as the Erdős Rényi random graph with high edge probability (see Fig. 7). Note that these performance gains over DCADMM are achieved without paying a substantial computational cost (just one averaging step), which speaks for the practical merits of HCADMM.
6.4 Tradeoff between FCs and performance gain
Finally, we explore the tradeoff between the number of LFCs and the corresponding convergence rate. We perform tests on several graphs with different properties and using only HCADMM with innetwork acceleration. In this test, we measure performance by the number of iterations needed to achieve a target accuracy of 10^{−8}, given a varying number of LFCs ranging from 1 to 25.
 For Erdős Rényi graphs with a relatively highedge probability, there is a small initial gain arising from the introduction of virtual LFCs, which diminishes fast as their number increases. This is not surprising since such graphs are “well connected,” and therefore, adding more virtual LFCs does not help much. On the other hand, for “badly connected” graphs, such as the line graph or the lollipop graph, there is a significant convergence boost as the number of LFCs increases.

For the line and the cycle graph, a significant change in convergence rate happens only after an initial threshold has been surpassed (in our case 5–6 LFCs). For the lollipop graph, there seems to exist a cutoff point above which adding more nodes does not lead to significant change in convergence rate.
7 Conclusion and future work
This paper introduces the novel HCADMM algorithm that generalizes the centralized and the decentralized CADMM, while also accelerating DCADMM with modified updates.
We establish linear convergence of HCADMM, and we also conduct a comprehensive set of numerical tests that validate our analytical findings and demonstrate the effectiveness of the proposed approach in practice.
A very promising direction we are currently pursuing involves the development of techniques leveraging the intrinsic hierarchical organization that is commonly found in distributed system architectures (see, e.g., [42, 43]) as well as realworld largescale network topologies (see, e.g., [44, 45]).
8 Appendix A: Algorithm 1 for l>2
Relative to (13)–(15), the computational complexity of (39) is clearly higher. To see this, consider the perstep complexity for l=1. Given that E is a diagonal matrix, naive matrixvector multiplication incurs complexity \({\mathcal {O}({MN})}\), which can be reduced to \({\mathcal {O}({M {E_{\text {max}}}})}\) by exploiting the sparsity of C, where E_{max} is the largest edge degree. When l≥2 however, perstep complexity grows to \( {\mathcal {O}({l^{2}M{E_{\text {max}}}})} \) which—being quadratic in l—would make it difficult for the method to handle highdimensional data. Thankfully, a compact form of (39) made possible by Proposition 3 reduces the complexity from quadratic to linear in l.
Proposition 3
Proof
which concludes the proof. □
Proposition 3 establishes that HCADMM can run using much smaller matrices, effectively reducing its computational complexity. To see this, consider the perstep complexity of (40). The difference with (39) is dominated by
C^{⊤}X, which leads to complexity \({\mathcal {O}({lMN})}\). This can be further reduced to \({\mathcal {O}({lM{E_{\text {max}}}})}\) by exploiting the sparsity of C, thus improving the complexity of (39) by a factor of l.
9 Proof of Lemma 1
from which we obtain A^{⊤}A=D and B^{⊤}B=E.
Consider now the dot product \({{\mathbf {{a}}}}_{i}^{\top }{{\mathbf {{b}}}_{j}}\). When the tth constraint reads x_{i}=z_{j}, it holds by construction that (a_{i})_{t}=1, and (b_{j})_{t}=1. Therefore, if node i and edge j are incident, we have \({{\mathbf {{a}}}}_{i}^{\top }{{\mathbf {{b}}}_{j}} = 1\); otherwise, \({{\mathbf {{a}}}}_{i}^{\top }{{\mathbf {{b}}}_{j}} = 0\). Thus, the incidence matrix definition implies that A^{⊤}B=C.
10 Proof of Lemma 3
where the last matrix is a valid Laplacian (\(d_{i} = \sum _{j=1}^{N} c_{ij}\)), which in turn implies that it is PSD.
11 Proof of Lemma 4
Feasibility of (25c) guarantees the existence of at least one solution, and therefore, there exists at least one optimal solution. The uniqueness of x^{⋆} and z^{⋆} follows from the strong convexity of F(·) and the dual feasibility (25c).
from which we find that x_{1}=x_{2}, which is a contradiction. Therefore, λ_{1}=λ_{2}, and thus, y=A^{⊤}λ is also unique.
12 Proof of Lemma 5
Combining this with (43) completes the proof.
13 Proof of Theorem 1
Footnotes
 1.
This should be expected since HCADMM reduces to DCADMM upon modeling each simple edge as an hyperedge.
Notes
Acknowledgements
The authors wish to thank Prof. M. Hong from University of Minnesota for insightful discussions on related subjects. Part of this work was supported by NSF grants 1442686, 1500713, 1509040, and 171141.
Authors’ contributions
All authors read and approved the final manuscript.
Competing interests
The authors declare that they have no competing interests.
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
References
 1.L. Xiao, S. Boyd, S. Lall, in Proceedings of the 4th International Symposium on Information Processing in Sensor Networks. IPSN ’05. A scheme for robust distributed sensor fusion based on average consensus (IEEELos Angeles, 2005), pp. 63–70.Google Scholar
 2.R. OlfatiSaber, in Proceedings of the 44th IEEE Conference on Decision and Control. Distributed Kalman Filter with Embedded Consensus Filters (IEEESeville, 2005), pp. 8179–8184.Google Scholar
 3.Q. Ling, Z. Tian, Decentralized sparse signal recovery for compressive sleeping wireless sensor networks. IEEE Trans. Signal Process.58(7), 3816–3827 (2010).MathSciNetzbMATHGoogle Scholar
 4.I. D. Schizas, A. Ribeiro, G. B. Giannakis, Consensus in ad hoc WSNs with noisy links—part I: distributed estimation of deterministic signals. IEEE Trans. Signal Process.56(1), 350–364 (2008).MathSciNetzbMATHGoogle Scholar
 5.W. Ren, R. W. Beard, E. M. Atkins, Information consensus in multivehicle cooperative control. IEEE Control Syst. Mag.27(2), 71–82 (2007).Google Scholar
 6.W. Ren, in 2006 American Control Conference. Consensus based formation control strategies for multivehicle systems (Minneapolis, 2006), pp. 4237–4242.Google Scholar
 7.V. Kekatos, G. B. Giannakis, Distributed robust power system state estimation. IEEE Trans. Power Syst.28(2), 1617–1626 (2013).Google Scholar
 8.P. A. Forero, A. Cano, G. B. Giannakis, Distributed clustering using wireless sensor networks. IEEE J. Sel. Top. Signal Process.5(4), 707–724 (2011).Google Scholar
 9.P. A. Forero, A. Cano, G. B. Giannakis, Consensusbased distributed support vector machines. J. Mach. Learn. Res.11(May), 1663–1707 (2010).MathSciNetzbMATHGoogle Scholar
 10.G. Mateos, J. A. Bazerque, G. B. Giannakis, Distributed sparse linear regression. IEEE Trans. Signal Process.58(10), 5262–5276 (2010).MathSciNetzbMATHGoogle Scholar
 11.R. OlfatiSaber, J. S. Shamma, in Proceedings of the 44th IEEE Conference on Decision and Control. Consensus Filters for Sensor Networks and Distributed Sensor Fusion (IEEESeville, 2005), pp. 6698–6703.Google Scholar
 12.H. Zhu, A. Cano, G. Giannakis, Distributed consensusbased demodulation: algorithms and error analysis. IEEE Trans. Wirel. Commun.9(6), 2044–2054 (2010).Google Scholar
 13.J. A. Bazerque, G. B. Giannakis, Distributed spectrum sensing for cognitive radio networks by exploiting sparsity. IEEE Trans. Signal Process.58(3), 1847–1862 (2010).MathSciNetzbMATHGoogle Scholar
 14.D. P. Bertsekas, J. N. Tsitsiklis, Parallel and distributed computation: numerical methods vol. 23 (PrenticeHall, Inc., Upper Saddle River, 1989).zbMATHGoogle Scholar
 15.S. Boyd, N. Parikh, E. Chu, B. Peleato, J. Eckstein, Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends® Mach. Learn.3(1), 1–122 (2011).zbMATHGoogle Scholar
 16.T. Erseghe, D. Zennaro, E. Dall’Anese, L. Vangelista, Fast consensus by the alternating direction multipliers method. IEEE Trans. Signal Process.59(11), 5523–5537 (2011).MathSciNetzbMATHGoogle Scholar
 17.T. Goldstein, B. O’Donoghue, S. Setzer, R. Baraniuk, Fast alternating direction optimization methods. SIAM J. Imaging Sci.7(3), 1588–1623 (2014).MathSciNetzbMATHGoogle Scholar
 18.M. Hong, Z. Q. Luo, On the linear convergence of the alternating direction method of multipliers. Math. Program.162(12), 165–199 (2017).MathSciNetzbMATHGoogle Scholar
 19.W. Deng, W. Yin, On the global and linear convergence of the generalized alternating direction method of multipliers. J Sci Comput. 66(3), 889–916 (2015).MathSciNetzbMATHGoogle Scholar
 20.G. B. Giannakis, Q. Ling, G. Mateos, I. D. Schizas, H. Zhu, in Splitting Methods in Communication, Imaging, Science, and Engineering, ed. by R. Glowinski, S. J. Osher, and W. Yin. Decentralized learning for wireless communications and networking (Springer International PublishingCham, 2016), pp. 461–497.zbMATHGoogle Scholar
 21.W. Shi, Q. Ling, K. Yuan, G. Wu, W. Yin, On the linear convergence of the ADMM in decentralized consensus optimization. IEEE Trans. Signal Process.62(7), 1750–1761 (2014).MathSciNetzbMATHGoogle Scholar
 22.J. N. Tsitsiklis, Problems in Decentralized Decision Making and Computation (Massachusetts Institute of Technology, Cambridge, 1984).Google Scholar
 23.A. Nedic, A. Ozdaglar, Distributed subgradient methods for multiagent optimization. IEEE Trans. Autom. Control. 54(1), 48–61 (2009).MathSciNetzbMATHGoogle Scholar
 24.A. Nedic, A. Ozdaglar, P. A. Parrilo, Constrained consensus and optimization in multiagent networks. IEEE Trans. Autom. Control. 55(4), 922–938 (2010).MathSciNetzbMATHGoogle Scholar
 25.H. Terelius, U. Topcu, R. M. Murray, Decentralized multiagent optimization via dual decomposition. IFAC Proc. Vol.44(1), 11245–11251 (2011).Google Scholar
 26.K. I. Tsianos, M. G. Rabbat, in 50th Annual Allerton Conference on Communication, Control, and Computing (Allerton). Distributed strongly convex optimization (IEEEMonticello, 2012), pp. 593–600.Google Scholar
 27.S. Sundhar Ram, A. Nedić, V. V. Veeravalli, Distributed Stochastic Subgradient Projection Algorithms for Convex Optimization. J. Optim. Theory. Appl.147(3), 516–545 (2010).MathSciNetzbMATHGoogle Scholar
 28.JC Duchi, A Agarwal, MJ Wainwright, Dual averaging for distributed optimization: convergence analysis and network scaling. IEEE Trans. Autom. Control.57(3), 592–606 (2012).MathSciNetzbMATHGoogle Scholar
 29.K. Tsianos, S. Lawlor, M. G. Rabbat, in Advances in Neural Information Processing Systems 25, ed. by F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger. Communication/Computation Tradeoffs in ConsensusBased Distributed Optimization (Curran Associates, Inc.New York, 2012), pp. 1943–1951.Google Scholar
 30.A. G. Dimakis, S. Kar, J. M. F. Moura, M. G. Rabbat, A. Scaglione, Gossip algorithms for distributed signal processing. Proc. IEEE. 98(11), 1847–1864 (2010).Google Scholar
 31.A. Makhdoumi, A. Ozdaglar, Convergence rate of distributed ADMM over networks. IEEE Trans. Autom. Control. 62(10), 5082–5095 (2017).MathSciNetzbMATHGoogle Scholar
 32.R. Glowinski, A. Marroco, Sur l’approximation, par éléments finis d’ordre un, et la résolution, par pénalisationdualité d’une classe de problèmes de Dirichlet non linéaires. Rev. Fr. Autom. Inform. Rech. Opérationnelle Anal. Numér.9(R2), 41–76 (1975).zbMATHGoogle Scholar
 33.D. Gabay, in Studies in Mathematics and Its Applications. Augmented Lagrangian Methods: Applications to the Numerical Solution of BoundaryValue Problems, vol. 15, ed. by M. Fortin, R. Glowinski. Applications of the method of multipliers to variational inequalities (NorthHolland, Amsterdam, 1983), pp. 299–331.Google Scholar
 34.J. Eckstein, D. P. Bertsekas, On the Douglas–Rachford splitting method and the proximal point algorithm for maximal monotone operators. Math. Program.55(13), 293–318 (1992).MathSciNetzbMATHGoogle Scholar
 35.D. Boley, Local linear convergence of the alternating direction method of multipliers on quadratic or linear programs. SIAM J. Optim.23(4), 2183–2207 (2013).MathSciNetzbMATHGoogle Scholar
 36.J. A. Bazerque, G. Mateos, G. B. Giannakis, GroupLasso on splines for spectrum cartography. Signal Process. IEEE Trans. On. 59(10), 4648–4663 (2011).MathSciNetzbMATHGoogle Scholar
 37.Q. Ling, Y. Liu, W. Shi, Z. Tian, in 2016 IEEE International Conference on Acoustics, Speech and Signal Processing. Communicationefficient weighted ADMM for decentralized network optimization (IEEEShanghai, 2016), pp. 4821–4825.Google Scholar
 38.J. B. Predd, S. R. Kulkarni, H. V. Poor, A collaborative training algorithm for distributed learning. IEEE Trans. Inf. Theory. 55(4), 1856–1871 (2009).MathSciNetzbMATHGoogle Scholar
 39.F. Iutzeler, P. Bianchi, P. Ciblat, W. Hachem, Explicit convergence rate of a distributed alternating direction method of multipliers. IEEE Trans. Autom. Control. 61(4), 892–904 (2016).MathSciNetzbMATHGoogle Scholar
 40.M. Ma, A. N. Nikolakopoulos, G. B. Giannakis, in Proceedings of 2018 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP). Fast decentralized learning via hybrid consensus ADMM (IEEECalgary, 2018).Google Scholar
 41.F. R. K. Chung, Spectral Graph Theory vol. no. 92 (American Mathematical Society, Providence, R.I, 1997).Google Scholar
 42.J. P. Gross, R. B. Pandit, C. G. Cook, T. H. Matson, Network with distributed shared memory. US9317469B2: (2016).Google Scholar
 43.A. N. Nikolakopoulos, J. D. Garofalakis, in Proceedings of the 2016 PCI Conference. PCI ’16. On the multilevel near complete decomposability of a class of multiprocessing systems (ACMNew York, 2016), pp. 14–1146.Google Scholar
 44.M. Faloutsos, P. Faloutsos, C. Faloutsos, in Proceedings of the Conference on Applications, Technologies, Architectures, and Protocols for Computer Communication, SIGCOMM ’99. On powerlaw relationships of the internet topology (ACMNew York, 1999), pp. 251–262.zbMATHGoogle Scholar
 45.M. E. Newman, The structure and function of complex networks. SIAM Rev. 45(2), 167–256 (2003).MathSciNetzbMATHGoogle Scholar
 46.A. K. Jain, Fundamentals of Digital Image Processing (PrenticeHall, Inc., Upper Saddle River, 1989).zbMATHGoogle Scholar
Copyright information
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.