Fast and stable nonconvex constrained distributed optimization: the ELLADA algorithm

Abstract

Distributed optimization using multiple computing agents in a localized and coordinated manner is a promising approach for solving large-scale optimization problems, e.g., those arising in model predictive control (MPC) of large-scale plants. However, a distributed optimization algorithm that is computationally efficient, globally convergent, amenable to nonconvex constraints remains an open problem. In this paper, we combine three important modifications to the classical alternating direction method of multipliers for distributed optimization. Specifically, (1) an extra-layer architecture is adopted to accommodate nonconvexity and handle inequality constraints, (2) equality-constrained nonlinear programming (NLP) problems are allowed to be solved approximately, and (3) a modified Anderson acceleration is employed for reducing the number of iterations. Theoretical convergence of the proposed algorithm, named ELLADA, is established and its numerical performance is demonstrated on a large-scale NLP benchmark problem. Its application to distributed nonlinear MPC is also described and illustrated through a benchmark process system.

Appendices

Appendix 1: Proof of Lemma 1

We first prove that

$$\begin{aligned} \begin{aligned}&L\left( x^{k,r+1}, {\bar{x}}^{k,r+1}, z^{k,r+1}, y^{k,r+1}\right) \le L\left( x^{k,r}, {\bar{x}}^{k,r}, z^{k,r}, y^{k,r}\right) \\&\quad - \beta ^k\left\| B{\bar{x}}^{k,r+1} - B{\bar{x}}^{k,r}\right\| ^2 - \frac{\beta ^k}{2}\left\| z^{k,r+1} - z^{k,r}\right\| ^2 \end{aligned} \end{aligned}$$

(62)

for $r=0, 1, 2, \dots$. First, since $x^{k,r+1}$ is chosen as the minimizer of the augmented Lagrangian with respect to x (Line 9, Algorithm 1), the update of x leads to a decrease in L:

$$\begin{aligned} L\left( x^{k,r+1}, {\bar{x}}^{k,r}, z^{k,r}, y^{k,r}\right) \le L\left( x^{k,r}, {\bar{x}}^{k,r}, z^{k,r}, y^{k,r}\right) . \end{aligned}$$

(63)

Then consider the decrease resulted from ${\bar{x}}$-update:

$$\begin{aligned} \begin{aligned}&L\left( x^{k,r+1}, {\bar{x}}^{k,r+1}, z^{k,r}, y^{k,r}\right) - L\left( x^{k,r+1}, {\bar{x}}^{k,r}, z^{k,r}, y^{k,r}\right) \\&\quad = g\left( {\bar{x}}^{k,r+1}\right) - g\left( {\bar{x}}^{k,r}\right) + y^{k,r\top }\left( B{\bar{x}}^{k,r+1} - B{\bar{x}}^{k,r}\right) \\&\qquad + \frac{\rho ^k}{2} \left\| Ax^{k,r+1} + B{\bar{x}}^{k,r+1} + z^{k,r}\right\| ^2 - \frac{\rho ^k}{2} \left\| Ax^{k,r+1} + B{\bar{x}}^{k,r} + z^{k,r}\right\| ^2 \\&\quad = g\left( {\bar{x}}^{k,r+1}\right) - g\left( {\bar{x}}^{k,r}\right) - \frac{\rho ^k}{2}\left\| B{\bar{x}}^{k,r+1}-B{\bar{x}}^{k,r}\right\| ^2 \\&\qquad -\rho ^k \left( {\bar{x}}^{k,r} - {\bar{x}}^{k,r+1}\right) ^\top B^\top \left( Ax^{k,r+1} + B{\bar{x}}^{k,r+1} + z^{k,r} + \frac{y^{k,r}}{\rho ^k} \right) . \end{aligned} \end{aligned}$$

(64)

The minimization of ${\bar{x}}$ (Line 10, Algorithm 1) should satisfy the optimality condition

$$\begin{aligned} 0 \in \rho ^k B^\top \left( Ax^{k,r+1} + B{\bar{x}}^{k,r+1} + z^{k,r} + \frac{y^{k,r}}{\rho ^k} \right) + \partial g\left( {\bar{x}}^{k,r+1}\right) + {\mathcal {N}}_{\bar{{\mathcal {X}}}} \left( {\bar{x}}^{k,r+1}\right) , \end{aligned}$$

(65)

i.e., there exist vectors $v_1\in \partial g\left( {\bar{x}}^{k,r+1}\right)$ and $v_2 \in {\mathcal {N}}_{\bar{{\mathcal {X}}}}\left( {\bar{x}}^{k,r+1}\right)$ with

$$\begin{aligned} \rho ^k B^\top \left( Ax^{k,r+1} + B{\bar{x}}^{k,r+1} + z^{k,r} + \frac{y^{k,r}}{\rho ^k} \right) = -v_1 - v_2. \end{aligned}$$

(66)

Since $v_1\in \partial g\left( {\bar{x}}^{k,r+1}\right)$ and g is convex, $v_1^\top \left( {\bar{x}}^{k,r} - {\bar{x}}^{k,r+1}\right) \le g\left( {\bar{x}}^{k,r}\right) - g\left( {\bar{x}}^{k,r+1}\right)$. And $v_2 \in {\mathcal {N}}_{\bar{{\mathcal {X}}}} \left( {\bar{x}}^{k,r+1}\right)$ implies $v_2^\top \left( {\bar{x}}^{k,r} - {\bar{x}}^{k,r+1}\right) \le 0$. Hence

$$\begin{aligned} \begin{aligned}&\rho ^k \left( {\bar{x}}^{k,r} - {\bar{x}}^{k,r+1}\right) ^\top B^\top \left( Ax^{k,r+1} + B{\bar{x}}^{k,r+1} + z^{k,r} + \frac{y^{k,r}}{\rho ^k} \right) \\&\quad = -v_1^\top \left( {\bar{x}}^{k,r} - {\bar{x}}^{k,r+1}\right) - v_2^\top \left( {\bar{x}}^{k,r} - {\bar{x}}^{k,r+1}\right) \\&\quad \ge -\left( g\left( {\bar{x}}^{k,r}\right) - g\left( {\bar{x}}^{k,r+1}\right) \right) . \end{aligned} \end{aligned}$$

(67)

Substituting the above inequality in (64), we obtain

$$\begin{aligned}&L\left( x^{k,r+1}, {\bar{x}}^{k,r+1}, z^{k,r}, y^{k,r}\right) \le L\left( x^{k,r+1}, {\bar{x}}^{k,r}, z^{k,r}, y^{k,r}\right) \nonumber \\&\quad -\frac{\rho ^k}{2} \left\| B{\bar{x}}^{k,r+1}-B{\bar{x}}^{k,r}\right\| ^2. \end{aligned}$$

(68)

Third, we consider the decrease resulted from z- and y-updates:

$$\begin{aligned} \begin{aligned}&L\left( x^{k,r+1}, {\bar{x}}^{k,r+1}, z^{k,r+1}, y^{k,r+1}\right) - L\left( x^{k,r+1}, {\bar{x}}^{k,r+1}, z^{k,r}, y^{k,r}\right) \\&\quad = \lambda ^{k\top } \left( z^{k,r+1}-z^{k,r} \right) + \frac{\beta ^k}{2} \left( \left\| z^{k,r+1}\right\| ^2 - \left\| z^{k,r}\right\| ^2 \right) \\&\qquad + y^{k,r+1\top } \left( Ax^{k,r+1} + B{\bar{x}}^{k,r+1} + z^{k,r+1}\right) \\&\qquad - y^{k,r\top } \left( Ax^{k,r+1} + B{\bar{x}}^{k,r+1} + z^{k,r}\right) \\&\qquad + \frac{\rho ^k}{2} \left\| Ax^{k,r+1} + B{\bar{x}}^{k,r+1} + z^{k,r+1}\right\| ^2 \\&\qquad - \frac{\rho ^k}{2} \left\| Ax^{k,r+1} + B{\bar{x}}^{k,r+1} + z^{k,r}\right\| ^2. \end{aligned} \end{aligned}$$

(69)

Since $\upsilon (z; \lambda , \beta )=\lambda ^\top z + \frac{\beta }{2}\Vert z\Vert ^2$ is a convex function, whose gradient is $\nabla \upsilon (z; \lambda , \beta ) = \lambda + \beta z$,

$$\begin{aligned} \upsilon \left( z^{k,r+1}; \lambda ^k, \beta ^k \right) - \upsilon \left( z^{k,r}; \lambda ^k, \beta ^k \right) \le \left( \lambda ^k + \beta ^k z^{k,r+1} \right) ^\top \left( z^{k,r+1}-z^{k,r} \right) , \end{aligned}$$

(70)

From Line 11 of Algorithm 1 it can be obtained

$$\begin{aligned} \lambda ^k + \beta z^{k,r+1} = -y^{k,r+1}. \end{aligned}$$

(71)

Substituting into (69), we obtain

$$\begin{aligned} \begin{aligned}&L\left( x^{k,r+1}, {\bar{x}}^{k,r+1}, z^{k,r+1}, y^{k,r+1}\right) - L\left( x^{k,r+1}, {\bar{x}}^{k,r+1}, z^{k,r}, y^{k,r}\right) \\&\quad \le \left( y^{k,r+1}- y^{k,r}\right) ^\top \left( Ax^{k,r+1} + B{\bar{x}}^{k,r+1} + z^{k,r}\right) \\&\qquad + \frac{\rho ^k}{2} \left\| Ax^{k,r+1} + B{\bar{x}}^{k,r+1} + z^{k,r+1}\right\| ^2 - \frac{\rho ^k}{2} \left\| Ax^{k,r+1} + B{\bar{x}}^{k,r+1} + z^{k,r}\right\| ^2 \\&\quad = \frac{\rho ^k}{2} \left( Ax^{k,r+1} + B{\bar{x}}^{k,r+1} + z^{k,r+1}\right) ^\top \left( Ax^{k,r+1} + B{\bar{x}}^{k,r+1} + z^{k,r}\right) \\&\qquad + \frac{\rho ^k}{2} \left\| Ax^{k,r+1} + B{\bar{x}}^{k,r+1} + z^{k,r+1}\right\| ^2 - \frac{\rho ^k}{2} \left\| Ax^{k,r+1} + B{\bar{x}}^{k,r+1} + z^{k,r}\right\| ^2 \\&\quad = -\frac{\rho ^k}{2} \left\| z^{k,r+1}-z^{k,r}\right\| ^2 + \rho ^k \left\| Ax^{k,r+1} + B{\bar{x}}^{k,r+1} + z^{k,r+1}\right\| ^2 \end{aligned} \end{aligned}$$

(72)

From (71),

$$\begin{aligned} Ax^{k,r+1} + B{\bar{x}}^{k,r+1} + z^{k,r+1} = \frac{1}{\rho _k} \left( y^{k,r+1} - y^{k,r}\right) = -\frac{\beta ^k}{\rho ^k} \left( z^{k,r+1}-z^{k,r}\right) . \end{aligned}$$

(73)

Then (72) becomes

$$\begin{aligned} \begin{aligned}&L\left( x^{k,r+1}, {\bar{x}}^{k,r+1}, z^{k,r+1}, y^{k,r+1}\right) - L\left( x^{k,r+1}, {\bar{x}}^{k,r+1}, z^{k,r}, y^{k,r}\right) \\&\quad \le -\left( \frac{\rho ^k}{2}-\frac{\left( \beta ^k\right) ^2}{\rho ^k}\right) \left\| z^{k,r+1}-z^{k,r}\right\| ^2 = -\frac{\beta ^k}{2} \left\| z^{k,r+1}-z^{k,r}\right\| ^2. \end{aligned} \end{aligned}$$

(74)

Summing up the inequalities (63), (68) and (74), we have proved the inequality (62).

Next, we show that the augmented Lagrangian is lower bounded, and hence is convergent towards some ${\underline{L}}^k\in {\mathbb {R}}$. We note that $\upsilon (z; \lambda , \beta )$ is a convex function of modulus $\beta$, it can be easily verified that

$$\begin{aligned}&\upsilon \left( z^{k,r}; \lambda ^k, \beta ^k \right) + \left( \lambda ^k + \beta ^k z^{k,r}\right) ^\top \left( z^\prime - z^{k,r}\right) \nonumber \\&\quad + \frac{\rho ^k}{2} \left\| z^\prime - z^{k,r}\right\| ^2 \ge \upsilon \left( z^\prime ; \lambda ^k, \beta ^k\right) \end{aligned}$$

(75)

for any $z^\prime$, i.e.,

$$\begin{aligned} \upsilon \left( z^{k,r}; \lambda ^k, \beta ^k\right) + y^{k,r\top } \left( z^{k,r} - z^\prime \right) \ge \upsilon \left( z^\prime ; \lambda ^k, \beta ^k\right) - \frac{\rho ^k}{2}\left\| z^\prime - z^{k,r}\right\| ^2. \end{aligned}$$

(76)

Let $z^\prime = -\left( Ax^{k,r} + B{\bar{x}}^{k,r}\right)$ and remove the last term on the right-hand side. Then

$$\begin{aligned}&\upsilon \left( z^{k,r}; \lambda ^k, \beta ^k\right) + y^{k,r\top } \left( Ax^{k,r} + B{\bar{x}}^{k,r} + z^{k,r}\right) \nonumber \\&\quad \ge \upsilon \left( - \left( Ax^{k,r} + B{\bar{x}}^{k,r}\right) ; \lambda ^k, \beta ^k\right) . \end{aligned}$$

(77)

Hence

$$\begin{aligned} \begin{aligned}&L\left( x^{k,r}, {\bar{x}}^{k,r+1}, z^{k,r}, y^{k,r}\right) \\&\quad = f\left( x^{k,r}\right) + g\left( {\bar{x}}^{k,r}\right) + \upsilon \left( z^{k,r}; \lambda ^k, \beta ^k\right) \\&\qquad + y^{k,r\top } \left( Ax^{k,r} + B{\bar{x}}^{k,r} + z^{k,r}\right) + \frac{\rho ^k}{2} \left\| Ax^{k,r} + B{\bar{x}}^{k,r} + z^{k,r}\right\| ^2 \\&\quad \ge f\left( x^{k,r}\right) + g\left( {\bar{x}}^{k,r}\right) + \upsilon \left( -\left( Ax^{k,r} + B{\bar{x}}^{k,r}\right) ; \lambda ^k, \beta ^k\right) . \\ \end{aligned} \end{aligned}$$

(78)

Since $\upsilon (z) = \lambda ^\top z+\frac{\beta }{2}\Vert z\Vert ^2 \ge -\Vert \lambda \Vert ^2/(2\beta )$, $\lambda$ is bounded in $\left[ {\underline{\lambda }}, {\overline{\lambda }} \right]$, $\beta ^k\ge \beta ^1$, and f and g are bounded below, L has a lower bound.

Taking the limit $r\rightarrow \infty$ on the both sides of inequality (62), it becomes obvious that $B{\bar{x}}^{k,r+1}-B{\bar{x}}^{k,r}$ and $z^{k,r+1}-z^{k,r}$ converge to 0. Due to (73), we have $Ax^{k,r} + B{\bar{x}}^{k,r} + z^{k,r} \rightarrow 0$. Hence there must exist a r such that (22) is met. At this time, the optimality conditions for $x^{k,r+1}$ is written as

$$\begin{aligned} 0 \in \partial f\left( x^{k,r+1}\right) + {\mathcal {N}}_{\mathcal {X}}\left( x^{k,r+1}\right) + A^\top y^{k,r} + \rho ^kA^\top \left( Ax^{k,r+1}+B{\bar{x}}^{k,r}+z^{k,r} \right) . \end{aligned}$$

(79)

According to the update rule of $y^{k,r}$, the above expression is equivalent to

$$\begin{aligned} 0\in & {} \partial f\left( x^{k,r+1}\right) + {\mathcal {N}}_{\mathcal {X}} \left( x^{k,r+1}\right) + A^\top y^{k,r+1} -\rho ^k A^\top \nonumber \\&\left( B{\bar{x}}^{k,r+1}+z^{k,r+1}-B{\bar{x}}^{k,r}-z^{k,r}\right) , \end{aligned}$$

(80)

i.e.,

$$\begin{aligned}&\rho ^kA^\top \left( B{\bar{x}}^{k,r+1}+z^{k,r+1}-B{\bar{x}}^{k,r}-z^{k,r}\right) \in \partial f\left( x^{k,r+1}\right) \nonumber \\&\quad +\,{\mathcal {N}}_{\mathcal {X}}\left( x^{k,r+1}\right) + A^\top y^{k,r+1}. \end{aligned}$$

(81)

According to the first inequality of (22), the norm of the left hand side above is not larger than $\epsilon _1^k$, which directly implies the first condition in (23). In a similar manner, the second condition in (23) can be established. The third one follows from (71) and the fourth condition is obvious.

Appendix 2: Proof of Lemma 2

We first consider the situation when $\beta ^k$ is unbounded. From (78), we have

$$\begin{aligned} {\overline{L}} \ge f\left( x^{k+1}\right) + g\left( x^{k+1}\right) - \lambda ^{k\top }\left( Ax^{k+1} + B{\bar{x}}^{k+1}\right) + \frac{\beta ^k}{2}\left\| Ax^{k+1} + B{\bar{x}}^{k+1}\right\| ^2. \end{aligned}$$

(82)

Since f and g are both lower bounded, as $\beta ^k\rightarrow \infty$, we have $Ax^{k+1} + B{\bar{x}}^{k+1} \rightarrow 0$. Combined with the first two conditions of (23) in the limit of $\epsilon _1^k$, $\epsilon _2^k$, $\epsilon _3^k \downarrow 0$, we have reached (25).

Then we suppose that $\beta ^k$ is bounded, i.e., the amplification step $\beta ^{k+1}=\gamma \beta ^k$ is executed for only a finite number of outer iterations. According to Lines 17–21 of Algorithm 1, expect for some finite choices of k, $\left\| z^{k+1}\right\| \le \omega \left\| z^k\right\|$ always hold. Therefore $z^{k+1}\rightarrow 0$. Apparently, (25) follows from the limit of (23).

Appendix 3: Proof of Lemma 3

From Lemma 1 one knows that within R inner iterations

$$\begin{aligned} \frac{{\overline{L}} - {\underline{L}}^k}{\beta ^k} \ge \sum _{r=1}^{R} \left( \left\| B{\bar{x}}^{k,r+1} - B{\bar{x}}^{k,r} \right\| ^2 + \frac{1}{2} \left\| {\bar{z}}^{k,r+1} - z^{k,r} \right\| ^2 \right) . \end{aligned}$$

(83)

Then

$$\begin{aligned} \left\| B{\bar{x}}^{k,R+1} - B{\bar{x}}^{k,R} \right\| , \left\| z^{k,R+1} - z^{k,R} \right\| \sim {\mathcal {O}}\left( 1/\sqrt{\beta ^k R}\right) . \end{aligned}$$

(84)

For the kth outer iteration, its inner iterations are terminated when (22) is met, which is translated into the following relations:

$$\begin{aligned} \begin{aligned}&{\mathcal {O}}\left( \rho ^k/\sqrt{\beta ^k R^k} \right) \le \epsilon _1^k, \epsilon _2^k \sim {\mathcal {O}}\left( \vartheta ^k \right) , \\&{\mathcal {O}}\left( 1/\sqrt{\beta ^k R^k} \right) \le \epsilon _3^k \sim {\mathcal {O}}\left( \vartheta ^k/\beta ^k \right) . \end{aligned} \end{aligned}$$

(85)

where the last relation uses (73) with $\rho ^k=2\beta ^k$. Therefore

$$\begin{aligned} R^k \sim {\mathcal {O}}\left( \beta ^k/\vartheta ^{2k} \right) . \end{aligned}$$

(86)

At the end of the kth iteration, suppose that Lines 19–20 and Lines 17–18 of Algorithm 1 have been executed for $k_1$ and $k_2$ times, respectively ($k_1+k_2=k$). Then the obtained $z^{k+1}$ satisfies $\left\| z^{k+1}\right\| \sim {\mathcal {O}}\left( \omega ^{k_1}\right)$, and $\left\| Ax^{k+1}+B{\bar{x}}^{k+1}+z^{k+1}\right\| \le \epsilon _3^k \sim {\mathcal {O}}\left( \vartheta ^k/\beta ^k\right)$, which imply

$$\begin{aligned} \left\| Ax^{k+1}+B{\bar{x}}^{k+1}\right\| \le {\mathcal {O}}\left( \vartheta ^k/\beta ^k\right) + {\mathcal {O}}\left( \omega ^{k_1}\right) . \end{aligned}$$

(87)

From (82),

$$\begin{aligned} \beta ^k \left\| Ax^{k+1}+B{\bar{x}}^{k+1}\right\| ^2 \sim \beta ^k \left( {\mathcal {O}} \left( \vartheta ^k/\beta ^k\right) + {\mathcal {O}}\left( \omega ^{k_1}\right) \right) ^2 \sim {\mathcal {O}}(1). \end{aligned}$$

(88)

Substituting (88) into (86), we obtain

$$\begin{aligned} R^k \sim {\mathcal {O}}\left( \frac{1}{\vartheta ^{2k}} \frac{1}{\left( {\mathcal {O}}\left( \vartheta ^k/\beta ^k\right) + {\mathcal {O}}\left( \omega ^{k_1}\right) \right) ^2} \right) . \end{aligned}$$

(89)

When $\vartheta \le \omega$, $\vartheta ^k \le \omega ^{k} \le \omega ^{k_1}\gamma ^{k_2}$, and hence $\gamma ^{k_2}\vartheta ^k \le \omega ^{k_1}$, i.e., $\omega ^{k_1}$ dominates over $\vartheta ^k/\beta ^k$, leading to

$$\begin{aligned} R^k \sim {\mathcal {O}}\left( 1/\vartheta ^{2k}\omega ^{2k_1} \right) \sim {\mathcal {O}}\left( 1/\vartheta ^{2k}\omega ^{2k} \right) . \end{aligned}$$

(90)

For K outer iterations, the total number of inner iterations is

$$\begin{aligned} R = \sum _{k=1}^K R^k \sim {\mathcal {O}}\left( \sum _{k=1}^K \frac{1}{\vartheta ^{2k}\omega ^{2k}} \right) \sim {\mathcal {O}}\left( \frac{1}{\vartheta ^{2K}\omega ^{2K}} \right) . \end{aligned}$$

(91)

The number of outer iterations needed to reach an $\epsilon$-approximate stationary point is obviously $K\sim {\mathcal {O}}\left( \log _\vartheta \epsilon \right)$. Then

$$\begin{aligned} R \sim {\mathcal {O}}\left( \epsilon ^{-2(1+\varsigma )} \right) . \end{aligned}$$

(92)

Appendix 4: Proof of Lemma 6

Through the inner iterations, only Anderson acceleration might lead to an increase in the barrier augmented Lagrangian. Combining Assumptions 3, 5, and the safeguarding criterion (41), we obtain

$$\begin{aligned} L_{b^k}\left( x^{k,r+1}, {\bar{x}}^{k,r+1}, z^{k,r+1}, y^{k,r+1}\right) \le {\overline{L}} + {\tilde{L}}_0\eta _L\sum _{r=0}^{\infty } \frac{1}{r^{1+\sigma }} < +\infty , \end{aligned}$$

(93)

Together with Assumptions 1 and 2, $L_{b^k}$ is also bounded below. Therefore $L_{b^k}$ is bounded in a closed interval and must have converging subsequences. Therefore we can choose a subsequence converging to the lower limit ${\underline{L}}$. For any $\varepsilon >0$ there exists an index R of inner iteration in this subsequence, such that ${\tilde{L}}_0\eta _L \sum _{r=R}^\infty r^{-(1+\sigma )} < \varepsilon /2$ and $L_{b^k}\left( x^{k,r+1},{\bar{x}}^{k,r+1},z^{k,r+1},y^{k,r+1}\right) < {\underline{L}}+\varepsilon /2$ for any $r\ge R$ on this subsequence. It then follows that for any $r\ge R$, whether on the subsequence or not, it holds that

$$\begin{aligned} L_{b^k}\left( x^{k,r+1},{\bar{x}}^{k,r+1},z^{k,r+1},y^{k,r+1}\right) < {\underline{L}}+\varepsilon . \end{aligned}$$

(94)

Hence the upper limit is not larger than ${\underline{L}}+\varepsilon$. Due to the arbitrariness of $\varepsilon >0$, the lower limit coincides with the upper limit, and hence the sequence of barrier augmented Lagrangian is convergent.

The convergence of the barrier augmented Lagrangian implies that as $r \rightarrow \infty$,

$$\begin{aligned} L_{b^k}\left( x^{k,r+1},{\bar{x}}^{k,r+1},z^{k,r+1},y^{k,r+1}\right) - L_{b^k}\left( x^{k,r},{\bar{x}}^{k,r},z^{k,r},y^{k,r}\right) \rightarrow 0. \end{aligned}$$

(95)

Suppose that r is not an accelerated iteration, then since this quantity does not exceed $-\beta ^k\left\| B{\bar{x}}^{k,r+1} - B{\bar{x}}^{k,r}\right\| ^2 - \left( \beta ^k/2\right) \left\| z^{k,r+1}-z^{k,r}\right\| ^2$, we must have $B{\bar{x}}^{k,r+1} - B{\bar{x}}^{k,r} \rightarrow 0$ and $z^{k,r+1}-z^{k,r}\rightarrow 0$. Otherwise if inner iteration r is accelerated, the convergence of $B{\bar{x}}^{k,r+1} - B{\bar{x}}^{k,r}$ and $z^{k,r+1}-z^{k,r}$ are automatically guaranteed by the second criterion (42) of accepting Anderson acceleration. The convergence properties of these two sequences naturally fall into the paradigm of Lemma 1 for establishing the convergence to approximate KKT conditions of the relaxed problem.