A Provably Convergent Scheme for Compressive Sensing Under Random Generative Priors

Abstract

Deep generative modeling has led to new and state of the art approaches for enforcing structural priors in a variety of inverse problems. In contrast to priors given by sparsity, deep models can provide direct low-dimensional parameterizations of the manifold of images or signals belonging to a particular natural class, allowing for recovery algorithms to be posed in a low-dimensional space. This dimensionality may even be lower than the sparsity level of the same signals when viewed in a fixed basis. What is not known about these methods is whether there are computationally efficient algorithms whose sample complexity is optimal in the dimensionality of the representation given by the generative model. In this paper, we present such an algorithm and analysis. Under the assumption that the generative model is a neural network that is sufficiently expansive at each layer and has Gaussian weights, we provide a gradient descent scheme and prove that for noisy compressive measurements of a signal in the range of the model, the algorithm converges to that signal, up to the noise level. The scaling of the sample complexity with respect to the input dimensionality of the generative prior is linear, and thus can not be improved except for constants and factors of other variables. To the best of the authors’ knowledge, this is the first recovery guarantee for compressive sensing under generative priors by a computationally efficient algorithm.

Appendix A: Supporting Lemmas

Lemma A.1 is used in proofs for Sect. 5.3 and Lemma 5.3.

Lemma A.1

Suppose that the WDC and RRIC holds with $\epsilon < 1/(16 \pi d^2)^2$ and that the noise e satisfies $\Vert e\Vert \le a_5 2^{-d/2} \Vert x_*\Vert$. Then, for all x and all $v_x \in \partial f(x)$,

$$\begin{aligned} \Vert {v}_x\Vert \le \frac{a_6 d}{2^d} \max (\Vert x\Vert , \Vert x_*\Vert ), \end{aligned}$$

(A.1)

where $a_5$ and $a_6$ are universal constants.

Proof

Define for convenience $\zeta _j=\prod _{i = j}^{d - 1} \frac{\pi - {\bar{\theta }}_{j, x, x_*}}{\pi }$. We have

$$\begin{aligned} \Vert {v}_x\Vert&\le \Vert h_{x}\Vert + \Vert h_{x} - {v}_x\Vert \\&\le \left\| \frac{1}{2^d} x - \frac{1}{2^d} \zeta _{0} x_* - \frac{1}{2^d} \sum _{i = 0}^{d - 1} \frac{\sin {\bar{\theta }}_{i,x}}{\pi } \zeta _{i + 1} \frac{\Vert x_*\Vert }{\Vert x\Vert } x \right\| \\&\quad + a_1 \frac{d^3 \sqrt{\epsilon }}{2^d} \max ( \Vert x\Vert , \Vert x_* \Vert ) + \frac{2}{2^{d/2}} \Vert e\Vert \\&\le \frac{1}{2^d} \Vert x\Vert + \left( \frac{1}{2^d} + \frac{d}{\pi 2^d} \right) \Vert x_*\Vert + a_1 \frac{d^3 \sqrt{\epsilon }}{2^d} \max (\Vert x\Vert , \Vert x_*\Vert ) + \frac{2}{2^{d/2}} \Vert e\Vert \\&\le \frac{a_6 d}{2^d} \max (\Vert x\Vert , \Vert x_*\Vert ), \end{aligned}$$

where the second inequality follows from the definition of $h_x$ and Lemma 5.2, the third inequality uses $| \zeta _j | \le 1$, and the last inequality uses the assumption $\Vert e\Vert \le a_5 2^{-d/2} \Vert x_*\Vert$. $\square$

Lemma A.2 is used in proofs for Lemma 5.1.

Lemma A.2

Suppose $a_i, b_i \in [0, \pi ]$ for $i = 1, \ldots , k$, and $|a_i - b_i| \le |a_j - b_j|, \forall i \ge j$. Then it holds that

$$\begin{aligned} \left| \prod _{i = 1}^k \frac{\pi - a_i}{\pi } - \prod _{i = 1}^k \frac{\pi - b_i}{\pi }\right| \le \frac{k}{\pi } |a_1 - b_1|. \end{aligned}$$

Proof

Prove by induction. It is easy to verify that the inequality holds if $k = 1$. Suppose the inequality holds with $k = t - 1$. Then

$$\begin{aligned} \left| \prod _{i = 1}^{t} \frac{\pi - a_i}{\pi } - \prod _{i = 1}^t \frac{\pi - b_i}{\pi }\right|&\le \left| \prod _{i = 1}^{t} \frac{\pi - a_i}{\pi } - \frac{\pi - a_t}{\pi } \prod _{i = 1}^{t-1} \frac{\pi - b_i}{\pi }\right| \\&\quad + \left| \frac{\pi - a_t}{\pi } \prod _{i = 1}^{t-1} \frac{\pi - b_i}{\pi } - \prod _{i = 1}^t \frac{\pi - b_i}{\pi }\right| \\&\le \frac{t-1}{\pi } |a_1 - b_1| + \frac{1}{\pi } |a_t - b_t| \le \frac{t}{\pi } |a_1 - b_1|. \end{aligned}$$

$\square$

Lemma A.3 is used in proofs for Lemmas 5.2, 5.3, and 5.5.

Lemma A.3

Suppose the WDC and RRIC hold with $\epsilon \le 1 / (16 \pi d^2)^2$. Then we have

$$\begin{aligned} \left| x^T q_x\right| \le \frac{2}{2^{d/2}} \Vert e\Vert \Vert x\Vert , \end{aligned}$$

where $q_x = \left( \prod _{i = d}^1 W_{i, +, x}\right) ^T A^T e$. In addition, if x is differentiable at G(x), then we have

$$\begin{aligned} \left\| q_x\right\| \le \frac{2}{2^{d/2}} \Vert e\Vert . \end{aligned}$$

Proof

We have

$$\begin{aligned} |x^T q_x|^2&= |e^T A G(x)|^2 \le \Vert A G(x)\Vert ^2 \Vert e\Vert ^2 \le (1 + \epsilon ) \Vert G(x)\Vert ^2 \Vert e\Vert ^2 \\&\le (1 + \epsilon ) \prod _{i = d}^1\Vert W_{i, +, x} \Vert ^2 \Vert e\Vert ^2 \Vert x\Vert ^2 \le (1 + \epsilon ) (1 + 2 \epsilon d)^2 \frac{1}{ 2^{d}} \Vert e\Vert ^2 \Vert x\Vert ^2, \end{aligned}$$

where the second inequality follows from RRIC and the last inequality follows from [12, (10)]. Therefore, $\left| x^T q_x\right| \le \frac{2}{2^{d/2}} \Vert e\Vert \Vert x\Vert$.

Suppose G is differentiable at x. Then the local linearity of G implies that $G(x + z) - G(x) = \left( \prod _{i = d}^1 W_{i, +, x}\right) z$ for any sufficiently small $z \in {\mathbb {R}}^k$. By the RRIC, we have

$$\begin{aligned}&\left| {\left\langle A \left( \prod _{i = d}^1 W_{i, +, x}\right) z ,A \left( \prod _{i = d}^1 W_{i, +, x}\right) z \right\rangle _{}} - {\left\langle \left( \prod _{i = d}^1 W_{i, +, x}\right) z ,\left( \prod _{i = d}^1 W_{i, +, x}\right) z \right\rangle _{}} \right| \\&\quad \le \epsilon \prod _{i = d}^1\Vert W_{i, +, x} \Vert ^2 \Vert z\Vert ^2, \end{aligned}$$

which implies

$$\begin{aligned} \left| {\left\langle A \left( \prod _{i = d}^1 W_{i, +, x}\right) z ,A \left( \prod _{i = d}^1 W_{i, +, x}\right) z \right\rangle _{}} \right| \le (1 + \epsilon ) \prod _{i = d}^1\Vert W_{i, +, x} \Vert ^2 \Vert z\Vert ^2. \end{aligned}$$

Therefore, we obtain

$$\begin{aligned} \left\| A \left( \prod _{i = d}^1 W_{i, +, x}\right) \right\| \le \sqrt{1 + \epsilon } \prod _{i = d}^1\Vert W_{i, +, x} \Vert . \end{aligned}$$

Combining above inequality with $\prod _{i = d}^1\Vert W_{i, +, x} \Vert \le (1 + 2 \epsilon d) / 2^{d/2} \le 1.5 / 2^{d/2}$ given in [12, (10)] yields

$$\begin{aligned} \left\| A \left( \prod _{i = d}^1 W_{i, +, x}\right) \right\| \le 1.5 \sqrt{1 + \epsilon } / 2^{d/2} \le 2 / 2^{d/2}, \end{aligned}$$

where the second inequality follows from the assumption on $\epsilon$. Therefore, we obtain

$$\begin{aligned} \Vert q_x\Vert = \left\| \left( \prod _{i = d}^1 W_{i, +, x}\right) ^T A^T e\right\| \le \left\| \left( \prod _{i = d}^1 W_{i, +, x}\right) ^T A^T\right\| \Vert e\Vert \le \frac{2}{2^{d/2}} \Vert e\Vert . \end{aligned}$$

$\square$

Lemma A.4 is used in proofs for Lemma 5.5.

Lemma A.4

For all $d\ge 2$, that

$$\begin{aligned} 1/\left( a_7(d + 2)^2\right) \le 1 - \rho _d \le 250/(d + 1), \end{aligned}$$

and $a_8 = \min _{d \ge 2} \rho _d > 0$.

Proof

It holds that

$$\begin{aligned}&\log (1+x) \le x&\forall x \in [-0.5, 1] \end{aligned}$$

(A.2)

$$\begin{aligned}&\log (1-x) \ge -2 x&\forall x \in [0, 0.75] \end{aligned}$$

(A.3)

where $\theta _{x, y} = \angle (x, y)$.

We recall the results in [12, (35), (36), and (49)]:

$$\begin{aligned}&{\check{\theta }}_i \le \frac{3 \pi }{i + 3}\;\;\;\;\; \;\;\; \hbox { and }\;\;\;\;\; \;\;\; {\check{\theta }}_i \ge \frac{\pi }{i + 1}\;\;\;\;\; \forall i \ge 0 \\&\quad 1 - \rho _d = \prod _{i = 1}^{d - 1} \left( 1 - \frac{{\check{\theta }}_{i}}{\pi } \right) + \sum _{i = 1}^{d-1} \frac{{\check{\theta }}_{i} - \sin {\check{\theta }}_{i}}{\pi } \prod _{j = i+1}^{d-1} \left( 1 - \frac{{\check{\theta }}_{j}}{\pi } \right) . \end{aligned}$$

Therefore, we have for all $0 \le i \le d - 2$,

$$\begin{aligned} \prod _{j = i+1}^{d-1} \left( 1 - \frac{{\check{\theta }}_{j}}{\pi } \right)&\le \prod _{j = i+1}^{d-1} \left( 1 - \frac{1}{j + 1} \right) = e^{\sum _{j = i + 1}^{d - 1} \log \left( 1 - \frac{1}{j + 1}\right) } \\&\le e^{- \sum _{j = i + 1}^{d - 1} \frac{1}{j + 1}} \le e^{- \int _{i + 1}^d \frac{1}{s + 1} d s} = \frac{i + 2}{d + 1}, \\ \prod _{j = i+1}^{d-1} \left( 1 - \frac{{\check{\theta }}_{j}}{\pi } \right)&\ge \prod _{j = i+1}^{d-1} \left( 1 - \frac{3}{j + 3} \right) = e^{\sum _{j = i + 1}^{d - 1} \log \left( 1 - \frac{3}{j + 3}\right) } \\&\ge e^{- \sum _{j = i + 1}^{d - 1} \frac{6}{j + 3}} \ge e^{- \int _{i}^{d - 1} \frac{6}{s + 3} d s} = \left( \frac{i + 3}{d + 2}\right) ^6, \end{aligned}$$

where the second and the fifth inequalities follow from (A.2) and (A.3) respectively. Since $\pi ^3 / (12 (i + 1)^3) \le {\check{\theta }}_{i}^3 / 12 \le {\check{\theta }}_{i} - \sin {\check{\theta }}_{i} \le {\check{\theta }}_{i}^3 / 6 \le 27 \pi ^3 / (6 (i + 3)^3)$, we have that for all $d \ge 3$

$$\begin{aligned} 1 - \rho _d&\le \frac{2}{d + 1} + \sum _{i = 1}^{d - 1} \frac{27 \pi ^3}{6 (i + 3)^3} \frac{i + 2}{d + 1} \le \frac{2}{d + 1} + \frac{3 \pi ^5}{4 (d + 1)} \le \frac{250}{d + 1} \end{aligned}$$

and

$$\begin{aligned} 1 - \rho _d&\ge \left( \frac{3}{(d+2)}\right) ^6 + \sum _{i = 1}^{d - 1} \frac{\pi ^3}{12 (i + 3)^3} \left( \frac{i + 3}{d + 2} \right) ^6 \ge \frac{1}{K_1 (d + 2)^2}, \end{aligned}$$

where we use $\sum _{i = 4}^\infty \frac{1}{i^2} \le \frac{\pi ^2}{6}$ and $\sum _{i = 1}^n i^3 = O(n^4)$. Since $\rho _d \ge 1 - 250 / (d+1)$ and $\rho _d > 0$ for all $d \ge 2$, we have $\min _{d \ge 2} \rho _d > 0$. $\square$

Lemma A.5 is used in proofs for Lemma 5.5.

Lemma A.5

Fix $0< a_9 < \frac{1}{4 d^2 \pi }$. For any $\phi _d \in [\rho _d, 1]$, it holds that

$$\begin{aligned} {f^E}(x)&< \frac{1}{2^{d+1}} \left( \phi _d^2 - 2 \phi _d + \frac{10}{a_8^3} d a_9 \right) \Vert x_*\Vert ^2 + \frac{\Vert x_*\Vert ^2}{2^{d+1}}, \forall x \in {\mathcal {B}}(\phi _d x_*, a_9 \Vert x_*\Vert ) \hbox { and } \\ {f^E}(x)&> \frac{1}{2^{d+1}} \left( \phi _d^2 - 2 \phi _d \rho _d - 10 d^3 a_9 \right) \Vert x_*\Vert ^2 + \frac{\Vert x_*\Vert ^2}{2^{d+1}}, \forall x \in {\mathcal {B}}(- \phi _d x_*, a_9 \Vert x_*\Vert ), \end{aligned}$$

where $a_8$ is defined in Lemma A.4.

Proof

If $x \in {\mathcal {B}}(\phi _d x_*, a_9 \Vert x_*\Vert )$, then we have $0 \le {\bar{\theta }}_{0, x, x_*} \le \arcsin (a_9 / \phi _d) \le \frac{\pi a_9}{2 \phi _d}$, $0 \le {\bar{\theta }}_{0, x, x_*} \le {\bar{\theta }}_{i, x, x_*} \le \frac{\pi a_9}{2 \phi _d}$, and $\phi _d \Vert x_*\Vert - a_9 \Vert x_*\Vert \le \Vert x\Vert \le \phi _d \Vert x_*\Vert + a_9 \Vert x_*\Vert$. Note that $\cos \theta \ge 1 - \frac{\theta ^2}{2}, \forall \theta \in [0, \pi ]$. We have

$$\begin{aligned}&{f^E}(x) - \frac{\Vert x_*\Vert ^2}{2^{d+1}} \le \frac{1}{2^{d+1}} \Vert x\Vert ^2 - \frac{1}{2^d} \left( \prod _{i = 0}^{d - 1} \frac{\pi - {\bar{\theta }}_{i, x, x_*}}{\pi } \right) x_*^T x \\&\quad \le \frac{1}{2^{d+1}} (\phi _d + a_9)^2 \Vert x_*\Vert ^2 - \frac{1}{2^d} \left( \prod _{i = 0}^{d - 1} \frac{\pi - \frac{\pi a_9}{2 \phi _d}}{\pi } \right) \Vert x_*\Vert \Vert x\Vert \cos {\bar{\theta }}_{0, x, x_*} \\&\quad \le \frac{1}{2^{d+1}} (\phi _d + a_9)^2 \Vert x_*\Vert ^2 - \frac{1}{2^d} \left( \prod _{i = 0}^{d - 1} \frac{\pi - \frac{\pi a_9}{2 \phi _d}}{\pi } \right) (\phi _d - a_9) \Vert x_*\Vert ^2 \left( 1 - \frac{\pi ^2 a_9^2}{8 \phi _d^2} \right) \\&\quad \le \frac{1}{2^{d+1}} \left( \phi _d^2 + 2 \phi _d a_9 + a_9^2 - 2 \left( 1 - \frac{d a_9}{\phi _d}\right) (\phi _d -a_9) \left( 1 - \frac{\pi ^2 a_9^2}{8 \phi _d^2}\right) \right) \Vert x_*\Vert ^2 \\&\quad \le \frac{1}{2^{d+1}} \left( \phi _d^2 - 2 \phi _d + \frac{10}{a_8^3} d a_9 \right) \Vert x_*\Vert ^2, \end{aligned}$$

where the last inequality is by Lemma A.4 and $a_9 < 1 / (4 \pi )$.

If $x \in {\mathcal {B}}(- \phi _d x_*, a_9 \Vert x_*\Vert )$, then we have $0 \le \pi - {{\bar{\theta }}}_{0, x, x_*} \le \arcsin (a_9 \pi ) \le \frac{\pi ^2}{2} a_9$, and $\phi _d \Vert x_*\Vert - a_9 \Vert x_*\Vert \le \Vert x\Vert \le \phi _d \Vert x_*\Vert + a_9 \Vert x_*\Vert$. It follows that

$$\begin{aligned} {f^E}(x) - \frac{\Vert x_*\Vert ^2}{2^{d+1}}&\ge \frac{1}{2^{d+1}} \Vert x\Vert ^2 - \frac{1}{2^d} \sum _{i = 0}^{d - 1} \frac{\sin {\bar{\theta }}_{i, x, x_*}}{\pi } \left( \prod _{j = i + 1}^{d-1} \frac{\pi - {\bar{\theta }}_{j, x, x_*}}{\pi } \right) \Vert x_*\Vert \Vert x\Vert \\&\ge \frac{1}{2^{d+1}} \Vert x\Vert ^2 - \frac{1}{2^d} \left( \rho _d + \frac{3 d^3a_9 \pi ^2}{2} \right) \Vert x_*\Vert \Vert x\Vert \;\;\;\;\; (\hbox {by }[12, (40)]) \\&\ge \frac{1}{2^{d+1}} \left( \phi _d - a_9 \right) ^2 \Vert x_*\Vert ^2 - \frac{1}{2^d} \left( \rho _d + \frac{3 d^3a_9 \pi ^2}{2} \right) \left( \phi _d + a_9\right) \Vert x_*\Vert ^2 \\&\ge \frac{1}{2^{d+1}} \left( \phi _d^2 - 2 \phi _d \rho _d - 10 d^3 a_9 \right) \Vert x_*\Vert ^2. \end{aligned}$$

$\square$

Lemma A.6 is used in proofs for Lemma 5.5.

Lemma A.6

If the WDC and RRIC hold with $\epsilon < 1 / (16 \pi d^2)^2$, then we have

$$\begin{aligned} |f(x) - {f^E}(x)|\le & {} \frac{\epsilon (1 + 4 \epsilon d)}{2^d} \Vert x\Vert ^2 + \frac{\epsilon (1 + 4 \epsilon d) + 48 d^3 \sqrt{\epsilon }}{2^{d+1}} \Vert x\Vert \Vert x_*\Vert \\&+ \frac{\epsilon (1 + 4 \epsilon d)}{2^d} \Vert x_*\Vert ^2. \end{aligned}$$

Proof

For brevity of notation, let $\Lambda _{z} = \prod _{i = d}^1 W_{i, +, z}$. We have

$$\begin{aligned} \left| f(x) - {f^E}(x) \right|&= \left| \frac{1}{2} x^T \left( \Lambda _{x}^T A^T A \Lambda _{x} - \Lambda _{x}^T \Lambda _{x} \right) x \right. + \frac{1}{2} x^T \left( \Lambda _{x}^T \Lambda _{x} - \frac{I_k}{2^d} \right) x \\&\quad - x^T \left( \Lambda _{x}^T A^T A \Lambda _{x_*} x_* - \Lambda _{x}^T \Lambda _{x_*} x_* \right) - x^T \left( \Lambda _{x}^T \Lambda _{x_*} x_* - h_{x, x_*} \right) \\&\quad + \frac{1}{2} x_*^T \left( \Lambda _{x_*}^T A^T A \Lambda _{x_*} - \Lambda _{x_*}^T \Lambda _{x_*} \right) x_* \\&\quad + \left. \frac{1}{2} x_*^T \left( \Lambda _{x_*}^T \Lambda _{x_*} - \frac{I_k}{2^d} \right) x_* \right| \\&\le \frac{\epsilon }{2} \frac{1 + 4 \epsilon d}{2^d} \Vert x\Vert ^2 + \frac{\epsilon }{2} \frac{1 + 4 \epsilon d}{2^d} \Vert x\Vert ^2 + \frac{\epsilon }{2} \frac{1 + 4 \epsilon d}{2^d} \Vert x\Vert \Vert x_*\Vert \\&\quad + \frac{24 d^3 \sqrt{\epsilon }}{2^d} \Vert x\Vert \Vert x_*\Vert \\&\quad + \frac{\epsilon }{2} \frac{1 + 4 \epsilon d}{2^d} \Vert x_*\Vert ^2 + \frac{\epsilon }{2} \frac{1 + 4 \epsilon d}{2^d} \Vert x_*\Vert ^2\\&= \frac{\epsilon (1 + 4 \epsilon d)}{2^d} \Vert x\Vert ^2 \\&\quad + \frac{\epsilon (1 + 4 \epsilon d) + 48 d^3 \sqrt{\epsilon }}{2^{d+1}} \Vert x\Vert \Vert x_*\Vert + \frac{\epsilon (1 + 4 \epsilon d)}{2^d} \Vert x_*\Vert ^2, \end{aligned}$$

where the first inequality uses the WDC, the RRIC, and [12, Lemma 8]. $\square$

Lemma A.7 is used in proofs for Lemma A.8.

Lemma A.7

Suppose $W \in {\mathbb {R}}^{n \times k}$ satisfies the WDC with constant $\epsilon$. Then for any $x, y \in {\mathbb {R}}^k$, it holds that

$$\begin{aligned} \Vert W_{+, x} x - W_{+, y} y\Vert \le \left( \sqrt{\frac{1}{2} + \epsilon } + \sqrt{2(2\epsilon + \theta )} \right) \Vert x - y\Vert , \end{aligned}$$

where $\theta = \angle (x, y)$.

Proof

We have

$$\begin{aligned}&\Vert W_{+, x} x - W_{+, y} y\Vert \le \Vert W_{+, x} x - W_{+, x} y\Vert + \Vert W_{+, x} y - W_{+, y} y\Vert \nonumber \\&\qquad = \Vert W_{+, x} (x - y)\Vert + \Vert (W_{+, x} - W_{+, y}) y\Vert \nonumber \\&\quad \le \Vert W_{+, x}\Vert \Vert x - y\Vert + \Vert (W_{+, x} - W_{+, y}) y\Vert . \end{aligned}$$

(A.4)

By WDC assumption, we have

$$\begin{aligned} \Vert W_{+, x}^T (W_{+, x} - W_{+, y})\Vert&\le \left\| W_{+, x}^T W_{+, x} - I / 2\right\| + \left\| W_{+, x}^T W_{+, y} - Q_{x, y}\right\| \nonumber \\&+ \left\| Q_{x, y} - I / 2 \right\| \le 2 \epsilon + \theta . \end{aligned}$$

(A.5)

We also have

$$\begin{aligned}&\Vert (W_{+, x} - W_{+, y}) y\Vert ^2\nonumber \\&\quad = \sum _{i = 1}^n (1_{w_i \cdot x> 0} - 1_{w_i \cdot y> 0})^2 (w_i \cdot y)^2 \nonumber \\&\quad \le \sum _{i = 1}^n (1_{w_i \cdot x> 0} - 1_{w_i \cdot y> 0})^2 ((w_i \cdot x)^2 + (w_i \cdot y)^2 - 2 (w_i \cdot x) (w_i \cdot y) ) \nonumber \\&\quad = \sum _{i = 1}^n (1_{w_i \cdot x> 0} - 1_{w_i \cdot y> 0})^2 (w_i \cdot (x - y) )^2 \nonumber \\&\quad = \sum _{i = 1}^n 1_{w_i \cdot x> 0} 1_{w_i \cdot y \le 0} (w_i \cdot (x - y))^2 + \sum _{i = 1}^n 1_{w_i \cdot x \le 0} 1_{w_i \cdot y > 0} (w_i \cdot (x - y))^2 \nonumber \\&\quad = (x - y)^T W_{+, x}^T (W_{+, x} - W_{+, y}) (x - y) + (x - y)^T W_{+, y}^T (W_{+, y} - W_{+, x}) (x - y) \nonumber \\&\quad \le 2 (2 \epsilon + \theta ) \Vert x - y\Vert ^2. \;\;\; (\hbox {by}~\hbox {A}.5) \end{aligned}$$

(A.6)

Combining (A.4), (A.6), and $\Vert W_{i, +, x}\Vert ^2 \le 1/2 + \epsilon$ given in [12, (9)] yields the result. $\square$

Lemma A.8 is used in proofs for Lemma 5.6 and Lemma A.9.

Lemma A.8

Suppose $x \in {\mathcal {B}}(x_*, d \sqrt{\epsilon } \Vert x_*\Vert )$, and the WDC holds with $\epsilon < 1/ (200)^4 / d^6$. Then it holds that

$$\begin{aligned} \left\| \prod _{i = j}^1 W_{i, +, x} x - \prod _{i = j}^1 W_{i, +, x_*} x_*\right\| \le \frac{1.2}{2^{\frac{j}{2}}} \Vert x - x_*\Vert . \end{aligned}$$

Proof

In this proof, we denote $\theta _{i, x, x_*}$ and ${\bar{\theta }}_{i, x, x_*}$ by $\theta _i$ and ${\bar{\theta }}_{i}$ respectively. Since $x \in {\mathcal {B}}(x_*, d \sqrt{\epsilon } \Vert x_*\Vert )$, we have

$$\begin{aligned} {\bar{\theta }}_{i} \le {\bar{\theta }}_{0} \le 2 d \sqrt{\epsilon }. \end{aligned}$$

(A.7)

By [12, (14)], we also have $|\theta _{i} - {\bar{\theta }}_{i}| \le 4 i \sqrt{\epsilon } \le 4 d \sqrt{\epsilon }$. It follows that

$$\begin{aligned} 2 \sqrt{\theta _i + 2 \epsilon }&\le 2 \sqrt{{\bar{\theta }}_i + 4 d \sqrt{\epsilon } + 2\epsilon } \le 2 \sqrt{2 d \sqrt{\epsilon } + 4 d \sqrt{\epsilon } + 2\epsilon } \nonumber \\&\quad \le 2 \sqrt{8 d \sqrt{\epsilon }} \le \frac{1}{30 d}. \, (\hbox {by the assumption on }\epsilon ) \end{aligned}$$

(A.8)

Note that $\sqrt{1 + 2 \epsilon } \le 1 + \epsilon \le 1 + \sqrt{d\sqrt{\epsilon }}$. We have

$$\begin{aligned} \prod _{i = d - 1}^0 \left( \sqrt{1+2\epsilon } + 2 \sqrt{ \theta _i + 2 \epsilon } \right)&\le \left( 1 + 7 \sqrt{d\sqrt{\epsilon }}\right) ^d \\&\le 1 + 14 d \sqrt{d\sqrt{\epsilon }} \le \frac{107}{100} < 1.2, \end{aligned}$$

where the second inequality is from that $(1+x)^d \le 1 + 2dx$ if $0< x d < 1$. Combining the above inequality with Lemma A.7 yields

$$\begin{aligned} \left\| \prod _{i = j}^1 W_{i, +, x}x - \prod _{i = j}^1 W_{i, +, x_*} x_*\right\|&\le \prod _{i = j - 1}^0 \left( \sqrt{\frac{1}{2}+\epsilon } + \sqrt{2} \sqrt{ \theta _i + 2 \epsilon } \right) \Vert x - x_*\Vert \\&\le \frac{1.2}{2^{\frac{j}{2}}} \Vert x - x_*\Vert . \end{aligned}$$

$\square$

Lemma A.9 is used in proofs for Lemma 5.6.

Lemma A.9

Suppose $x \in {\mathcal {B}}(x_*, d \sqrt{\epsilon } \Vert x_*\Vert )$, and the WDC holds with $\epsilon < 1/ (200)^4 / d^6$. Then it holds that

$$\begin{aligned}&\left( \prod _{i = d}^1 W_{i, +, x}\right) ^T\left[ \left( \prod _{i = d}^1 W_{i, +, x}\right) x - \left( \prod _{i = d}^1 W_{i, +, x_*}\right) x_*\right] \\&\quad = \frac{1}{2^d} (x - x_*) + \frac{1}{2^d} \frac{1}{16} \Vert x - x_*\Vert O_1(1). \end{aligned}$$

Proof

For brevity of notation, let $\Lambda _{j, k, z} = \prod _{i = j}^k W_{i, +, z}$. We have

$$\begin{aligned}&\Lambda _{d, 1, x}^T\left( \Lambda _{d, 1, x} x - \Lambda _{d, 1, x_*} x_*\right) \nonumber \\&\quad = \Lambda _{d, 1, x}^T\left[ \Lambda _{d, 1, x} x - \sum _{j = 1}^d \left( \Lambda _{d, j, x} \Lambda _{j-1, 1, x_*} x_*\right) \right. \nonumber \\&\qquad + \left. \sum _{j = 1}^d \left( \Lambda _{d, j, x} \Lambda _{j-1, 1, x_*} x_*\right) - \Lambda _{d, 1, x_*} x_*\right] \nonumber \\&\quad = \underbrace{\Lambda _{d, 1, x}^T \Lambda _{d, 1, x} (x - x_*)}_{T_1} + \underbrace{\Lambda _{d, 1, x}^T \sum _{j = 1}^d \Lambda _{d, j + 1, x} \left( W_{j, +, x} - W_{j, +, x_*} \right) \Lambda _{j-1, 1, x_*} x_*}_{T_2}. \end{aligned}$$

(A.9)

For $T_1$, we have

$$\begin{aligned} T_1 = \frac{1}{2^d} (x - x_*) + \frac{4 d}{2^d} \Vert x - x_*\Vert O_1(\epsilon ). \;\; ([12, (10)]) \end{aligned}$$

(A.10)

For $T_2$, we have

$$\begin{aligned} T_2&= O_1(1) \sum _{j = 1}^d \left( \frac{1}{2^{d - \frac{j}{2}}} + \frac{(4d - 2j)\epsilon }{2^{d - \frac{j}{2}}} \right) \left\| (W_{j, +, x} - W_{j, +, x_*}) \Lambda _{j-1, 1, x_*}x_* \right\| \nonumber \\&= O_1(1) \sum _{j = 1}^d \left( \frac{1}{2^{d - \frac{j}{2}}} + \frac{(4d - 2j)\epsilon }{2^{d - \frac{j}{2}}} \right) \left\| (\Lambda _{j-1, 1, x} x - \Lambda _{j-1, 1, x_*} x_*) \right\| \nonumber \\&\qquad \sqrt{2 (\theta _{i, x, x_*} + 2 \epsilon )} \nonumber \\&= O_1(1) \sum _{j = 1}^d \left( \frac{1}{2^{d - \frac{j}{2}}} + \frac{(4d - 2j)\epsilon }{2^{d - \frac{j}{2}}} \right) \frac{1.2}{2^{\frac{j}{2}}} \Vert x - x_*\Vert \frac{1}{ 30 \sqrt{2} d} \nonumber \\&= \frac{1}{16} \frac{1}{2^d} \Vert x - x_*\Vert O_1(1), \end{aligned}$$

(A.11)

where the first equation is by [12, (10)]; the second equation is by (A.6); the third equation is by Lemma A.8 and (A.8). The result follows from (A.9), (A.10) and (A.11). $\square$