A weighted randomized sparse Kaczmarz method for solving linear systems

Abstract

The randomized sparse Kaczmarz method, designed for seeking the sparse solutions of the linear systems $Ax=b$, selects the i-th projection hyperplane with likelihood proportional to $\Vert a_{i}\Vert _2^2$, where $a_{i}^{\mathrm{T}}$ is the i-th row of A. In this work, we propose a weighted randomized sparse Kaczmarz method, which selects the i-th projection hyperplane with probability proportional to $|\langle a_{i},x_{k}\rangle -b_{i}|^p$, where $0<p<\infty$, for possible acceleration. It bridges the randomized Kaczmarz and greedy Kaczmarz by parameter p. Theoretically, we show its linear convergence rate in expectation with respect to the Bregman distance in the noiseless and noisy cases, which is at least as efficient as the randomized sparse Kaczmarz method. The superiority of the proposed method is demonstrated via a group of numerical experiments.

Appendices

Appendix A Proof of Theorem 1

Proof

The proof is divided into two parts: we deduce the convergence rate of WRaSK in the first part and compare the convergence rate between WRaSK and RaSK in the second part.

First, we derive the convergence rate of WRaSK. By Theorem 2.8 in Lorenz et al. (2014b), we know that (11) in Lemma 2 holds for both the exact and inexact step sizes. Note that f is 1-strongly convex and $\Vert a_{i_k}\Vert _2=1$; it follows that

$$\begin{aligned} D_f^{x_{k+1}^*}(x_{k+1},{\hat{x}}) \le D_f^{x_{k}^*}(x_{k},{\hat{x}})-\frac{1}{2}\left( \langle a_{i_k},x_k\rangle -b_{i_k}\right) ^2, \end{aligned}$$

(A1)

we fix the values of the indices $i_0,\ldots ,i_{k-1}$ and only consider $i_k$ as a random variable. Taking the conditional expectation on both sides, we derive that

$$\begin{aligned}&{\mathbb {E}} \left( D_{f}^{x_{k+1}^*}(x_{k+1},{\hat{x}})|i_{0},\ldots ,i_{k-1}\right) \\&\quad \le D_{f}^{x_{k}^*}(x_{k},{\hat{x}})-\frac{1}{2}\sum _{i=1}^{m}(\langle a_{i},x_{k}\rangle -b_{i})^2\cdot \frac{|\langle a_{i},x_{k}\rangle -b_{i}|^p}{\Vert Ax_{k}-b\Vert _{l_p}^p}\\&\quad =D_f^{x_{k}^*}(x_{k},{\hat{x}})-\frac{1}{2}\frac{\Vert Ax_k-b\Vert _{l_{p+2}}^{p+2}}{\Vert Ax_k-b\Vert _{l_{p}}^{p}\Vert Ax_k-b\Vert _2^{2}}\cdot \Vert Ax_k-b\Vert _2^{2} \\&\quad \le D_f^{x_{k}^*}(x_{k},{\hat{x}})-\frac{1}{2}\mathop {\inf }\limits _{x\ne {\hat{x}}}\frac{\Vert Ax-b\Vert _{l_{p+2}}^{p+2}}{\Vert Ax-b\Vert _{l_{p}}^{p}\Vert Ax-b\Vert _2^{2}}\cdot \Vert Ax_k-b\Vert _{2}^{2} \\&\quad \le \left( 1-\frac{1}{2} {\widetilde{\sigma }}^2_{\min }(A)\cdot \frac{|{\hat{x}}|_{\min }}{|{\hat{x}}|_{\min }+2\lambda }\cdot \mathop {\inf }\limits _{x\ne {\hat{x}}}\frac{\Vert Ax-b\Vert _{l_{p+2}}^{p+2}}{\Vert Ax-b\Vert _{l_{p}}^{p}\Vert Ax-b\Vert _2^{2}}\right) D_{f}^{x_{k}^*}(x_{k},{\hat{x}}). \end{aligned}$$

The last inequality follows by invoking Lemma 3. Now considering all indices $i_0,\ldots ,i_k$ as random variables and taking the full expectation on both sides, we have that

$$\begin{aligned} {\mathbb {E}}\left( D_f^{x_{k+1}^*}(x_{k+1},{\hat{x}})\right) \le \left( 1-\frac{1}{2} {\widetilde{\sigma }}^2_{\min }(A)\cdot \frac{|{\hat{x}}|_{\min }}{|{\hat{x}}|_{\min }+2\lambda }\cdot \mathop {\inf } \limits _{z\ne 0}\frac{\Vert Az\Vert _{l_{p+2}}^{p+2}}{\Vert Az\Vert _{l_{p}}^{p}\Vert Az\Vert _2^{2}}\right) {\mathbb {E}}\left( D_f^{x_{k}^*}(x_{k},{\hat{x}})\right) , \end{aligned}$$

where $z=x-{\hat{x}}$. According to Lemma 1 and f is 1-strongly convex, we can obtain

$$\begin{aligned} D_f^{x_{k}^*}(x_{k},{\hat{x}})\ge \frac{1}{2}\Vert x_{k}-{\hat{x}}\Vert _2^2. \end{aligned}$$

Thus, we get

$$\begin{aligned} {\mathbb {E}}\Vert x_{k}-{\hat{x}}\Vert _2\le \left( 1-\frac{1}{2} {\widetilde{\sigma }}^2_{\min }(A)\cdot \frac{|{\hat{x}}|_{\min }}{|{\hat{x}}|_{\min }+2\lambda }\cdot \mathop {\inf } \limits _{z\ne 0}\frac{\Vert Az\Vert _{l_{p+2}}^{p+2}}{\Vert Az\Vert _{l_{p}}^{p}\Vert Az\Vert _2^{2}}\right) ^{\frac{k}{2}} \sqrt{2\lambda \Vert {\hat{x}}\Vert _1+\Vert {\hat{x}}\Vert _2^2}. \end{aligned}$$

Next, we compare the convergence rates between RaSK and WRaSK. Hölder’s inequality implies that for any $0\ne x\in {\mathbb {R}}^m,$

$$\begin{aligned} \Vert x\Vert _{l_p}^p=\sum _{i=1}^{m}|x_i|^p\le \left( \sum _{i=1}^{m}|x_i|^{p+2}\right) ^{\frac{p}{p+2}}\left( \sum _{i=1}^{m}1\right) ^{\frac{2}{p+2}}=\Vert x\Vert _{l_{p+2}}^pm^{\frac{2}{p+2}}, \end{aligned}$$

(A2)

and

$$\begin{aligned} \Vert x\Vert _{2}^2=\sum _{i=1}^{m}|x_i|^{2}\le \left( \sum _{i=1}^{m}|x_i|^{p+2}\right) ^{\frac{2}{p+2}}\left( \sum _{i=1}^{m}1\right) ^{\frac{p}{p+2}}=\Vert x\Vert _{l_{p+2}}^2m^{\frac{p}{p+2}}. \end{aligned}$$

(A3)

Based on (A2) and (A3), for $0\ne Az\in {\mathbb {R}}^m$ we deduce that

$$\begin{aligned} \frac{\Vert Az\Vert _{l_{p+2}}^{p+2}}{\Vert Az\Vert _{l_{p}}^{p}} \ge \frac{\Vert Az\Vert _{l_{p+2}}^{2}}{m^{\frac{2}{p+2}}} \ge \frac{1}{m}\Vert Az\Vert _2^2. \end{aligned}$$

(A4)

Hence,

$$\begin{aligned} \frac{\Vert Az\Vert _{l_{p+2}}^{p+2}}{\Vert Az\Vert _{l_{p}}^{p}\Vert Az\Vert _2^{2}}\ge \frac{1}{m}. \end{aligned}$$

(A5)

It follows that

$$\begin{aligned} \mathop {\inf }\limits _{z\ne 0}\frac{\Vert Az\Vert _{l_{p+2}}^{p+2}}{\Vert Az\Vert _{l_{p}}^{p}\Vert Az\Vert _2^{2}}\ge \frac{1}{m}, \end{aligned}$$

with which we further derive that

$$\begin{aligned} \frac{1}{2}\cdot {\widetilde{\sigma }}^2_{\min }(A)\cdot \frac{|{\hat{x}}|_{\min }}{|{\hat{x}}|_{\min }+2\lambda }\cdot \mathop {\inf } \limits _{z\ne 0}\frac{\Vert Az\Vert _{l_{p+2}}^{p+2}}{\Vert Az\Vert _{l_{p}}^{p}\Vert Az\Vert _2^{2}}\ge \frac{1}{2}\cdot \frac{1}{m}\cdot {\widetilde{\sigma }}^2_{\min }(A)\cdot \frac{|{\hat{x}}|_{\min }}{|{\hat{x}}|_{\min }+2\lambda }. \end{aligned}$$

Thereby, we conclude that the convergence rate of WRaSK is at least as efficient as RaSK.

As we all know, Hölder’s inequality takes the equal sign if and only if one of the two vectors is the constant multiple of the other. Since uses Hölder’s inequality twice, (A5) with equality holds if and only if Az is a constant multiple of the unit vector. The proof is completed. $\square$

Appendix B Proof of Theorem 2

Proof

We make use of the observation in Needell (2010) that

$$\begin{aligned} x_k^{\delta }:={\hat{x}}+\frac{b_{i_k}^{\delta }-b_{i_k}}{\Vert a_{i_k}\Vert _2^2}a_{i_k} \in H\left( a_{i_k},b_{i_k}^{\delta }\right) . \end{aligned}$$

(B6)

Note that f is 1-strongly convex and $\Vert a_{i_k}\Vert _2=1$; hence according to Lemma 2, we deduce that

$$\begin{aligned} D_f^{x_{k+1}^*}(x_{k+1},x_k^{\delta })\le D_f^{x_{k}^*}(x_{k},x_k^{\delta })-\frac{1}{2}\left( \langle a_{i_k},x_k\rangle -b_{i_k}^{\delta }\right) ^2. \end{aligned}$$

(B7)

Reformulating (B7) by (B6), we derive that

$$\begin{aligned} D_f^{x_{k+1}^*}\left( x_{k+1},{\hat{x}}\right) \le D_f^{x_{k}^*}(x_{k},{\hat{x}})-\frac{1}{2}\left( \langle a_{i_k},x_k\rangle -b_{i_k}^{\delta }\right) ^2+\left\langle x_{k+1}^*-x_k^*,x_k^{\delta }-{\hat{x}}\right\rangle . \end{aligned}$$

(B8)

(a) In the WRaSK method, we have

$$\begin{aligned} x_{k+1}^*-x_k^*=-\left( \langle a_{i_k},x_k\rangle -b_{i_k}^{\delta }\right) a_{i_k}. \end{aligned}$$

Recall that $x_k^{\delta }-{\hat{x}}=(b_{i_k}^{\delta }-b_{i_k})a_{i_k}$, we get

$$\begin{aligned} \langle x_{k+1}^*-x_k^*,x_k^{\delta }-{\hat{x}}\rangle =\left( b_{i_k}^{\delta }-b_{i_k}\right) ^2 -\left( b_{i_k}^{\delta }-b_{i_k}\right) \cdot \left( \langle a_{i_k},x_k\rangle -b_{i_k}\right) , \end{aligned}$$

(B9)

and

$$\begin{aligned}&-\frac{1}{2}\left( \langle a_{i_k},x_k\rangle -b_{i_k}^{\delta }\right) ^2\nonumber \\&\quad = -\frac{1}{2}\left( \langle a_{i_k},x_k\rangle -b_{i_k}+b_{i_k}-b_{i_k}^{\delta }\right) ^2\nonumber \\&\quad =-\frac{1}{2}\left( \langle a_{i_k},x_k\rangle -b_{i_k}\right) ^2+ \left( b_{i_k}^{\delta }-b_{i_k}\right) \cdot \left( \langle a_{i_k},x_k\rangle -b_{i_k}\right) -\frac{1}{2}\left( b_{i_k}-b_{i_k}^{\delta }\right) ^2. \end{aligned}$$

(B10)

Plugging the reformulations (B9) and (B10) into (B8), we have

$$\begin{aligned} D_f^{x_{k+1}^*}\left( x_{k+1},{\hat{x}}\right) \le D_f^{x_{k}^*}\left( x_{k},{\hat{x}}\right) -\frac{1}{2}\left( \langle a_{i_k},x_k\rangle -b_{i_k}\right) ^2 +\frac{1}{2}\left( b_{i_k}-b_{i_k}^{\delta }\right) ^2. \end{aligned}$$

We fix the values of the indices $i_0,\ldots ,i_{k-1}$ and only consider $i_k$ as a random variable. Taking the conditional expectation on both sides, we get

$$\begin{aligned} \begin{aligned}&{\mathbb {E}}\left( D_f^{x_{k+1}^*}\left( x_{k+1},{\hat{x}}\right) |i_0,\ldots ,i_{k-1}\right) \\&\quad \le D_f^{x_{k}^*}\left( x_{k},{\hat{x}}\right) - \frac{1}{2}\sum _{i=1}^{m}\left( \langle a_i,x_k\rangle -b_i\right) ^2\frac{|\langle a_i,x_k\rangle -b_i|^p}{\Vert Ax_k-b\Vert _{l_p}^p}+ \frac{1}{2}\sum _{i=1}^{m}\left( b_i-b_i^{\delta }\right) ^2\frac{|\langle a_i,x_k\rangle -b_i|^p}{\Vert Ax_k-b\Vert _{l_p}^p}\\&\quad \le qD_f^{x_{k}^*}\left( x_{k},{\hat{x}}\right) +\frac{1}{2}\Vert b-b^{\delta }\Vert _{l_{p+2}}^2\cdot \frac{\Vert Ax_k-b\Vert _{l_{p+2}}^p}{\Vert Ax_k-b\Vert _{l_{p}}^p}. \end{aligned} \end{aligned}$$

The last inequality can be deduced by using the conclusion of Theorem 1 and Hölder’s inequality

$$\begin{aligned} \sum _{i=1}^{m}(b_i-b_i^{\delta })^2\cdot |\langle a_i,x_k\rangle -b_i|^p\le \Vert b-b^{\delta }\Vert _{l_{p+2}}^2\cdot \Vert Ax_k-b\Vert _{l_{p+2}}^p. \end{aligned}$$

Now considering all indices $i_0,\ldots ,i_k$ as random variables and taking the full expectation on both sides, we can derive that

$$\begin{aligned} \begin{aligned} {\mathbb {E}}\left( D_f^{x_{k+1}^*}(x_{k+1},{\hat{x}})\right)&\le q^{k+1}\left( \lambda \Vert {\hat{x}}\Vert _1+\frac{1}{2} \Vert {\hat{x}}\Vert _2^2\right) +\frac{1}{2}\Vert b-b^{\delta }\Vert _{l_{p+2}}^2\sum _{i=0}^{k}q^{k-i} \frac{\Vert Ax_i-b\Vert _{l_{p+2}}^p}{\Vert Ax_i-b\Vert _{l_{p}}^p}. \end{aligned} \end{aligned}$$

According to the equivalence of vector norms in ${\mathbb {R}}^m$, there is a constant $c\in {\mathbb {R}}$ such that for any vector $z\in {\mathbb {R}}^m$ we have that

$$\begin{aligned} \Vert z\Vert _{l_{p+2}} \le c\Vert z\Vert _{l_{p}}. \end{aligned}$$

Thus,

$$\begin{aligned} \begin{aligned} {\mathbb {E}}(D_f^{x_{k}^*}(x_{k},{\hat{x}}))&\le q^k(\lambda \Vert {\hat{x}}\Vert _1+\frac{1}{2} \Vert {\hat{x}}\Vert _2^2)+\frac{1}{2}\Vert b-b^{\delta }\Vert _{l_{p+2}}^2\cdot \frac{c^pq}{1-q}. \end{aligned} \end{aligned}$$

Using $\sqrt{u+v}\le \sqrt{u}+\sqrt{v}$ and f is 1-strongly convex, we further deduce that

$$\begin{aligned} \begin{aligned} {\mathbb {E}}\Vert x_{k}-{\hat{x}}\Vert&\le q^{\frac{k}{2}}\sqrt{2\lambda \Vert {\hat{x}}\Vert _1+ \Vert {\hat{x}}\Vert _2^2}+ \delta \sqrt{ \frac{c^pq}{1-q}}. \end{aligned} \end{aligned}$$

(b) In the EWRaSK method, according to Example 1 we have $x_{k+1}^*=x_k+\lambda \cdot s_k$, where $\Vert s_k\Vert _{\infty },\Vert s_{k+1}\Vert _{\infty }\le 1$. The exact linesearch guarantees $\langle x_{k+1},a_{i_k}\rangle =b_{i_k}^{\delta };$ thus,

$$\begin{aligned} \langle x_{k+1}^{*}-x_{k}^{*},x_k^{\delta }-{\hat{x}}\rangle&=\frac{b_{i_{k}}^{\delta }-b_{i_{k}}}{\Vert a_{i_{k}}\Vert _2^2}(\langle x_{k+1}-x_{k},a_{i_{k}}\rangle +\lambda \langle s_{k+1}-s_{k},a_{i_k}\rangle ),\nonumber \\&\le \frac{(b_{i_{k}}^{\delta }-b_{i_{k}})^{2}}{\Vert a_{i_{k}}\Vert _2^{2}} -\frac{(b_{i_{k}}^{\delta }-b_{i_{k}})(\langle a_{i_{k}},x_{k}\rangle -b_{i_{k}})}{\Vert a_{i_{k}}\Vert _2^{2}}+ \frac{2\lambda |b_{i_{k}}^{\delta }-b_{i_{k}}|\cdot \Vert a_{i_{k}}\Vert _{1}}{\Vert a_{i_{k}}\Vert _{2}^{2}}. \end{aligned}$$

(B11)

Bringing (B10) and (B11) into (B8), note that $\Vert a_{i_k}\Vert _2=1$ we derive

$$\begin{aligned} D_f^{x_{k+1}^*}(x_{k+1},{\hat{x}})&\le D_f^{x_{k}^*}(x_{k},{\hat{x}})-\frac{1}{2}(\langle a_{i_k},x_k\rangle -b_{i_k})^2 +\frac{1}{2}(b_{i_k}-b_{i_k}^{\delta })^2+ 2\lambda |b_{i_k}^{\delta }-b_{i_k}|\cdot \Vert a_{i_k}\Vert _1. \end{aligned}$$

(B12)

Use Hölder’s inequality to reformulate

$$\begin{aligned}&2\lambda \sum _{i=1}^{m}|b_{i}^{\delta }-b_{i}|\cdot \Vert a_{i}\Vert _1\cdot \frac{|\langle a_i,x_k\rangle -b_i|^p}{\Vert Ax_{k}-b\Vert _{l_p}^p}&\le 2\lambda \Vert b^{\delta }-b\Vert _{l_{p+2}}\cdot \Vert A\Vert _{1,p+2}\cdot \frac{\Vert Ax_k-b\Vert _{l_{p+2}}^p}{\Vert Ax_k-b\Vert _{l_p}^p}. \end{aligned}$$

(B13)

Similar to (a), we get

$$\begin{aligned} \begin{aligned} {\mathbb {E}}[\Vert x_k-{\hat{x}}\Vert _2]&\le q^{\frac{k}{2}}\sqrt{2\lambda \Vert {\hat{x}}\Vert _1+\Vert {\hat{x}}\Vert _2^2}+\delta \sqrt{\left( 1+\frac{4\lambda \Vert A\Vert _{1,p+2}}{\delta }\right) \cdot \frac{c^pq}{1-q}}. \end{aligned} \end{aligned}$$

The proof is completed. $\square$

Appendix C Proof of Lemma 4

Proof

First, we compute the derivative of the function f(x) as follows:

$$\begin{aligned} \begin{aligned} f'(x)&= \frac{1}{\left( \sum _{i=1}^{n}e^{d_ix}\right) ^2} \left[ \left( \sum _{i=1}^{n}d_ie^{2d_i}e^{d_ix}\right) \left( \sum _{j=1}^{n}e^{d_jx}\right) - \left( \sum _{i=1}^{n}e^{2d_i}e^{d_ix}\right) \left( \sum _{j=1}^{n}d_je^{d_jx}\right) \right] ,\\&= \frac{1}{\left( \sum _{i=1}^{n}e^{d_ix}\right) ^2} \left[ \sum _{i=1}^{n}\sum _{j=1}^{n}d_ie^{2d_i}e^{(d_i+d_j)x}- \sum _{i=1}^{n}\sum _{j=1}^{n}d_je^{2d_i}e^{(d_i+d_j)x}\right] ,\\&=\frac{1}{\left( \sum _{i=1}^{n}e^{d_ix}\right) ^2} \left[ \sum _{i=1}^{n}\sum _{j=1}^{n}(d_i-d_j)e^{2d_i}e^{(d_i+d_j)x}\right] . \end{aligned} \end{aligned}$$

Denote

$$\begin{aligned} h(x):= \sum _{i=1}^{n}\sum _{j=1}^{n}(d_i-d_j)e^{2d_i}e^{(d_i+d_j)x}. \end{aligned}$$

It follows that $f^{'}(x)=\frac{h(x)}{\left( \sum _{i=1}^{n}e^{d_ix}\right) ^2}$. Let $t_{ij}=e^{2d_i}e^{(d_i+d_j)x}$; then we have

$$\begin{aligned} h(x)=\sum _{i=1}^{n}\sum _{j=1}^{n}(d_i-d_j)t_{ij}. \end{aligned}$$

(C14)

Exchanging the role of i and j, we obtain

$$\begin{aligned} h(x)=\sum _{i=1}^{n}\sum _{j=1}^{n}(d_j-d_i)t_{ji}. \end{aligned}$$

(C15)

Based on (C14) and (C15), we deduce that

$$\begin{aligned} \begin{aligned} 2h(x) = \sum _{i=1}^{n}\sum _{j=1}^{n}(d_i-d_j)(t_{ij}-t_{ji}) = \sum _{i=1}^{n}\sum _{j=1}^{n}(d_i-d_j)(e^{2d_i}-e^{2d_j})e^{(d_i+d_j)x}. \end{aligned} \end{aligned}$$

It follows that $h(x)\ge 0$ and hence $f^{'}(x)\ge 0,\, \forall x\in (0,\infty )$. Therefore, f(x) is a monotonic increasing function. The proof is completed. $\square$