Recovering Structured Signals in Noise: Least-Squares Meets Compressed Sensing

Abstract

The typical scenario that arises in most “big data” problems is one where the ambient dimension of the signal is very large (e.g., high resolution images, gene expression data from a DNA microarray, social network data, etc.), yet is such that its desired properties lie in some low dimensional structure (sparsity, low-rankness, clusters, etc.). In the modern viewpoint, the goal is to come up with efficient algorithms to reveal these structures and for which, under suitable conditions, one can give theoretical guarantees. We specifically consider the problem of recovering such a structured signal (sparse, low-rank, block-sparse, etc.) from noisy compressed measurements. A general algorithm for such problems, commonly referred to as generalized LASSO, attempts to solve this problem by minimizing a least-squares cost with an added “structure-inducing” regularization term (ℓ ₁ norm, nuclear norm, mixed ℓ ₂/ℓ ₁ norm, etc.). While the LASSO algorithm has been around for 20 years and has enjoyed great success in practice, there has been relatively little analysis of its performance. In this chapter, we will provide a full performance analysis and compute, in closed form, the mean-square-error of the reconstructed signal. We will highlight some of the mathematical vignettes necessary for the analysis, make connections to noiseless compressed sensing and proximal denoising, and will emphasize the central role of the “statistical dimension” of a structured signal.

Appendix

The upper bounds on the NSE of the generalized LASSO presented in Sections 4.5–4.7 are in terms of the summary parameters $\mathbf{D}(\lambda \partial f(\mathbf{x}_{0}))$ and $\mathbf{D}(\mathop{\mathrm{cone}}\nolimits (\partial f(\mathbf{x}_{0})))$. While the bounds are simple, concise, and nicely resemble the corresponding ones in the case of OLS, it may appear to the reader that the formulae are rather abstract, because of the presence of $\mathbf{D}(\mathop{\mathrm{cone}}\nolimits (\partial f(\mathbf{x}_{0})))$ and $\mathbf{D}(\lambda \partial f(\mathbf{x}_{0}))$.

However, as discussed here, for a large number of widely used convex regularizers f(⋅ ), one can calculate (tight) upper bounds or even explicit formulae for these quantities. For example, for the estimation of a k-sparse signal x ₀ with $f(\cdot ) =\| \cdot \|_{1}$, it has been shown that $\mathbf{D}(\mathop{\mathrm{cone}}\nolimits (\partial f(\mathbf{x}_{0}))) \lesssim 2k(\log \frac{n} {k} + 1)$. Substituting this into Theorems 1 and 4 results in the “closed-form” upper bounds given in (4.20) and (4.22), i.e. ones expressed only in terms of m, n, and k. Analogous results have been derived [14, 31, 44, 54] for other well-known signal models as well, including low rankness and block-sparsity¹⁵. The first column of Table 4.1 summarizes some of the results for $\mathbf{D}(\mathop{\mathrm{cone}}\nolimits (\partial f(\mathbf{x}_{0})))$ found in the literature [14, 31]. The second column provides closed form results on $\mathbf{D}(\lambda \partial f(\mathbf{x}_{0}))$ when λ is sufficiently large [47]. Note that, by setting λ to its lower bound in the second row, one approximately obtains the corresponding result in the first row. This should not be surprising due to (4.29). Also, this value of λ is a good proxy for the optimal regularizer λ _best of the ℓ ₂-LASSO as was discussed in Sections 4.5.3.4 and 4.6.2.1.

Table 4.1 Closed form upper bounds for $\mathbf{D}(\mathop{\mathrm{cone}}\nolimits (\partial f(\mathbf{x}_{0})))$ and $\mathbf{D}(\lambda \partial f(\mathbf{x}_{0}))$.

We refer the reader to [1, 14, 31, 47] for the details and state-of-the-art bounds on $\mathbf{D}(\mathop{\mathrm{cone}}\nolimits (\partial f(\mathbf{x}_{0})))$ and $\mathbf{D}(\lambda \partial f(\mathbf{x}_{0}))$. Identifying the subdifferential ∂ f(x ₀) and calculating D(λ ∂ f(x ₀)) for all λ ≥ 0 are the critical steps. Once those are available, computing $\min _{\lambda \geq 0}\mathbf{D}(\lambda \partial f(\mathbf{x}_{0}))$ provides upper approximation formulae for $\mathbf{D}(\mathop{\mathrm{cone}}\nolimits (\partial f(\mathbf{x}_{0})))$. This idea was first introduced by Stojnic [55] and was subsequently refined and generalized in [14]. Most recently [1, 31] proved (4.29), thus showing that the resulting approximation on $\mathbf{D}(\mathop{\mathrm{cone}}\nolimits (\partial f(\mathbf{x}_{0})))$ is in fact highly accurate. Section 4 of [1] is an excellent reference for further details and the notation used there is closer to ours.

We should emphasize that examples of regularizers are not limited to the ones discussed here and presented in Table 4.1. There are increasingly more signal classes that exhibit low-dimensionality and to which the theorems of Sections 4.5–4.7 would apply. Some of these are as follows.

• Non-negativity constraint: x ₀ has non-negative entries, [20].

• Low-rank plus sparse matrices: x ₀ can be represented as sum of a low-rank and a sparse matrix, [65].

• Signals with sparse gradient: Rather than x ₀ itself, its gradient $\mathbf{d} _{\mathbf{x}_{0}}(i) = \mathbf{x}_{0}(i) -\mathbf{x}_{0}(i - 1)$ is sparse, [8].

• Low-rank tensors: x ₀ is a tensor and its unfoldings are low-rank matrices, [32, 36].

• Simultaneously sparse and low-rank matrices: For instance, x ₀ = s s ^T for a sparse vector s, [45, 51].

Establishing new and tighter analytic bounds for D(λ ∂ f(x ₀)) and $\mathbf{D}(\mathop{\mathrm{cone}}\nolimits (\partial f(\mathbf{x}_{0})))$ for more regularizers f is certainly an interesting direction for future research. In the case where such analytic bounds do not already exist in literature or are hard to derive, one can numerically estimate D(λ ∂ f(x ₀)) and $\mathbf{D}(\mathop{\mathrm{cone}}\nolimits (\partial f(\mathbf{x}_{0})))$ once there is an available characterization of the subdifferential ∂ f(x ₀). Using the concentration property of $\mathop{\mathrm{dist}}\nolimits ^{2}(\mathbf{h},\lambda \partial f(\mathbf{x}_{0}))$ around $\mathbf{D}(\lambda \partial f(\mathbf{x}_{0}))$, when $\mathbf{h} \sim \mathcal{N}(0,\mathbf{I}_{n})$, we can compute $\mathbf{D}(\lambda \partial f(\mathbf{x}_{0}))$, as follows:

• 1. draw a vector $\mathbf{h} \sim \mathcal{N}(0,\mathbf{I}_{n})$,

• 2. return the solution of the convex program $\min _{\mathbf{s}\in \partial f(\mathbf{x}_{0})}\|\mathbf{h} -\lambda \mathbf{s}\|^{2}$.

Computing $\mathbf{D}(\mathop{\mathrm{cone}}\nolimits (\partial f(\mathbf{x}_{0})))$ can be built on the same recipe by recognizing $\mathop{\mathrm{dist}}\nolimits ^{2}(\mathbf{h},\text{cone}(\partial f(\mathbf{x}_{0})))$ as $\min _{\lambda \geq 0,\mathbf{s}\in \partial f(\mathbf{x}_{0})}\|\mathbf{h} -\lambda \mathbf{s}\|^{2}$.

To sum up, any bound on D(λ ∂ f(x ₀)) and $\mathbf{D}(\mathop{\mathrm{cone}}\nolimits (\partial f(\mathbf{x}_{0})))$ translates, through Theorems 1–6, into corresponding upper bounds on the NSE of the generalized LASSO. For purposes of illustration and completeness, we review next the details of computing $\mathbf{D}(\mathop{\mathrm{cone}}\nolimits (\partial f(\mathbf{x}_{0})))$ and $\mathbf{D}(\lambda \partial f(\mathbf{x}_{0}))$ for the celebrated case where x ₀ is sparse and the ℓ ₁-norm is used as the regularizer.

4.1.1 Sparse signals

Suppose x ₀ is a k-sparse signal and $f(\cdot ) =\| \cdot \|_{1}$. Denote by S the support set of x ₀, and by S ^c its complement. The subdifferential at x ₀ is [52],

$$\displaystyle{\partial f(\mathbf{x}_{0}) =\{ \mathbf{s} \in \mathbb{R}^{n}\ \vert \ \|\mathbf{s}\|_{ \infty }\leq 1\text{ and }\mathbf{s}_{i} = \text{sign}((\mathbf{x}_{0})_{i}),\forall i \in S\}.}$$

Let $\mathbf{h} \in \mathbb{R}^{n}$ have i.i.d $\mathcal{N}(0,1)$ entries and define

$$\displaystyle{\text{shrink}(\chi,\lambda ) = \left \{\begin{array}{@{}l@{\quad }l@{}} \chi -\lambda \quad &,\chi> \lambda, \\ 0 \quad &,-\lambda \leq \chi \leq \lambda, \\ \chi +\lambda \quad &,\chi <-\lambda. \end{array} \right.}$$

Then, D(λ ∂ f(x ₀)) is equal to ([1, 14])

$$\displaystyle\begin{array}{rcl} \mathbf{D}(\lambda \partial f(\mathbf{x}_{0}))& =& \mathbb{E}[\mathop{\mathrm{dist}}\nolimits ^{2}(\mathbf{h},\lambda \partial f(\mathbf{x}_{ 0}))] \\ & =& \sum _{i\in S}\mathbb{E}[(\mathbf{h}_{i} -\lambda \text{sign}((\mathbf{x}_{0})_{i}))^{2}] +\sum _{ i\in S^{c}}\mathbb{E}[\text{shrink}^{2}(\mathbf{h}_{ i},\lambda )] = \\ & =& k(1 + \lambda ^{2}) + (n - k)\sqrt{\frac{2} {\pi }} \left [(1 + \lambda ^{2})\int _{\lambda }^{\infty }e^{-t^{2}/2 }\mathrm{d}t -\lambda \exp (-\lambda ^{2}/2)\right ].{}\end{array}$$

(4.50)

Note that D(λ ∂ f(x ₀)) depends only on n, λ and k = | S | , and not explicitly on S itself (which is not known). Substituting the expression in (4.50) in place of the D(λ ∂ f(x ₀)) in Theorems 2 and 5 yields explicit expressions for the corresponding upper bounds in terms of n, m, k, and λ.

We can obtain an even simpler upper bound on D(λ ∂ f(x ₀)) which does not involve error functions as we show below. Denote $Q(t) = \frac{1} {\sqrt{2\pi }}\int _{t}^{\infty }e^{-\tau ^{2}/2 }\mathrm{d}\tau$ the complementary c.d.f. of a standard normal random variable. Then,

$$\displaystyle\begin{array}{rcl} \frac{1} {2}\mathbb{E}[\text{shrink}^{2}(\mathbf{h}_{ i},\lambda )]& =& \int _{\lambda }^{\infty }(t -\lambda )^{2}\mathrm{d}(-Q(t)) \\ & =& -\left [(t -\lambda )^{2}Q(t)\right ]_{\lambda }^{\infty } + 2\int _{\lambda }^{\infty }(t -\lambda )Q(t)\mathrm{d}t \\ & \leq & \int _{\lambda }^{\infty }(t -\lambda )e^{-t^{2}/2 }\mathrm{d}t {}\end{array}$$

(4.51)

$$\displaystyle\begin{array}{rcl} & \leq & e^{-\lambda ^{2}/2 } - \frac{\lambda ^{2}} {\lambda ^{2} + 1}e^{-\lambda ^{2}/2 } \\ & =& \frac{1} {\lambda ^{2} + 1}e^{-\lambda ^{2}/2 }. {}\end{array}$$

(4.52)

(4.51) and (4.52) follow from standard upper and lower tail bounds on normal random variables, namely $\frac{1} {\sqrt{2\pi }} \frac{t} {t^{2}+1}e^{-t^{2}/2 } \leq Q(t) \leq \frac{1} {2}e^{-t^{2}/2 }$. From this, we find that

$$\displaystyle\begin{array}{rcl} \mathbf{D}(\lambda \partial f(\mathbf{x}_{0})) \leq k(1 + \lambda ^{2}) + (n - k) \frac{2} {\lambda ^{2} + 1}e^{-\lambda ^{2}/2 }.& & {}\\ \end{array}$$

Letting $\lambda \geq \sqrt{2\log (\frac{n} {k})}$ in the above expression recovers the corresponding entry in Table 4.1:

$$\displaystyle\begin{array}{rcl} \mathbf{D}(\lambda \partial f(\mathbf{x}_{0})) \leq (\lambda ^{2} + 3)k,\text{ when }\lambda \geq \sqrt{2\log (\frac{n} {k})}.& &{}\end{array}$$

(4.53)

Substituting (4.53) in Theorems 2 and 5 recovers the bounds in (4.21) and (4.23), respectively.

Setting $\lambda = \sqrt{2\log (\frac{n} {k})}$ in (4.53) provides an approximation to $\mathbf{D}(\mathop{\mathrm{cone}}\nolimits (\partial f(\mathbf{x}_{0})))$. In particular, $\mathbf{D}(\mathop{\mathrm{cone}}\nolimits (\partial f(\mathbf{x}_{0}))) \leq 2k(\log (\frac{n} {k}) + 3/2)$. [14] obtains an even tighter bound ($\mathbf{D}(\mathop{\mathrm{cone}}\nolimits (\partial f(\mathbf{x}_{0}))) \leq 2k(\log (\frac{n} {k}) + 3/4)$ starting again from (4.50), but using different tail bounds for Gaussians. We refer the reader to Proposition 3.10 in [14] for the exact details.