Matrix completion with nonconvex regularization: spectral operators and scalable algorithms

Abstract

In this paper, we study the popularly dubbed matrix completion problem, where the task is to “fill in” the unobserved entries of a matrix from a small subset of observed entries, under the assumption that the underlying matrix is of low rank. Our contributions herein enhance our prior work on nuclear norm regularized problems for matrix completion (Mazumder et al. in J Mach Learn Res 1532(11):2287–2322, 2010) by incorporating a continuum of nonconvex penalty functions between the convex nuclear norm and nonconvex rank functions. Inspired by Soft-Impute (Mazumder et al. 2010; Hastie et al. in J Mach Learn Res, 2016), we propose NC-Impute—an EM-flavored algorithmic framework for computing a family of nonconvex penalized matrix completion problems with warm starts. We present a systematic study of the associated spectral thresholding operators, which play an important role in the overall algorithm. We study convergence properties of the algorithm. Using structured low-rank SVD computations, we demonstrate the computational scalability of our proposal for problems up to the Netflix size (approximately, a 500,000 $\times$ 20,000 matrix with $10^8$ observed entries). We demonstrate that on a wide range of synthetic and real data instances, our proposed nonconvex regularization framework leads to low-rank solutions with better predictive performance when compared to those obtained from nuclear norm problems. Implementations of algorithms proposed herein, written in the R language, are made available on github.

Appendix

1.1 Additional technical material

Lemma 1

(Marchenko–Pastur law (Bai and Silverstein 2010)). Let $X\in {\mathbb {R}}^{m \times n}$, where $X_{ij}$ are iid with ${\mathbb {E}}(X_{ij})=0, {\mathbb {E}}(X_{ij}^2)=1$, and $m>n$. Let $\lambda _1\le \lambda _2 \le \dots \le \lambda _n$ be the eigenvalues of $Q_m=\frac{1}{m}X'X$. Define the random spectral measure

$$\begin{aligned} \mu _n=\frac{1}{n}\sum _{i=1}^n\delta _{\lambda _i}\,. \end{aligned}$$

Then, assuming $n/m \rightarrow \alpha \in (0,1]$, we have

$$\begin{aligned} \mu _n(\cdot , \omega ) \rightarrow \mu ~~a.s., \end{aligned}$$

where $\mu$ is a deterministic measure with density

$$\begin{aligned} \frac{\mathrm{d}\mu }{\mathrm{d}x}=\frac{\sqrt{(\alpha _+-x)(x-\alpha _-)}}{2\pi \alpha x}I(\alpha _-\le x \le \alpha _+). \end{aligned}$$

Here, $\alpha _+=(1+\sqrt{\alpha })^2\,$ and $\, \alpha _-=(1-\sqrt{\alpha })^2$.

1.1.1 Proof of Proposition 5.

Proof

In the following proof, we make use of the notation: $\varTheta _1(\cdot )$ and $\varTheta _2(\cdot )$, defined as follows. For two positive sequences $a_{k}$ and $b_k$, we say $a_k= \varTheta _2(b_k)$ if there exists a constant $c>0$ such that $a_k \ge c b_k$ and we say $a_k = \varTheta _1(b_k)$, whenever, $a_k = \varTheta _2(b_k)$ and $b_k=\varTheta _2(a_k)$.

We first consider the case $\lambda _n=\varTheta _1(\sqrt{m})$ For simplicity, we assume $\lambda _n=\zeta \sqrt{m}$ for some constant $\zeta >0$. Denote $df(S_{\lambda _n,\gamma }(Z))= D_{\lambda _n,\gamma }$, and use ${\mathcal {T}}_{t_1,t_2}$ to represent

$$\begin{aligned} \frac{\sqrt{mt_1} s_{\lambda _n,\gamma }(\sqrt{mt_1}) - \sqrt{mt_2} s_{\lambda _n,\gamma }(\sqrt{mt_2})}{mt_1-mt_2} \mathbb {1}(t_1\ne t_2). \end{aligned}$$

Adopting the notation from Lemma 1, it is not hard to verify that

$$\begin{aligned} D_{\lambda _n,\gamma }&= n {\mathbb {E}}_{\mu _n} \bigg \{ s'_{\lambda _n,\gamma }(\sqrt{mt_1}) + |m-n|\frac{s_{\lambda _n,\gamma }(\sqrt{mt_1})}{\sqrt{mt_1}} \bigg \} \\&\quad + n^2 {\mathbb {E}}_{\mu _n} ({\mathcal {T}}_{t_1,t_2})\,, \end{aligned}$$

where $t_1, t_2 \overset{\text {iid}}{\sim } \mu _n$. A quick check of the relation between $s_{\lambda _n,\gamma }$ and $g_{\zeta ,\gamma }$ yields

$$\begin{aligned} \frac{D_{\lambda _n,\gamma }}{mn}= & {} \frac{1}{m}{\mathbb {E}}_{\mu _n}s'_{\lambda _n, \gamma }(\sqrt{mt_1})+\left( 1-\frac{n}{m}\right) {\mathbb {E}}_{\mu _n}g_{\zeta , \gamma }(t_1)\\&+\frac{n}{m} {\mathbb {E}}_{\mu _n} \left\{ \frac{t_1g_{\zeta ,\gamma }(t_1)-t_2g_{\zeta ,\gamma }(t_2)}{t_1-t_2} \mathbb {1}(t_1\ne t_2) \right\} \,. \end{aligned}$$

Due to the Lipschitz continuity of the functions $s_{\lambda _n,\gamma }(x)$ and $xg_{\zeta , \gamma }(x)$, we obtain

$$\begin{aligned} \Big | \frac{D_{\lambda _n,\gamma }}{mn} \Big | \le \frac{\gamma }{m(\gamma -1)}+\left( 1-\frac{n}{m}\right) + \frac{n}{m}\left( \frac{2\gamma -1}{2\gamma -2}\right) \,. \end{aligned}$$

Hence, there exists a positive constant $C_{\alpha }$, such that for sufficiently large n,

$$\begin{aligned} \Big | \frac{D_{\lambda _n,\gamma }}{mn} \Big | \le C_{\alpha }, \quad \, a.s. \end{aligned}$$

Let $T_1,T_2$ be two independent random variables generated from the Marchenko–Pastur distribution $\mu$. If we can show

$$\begin{aligned}&\frac{D_{\lambda _n,\gamma }}{mn} \overset{a.s.}{\rightarrow } \\&\quad (1-\alpha ){\mathbb {E}}g_{\zeta ,\gamma }(T_1) + \alpha {\mathbb {E}}\left( \frac{T_1g_{\zeta ,\gamma }(T_1)-T_2g_{\zeta ,\gamma } (T_2)}{T_1-T_2}\right) , \end{aligned}$$

then by the dominated convergence theorem (DCT), we conclude the proof in the $\lambda _n=\varTheta _1(\sqrt{m})$ regime. Note immediately that

$$\begin{aligned} \frac{1}{m} {\mathbb {E}}_{\mu _n}s'_{\lambda _n,\gamma }(\sqrt{mt_1}) \rightarrow 0 \quad a.s. \end{aligned}$$

(34)

Moreover, given that $g_{\zeta ,\gamma }(\cdot )$ is bounded and continuous, the Marchenko–Pastur theorem in Lemma 1 implies

$$\begin{aligned} \left( 1-\frac{n}{m}\right) {\mathbb {E}}_{\mu _n} g_{\zeta ,\gamma }(t_1) \rightarrow (1-\alpha ) {\mathbb {E}}_{\mu } g_{\zeta ,\gamma }(T_1) \quad a.s. \end{aligned}$$

(35)

Since $(t_1, t_2) \overset{d}{\rightarrow } (T_1, T_2)$, and the discontinuity set of the function $\frac{t_1g_{\zeta ,\gamma }(t_1)-t_2g_{\zeta ,\gamma }(t_2)}{t_1-t_2}\mathbb {1}(t_1\ne t_2)$ has zero probability under the measure induced by $(T_1,T_2)$, by the continuous mapping theorem,

$$\begin{aligned}&\frac{t_1g_{\zeta ,\gamma }(t_1)-t_2g_{\zeta ,\gamma }(t_2)}{t_1-t_2}\mathbb {1}(t_1\ne t_2) \overset{d}{\rightarrow } \nonumber \\&\quad \frac{T_1g_{\zeta ,\gamma }(T_1)-T_2g_{\zeta ,\gamma }(T_2)}{T_1-T_2}\mathbb {1}(T_1 \ne T_2) \quad \text {as } \, n \rightarrow \infty \,. \end{aligned}$$

Also, due to the boundedness of $\frac{t_1g_{\zeta ,\gamma }(t_1)-t_2g_{\zeta ,\gamma }(t_2)}{t_1-t_2}\mathbb {1}(t_1\ne t_2)$, it holds that

$$\begin{aligned}&{\mathbb {E}}_{\mu _n} \left\{ \frac{t_1g_{\zeta ,\gamma }(t_1)-t_2g_{\zeta ,\gamma }(t_2)}{t_1-t_2}\mathbb {1}(t_1\ne t_2) \right\} \overset{a.s.}{\rightarrow } \nonumber \\&\quad {\mathbb {E}}_{\mu } \left\{ \frac{T_1g_{\zeta ,\gamma }(T_1)-T_2g_{\zeta ,\gamma }(T_2)}{T_1-T_2}\mathbb {1}(T_1 \ne T_2)\right\} . \end{aligned}$$

(36)

Combining (34)–(36) completes the proof for the $\lambda _n=\varTheta _1(\sqrt{m})$ case.

When $\lambda _n=o(\sqrt{m})$, we can readily see that

$$\begin{aligned} {\mathbb {E}}_{\mu _n}\mathbb {1}(\sqrt{mt_1} \ge \lambda _n \gamma ) \rightarrow 1, a.s. \end{aligned}$$

Using that both $\frac{s_{\lambda _n,\gamma }(\sqrt{mt_1})}{\sqrt{mt_1}}\,$ and ${\mathcal {T}}_{t_1,t_2}$ are bounded, we have, almost surely

$$\begin{aligned}&{\mathbb {E}}_{\mu _n}\frac{s_{\lambda _n,\gamma }(\sqrt{mt_1})}{\sqrt{mt_1}}\\&\quad = {\mathbb {E}}_{\mu _n}\mathbb {1}(\sqrt{mt_1}\ge \lambda _n \gamma ) \\&\quad \quad +{\mathbb {E}}_{\mu _n} \left\{ \frac{s_{\lambda _n, \gamma }(\sqrt{mt_1})}{\sqrt{mt_1}} \mathbb {1}(\sqrt{mt_1}< \lambda _n \gamma ) \right\} \rightarrow 1 \end{aligned}$$

and

$$\begin{aligned} {\mathbb {E}}_{\mu _n}({\mathcal {T}}_{t_1,t_2})&={\mathbb {E}}_{\mu _n} \mathbb {1}(\sqrt{mt_1} \ge \lambda _n \gamma ) \mathbb {1}(\sqrt{mt_2} \ge \lambda _n \gamma ) + o(1)\\&\rightarrow 1. \end{aligned}$$

Invoking DCT completes the proof. Similar arguments hold for the case $\lambda _n=\varTheta _2(\sqrt{m})$. $\square$

Fig. 9

Random orthogonal model (ROM) simulations with $\text {SNR}=1$. The optimal nonconvex penalties are obtained at $\gamma =30$ and $\gamma =20$ under the two scenarios, respectively. The integers from 1 to 100 on the x-axis index the grid of 100 values of $\lambda$ (from largest to smallest) as described in Sect. 4.1

Fig. 10

Random orthogonal model (ROM) simulations with $\text {SNR}=5$. The optimal nonconvex penalties are obtained at $\gamma =30$ and $\gamma =5$ under the two scenarios, respectively. The integers from 1 to 100 on the x-axis index the grid of 100 values of $\lambda$ (from largest to smallest) as described in Sect. 4.1

Fig. 11

Coherent and nonuniform sampling (NUS) simulations with $\text {SNR}=10$. The optimal nonconvex penalties are both obtained at $\gamma =5$ under the two scenarios, respectively. The integers from 1 to 100 on the x-axis index the grid of 100 values of $\lambda$ (from largest to smallest) as described in Sect. 4.1

1.1.2 Proof of Proposition 10

Proof

Observe that R as defined in Proposition 9 can be written as:

$$\begin{aligned} \begin{aligned} R= & {} {\widetilde{A}}{\widetilde{V}}_{1} - {\widetilde{U}}_{1}{\widetilde{\varSigma }}_{1} + (A - {\widetilde{A}}){\widetilde{V}}_{1}= & {} (A - {\widetilde{A}}){\widetilde{V}}_{1}, \end{aligned} \end{aligned}$$

(37)

where above we have used the fact that ${\widetilde{A}}{\widetilde{V}}_{1} = {\widetilde{U}}_{1}{\widetilde{\varSigma }}_{1}$, which follows from the definition of the SVD of ${\widetilde{A}}$. By a simple inequality, it follows that

$$\begin{aligned} \Vert R \Vert _2 \le \Vert (A - {\widetilde{A}})\Vert _2 \Vert {\widetilde{V}}_{1}\Vert _2 = \Vert (A - {\widetilde{A}})\Vert _2, \end{aligned}$$

(38)

where we have used the fact that $\Vert {\widetilde{V}}_{1}\Vert _2 = 1$. Similarly, we have an analogous result for Q:

$$\begin{aligned} \Vert Q \Vert _2 \le \Vert (A - {\widetilde{A}})\Vert _2 \Vert {\widetilde{U}}_{1}\Vert _2 = \Vert (A - {\widetilde{A}})\Vert _2. \end{aligned}$$

(39)

Note that (38) and (39) together imply that if $\Vert {\widetilde{A}} - A \Vert _2$ is small, then so are $\Vert R\Vert _2, \Vert Q\Vert _2$.

We now apply (31) (Proposition 9) with $A = X_{k}$ and ${\widetilde{A}} = X_{k+1}$ and $r_{1} = p$, to arrive at the proof of Proposition 10. $\square$

1.1.3 Proof of Proposition 11

Proof

Proof of Part (a):

Let us write the stationary conditions for every update:

$$\begin{aligned} X_{k+1} = \mathop {\hbox {arg min}}\limits _{X} \; F_{\ell }(X;X_{k}). \end{aligned}$$

We set the subdifferential of the map $X \mapsto F_{\ell }(X;X_{k})$ to zero at $X = X_{k+1}$:

$$\begin{aligned}&\left( X_{k+1} - \left( {\mathcal {P}}_{\varOmega }(Y)+ {\mathcal {P}}_{\varOmega }^\perp (X_{k}) \right) \right) \nonumber \\&\quad + \ell (X_{k+1} - X_{k}) + U_{k+1} \nabla _{k+1} V_{k+1}' = 0, \end{aligned}$$

(40)

where $X_{k+1} = U_{k+1} \mathrm {diag}(\varvec{\sigma }_{k+1})V'_{k+1}$ is the SVD of $X_{k+1}$. Note that the term, $U_{k+1} \nabla _{k+1} V_{k+1}'$ in (40), is a subdifferential (Lewis 1995) of the spectral function:

$$\begin{aligned} X \mapsto \sum _{i} P(\sigma _{i}(X); \lambda , \gamma ), \end{aligned}$$

where $\nabla _{k+1}$ is a diagonal matrix with the ith diagonal entry being a derivative of the map $\sigma _{i} \mapsto P(\sigma _{i}; \lambda ,\gamma )$ (on $\sigma _{i} \ge 0$), denoted by $\partial P(\sigma _{k+1, i}; \lambda ,\gamma )/\partial \sigma _{i}$ for all i. Note that (40) can be rewritten as:

$$\begin{aligned}&{\mathcal {P}}_{\varOmega }(X_{k+1}) - {\mathcal {P}}_{\varOmega }(Y) +U_{k+1} \nabla _{k+1} V_{k+1}' \\&\quad + \underbrace{\left( {\mathcal {P}}_{\varOmega }^\perp (X_{k+1} - X_{k}) + \ell (X_{k+1} - X_{k}) \right) }_{(a)} = 0. \end{aligned}$$

As $k \rightarrow \infty$, term (a) converges to zero (See Proposition 7), and thus, we have:

$$\begin{aligned} {\mathcal {P}}_{\varOmega }(X_{k+1}) - {\mathcal {P}}_{\varOmega }(Y) + U_{k+1} \nabla _{k+1} V_{k+1}' \rightarrow 0. \end{aligned}$$

Let us denote the ith column of $U_{k}$ by ${u}_{k,i}$, and use a similar notation for $V_{k}$ and $v_{k,i}$. Let $r_{k+1}$ denote the rank of $X_{k+1}$. Hence, we have:

$$\begin{aligned}&\sum _{i=1}^{r_{k+1}}\sigma _{k+1,i} {\mathcal {P}}_{\varOmega }(u_{k+1,i} v_{k+1, i}') \\&\quad - {\mathcal {P}}_{\varOmega }(Y) + U_{k+1} \nabla _{k+1} V_{k+1}' \rightarrow 0. \end{aligned}$$

Multiplying the left- and right-hand sides of the above by $u'_{k+1,j}$ and $v_{k+1,j}$, we have the following:

$$\begin{aligned}&\sum _{i=1}^{r_{k+1}} \sigma _{k+1,i} u'_{k+1,j}{\mathcal {P}}_{\varOmega }(u_{k+1,i}v'_{k+1,i})v_{k+1,j}\\&\quad - u'_{k+1,j}{\mathcal {P}}_{\varOmega }(Y)v_{k+1,j} + \nabla _{k+1,j} \rightarrow 0, \end{aligned}$$

for $j = 1, \ldots , r_{k+1}.$ Let $\left\{ {\bar{U}}, {\bar{V}} \right\}$ denote a limit point of the sequence $\left\{ U_{k},V_{k}\right\}$ (which exists since the sequence is bounded), and let r be the rank of ${\bar{U}}$ and ${\bar{V}}$. Let us now study the following equations:⁸

$$\begin{aligned}&\sum _{i=1}^{r} {\bar{\sigma }}_{j} {\bar{u}}'_{j}{\mathcal {P}}_{\varOmega }({\bar{u}}_{i}{\bar{v}}'_{i}){\bar{v}}_{j}\nonumber \\&\quad - {\bar{u}}'_{j}{\mathcal {P}}_{\varOmega }(Y){\bar{v}}_{j} + {\bar{\nabla }}_{j} = 0, \;\;\; j = 1, \ldots , r. \end{aligned}$$

(41)

Using the notation ${\bar{\theta }}_{j} = \text {vec} \left( {\mathcal {P}}_{\varOmega }({\bar{u}}_{j}{\bar{v}}'_{j}) \right)$ and ${\bar{y}} = \text {vec}({\mathcal {P}}_{\varOmega }(Y))$, we note that (41) are the first-order stationary conditions for a point $\bar{\varvec{\sigma }}$ for the following penalized regression problem:

$$\begin{aligned} \mathop {\hbox {min}}\limits _{\varvec{\sigma }} \;\; \frac{1}{2} \Vert \sum _{j=1}^{r} \sigma _{j} {\bar{\theta }}_{j} - {\bar{y}} \Vert _{2}^2 + \sum _{j=1}^{r} P(\sigma _{j}; \lambda ,\gamma ), \end{aligned}$$

(42)

with $\varvec{\sigma } \ge \mathbf {0}$.

If the matrix ${\bar{\varTheta }} = [{{\bar{\theta }}}_{1}, \ldots , {{\bar{\theta }}}_{r}]$ (note that ${\bar{\varTheta }} \in {\mathbb {R}}^{mn \times r}$) has rank r, then any $\varvec{\sigma }$ that satisfies (41) is finite—in particular, the sequence $\varvec{\sigma }_{k}$ is bounded and has a limit point: $\bar{\varvec{\sigma }}$ which satisfies the first-order stationary condition (41).

Proof of Part (b):

Furthermore, if we assume that

$$\begin{aligned} \lambda _{\min }( {\bar{\varTheta }}'{\bar{\varTheta }}) + \phi _P > 0, \end{aligned}$$

then (42) admits a unique solution $\bar{\varvec{\sigma }}$, which implies that $\varvec{\sigma }_{k}$ has a unique limit point, and hence, the sequence $\varvec{\sigma }_{k}$ necessarily converges. $\square$

1.2 Additional simulation results

Fig. 12

The y-axis denotes the number of iterations NC-Impute takes to stabilize the rank. The integers on the x-axis index some values on a grid of $\lambda$ (from largest to smallest) as described in Sect. 4.1. The six plots represent the six scenarios considered in Sect. 4.1: (a)–(d) correspond to the four scenarios of Example A; (e) covers Example B; (f) is for Example C. Each procedure is repeated 10 times

This section contains additional numerical results from the simulation study in Sect. 4.1.

• To demonstrate the variation of the procedures in the experiments, we plot the averaged value and standard error of both test error and rank for some representative nonconvex penalty functions. Specifically, under each scenario considered in Sect. 4.1, we pick the nonconvex penalty that yields the best prediction and rank estimation performance. For each picked penalty, we plot the averaged value of test error and rank along with the associated standard error, against the tuning parameter $\lambda$. The results are shown in Figs. 10, 11, and 12. As is clear form the figures, the standard error is typically (at least) one order of magnitude smaller than the average. Moreover, the general patterns of test error and rank on the solution path are expected, except for a few points corresponding to very small values of $\lambda$. The irregularity of these few points occurs probably because the solutions are getting unstable as the nonconvex regularization becomes weak when $\lambda$ is significantly small.

• To examine the rank dynamics of the updates in NC-Impute, we compute the number of iterations that the algorithm takes for the convergence of the rank. We choose the same six nonconvex penalties as above and evaluate the rank stabilization for several values of $\lambda$. The results are summarized in Fig. 9. One clearly observes that except for few instances, it takes less than 10 iterations for the rank to stabilize. Moreover, when the penalty is more “nonconvex” (i.e., $\gamma$ is smaller), the rank stabilization occurs earlier. These empirical results provide complementary information on rank stabilization that has been theoretically investigated in Sect. 3.1.1.