Max-norm optimization for robust matrix recovery

Abstract

This paper studies the matrix completion problem under arbitrary sampling schemes. We propose a new estimator incorporating both max-norm and nuclear-norm regularization, based on which we can conduct efficient low-rank matrix recovery using a random subset of entries observed with additive noise under general non-uniform and unknown sampling distributions. This method significantly relaxes the uniform sampling assumption imposed for the widely used nuclear-norm penalized approach, and makes low-rank matrix recovery feasible in more practical settings. Theoretically, we prove that the proposed estimator achieves fast rates of convergence under different settings. Computationally, we propose an alternating direction method of multipliers algorithm to efficiently compute the estimator, which bridges a gap between theory and practice of machine learning methods with max-norm regularization. Further, we provide thorough numerical studies to evaluate the proposed method using both simulated and real datasets.

Extensions

In this section, we consider solving the max-norm constrained version of the optimization problem (2.3). In particular, we consider

$$\begin{aligned} \min _{M \in \mathbb {R}^{d_1\times d_2} } \frac{1}{2}\sum _{t=1}^n \big ( Y_{i_t,j_t} - M_{i_t,j_t} \big )^2 + \langle M, I\rangle , \text { subject to }\Vert M\Vert _\infty \le \alpha , \Vert M\Vert _{\max }\le R. \end{aligned}$$

(7.1)

This problem can be formulated as an SDP problem as follows:

$$\begin{aligned} \begin{aligned} \min _{Z\in \mathbb {R}^{d \times d}}&~\frac{1}{2}\sum _{t=1}^n(Y_{i_t,j_t} - Z^{12}_{i_t,j_t})^2 + {\mu \langle I,\, Z\rangle }, \\ \text {subject to }&~ \Vert Z^{12}\Vert _\infty \le \alpha , \ \Vert \mathrm{diag}(Z)\Vert _\infty \le R, \ \ Z\succeq 0. \end{aligned} \end{aligned}$$

(7.2)

Let the loss function be

$$\begin{aligned} \mathcal {L}(Z) = \frac{1}{2} \sum _{t=1}^n \big ( Y_{i_t,j_t} - Z^{12}_{i_t,j_t} \big )^2 + \mu \langle I, Z\rangle . \end{aligned}$$

We define the set

$$\begin{aligned} \mathcal {P}= \{Z\in {\mathcal {S}}^d: \mathrm{diag}(Z)\ge 0, \Vert Z^{11}\Vert _{\infty } \le R, \Vert Z^{22}\Vert _\infty \le R, \Vert Z^{12}\Vert _\infty <\alpha \}. \end{aligned}$$

Thus, we have an equivalent formulation of (7.2) below, which is more conducive for computation:

$$\begin{aligned} \min _{X,Z}\mathcal {L}(Z) + \mu \langle X, I\rangle , \text { subject to }X\succeq 0, \ Z\in \mathcal {P}, X-Z=0. \end{aligned}$$

(7.3)

We consider the augmented Lagrangian function of (7.3) defined by

$$\begin{aligned} L(X,Z;W) =\mathcal {L}(Z) + \langle W,X-Z\rangle + \frac{\rho }{2} \Vert X-Z\Vert _F^2, \ X\in {\mathcal {S}}^d_+, \ Z\in \mathcal {P}, \end{aligned}$$

where W is the dual variable. Then, it is natural to apply the ADMM to solve the problem (7.3). At the t-th iteration, we update (X, Z; W) by

$$\begin{aligned} \begin{aligned} X^{t+1}&= \mathop {\mathrm {argmin}}_{X\in {\mathcal {S}}^d_+} L(X,Z^t;W^t) = \Pi _{{\mathcal {S}}_+^{d}}\big \{Z^t -\rho ^{-1}{(W^t+\mu I)}\big \},\\ Z^{t+1}&= \mathop {\mathrm {argmin}}_{Z\in \mathcal {P}} L(X^{t+1},Z;W^t) = \mathop {\mathrm {argmin}}_{Z\in \mathcal {P}} \mathcal {L}(Z) +\frac{\rho }{2} \Vert Z - X^{t+1}- \rho ^{-1}W^t\Vert _F^2,\\ W^{t+1}&= W^{t} + {\tau } \rho (X^{t+1}-Z^{t+1}), \end{aligned} \end{aligned}$$

(7.4)

The next proposition provides a closed-form solution for the Z-subproblem in (7.4).

Proposition 7.1

Denote the observed set of indices of $M^0$ by $\Omega = \{(i_t,j_t)\}_{t=1}^n$. For a given matrix $C\in \mathbb {R}^{d\times d}$, we have

(7.5)

where

and $\Pi _{[a,b]}(x) = \min \{b,\max (a,x) \}$ projects $x\in \mathbb {R}$ to the interval [a, b].

We summarize the algorithm for solving the problem (7.2) below.