An inexact Riemannian proximal gradient method

Abstract

This paper considers the problem of minimizing the summation of a differentiable function and a nonsmooth function on a Riemannian manifold. In recent years, proximal gradient method and its variants have been generalized to the Riemannian setting for solving such problems. Different approaches to generalize the proximal mapping to the Riemannian setting lead different versions of Riemannian proximal gradient methods. However, their convergence analyses all rely on solving their Riemannian proximal mapping exactly, which is either too expensive or impracticable. In this paper, we study the convergence of an inexact Riemannian proximal gradient method. It is proven that if the proximal mapping is solved sufficiently accurately, then the global convergence and local convergence rate based on the Riemannian Kurdyka–Łojasiewicz property can be guaranteed. Moreover, practical conditions on the accuracy for solving the Riemannian proximal mapping are provided. As a byproduct, the proximal gradient method on the Stiefel manifold proposed in Chen et al. [SIAM J Optim 30(1):210–239, 2020] can be viewed as the inexact Riemannian proximal gradient method provided the proximal mapping is solved to certain accuracy. Finally, numerical experiments on sparse principal component analysis are conducted to test the proposed practical conditions.

Appendix: Implementations of \(B_x^T\) and \(B_x\)

In this section, the implementations of the functions $B_x^T: \mathbb {R}^n \rightarrow \mathbb {R}^{n - d}$ and $B_x: \mathbb {R}^{n - d} \rightarrow \mathbb {R}^n$ are given for Grassmann manifold, manifold of fixed-rank matrices, manifold of symmetric positive definite matrices, and products of manifolds. Note that the Riemannian metric is chosen to be the Euclidean metric in this section.

Grassmann manifold We consider the representation of Grassmann manifold by

$$\begin{aligned} {{\,\mathrm{\mathrm {Gr}}\,}}(p, n) = \{[X] : X \in {{\,\mathrm{\mathrm {St}}\,}}(p, n)\}, \end{aligned}$$

where $[X] = \{X O : O^T O = I_p\}$. The ambient space of ${{\,\mathrm{\mathrm {Gr}}\,}}(p, n)$ is $\mathbb {R}^{n \times p}$ and the orthogonal complement space of the horizontal space $\mathcal {H}_X$ at $X \in {{\,\mathrm{\mathrm {St}}\,}}(p, n)$ is given by

$$\begin{aligned} \mathcal {H}_X^{\perp } = \{X M : M \in \mathbb {R}^{p \times p} \}. \end{aligned}$$

Therefore, we have

$$\begin{aligned}&B_X^T: \mathbb {R}^{n \times p} \rightarrow \mathbb {R}^{p \times p}: Z \rightarrow X^T Z,\quad \hbox {and} \\&B_X: \mathbb {R}^{p \times p} \rightarrow \mathbb {R}^{n \times p}: M \rightarrow X M. \end{aligned}$$

Manifold of fixed-rank matrices The manifold is given by

$$\begin{aligned} \mathbb {R}_r^{m \times n} = \{X \in \mathbb {R}^{m \times n} : \mathrm {rank}(X) = r\}. \end{aligned}$$

The ambient space is therefore $\mathbb {R}^{m \times n}$. Given $X \in \mathbb {R}_r^{m \times n}$, let $X = U S V$ be a thin singular value decomposition. The normal space at X is given by

$$\begin{aligned} {{\,\mathrm{\mathrm {N}}\,}}_X \mathbb {R}_r^{m \times n} = \{U_{\perp } M V_{\perp }^T : M \in \mathbb {R}^{(m - r) \times (n - r)} \}, \end{aligned}$$

where $U_{\perp } \in \mathbb {R}^{m \times (m - r)}$ forms an orthonormal basis of the perpendicular space of $\mathrm {span}(U)$ and $V_{\perp } \in \mathbb {R}^{n \times (n - r)}$ forms an orthonormal basis of the perpendicular space of $\mathrm {span}(V)$. It follows that

$$\begin{aligned}&B_X^T: \mathbb {R}^{m \times n} \rightarrow \mathbb {R}^{(m - r) \times (n - r)}: Z \mapsto U_{\perp }^T Z V_{\perp },\quad \hbox {and} \\&B_X: \mathbb {R}^{(m - r) \times (n - r)} \rightarrow \mathbb {R}^{m \times n}: M \mapsto U_{\perp } M V_{\perp }^T. \end{aligned}$$

Note that it is not necessary to form the matrices $U_{\perp }$ and $V_{\perp }$. One can use [53, Algorithms 4 and 5] to implement the actions of $U_{\perp }$, $U_{\perp }^T$, $V_{\perp }$, and $V_{\perp }^T$.

Manifold of symmetric positive semi-definite matrices The manifold is

$$\begin{aligned} \mathbb {S}_r^{n \times n} = \{X \in \mathbb {R}^{n \times n} : X = X^T, X \succeq 0, \mathrm {rank}(X) = r \}. \end{aligned}$$

The ambient space is $\mathbb {R}^{n \times n}$. Given $X \in \mathbb {S}_r^{n \times n}$, let $X = H H^T$, where $H \in \mathbb {R}^{n \times r}$ is full rank. The normal space at X is

$$\begin{aligned} {{\,\mathrm{\mathrm {N}}\,}}_X \mathbb {S}_r^{n \times n} = \{H_{\perp } M H_{\perp }^T: M \in \mathbb {R}^{(n - r) \times (n - r)}, M = M^T \}, \end{aligned}$$

where $H_{\perp } \in \mathbb {R}^{n \times (n - r)}$ forms an orthonormal basis of the perpendicular space of $\mathrm {span}(H)$. Therefore, we have

$$\begin{aligned}&B_X^T: \mathbb {R}^{n \times n} \rightarrow \mathbb {R}^{\frac{(n - r) (n - r + 1)}{2}}: Z \mapsto \mathrm {vec}\left( \frac{1}{2} H_{\perp }^T (Z + Z^T) H_{\perp }\right) ,\quad \hbox {and} \\&B_X: \mathbb {R}^{\frac{(n - r) (n - r + 1)}{2}} \rightarrow \mathbb {R}^{n \times n}: v \mapsto H_{\perp } \mathrm {vec}^{-1}(v) H_{\perp }^T, \end{aligned}$$

where $\mathrm {vec}(M) = (M_{11}, M_{22}, \ldots , M_{ss}, \sqrt{2} M_{12}, \sqrt{2} M_{13}, \sqrt{2} M_{1s}, \ldots , \sqrt{2} M_{(s - 1) s})^T$ for $M \in \mathbb {R}^{s \times s}$ being a symmetric matrix, and $\mathrm {vec}^{-1}$ is the inverse function of $\mathrm {vec}$.

Product of manifolds Let the product manifold $\mathcal {M}$ be denoted by $\mathcal {M}_1 \times \mathcal {M}_2 \times \ldots \times \mathcal {M}_t$. Let the ambient space of $\mathcal {M}_i$ be $\mathbb {R}^{n_i}$ and the dimension of $\mathcal {M}_i$ be $d_i$. For any $X = (X_1, X_2, \ldots , X_t) \in \mathcal {M}$, the mappings $B_X^T$ and $B_X$ are given by

$$\begin{aligned} B_X^T&: \mathbb {R}^{n_1} \times \mathbb {R}^{n_2} \times \ldots \times \mathbb {R}^{n_t} \rightarrow \mathbb {R}^{(n_1 - d_1 + n_2 - d_2 + \ldots + n_t - d_t)} \\&: (Z_1, Z_2, \ldots , Z_t) \mapsto \left( (B_{X_1}^T Z_1)^T, (B_{X_2}^T Z_2)^T, \ldots , (B_{X_t}^T Z_t)^T \right) ^T, \hbox { and } \\ B_X&: \mathbb {R}^{(n_1 - d_1 + n_2 - d_2 + \ldots + n_t - d_t)} \rightarrow \mathbb {R}^{n_1} \times \mathbb {R}^{n_2} \times \ldots \times \mathbb {R}^{n_t} \\&: (v_1^T, v_2^T, \ldots , v_t^T)^T \mapsto (B_{X_1} v_1, B_{X_2} v_2, \ldots , B_{X_t} v_t), \end{aligned}$$

where $B_{X_i}^T$ and $B_{X_i}$ denote the mappings for manifold $\mathcal {M}_i$ at $X_i$, and $v_i \in \mathbb {R}^{n_i - d_i}$, $i = 1, \ldots , t$.