Riemannian optimization on unit sphere with p-norm and its applications

Abstract

This study deals with Riemannian optimization on the unit sphere in terms of p-norm with general $p> 1$. As a Riemannian submanifold of the Euclidean space, the geometry of the sphere with p-norm is investigated, and several geometric tools used for Riemannian optimization, such as retractions and vector transports, are proposed and analyzed. Applications to Riemannian optimization on the sphere with nonnegative constraints and $\textit{L}_{\textit{p}}$-regularization-related optimization are also discussed. As practical examples, the former includes nonnegative principal component analysis, and the latter is closely related to the Lasso regression and box-constrained problems. Numerical experiments verify that Riemannian optimization on the sphere with p-norm has substantial potential for such applications, and the proposed framework provides a theoretical basis for such optimization.

Appendix A: Proof of Proposition 10

In this section, we provide a complete proof of Proposition 10 for self-containedness.

Proof of Proposition 10

First, we fix $\lambda \ge 0$ and let $w_*$ be an optimal solution to Problem (46). Then, for any $w \in {\mathbb {R}}^n$, we have

$$\begin{aligned} L(w_*) + \lambda \Vert w_*\Vert _p \le L(w) + \lambda \Vert w\Vert _p. \end{aligned}$$

(A1)

We show that $w_*$ is an optimal solution to Problem (47) with $C :=\Vert w_*\Vert _p$. For any feasible solution $w \in {\mathbb {R}}^n$ to Problem (47), we have $\Vert w\Vert _p \le C = \Vert w_*\Vert _p$. Combining this and (A1), we have $L(w_*) + \lambda \Vert w_*\Vert _p \le L(w) + \lambda \Vert w_*\Vert _p$, which means $L(w_*) \le L(w)$. Furthermore, $w_*$ is clearly a feasible solution to Problem (47) since $\Vert w_*\Vert _p = C$. Therefore, $w_*$ is an optimal solution to Problem (47).

Conversely, we fix $C \ge 0$ and let $w_*$ be an optimal solution to Problem (47). Here, we additionally consider the Lagrange dual problem of (47):

$$\begin{aligned}&\text{ maximize }\qquad \inf _{w \in {\mathbb {R}}^n} (L(w) + \mu (\Vert w\Vert _p - C))\nonumber \\ {}&\text{ subject } \text{ to }\qquad \mu \ge 0, \; \mu \in {\mathbb {R}}. \end{aligned}$$

(A2)

If $C > 0$, then Slater’s condition for Problem (47), which is that there exists $w \in {\mathbb {R}}^n$ with $\Vert w\Vert _p < C$, clearly holds with $w = 0$. If $C = 0$, then the constraint $\Vert w\Vert _p \le C$ in Problem (47) is rewritten as the equality constraint $w = 0$, and Slater’s condition (which in this case is that a feasible solution exists) holds by taking $w = 0$. In each case, Slater’s condition for Problem (47) holds. Furthermore, Problem (47) is a convex optimization problem. Therefore, it follows from Slater’s theorem [5, Section 5.2.3] that strong duality holds. Hence, the optimal value $L(w_*)$ of Problem (47) and the optimal value of the dual problem (A2) coincide. Letting $\mu _* \ge 0$ be an optimal solution to (A2), we have

$$\begin{aligned} L(w_*) = \inf _{w \in {\mathbb {R}}^n}(L(w) + \mu _*(\Vert w\Vert _p - C)). \end{aligned}$$

Since $\Vert w_*\Vert _p \le C$ and $\mu _* \ge 0$, we obtain $L(w_*) \le L(w_*) + \mu _*(\Vert w_*\Vert _p - C) \le L(w_*)$. Thus, $L(w_*) = L(w_*) + \mu _*(\Vert w_*\Vert _p - C)$ holds, and $w = w_*$ attains the minimum value of $L(w) + \mu _*(\Vert w\Vert _p - C)$ over all $w \in {\mathbb {R}}^n$. Since $\mu _* C$ is a constant, $w = w_*$ also attains the minimum value of $L(w) + \mu _*\Vert w\Vert _p$ over ${\mathbb {R}}^n$. This implies that $w_*$ is an optimal solution to Problem (46) with $\lambda = \mu _*$, thereby completing the proof. $\square$

Remark 8

Although we focus on the p-norm here, Proposition 10 can be further straightforwardly generalized to the case with a general norm in ${\mathbb {R}}^n$. Indeed, in the proof of Proposition 10, we do not need any specific property of the p-norm but properties of a general norm.