Image Segmentation with Superpixel Based Covariance Descriptor

Abstract

This paper investigates the problem of image segmentation using superpixels. We propose two approaches to enhance the discriminative ability of the superpixel’s covariance descriptors. In the first one, we employ the Log-Euclidean distance as the metric on the covariance manifolds, and then use the RBF kernel to measure the similarities between covariance descriptors. The second method is focused on extracting the subspace structure of the set of covariance descriptors by extending a low rank representation algorithm on to the covariance manifolds. Experiments are carried out with the Berkly Segmentation Dataset, and compared with the state-of-the-art segmentation algorithms, both methods are competitive.

Appendix: Solution of Eq. (4)

The solution of Eq. (4) is partly refer to the work of Wang et al. [24], but the distance induced by Frobenius norm is not geodesic. The problem is rephrased as follows.

Find a matrix Z that satisfied,

$$\begin{aligned} {\begin{matrix} &{}\min _{E,Z}{\Vert E\Vert _F^2 + \lambda \Vert Z\Vert _{*}},\\ &{}s.t. \quad \mathcal {X} = \mathcal {X}_{\times _3}Z+E \end{matrix}} \end{aligned}$$

where, $\mathcal {X}$ is a 3-order tensor stacking by covariance matrices $(X_i)_{d\times d}$, $i=1,2,...,n$; $\Vert \cdot \Vert _F$ is the Frobenius norm; $\Vert \cdot \Vert _{*}$ is the nuclear norm; $\lambda$ is the balance parameter; $\times _3$ means mode-3 multiplication of a tensor and matrix [15].

For the error term E, we have $\Vert E\Vert _F^2 = \Vert \mathcal {X}-\mathcal {X}_{\times 3}Z\Vert _{F}^2$, and we can rewrite $\Vert E\Vert _F^2$ as,

$$\begin{aligned} \Vert E\Vert _F^2 = \sum _{i}^{N}\Vert E_i\Vert _F^2, \end{aligned}$$

where, $E_i = X_i - \sum _j^Nz_{ij}X_j$, i.e. the i-th slice of E.

Note that for matrix A, it holds $\Vert A\Vert _{F}^2 = tr(A^TA)$, and $X_i$ is symmetric, so, the above equation can be expanded as,

$$\begin{aligned} {\begin{matrix} &{}\Vert E_i\Vert _F^2 = tr[(X_i - \sum _j^Nz_{ij}X_j)^T(X_i-\sum _j^Nz_{ij}X_j)]\\ &{}=tr(X_i^TX_i) - tr(X_i^T\sum _j^Nz_{ij}X_j) -tr(\sum _j^Nz_{ij}X_j^TX_i)\\ &{}+tr(\sum _{j_1}^Nz_{ij_1}X_{j_1}^T\sum _{j_2}^Nz_{ij_2}X_{j_2})\\ &{}=tr(X_iX_i) - 2tr(\sum _{j}^Nz_{ij}X_iX_j) + tr(\sum _{j_1,j_2}^Nz_{ij_1}z_{ij_2}X_{j_1}X_{j_2}). \end{matrix}} \end{aligned}$$

Let $\varDelta$ be a symmetric matrix of size $N\times N$, whose entries are $\varDelta _{ij}=\varDelta _{ji}=tr(X_iX_j)$. Because $X_i$ is a symmetric matrix, $\varDelta _{ij}$ can be written as $\varDelta _{ij}=vec(X_i)^Tvec(X_j)$, where $vec(\cdot )$ is an operator that vectorized a matrix. As a Gram matrix, $\varDelta$ is positive semidefinite. So, we have,

$$\begin{aligned} {\begin{matrix} \Vert E_i\Vert _F^2 &{}= \varDelta _{ii} - 2\sum _{j=1}^{N}z_{ij}\varDelta _{ij} + \sum _{j_1}^{N}\sum _{j_2}^{N}z_{ij_1}z_{ij_2}\varDelta _{j_1j_2}\\ &{}= \varDelta {ii} - 2\sum _{j=1}^{N}z_{ij}\varDelta _{ij} + \mathbf z _i\varDelta \mathbf z _i^T. \end{matrix}} \end{aligned}$$

For $\varDelta = PP^T$,

$$\begin{aligned} {\begin{matrix} \Vert E\Vert _F^2 &{}= \sum _{i=1}^{N}\varDelta _{ii} - 2tr[Z\varDelta ] + tr[Z\varDelta Z^T]\\ &{}= C + \Vert ZP-P\Vert _{F}^2, \end{matrix}} \end{aligned}$$

Then, the optimization is equivalent to:

$$\begin{aligned} \min _{Z}\Vert ZP-P\Vert _F^2 +\lambda \Vert Z\Vert _{*}. \end{aligned}$$

Let $\varDelta$ be a symmetric matrix, whose entries are $\varDelta _{ij}=\varDelta _{ji}=tr(X_iX_j)$, and $P = \varDelta ^{\frac{1}{2}}$. First, we transform the above equation into an equivalent formulation

$$\begin{aligned} {\begin{matrix} &{}\min _{Z}\frac{1}{\lambda }\Vert ZP-P\Vert _{F}^2 +\Vert J\Vert _{*},\\ &{}s.t. \qquad J=Z. \end{matrix}} \end{aligned}$$

Then by ALM, we have,

$$\begin{aligned} \min _{Z,J}\frac{1}{\lambda }\Vert ZP-P\Vert _{F}^2 +\Vert J\Vert _{*} + <Y,Z-J>+\frac{\mu }{2}\Vert Z-J\Vert _{F}^2, \end{aligned}$$

where, Y is the Lagrange coefficient, $\lambda$ and $\mu$ are scale parameters.

The above problem can be solved by the following two subproblems [17],

$$\begin{aligned} J_{k+1}=\min _{J}(\Vert J\Vert _{*}+<Y,Z_k-J> +\frac{\mu }{2}\Vert Z_k-J\Vert _F^2) \end{aligned}$$

and,

$$\begin{aligned} Z_{k+1} = \min _{Z}(\frac{1}{\lambda }\Vert ZP-P\Vert _F^2+<Y,Z-J_k>+\frac{\mu }{2}\Vert Z-J\Vert _F^2). \end{aligned}$$

Fortunately according to [3], the solutions for the above subproblems have the following close forms,

$$\begin{aligned} J = \varTheta (Z+\frac{Y}{\mu }), \end{aligned}$$

$$\begin{aligned} Z = (\lambda \mu J -\lambda Y +2\varDelta )(2\varDelta +\lambda \mu I)^{-1}, \end{aligned}$$

where, $\varTheta (\cdot )$ is the singular value thresholding operator [3].

Thus, by iteratively updating J and Z until the converge conditions are satisfied, a solution for Eq. (4) can be found.