A Bimodal Co-sparse Analysis Model for Image Processing

Abstract

The success of many computer vision tasks lies in the ability to exploit the interdependency between different image modalities such as intensity and depth. Fusing corresponding information can be achieved on several levels, and one promising approach is the integration at a low level. Moreover, sparse signal models have successfully been used in many vision applications. Within this area of research, the so-called co-sparse analysis model has attracted considerably less attention than its well-known counterpart, the sparse synthesis model, although it has been proven to be very useful in various image processing applications. In this paper, we propose a bimodal co-sparse analysis model that is able to capture the interdependency of two image modalities. It is based on the assumption that a pair of analysis operators exists, so that the co-supports of the corresponding bimodal image structures have a large overlap. We propose an algorithm that is able to learn such a coupled pair of operators from registered and noise-free training data. Furthermore, we explain how this model can be applied to solve linear inverse problems in image processing and how it can be used as a prior in bimodal image registration tasks. This paper extends the work of some of the authors by two major contributions. Firstly, a modification of the learning process is proposed that a priori guarantees unit norm and zero-mean of the rows of the operator. This accounts for the intuition that local texture carries the most important information in image modalities independent of brightness and contrast. Secondly, the model is used in a novel bimodal image registration algorithm, which estimates the transformation parameters of unregistered images of different modalities.

Appendix I

1.1 Derivation of the Riemannian Gradient in Sect. 5

In this section we derive the Riemannian gradient in Eq. (53) for the bimodal alignment algorithm. We make use of the following criterion for its derivation. Let $\langle \cdot ,\cdot \rangle _\mathbf {P}$ be the Riemannian metric on the Lie group $\mathcal {G}$ inherited from (52) and let $F(\cdot )$ be a smooth real valued function on $\mathcal {G}$. Then the Riemannian gradient of $F$ at $\delta \in \mathcal {G}$ is the unique vector $\mathbf {G} \in T_\delta \mathcal {G}$, the tangent space at $\delta$, such that

$$\begin{aligned} \frac{\mathrm{d} }{\mathrm{d} t } \Big |_{t=0} F(e^{t \mathbf {H}}\delta ) = \langle \mathbf {H},\mathbf {G} \rangle _\mathbf {P} \end{aligned}$$

(56)

holds for all tangent elements $\mathbf {H} \in T_\delta \mathcal {G}$.

For our purpose, we compute the gradient at $\delta = \mathrm{id}$. Now let $B$ be the image region in which we want to align the modalities $I_U$ and $I_V$. We assume that $B$ is rectangular and denote by

$$\begin{aligned} I(\mathbf {x})_{\mathbf {x} \in B} \end{aligned}$$

(57)

the vectorized version of $I$ over the domain $B$. Using Eq. (51) and the fact that $\mathbf {c}:= \mathbf {{\varvec{\Omega }}}^F_U I_U$ is a constant vector, we compute by the chain rule that

$$\begin{aligned}&\tfrac{\mathrm{d} }{\mathrm{d} t } \Big |_{t=0} F(e^{t \mathbf {H}}\tau ) = \tfrac{\mathrm{d}}{\mathrm{d} t } \Big |_{t=0} g\left( \mathbf {c}, \mathbf {{\varvec{\Omega }}}^F_V \left[ (e^{t \mathbf {H}}\tau ) \circ I_V \right] \right) \nonumber \\&\quad = \nabla g\left( \mathbf {c}, \mathbf {{\varvec{\Omega }}}^F_V I_V (\tau \mathbf {x})_{\mathbf {x} \in B} \right) ^\top \mathbf {{\varvec{\Omega }}}^F_V \left[ \tfrac{\mathrm{d} }{\mathrm{d} t } \Big |_{t=0} I_V (e^{t \mathbf {H}} \tau \mathbf {x})_{\mathbf {x} \in B}\right] . \end{aligned}$$

(58)

The last bracket is a vector where each of its entries is computed as

$$\begin{aligned} \tfrac{\mathrm{d} }{\mathrm{d} t } \Big |_{t=0} I_V (e^{t \mathbf {H}}\tau \mathbf {x})&= \nabla I_V(\tau \mathbf {x})^\top \mathbf {H} \tau \mathbf {x} \nonumber \\&= \mathrm{vec} (\tau \mathbf {x} \otimes \nabla I_V(\tau \mathbf {x}))^\top \mathrm{vec}(\mathbf {H}), \end{aligned}$$

(59)

where, as usual, $\mathrm{vec}(\cdot )$ denotes the linear operator that stacks the columns of a matrix among each other and $\otimes$ is the Kronecker product. Note, that since we stick to the representation with homogeneous coordinates, $\nabla I_V(\mathbf {x}) \in \mathbb {R}^3$ is the common image gradient of $I_V$ with an additional $0$ in the third component.

Thus, with

$$\begin{aligned}&\mathbf {r}^\top \nonumber \\&\quad :=\nabla g\left( \mathbf {c}, \mathbf {{\varvec{\Omega }}}_V^F I_V (\tau \mathbf {x})_{\mathbf {x} \in B} \right) ^\top \mathbf {{\varvec{\Omega }}}_V^F \left( \mathrm{vec} (\tau \mathbf {x} \otimes \nabla I_V(\tau \mathbf {x}))^\top \right) _{\mathbf {x} \in B}, \end{aligned}$$

(60)

we have

$$\begin{aligned} \frac{\mathrm{d} }{\mathrm{d} t } \Big |_{t=0} F(e^{t \mathbf {H}}\delta ) = \mathbf {r}^\top \mathrm{vec}(\mathbf {H})&= {{\mathrm{tr}}}(\mathrm{vec}^{-1}(\mathbf {r}) \mathbf {H}^\top ) \nonumber \\&= \langle \mathrm{vec}^{-1}(\mathbf {r})\odot \hat{\mathbf {P}} , \mathbf {H} \rangle _{\mathbf {P}}, \end{aligned}$$

(61)

where the entries of $\hat{\mathbf {P}}$ are the inverse of the entries of $\mathbf {P}$.

Using Eq. (56), the Riemannian gradient is therefore the orthogonal projection of $\mathrm{vec}^{-1}(\mathbf {r})\odot \hat{\mathbf {P}}$ with respect to $\langle \cdot , \cdot \rangle _{\mathbf {P}}$ onto the tangent space of $\delta = \mathrm{id}$, which is nothing else than the Lie algebra $\mathfrak {g}$, i.e.

$$\begin{aligned} \mathrm{grad}_\delta F(\delta \circ \tau ) = \Pi _{\mathfrak g}\left( \mathrm{vec}^{-1}(\mathbf {r})\odot \hat{\mathbf {P}} \right) . \end{aligned}$$

(62)

If we further assume for the entries $p_{ij}$ of $\mathbf {P}$ that

$$\begin{aligned} p_{11}=p_{22} \text { and } p_{12}=p_{21}, \end{aligned}$$

(63)

then, for the considered Lie groups, these projections are explicitly given by

$$\begin{aligned} \Pi _{SO}(\mathbf {X})&= \begin{bmatrix} \tfrac{1}{2}(\mathbf {X}_{11}-\mathbf {X}_{11}^\top )&0 \\ 0&0 \end{bmatrix} \end{aligned}$$

(64)

$$\begin{aligned} \Pi _{SE}(\mathbf {X})&= \begin{bmatrix} \tfrac{1}{2}(\mathbf {X}_{11}-\mathbf {X}_{11}^\top )&\mathbf {x}_{12} \\ 0&0 \end{bmatrix} \end{aligned}$$

(65)

$$\begin{aligned} \Pi _{SA}(\mathbf {X})&= \begin{bmatrix} (\mathbf {X}_{11}-\tfrac{1}{2}{{\mathrm{tr}}}(\mathbf {X}_{11}) \mathbf {I}_2)&\mathbf {x}_{12} \\ 0&0 \end{bmatrix}\end{aligned}$$

(66)

$$\begin{aligned} \Pi _{A}(\mathbf {X})&= \begin{bmatrix} \mathbf {X}_{11}&\mathbf {x}_{12} \\ 0&0 \end{bmatrix}, \end{aligned}$$

(67)

where $\mathbf {X} \in \mathbb {R}^{3 \times 3}$ is partitioned as

$$\begin{aligned} \mathbf {X} = \begin{bmatrix} \mathbf {X}_{11}&\mathbf {x}_{12} \\ \mathbf {x}_{21}^\top&x_{22} \end{bmatrix}. \end{aligned}$$

(68)