Rationalizing Efficient Compositional Image Alignment

Abstract

We study the issue of computational efficiency for Gauss-Newton (GN) non-linear least-squares optimization in the context of image alignment. We introduce the Constant Jacobian Gauss-Newton (CJGN) optimization, a GN scheme with constant Jacobian and Hessian matrices, and the equivalence and independence conditions as the necessary requirements that any function of residuals must satisfy to be optimized with this efficient approach. We prove that the Inverse Compositional (IC) image alignment algorithm is an instance of a CJGN scheme and formally derive the compositional and extended brightness constancy assumptions as the necessary requirements that must be satisfied by any image alignment problem so it can be solved with an efficient compositional scheme. Moreover, in contradiction with previous results, we also prove that the forward and inverse compositional algorithms are not equivalent. They are equivalent, however, when the extended brightness constancy assumption is satisfied. To analyze the impact of the satisfaction of these requirements we introduce a new image alignment evaluation framework and the concepts of short- and wide-baseline Jacobian. In wide-baseline Jacobian problems the optimization will diverge if the requirements are not satisfied. However, with a good initialization, a short-baseline Jacobian problem may converge even if the requirements are not satisfied.

Appendices

Appendix 1: Derivative of Inverse Warps

Let $f({\varvec{x}},\varvec{\phi })$ be a warp function and $f^{-1}({\varvec{x}}, \varvec{\phi })$ its inverse, such that $f(f^{-1}({\varvec{x}}, \varvec{\phi }), \varvec{\phi }) = {\varvec{x}}$, where $\varvec{\phi }$ is a small disturbance around the identity warp, $\varvec{\phi }_0$. The derivative of this expression with respect to $\varvec{\phi }$ is

$$\begin{aligned} \left. \dfrac{\partial f(f^{-1}({\varvec{x}}, \varvec{\phi }), \varvec{\phi })}{\partial \varvec{\phi }} \right| _{\varvec{\phi }= \varvec{\phi }_0} = \dfrac{\partial {\varvec{x}}}{\partial \varvec{\phi }} = \mathbf 0, \end{aligned}$$

that can be expanded using the chain rule:

$$\begin{aligned}&\left. \dfrac{\partial f({\varvec{x}},\varvec{\phi })}{\partial \varvec{\phi }}\right| _{\varvec{\phi }=\varvec{\phi }_0} + \left. \dfrac{\partial f({\varvec{x}}',\varvec{\phi }_0)}{\partial {\varvec{x}}'}\right| _{{\varvec{x}}'={\varvec{x}}}\cdot \left. \dfrac{\partial f^{-1}({\varvec{x}}, \varvec{\phi })}{\partial \varvec{\phi }} \right| _{\varvec{\phi }=\varvec{\phi }_0}\\&= \mathbf 0. \end{aligned}$$

As $f({\varvec{x}}, \varvec{\phi }_0)$ is the identity warp,

$$\begin{aligned} \left. \dfrac{\partial f({\varvec{x}},\varvec{\phi })}{\partial \varvec{\phi }}\right| _{\varvec{\phi }=\varvec{\phi }_0} + \left. \dfrac{\partial f^{-1}({\varvec{x}}, \varvec{\phi })}{\partial \varvec{\phi }} \right| _{\varvec{\phi }=\varvec{\phi }_0} = \mathbf 0. \end{aligned}$$

Finally,

$$\begin{aligned} \left. \dfrac{\partial f({\varvec{x}},\varvec{\phi })}{\partial \varvec{\phi }}\right| _{\varvec{\phi }=\varvec{\phi }_0} = - \left. \dfrac{\partial f^{-1}({\varvec{x}}, \varvec{\phi })}{\partial \varvec{\phi }} \right| _{\varvec{\phi }=\varvec{\phi }_0}. \end{aligned}$$

Appendix 2: In-plane Translation

We consider the case of a plane $\varvec{\pi }$ that moves perpendicular to its normal $\mathbf {n}$ at a distance $d$ from the origin. The set of points of the plane are $\mathcal {V} = \{{\varvec{x}}\in \mathbb {R}^3 : \mathbf {n}^\top {\varvec{x}}+d=0\}$, that is a two-dimensional surface embedded in $\mathbb {R}^3$, and therefore, it is a closed set. The support set is a finite subset of $\mathcal {V}, \mathcal {X}\subset \mathcal {V}$.

The in-plane translation has two degrees of freedom. Thus, the pair of warps $\mathbf{f}$ and $\mathbf{g}$ are parametrized respectively by ${\varvec{\mu }}\in \mathbb {R}^3$ and $\varDelta {\varvec{\phi }}\in \mathbb {R}^2$. The two warps are $\mathbf{f}({\varvec{x}},{\varvec{\mu }})={\varvec{x}}+ {\varvec{\mu }}$ and $\mathbf{g}({\varvec{x}},\varDelta {\varvec{\phi }}) = {\varvec{x}}+ [{\varvec{u}}\ {\varvec{v}}]\cdot \varDelta {\varvec{\phi }}$, where ${\varvec{u}},{\varvec{v}}\in \mathbb {R}^3$ are two independent vectors perpendicular to $\mathbf {n}$.

We will prove that this system satisfies both the CA and the EBCA. The CA states that, for any ${\varvec{\mu }}$ and $\varDelta {\varvec{\phi }}$, there exists a ${\varvec{\mu }}'$ such that $\mathbf{f}({\varvec{x}},{\varvec{\mu }}') = \mathbf{f}(\mathbf{g}({\varvec{x}},\varDelta {\varvec{\phi }}),{\varvec{\mu }})$. This is trivially proved taking ${\varvec{\mu }}' = {\varvec{\mu }}+ [{\varvec{u}}\ {\varvec{v}}]\cdot \varDelta {\varvec{\phi }}$. The identity $\mathbf{g}$-warp is obtained for $\varDelta {\varvec{\phi }}_0=[0\ 0]^T$.

To prove the EBCA, we write the expression for Requirement 2

$$\begin{aligned}&\left. \dfrac{\partial I[{\varvec{x}}, t]}{\partial {\varvec{x}}} \right| _{{\varvec{x}}=f(\mathcal {X}, {\varvec{\mu }})} \cdot \left. \dfrac{\partial \mathbf{f}({\varvec{x}},{\varvec{\mu }})}{\partial {\varvec{x}}}\right| _{{\varvec{x}}=\mathcal {X}} \cdot \left. \dfrac{\partial \mathbf{g}(\mathcal {X},\varvec{\phi })}{\partial \varvec{\phi }}\right| _{\varvec{\phi }=\varvec{\phi }_0}\nonumber \\&= \left. \dfrac{\partial T[{\varvec{x}}]}{\partial {\varvec{x}}} \right| _{{\varvec{x}}=\mathcal {X}} \cdot \left. \dfrac{\partial \mathbf{g}(\mathcal {X},\varvec{\phi })}{\partial \varvec{\phi }}\right| _{\varvec{\phi }=\varvec{\phi }_0}. \end{aligned}$$

(22)

The derivative of the $\mathbf{f}$-warp with respect to ${\varvec{x}}$ is the $3\times 3$ identity. Also, the derivative of the $\mathbf{g}$-warp with respect to $\varDelta {\varvec{\phi }}$ is $[{\varvec{u}}\ {\varvec{v}}]$. Therefore, (22) becomes

$$\begin{aligned} \left. \dfrac{\partial I[{\varvec{x}}, t]}{\partial {\varvec{x}}} \right| _{{\varvec{x}}=f(\mathcal {X}, {\varvec{\mu }})} \cdot [{\varvec{u}}\ {\varvec{v}}] = \left. \dfrac{\partial T[{\varvec{x}}]}{\partial {\varvec{x}}} \right| _{{\varvec{x}}=\mathcal {X}} \cdot [{\varvec{u}}\ {\varvec{v}}]. \end{aligned}$$

(23)

We do not know $I$ nor $T$ but, thanks to the brightness constancy assumption, we know a relation between them $I[f({\varvec{x}},{\varvec{\mu }}),t] = T[{\varvec{x}}],\ \forall {\varvec{x}}\in \mathcal {V}$. The partial derivatives of two functions that are equal in a closed subset $\mathcal {V}$ of their domain are not, in general, equal in that subset. However, the partial derivatives projected onto $\mathcal {V}$ are equal. Thus, given a projection matrix $\mathbf {\Pi }$ onto the plane $\mathcal {V}$ we have that:

$$\begin{aligned} \left. \dfrac{\partial I[{\varvec{x}}, t]}{\partial {\varvec{x}}} \right| _{{\varvec{x}}=f(\mathcal {X}, {\varvec{\mu }}_t)} \cdot \mathbf {\Pi } = \left. \dfrac{\partial T[{\varvec{x}}]}{\partial {\varvec{x}}} \right| _{{\varvec{x}}=\mathcal {X}} \cdot \mathbf {\Pi }. \end{aligned}$$

Since we can choose $\mathbf {\Pi }=[{\varvec{u}}\ {\varvec{v}}]$, expression (23) is true. This proves the EBCA.