Verifying Global Minima for L ₂ Minimization Problems in Multiple View Geometry

Abstract

We consider the least-squares (L2) minimization problems in multiple view geometry for triangulation, homography, camera resectioning and structure-and-motion with known rotation, or known plane. Although optimal algorithms have been given for these problems under an L-infinity cost function, finding optimal least-squares solutions to these problems is difficult, since the cost functions are not convex, and in the worst case may have multiple minima. Iterative methods can be used to find a good solution, but this may be a local minimum. This paper provides a method for verifying whether a local-minimum solution is globally optimal, by providing a simple and rapid test involving the Hessian of the cost function. The basic idea is that by showing that the cost function is convex in a restricted but large enough neighbourhood, a sufficient condition for global optimality is obtained.

The method is tested on numerous problem instances of real data sets. In the vast majority of cases we are able to verify that the solutions are optimal, in particular, for small to medium-scale problems.

Appendices

Appendix A: Minimizing Squared Angle

In the derivations in Sect. 2, the error function used was the sum of squares of the tangents of the angles, as given by (4). This leads to a relatively simple result in terms of computing and bounding the Hessian. On the other hand, it would be more natural to wish to minimize the sum of squares of the error angles, and not their tangents. The difference is very small, but for exactness, we now derive a similar result for this error function. It is worth noting here that although in many computer-vision applications, pixel error is the natural metric for evaluating a solution, this is not clearly true of omnidirectional or wide-angle images.

In a different application, target tracking and localization, the position of a target may be measured from several observation points resulting in angular (alt-azimuth) readings. The task of determining the position of the target is precisely the orientation problem. In this context, it is natural to wish to minimize squared angular error, which is what we do in this appendix.

In the following discussion, we give the outline of the argument. To verify the details, the reader may need to use a computer algebra system, such as Mathematica. Now, under the same conditions as before, we define the error function

$$ f^2(x, y, z) = \phi^2 = \arctan\biggl( \frac{\sqrt{x^2 + y^2}}{z} \biggr)^2 . $$

(25)

We compute the Hessian at a point with (x,y,z) with respect to the coordinates x, y and z. Subsequently, evaluating with x≥0 and y=0, and making the substitutions ϕ for arctan(x/z), radial distance r for $\sqrt{x^{2} + z^{2}}$ and tanϕ for x/z, we arrive after some computation at the expression for the Hessian given in Fig. 3. This matrix has eigenvalues 2ϕ/(r ²tan(ϕ)) and $(1 \pm\sqrt{1 + 4 \phi^{2}} ) / r^{2}$, namely two positive and one negative eigenvalue.

Fig. 3

Hessian for cost function (25)

Similarly as before,¹ let H′ be the matrix $(2 / r^{2}) \operatorname {diag}(1/4,\allowbreak 1/4, -4 \phi^{2})$. We claim that H−H′ is positive semi-definite. Unfortunately, the proof is a little more intricate than before.

First, we observe that (0,1,0)^⊤ is an eigenvector of H−H′, with eigenvalue (2/r ²)(ϕ/tan(ϕ)−1/4), which is positive at least for ϕ<1. The other two eigenvalues are the eigenvalues of the reduced-size matrix obtained by eliminating the second row and column from H−H′. This matrix will have two positive eigenvalues, as long as its trace and determinant are both positive. Apart from the factor 1/r ² the trace is equal to 8ϕ ²+3/2 which is positive. The determinant is equal to

$$D(\phi) = -1 + \bigl(1 + 16 \phi^2\bigr) \bigl(\cos(2 \phi) - 2 \phi \sin(2 \phi)\bigr) . $$

This function is positive for ϕ<0.3 as may be shown by plotting the function D(ϕ)/ϕ ² (see Fig. 4). A more formal proof can be given by computing the series expansion of this function.

Fig. 4

Plot of the function D(ϕ)/ϕ ² demonstrating that the product of the eigenvalues of H−H′ is positive for ϕ<0.3

The result of this computation is the following result.

Lemma 1

If H is the Hessian of the error function (25) evaluated at a point with error less than angle ϕ<0.3, then

$$\mathtt{H} \succeq \bigl(2/r^2\bigr) \operatorname {diag}\bigl(1/4, 1/4, -4 \phi^2\bigr) . $$

From here on, conditions for convexity of the error function in the intersection of a set of cones with angle bounded by ϕ _max proceeds just the same as previously.

Appendix B: Minimum Bounds for Hessians

The matrix H′ given in (7) was presented without motivation. In this section, we will give some theory that justifies the choice. This will be developed in a more general context, to allow for possible applications to other problems. In what follows, all matrices are assumed to be symmetric, even if not stated.

Notation

We need to develop some properties of the relationship H⪰0 which has been used throughout this paper. It means that H is positive semi-definite, possibly zero. The notation H⪰H′ means H−H′⪰0. This defines a partial ordering relationship between matrices. We use the notation H≻0 to mean that H is positive semi-definite, but not zero. It could be interpreted as meaning that H is positive definite, but not semi-definite, but that is not what we mean. If we need to specify that H is strictly positive-definite, this will be specifically stated.

Consider a family of symmetric matrices H _v defined in terms of a parameter vector v. We say that a matrix G is a lower bound for this set if H _v ⪰G for all v, and it is a maximal lower bound if there is no other lower bound G′ such that G′≻G. There may be more than one maximal lower bound for a family of matrices. Clearly if we want to define a lower bound for a set of Hessian matrices, it is advantageous to identify the maximal lower bounds.

We are particularly interested in the lower bounds for matrices of the form

$$ \mathtt{H}_\mathbf {v}= \left [ \arraycolsep=5pt \begin{array}{@{}cc@{}} \mathtt{I}_{n\times n} & -2\mathbf{v} \\ -2\mathbf{v}^{\top}& 3 \tau _{\mathrm{max}}^2 \end{array} \right ] $$

(26)

as in (5) or (20). We may determine the complete set of maximal lower bounds for the set of all matrices H _v where ∥v∥ is bounded.

Theorem 4

Consider the set of matrices

$$ \mathcal{H} = \bigl\{ \mathtt{H}_\mathbf {v}~|~\| \mathbf{v} \|^2 \le\tau_{\mathrm{max}}^2 \bigr\} . $$

(27)

The maximal lower bounds for this set are the matrices of the form $\mathtt{G}_{\lambda}= \operatorname {diag}(1-1/\lambda, 1-1/\lambda, (3-4\lambda) \tau_{\mathrm{max}}^{2})$ with λ>0.

This theorem will follow from a more general theorem, which we will state shortly, in terms of maximal ellipsoids in ℝⁿ.

The matrices of interest are of the form

(28)

where v ranges over a set V of vectors in ℝⁿ, and A ₀ is a symmetric matrix. Compare this with the form of (26).

The following simple observation reduces finding lower bounds of such matrices A ₀−S(v) to finding bounds for the set S(v).

Lemma 2

The (maximal) lower bounds of A ₀−S(v) for all v∈V are exactly those matrices A ₀−G, where G is a (minimal) upper bound for the matrices S(v).

We now consider a connection between matrices and ellipsoids in ℝⁿ that will help us identify a connection between maximal lower bounds for sets of matrices and ellipsoids. Let G be a matrix of the form

$$ \mathtt{G} = \left [ \arraycolsep=5pt \begin{array}{@{}cc@{}} \mathtt{A} & \mathbf{r} \\ \mathbf{r}^{\top} & q \end{array} \right ]\quad\mbox{with } \mathtt{A} \succeq 0 \mbox{ and } q \ge0 $$

(29)

Define also

$$ \mathtt{G}_{\lambda}= \left [ \arraycolsep=5pt \begin{array}{@{}cc@{}} \mathtt{A}/\lambda& \mathbf{r} \\ \mathbf{r}^{\top} & q \lambda \end{array} \right ]\quad\mbox{for } \lambda> 0. $$

(30)

Definition 1

Let G be a matrix of the form (29). We define the ellipsoid C(G) associated with G to be the set

$$C(\mathtt{G}) = \bigl\{ \mathtt{K} \mathbf{w} + \mathbf{r}~|~\| \mathbf{w} \|^2 \le q \bigr\} $$

where K is a real symmetric n×n matrix such that A=K ².

Thus, C(G) is the image of a ball in ℝⁿ under an affine mapping, and hence an ellipsoid, possibly degenerate (lying in a subspace of dimension less than n) if A does not have full rank. It is clear that C(G) and C(G _λ ) define the same ellipsoid for all λ.

Even when A is not of full rank such a matrix K exists, since writing A=UD ² U ^⊤, where D is diagonal and U orthogonal, we can choose K=UDU ^⊤, which is symmetric. It is easy to verify that the set C(G) is independent of the choice of K, depending only on A=K ².

We give another useful characterization of ellipsoids. For a matrix A, define the span of A to be $\operatorname {Sp}(\mathtt{A}) = \{ \mathtt{A} \mathbf{w} \}$ for all vectors w. Further, given a symmetric matrix A with Singular Value Decomposition A=UDU ^⊤, define the pseudo-inverse A ⁺=UD ⁺ U ^⊤, where $\mathtt{D}^{+}_{ii} = \mathtt{D}_{ii}^{-1}$ when D _ii =0 and $\mathtt{D}^{+}_{ij} = 0$ otherwise. Then, for a matrix of the form (29),

$$ C(\mathtt{G}) = \bigl\{ \mathbf{x}~|~( \mathbf{x} - \mathbf{r}) \in \operatorname {Sp}(\mathtt{A}) \mbox{ and } (\mathbf{x} - \mathbf{r})^{\top} \mathtt{A}^+ (\mathbf{x} - \mathbf{r}) \le q \bigr\}. $$

(31)

Our main theorem on lower bounds of matrices can now be stated.

Theorem 5

Let V be a bounded set of points in ℝⁿ. The maximal lower bounds for the set of matrices A ₀−S(v) with v∈V, are the matrices of the form A ₀−G, where G is of the form (29) and C(G) is a minimal ellipsoid containing the set V.

Application

Before giving the proof of this theorem, we show how it may be used to prove Theorem 4 as a special case. Thus, let $\mathtt{A}_{0} = \operatorname {diag}(\mathtt{I}_{n\times n}, 3 \tau_{\mathrm{max}}^{2})$. According to Theorem 5, the maximal lower bounds for the set of matrices $\mathcal{H}$ are those matrices of the form A ₀−G _λ where C(G) is a minimal ellipsoid containing the set of vectors V={2v | ∥v∥≤τ ²}. The minimal ellipsoid containing V is clearly a sphere of radius 2τ, having equation x ^⊤ Ix ^⊤≤4τ ². Therefore, $\mathtt{G}_{\lambda}= \operatorname {diag}(\mathtt{I} / \lambda, 4 \lambda\tau^{2})$. Consequently, the least lower bounds for the set $\mathcal{H}$ are the matrices $\operatorname {diag}(1-1/\lambda, \ldots, 1 - 1/\lambda, (3 - 4\lambda) \tau ^{2})$ with λ>0, which is the conclusion of Theorem 4.

If λ<1, these lower bounds are not very useful, since the eigenvalues 1−1/λ are negative. Our strategy in this paper is to select the value of λ that leads to the largest absolute value of the ratio of the largest to smallest eigenvalues of this matrix. Hence, we seek to maximize (1−1/λ)/(4λ−3)τ ² with λ>1. The maximum occurs when λ=3/2, and the corresponding optimal bound is $\operatorname {diag}(1/3, 1/3, - 3 \tau^{2})$. This is the value that is used in (7).

Proof of Theorem

We now start the proof of Theorem 5, which will be completed through a sequence of lemmas. The connection of ellipsoids with lower bounds of sets of matrices is given by the following lemma.

Lemma 3

G⪰S(v) if and only if G is of the form (29) and v∈C(G). Consequently, G is an upper bound for a set of matrices S(v), v∈V if and only if C(G) contains the set V.

Proof

First, we prove the if part of this lemma. Writing G as in (29), choose K such that K ²=A with K symmetric. If v∈C(G), then we can write v−r=K w with w ^⊤ w≤q. Substituting for v−r in G−S(v) we see

Therefore x ^⊤(G−S(v))x≥0 for any x, and it follows that G−S(v)⪰0.

For the converse, assume that G−S(v)⪰0. Then G can be written as in (29) where A⪰0 and q≥0, since any principal minor of a positive semi-definite matrix must be semi-definite.

If v−r does not lie in the span of the columns of K, then there exists a vector x such that K x=0 and of suitable length such that (v−r)^⊤ x>q. Writing $\hat{\mathbf{x}} = (\mathbf{x}^{\top}, 1)^{\top}$, we see that

$$\hat{\mathbf{x}}^{\top}\bigl(\mathtt{G} - \mathtt{S}(\mathbf{v})\bigr) \hat{\mathbf{x}} = q - ( \mathbf{v} - \mathbf{r})^{\top} \mathbf{x} < 0 , $$

so G−S(v) can not be positive semi-definite.

Consequently, v−r must lie in $\operatorname {Sp}(\mathtt{K})$ and we may write v−r=K w. Select the w of minimum norm that satisfies this equation. This is given by w=K ⁺(v−r). Then

$$\mathbf{w} = \mathtt{K}^+ (\mathbf{v} - \mathbf{r}) = \mathtt{K} \bigl(\mathtt{K}^+ \mathtt{K}^+ ( \mathbf{v} - \mathbf{r})\bigr) = \mathtt{K} \mathbf{x} $$

where x is defined by this formula. Let $\hat{\mathbf{x}} = (\mathbf{x}^{\top}, 1)^{\top}$. Then, and it follows that $\hat{\mathbf{x}}^{\top}(\mathtt{G} - \mathtt{S}(\mathbf{v})) \hat{\mathbf{x}} = q - \mathbf{w}^{\top}\mathbf{w}$. Since by assumption G−S(v)⪰0, it follows that ∥w∥²≤q, and so v=K w+r is in the ellipsoid C(G), and the proof is complete. □

This identifies the upper bounds for a set of matrices S(v). We need to identify the least upper bounds. This will be done by the following lemma.

Lemma 4

Let G and G′ be matrices as in (29). If G′≺G then C(G′)⊂C(G). Conversely, if C(G′) is a proper subset of C(G) then there exists a number λ such that $\mathtt{G}'_{\lambda} \prec \mathtt{G}$.

Note that the ellipsoids represented by $\mathtt{G}'_{\lambda}$ are the same for all λ≠0. It is clearly not true that $\mathtt{G}'_{\lambda} \preceq \mathtt{G}$ for all λ, since by making λ large or small enough, this condition will surely be broken.

A corollary of this lemma is

Lemma 5

Let V be a bounded set of points in ℝⁿ. Matrix G is a least upper bound for matrices S(v), v∈V if and only if G is of the form (29) and C(G) is a minimal ellipsoid containing the set V.

Proof of the first part of Lemma 4 is easy. Let v be a point in C(G′). Then, by Lemma 3, G′⪰S(v). If G≻G′ then G⪰S(v). Again by Lemma 3, this implies that v∈C(G). Hence C(G′)⊂C(G).

Now, we turn to the converse statement, which is considerably more difficult. First, note that it is sufficient to prove this result in the case where the matrices A and A′ belonging to G and G′ are invertible. If this is not the case, then the ellipsoids they define are degenerate, lying in a lower-dimensional subspace. By an appropriate affine transformation of the matrices, we can reduce them to matrices of lower dimension after eliminating rows and columns of zeros, and subsequently assume that the matrices are invertible.

We simplify this problem a little by simplifying the form of the matrices. The first observation is that we can assume that one of the two vectors r and r′ is zero. Thus, we can assume that r′ is replaced by r′−r, and r is replaced by the zero vector.

The second simplification is to apply an affine transformation to reduce to the case where A′=I. There exists a matrix P such that P ^⊤ A′P=I. Essentially, this affine transformation rescales non-isometrically so that the second ellipsoid becomes a spherical ball. We may also assume that q=q′=1.

It is now easily verified that if we prove Lemma 4 in the special case where A′ is the identity and r=0, then it will hold generally. Thus, given

$$\mathtt{G} = \left [ \arraycolsep=5pt \begin{array}{@{}cc@{}} \mathtt{A} & \mathbf{0} \\ \mathbf{0}^{\top} & 1 \end{array} \right ];\qquad \mathtt{G}' = \left [ \arraycolsep=5pt \begin{array}{@{}cc@{}} \mathtt{I} & \mathbf{r}' \\ \mathbf{r}^{\prime \top} & 1 \end{array} \right ] $$

our task is to show that

$$ \mathtt{G} - \mathtt{G}'_{\lambda}= \left [ \arraycolsep=5pt \begin{array}{@{}cc@{}} \mathtt{A} - \mathtt{I}/\lambda& - \mathbf{r}' \\ - \mathbf{r}^{\prime\top} & 1 - \lambda \end{array} \right ]\succ \mathtt{0} $$

(32)

for some λ, under the assumption that the ellipsoid C(G) contains the sphere C(G′).

However, this is equivalent, according to Lemma 3 to showing that A−I/λ⪰0, and 1−λ≥0, and that the ellipsoid x ^⊤(A−I/λ)⁺ x≤(1−λ) contains the point r′. This is shown in Fig. 5.

Fig. 5

A result about ellipsoids. If a sphere is contained in an ellipsoid, then there exists an ellipsoid represented by a matrix of the form (32) that contains the centre of the sphere

In the worst case, we may assume that the sphere C(G′) is touching the ellipsoid C(G) tangentially at some point, which we denote by y. The inward-pointing normal to the ellipsoid at y is obtained by differentiating y ^⊤ A ⁻¹ y. It is −2A ⁻¹ y and the unit vector is −A ⁻¹ y/λ where we define λ=∥A ⁻¹ y∥. Therefore, the centre of the unit sphere touching the ellipsoid at point y is given by

$$ \mathbf{r}' = \mathbf{y} - \mathtt{A}^{-1} \mathbf{y} / \lambda= (\mathtt{A} - \mathtt{I}/\lambda) \mathtt{A}^{-1} \mathbf{y} $$

(33)

where we define λ=∥A ⁻¹ y∥. This shows that $\mathbf{r}' \in \operatorname {Sp}(\mathtt{A} - \mathtt{I} / \lambda)$. We need to show that r′ lies in the ellipsoid x ^⊤(A−I/λ)⁺ x≤(1−λ). We compute

(34)

as required.

We also need to show that the matrix A−I/λ is positive semi-definite. This will be true if and only if 1/λ≤λ _min(A), the smallest eigenvalue of A. Equivalently, our task is to show that

$$ \lambda\ge\lambda_{\mathrm{max}} \bigl( \mathtt{A}^{-1}\bigr) $$

(35)

under the assumption that the unit sphere touching the ellipsoid x ^⊤ A ⁻¹ x=1 at the point y lies entirely inside the ellipsoid. Now, let u be the eigenvector of A ⁻¹ such that A ⁻¹ u=λ _max u, and suppose that the condition (35) is violated. With the centre r′ of the sphere given by (33) we verify that

with c<0. This means that r′ and y are on opposite sides of the principal plane of the ellipsoid defined by points x such that u ^⊤ x=0. We argue that this is impossible.

Let y′ be the point symmetrically opposite to y, reflected across this plane. This point also lies on the boundary of the ellipsoid C(G). Since y′ is on the same side of the plane as r′, we claim that r′ is closer to y′ than it is to y. This means that y′ lies closer than unit distance from r′ and so the unit ball centred at r′ is not contained completely inside the ellipsoid. This contradiction indicates that (35) must be true, and hence A−I/λ is positive semi-definite, as required. This being so, it follows directly from (34) that also (1−λ)≥0, as required.

This completes the proof of Lemma 4. The truth of Theorem 5 follows from Lemmas 2 and 5.

Remarks on the Lower Bound Matrix

It was assumed throughout most of this paper that the errors in any particular are bounded within a circle of radius τ _max. This allows an easy derivation of a good maximal lower bound for the Hessians. The justification for this is that the region $\mathcal {D}$, formed as an intersection of cones, must project to points within the stated error bound. However, in reality the projection of the region $\mathcal {D}$ in each image may be significantly smaller, and not circular in shape. In this case, we could think of using the more general form of the maximal lower bound Hessians for each image given by Theorem 5. This would be expected to give better results, at the cost of greater computation. The improvement may, however, not be worth the extra computational effort.