Abstract
Why is it that semidefinite relaxations have been so successful in numerous applications in computer vision and robotics for solving nonconvex optimization problems involving rotations? In studying the empirical performance, we note that there are few failure cases reported in the literature, in particular for estimation problems with a single rotation, motivating us to gain further theoretical understanding. A general framework based on tools from algebraic geometry is introduced for analyzing the power of semidefinite relaxations of problems with quadratic objective functions and rotational constraints. Applications include registration, hand–eye calibration, and rotation averaging. We characterize the extreme points and show that there exist failure cases for which the relaxation is not tight, even in the case of a single rotation. We also show that some problem classes are always tight given an appropriate parametrization. Our theoretical findings are accompanied with numerical simulations, providing further evidence and understanding of the results.
Similar content being viewed by others
Avoid common mistakes on your manuscript.
1 Introduction
Optimization over the special orthogonal group of the orthogonal matrices with determinant one occurs in many geometric vision problems where rigidity of a model needs to be preserved under transformations. While the objective functions are often simple least squares costs, constraining a matrix to be a rotation requires a number of quadratic equality constraints on the elements which makes the problem nonconvex. On the other hand, since both the objective and the constraints are quadratic, the Lagrange dual function can be computed (in closed form) and therefore optimization of the dual problem can be considered. It turns out that this is a linear semidefinite program (SDP), which can be reliably solved with standard solvers in polynomial time.
Recent studies [9, 12, 14, 18, 28, 32] have observed that in many practical applications the lower bound provided by the dual problem is often the same as the optimal value of the primal one. In such cases, duality offers a way of solving the original nonconvex problem using a tight convex relaxation. For different problem classes, the prevalence of problem instances with tight relaxations varies, as illustrated by the synthetically generated instances in Fig. 1. Furthermore, finding conditions that can be rigorously proven to be sufficient for a tight relaxation and strong duality for a given problem class remains a challenging research area. In this paper, we focus on the converse question: For what problem classes can we find objective functions that give a nonzero duality gap and a nontight relaxation? We use tools from algebraic geometry for analyzing when this happens in the case of general quadratic objective functions over rotational constraints. In particular, we consider the three most commonly occurring parametrizations that can be realized with quadratic constraints, namely 3Drotations represented either by matrices from \(\mathrm{SO}(3)\) or by unit norm quaternions, and planar rotations represented by matrices from \(\mathrm{SO}(2)\).
We consider the dual of the dual wherein all quadratic terms of the primal problem are replaced by linear terms over a set of ‘lifted’ variables subject to semidefinite constraints. The quadratic objective functions are then replaced by linear functions in the new variables, which are known to attain their optimum in extreme points of the feasible set. By studying the extreme points of this relaxation, we show that the situation is not as favorable as one might expect from the literature: Even for applications with relatively few rotations, we prove the existence of extreme points with rank strictly greater than one, and objective functions that are minimized at such points. The larger rank then prevents us from extracting a solution to the primal problem from the ‘lifted’ variables and shows that there is a duality gap.
Our main contributions are:

We present a novel analysis of the duality properties for quadratic minimization problems subject to rotational constraints based on algebraic geometry.

We characterize for several applications when the standard SDP relaxation is tight and when it is not. For instance, we give counterexamples for the registration problem with \(\mathrm{SO}(3)\)parametrization, showing that the SDP relaxation is not always tight since its solution may be an extreme point of rank 6. Similarly, we generate counterexamples which show that averaging of four planar rotations is not necessarily tight.

We show that the registration problem and the hand–eye calibration problem with \(\mathrm{SO}(2)\)parametrization or quaternion parametrization are always guaranteed to produce tight SDPrelaxations.
1.1 Related Work
It is well known that finding the optimal rigid transformation that registers two point clouds can be done in closed form [27]. This is a key subroutine used in many different applications, for example, in the ICP algorithm [2]. However, registering other geometric primitives is a much harder problem. In [34, 35], a branchandbound approach is proposed for finding the 3D rigid transformation for corresponding points, lines, and planes. The same problem is solved in [9] by first eliminating the translation and then using SDP relaxations for estimating the rotation. Empirically, it was noted that the relaxations were always tight, but no theoretical analysis was given. The problem of registering multiple point clouds was solved using SDP relaxations and Lagrangian duality in [14]. The problem was further studied in [28] where it was shown that for low noise regimes, the SDP relaxation is always tight.
In robotics, SDP relaxations for estimating rigid transformations in simultaneous localization and mapping (SLAM) have been explored in a number of recent papers [8, 11, 12, 19, 32]. Again, the empirical performance is generally good, the optimal solutions can be efficiently computed [5], and the relaxations are shown to be tight for bounded noise levels. Nontight counterexamples are also provided. In computer vision, there are many structurefrommotion (SfM) pipelines that rely on solving the socalled rotation averaging problem, see [1, 13, 17, 18, 20, 23, 24, 40]. One of the first approaches to use convex relaxations and duality in this setting was [20] where it was empirically observed that the relaxations are tight. A theoretical analysis and proof that for low noise regimes, SDP relaxations are indeed tight (no duality gap) have been derived in [15, 18, 36] for the problem of rotation averaging. The recent paper [17] explores this analysis to develop an efficient algorithm with optimality guarantees.
Estimating the pose of a camera also involves optimization over the special orthogonal group, [10, 33, 38, 41]. Approaches based on minimal solvers and Gröbner bases are often used. Alternatively, we show that the camera pose problem can be solved with SDP relaxations and convex optimization. Another classical problem that involves rotations is the hand–eye calibration problem [26]. In a recent paper [21], an SDP relaxation is proposed, again, with seemingly good empirical performance.
There are several previous works with similar aims as ours, but for more general problem settings. For a general, geometric overview of the problem at hand, we recommend [37] where orbitopes are studied. An orbitope is the convex hull of an orbit of a compact group acting linearly on a vector space. The dual of the dual formulation that we study corresponds to the firstorder relaxation of the momentSOS hierarchy [25], pioneered by Lasserre [30]. The approach of Lasserre has previously been applied to multiview geometry [29], but without any tightness guarantees. In [15], SDP relaxations for quadratically constrained quadratic programs (QCQP) are analyzed. Given that the SDP relaxation correctly solves a problem under noiseless observations (which is the case for the problems that we analyze), conditions are given which guarantee strong duality in the low noise regime. The size of this neighborhood is, however, not explicitly given. Further, a geometric interpretation of the relaxation is provided in [16]. We base our framework on the mathematical results proved in [4] where a deep connection is established between, on the one hand, algebraic varieties of minimal degree and on the other hand, the study of nonnegativity and its relation with sums of squares (SOS).
1.2 Contents of the Paper
In the next section, we present our general problem formulation. In Sect. 3, applications are presented and formulated on the standard form, of which the SDP relaxation is given in Sect. 4. In Sects. 5–7, we relate our problem to duality and analyze it using results from algebraic geometry. Our main result, a complete classification of SDP tightness for our example applications, is presented in Sect. 8.
2 Problem Formulation
The class of problems that we are interested in analyzing are problems involving rotations, parametrized either by

(i)
\(p\times p\) matrices belonging to the special orthogonal group, denoted \(\mathrm{SO}(p)\), where \(p=2\) or \(p=3\) for planar rotations and 3Drotations, respectively.

(ii)
4vectors of unit length representing quaternions, denoted \({{\mathcal {Q}}}\) for 3Drotations.
In addition, we require that the objective function is quadratic in the variables of the chosen parametrization.
Let \( R= [R_1, \ldots \,R_n] \) where each \(R_i \in \mathrm{SO}(p)\) with \(p=2\) or \(p=3\) and let \(\text {vec}\left( R\right) \) denote the columnwise stacked vector of the \(p\times pn\) matrix R. Now let M be a real, symmetric \((p^2n+1)\times (p^2n+1)\) matrix; then, we would like to solve the following nonconvex optimization problem
Alternatively we model 3Drotations with unit quaternions, \(q= [q_1, \ldots \,q_n]\), and consider the similar formulation
Note that not every problem with 3Drotations may be straightforward to model on both of the formulations (1) and (2), and furthermore, their residual errors have different interpretation, and therefore, the formulations are not equivalent. Also, the set of quaternions forms a double covering of the set of rotations, meaning that q and \(q\) represent the same rotations [24].
Both of these problem formulations can be put in the following standard form:
The l quadratic equations \(r^T A_i r = 0\) enforce the rotational constraints and \(r^T e = 1\) with \(e=[0,\ldots ,0,1]^T\) forces the final element of r to be one. The rest of the paper will be devoted to this standard form, and we will analyze it in detail.
3 Applications
There are several application problems that can be modeled and solved using the above formulation. Often, one would like to solve for one or several rigid transformations (a rotation and a translation). However, in many cases, one can directly eliminate the translation and concentrate on the more difficult part of finding the rotations.
Next we give several examples of rotation problems appearing in the literature.
Example 1
Registration with pointtopoint, pointtoline and pointtoplane correspondences can be written as in (3). The residuals are all of the form
If point \(x_i\) corresponds to point \(y_i\), then set \(P_i = I_3\). If \(x_i\) is a measurement known to lie on a line, then set \(P_i = I  v_i v_i^T\), where \(v_i\) is a unit directional vector and \(y_i\) is a point on the line. Similarly, if \(x_i\) lies on a plane, then set \(P_i = n_i^T\), where \(n_i\) is a unit normal and \(y_i\) is a point on the plane.
Minimizing over t gives the closedform solution
which is linear in \(\text {vec}\left( R\right) \). Inserting this into the objective function thus gives an expression which is quadratic in \(\text {vec}\left( R\right) \) and therefore can be reshaped into (3).
In [27], it is shown that the registration problem with pointtopoint correspondences can be formulated as a quadratic optimization problem in the quaternion representation. We are not aware of any quadratic quaternion formulation for the pointtoline and pointtoplane cases.
Example 2
Resectioning is the problem of recovering the position and orientation of a camera given 2Dto3D correspondences. Geometrically this can be done by aligning the viewing rays from the camera with the 3D points. This reduces the problem to a special case of pointtoline registration where all of the lines intersect in the camera center.
Example 3
Hand–eye calibration is the problem of determining the transformation between a sensor (often a camera) and a robot hand on which the sensor is mounted. Given rotation measurements \(U_i\) and \(V_i\), \(i=1,\ldots ,m\) relative to a global frame, for the sensor and the robot hand, respectively, the objective is to find the relative rotation R between them by solving the following optimization problem:
where \(M_i = 2I V_i\otimes U_iV_i^T\otimes U_i^T\). Finally, set M as
An alternative formulation using quaternions can also be employed. If the unit quaternions \(u,v \in {{\mathcal {Q}}}\) represent the rotations \(U,V \in \mathrm{SO}(3)\), then the quaternion representing the composition UV can be written Q(u)v, where
Therefore, the optimization problem
can also be used and turned into the standard form in order to solve hand–eye calibration. Note, however, that due to the double covering the signs of \(u_i\) and \(v_i\) have to be selected consistently in order for q to give a low objective value.
Example 4
Rotation averaging aims to determine a set of absolute orientations \(R_i\), \(i=1,\ldots ,n\) from a number of measured relative rotations \(R_{ij}\) by minimizing
Since \(\Vert R_i\Vert _F^2 = 3\), the problem is (ignoring constants) equivalent to minimizing
Letting \(M = \text {blkdiag}\left( M_0 \otimes I_3, 0\right) \) now gives an optimization problem of form (3). \(I_3\) is a \(3 \times 3\) identity matrix, and the \(\text {blkdiag}\left( \cdot \right) \) operation constructs a blockdiagonal matrix.
Similar to the hand–eye calibration problem, rotation averaging can be formulated with quaternions [22] using the objective function \(\sum _{i\ne j} \Vert Q(r_i)r_{ij}  r_j\Vert ^2\), which after simplifications yields an expression similar to (11) and hence can be put in the standard form (3).
Example 5
Pointset averaging is the problem of registering a number of point sets, measured in different coordinate systems, to an unknown average model. If \(X_i\), \(i=1,\ldots ,n\) are \(3 \times n\) matrices containing measurements of corresponding 3D points, we want to find a \(3 \times n\) matrix Y, rotations \(R_i\) and translations \(t_i\) such that
where \({\mathbbm {1}}\) is a column vector with all ones, which is minimized. Since the variables Y and \(t_i\), \(i=1,\ldots ,n\) are unconstrained, they can be solved for as a function of the rotations \(R_i\), \(i=1,\ldots ,n\). Assuming that the centroid of the points in \(X_i\) is the origin for all i, backsubstitution allows us to write the problem solely in terms of the rotations as
where
Letting \(M = \text {blkdiag}\left( M_0 \otimes I_3, 0\right) \) now gives an optimization problem of form (3).
In the above examples, we only considered the case of 3Drotations. Note, however, that all of these problems also have meaningful versions in the plane for which parametrization using \(\mathrm{SO}(2)\) yields the same type of problem.
4 SDP Relaxation
Let us now derive the standard convex SDP relaxation for our standard form (3). Consider the objective function, which can be rewritten using trace notation as
where \(X=rr^T\). Note that \(\text {rank}\left( X\right) =1\) and \(X \succeq 0\).
For \(\mathrm{SO}(3)\), the condition that \(R_i\) belongs to the special orthogonal group can be expressed by quadratic constraints in the entries of \(R_i\), for instance \(R_i^TR_iI=0\). Similarly, that the cross product of the first and second rows should equal the third, which ensures \(\det {R_i}=1\) is also a quadratic constraint. Consequently, the same constraints can be expressed by linear equations in the entries of X in the form \(\text {tr}\left( A_iX\right) =0\). It can be checked that there are 20 linearly independent such constraints for each \(R_i\).
The corresponding program for \(\mathrm{SO}(2)\) is similar to that of \(\mathrm{SO}(3)\), but it only requires one constraint per rotation. We represent a rotation by \( R_i = \left( \begin{matrix} c_i &{} s_i \\ s_i &{} c_i \end{matrix} \right) , \) where \(c_i^2+s_i^2 = 1\). Hence, for n rotations, only 2n variables are needed in the vector r and the unit length constraint becomes linear in the entries of X. Similarly for quaternions \({{\mathcal {Q}}}\), the unit length constraint for each \(q_i\) can be written as a linear constraint.
If we ignore the nonconvex constraint that \(\text {rank}\left( X\right) =1\), then we get a semidefinite program problem over X: The objective function is linear in X subject to linear equality constraints and a positive semidefinite constraint, \(X\succeq 0\). This leads to the following convex relaxation:
Note that as we have relaxed (ignored) \(\text {rank}\left( X\right) =1\), the optimal value will be a lower bound on the original nonconvex problem (3). Further, if the optimal solution \(X^*\) has rank one, then we say that the relaxation is tight and the globally optimal solution is obtained.
5 Duality and Sums of Squares
Consider again the objective function in (3) and the Lagrangian dual problem of (15)
where \((\gamma ,\lambda )\) are the dual variables. By construction, this problem gives the same objective value as (15) and therefore a lower bound on the original (3). We are interested in knowing when this lower bound is tight.
Let I be the ideal of the polynomials defining the constraint set, that is, a polynomial p is in I when
where v and \(v_i\) are any polynomials. The variety V(I) consists of the feasible points \(\{r \,  \, p(r) = 0\ \forall p\in I\}\). Let \(R_2\) denote the set of quadratic polynomials modulo I, that is, two polynomials \(f, g \in R_2\) are considered equal if \(fg \in I\).
The question of tightness between the original problem (3) and the relaxation (15) and its dual (16) is related to the two convex, closed cones
and
Note that the cones are defined to be dependent on the constraint set of (3) and not on the actual form of the objective function \(r^TMr\). As any quadratic polynomial f in \(\Sigma \) is a sum of linear squares on the feasible set V(I) and hence nonnegative, it follows that \(\Sigma \subseteq P\).
Consider again our original problem in (3), written as
It follows that \(r^T(M\eta ^*ee^T)r \in P\). If \(\gamma ^*\) is the optimal value of (16) with dual variables \(\lambda ^*\), then the matrix \(M\sum _i \lambda ^*_i A_i  \gamma ^* ee^T\) is positive semidefinite, and we can factor it into a sum of rank1 matrices \(\sum _j a_j a_j^T\). Therefore,
Now, \(r^T\left( \sum _i \lambda ^*_i A_i\right) r\) lies in I, and we can conclude that the quadratic polynomial \(r^T(M  \gamma ^*ee^T)r\) belongs to \(\Sigma \) when \((\gamma ^*,\lambda ^*)\) is the solution to (16).
In view of the above discussion, it is clear that the convex formulations (15) and (16) can only give the same objective value as (3) when \(r^T M r  \eta ^*\) is a sum of squares, where \(\eta ^*\) is the optimal value of (3). The question we are interested in answering is hence when is it possible to find an SOS for this nonnegative quadratic form? If the cones are not equal, that is, \(\Sigma \subsetneq P\), then there may exist objective functions for which the relaxation is not tight. We shall investigate this further in a constructive manner. First, we need some more tools from algebraic geometry. See also book [3] for a general introduction.
6 The Varieties of Rotations
An algebraic variety V is the set of solutions of a system of polynomial equations over the reals. In this paper, we analyze three varieties that are commonly used in computer vision applications: \(\mathrm{SO}(2)^n\), \(\mathrm{SO}(3)^n\) and \({{\mathcal {Q}}}^n\). These varieties can be defined by a system of polynomial equations in the entries of \(2\times 2\), \(3\times 3\) matrices and 4vectors, respectively (cf. Sect. 4). The dimensions and codimensions of these varieties are wellknown, and \(\dim {\mathrm{SO}(2)} = 1\), \({{\,\mathrm{codim}\,}}{\mathrm{SO}(2)} = 1\), \(\dim {\mathrm{SO}(3)} = 3\), \({{\,\mathrm{codim}\,}}{\mathrm{SO}(3)} = 6\), \(\dim {{{\mathcal {Q}}}} = 3\), and \({{\,\mathrm{codim}\,}}{{{\mathcal {Q}}}} = 1\). The degree of V is by definition the number of intersection points of the variety with \(\dim {V}\) general hyperplanes, and we have that \(\deg {\mathrm{SO}(2)} = 2\), \(\deg {\mathrm{SO}(3)} = 8\) (see [7] for a derivation) and \(\deg {{{\mathcal {Q}}}} = 2\).
For n copies of V, it is straightforward to show that the variety of \(V^n\) has dimension \(n\dim {V}\), codimension \(n {{\,\mathrm{codim}\,}}{V}\), and degree \((\deg {V})^n\). For instance, for the case of \(\mathrm{SO}(3)^n\), we have that \(\dim {\mathrm{SO}(3)^n} = 3n\), \({{\,\mathrm{codim}\,}}{\mathrm{SO}(3)^n} = 6n\), and \(\deg {\mathrm{SO}(3)^n} = 8^n\).
For any variety V, \(\deg {V} \ge {{\,\mathrm{codim}\,}}{V}+1\). A variety is called minimal if it is nondegenerate (that is, not contained in a hyperplane) and \(\deg {V} = {{\,\mathrm{codim}\,}}{V}+1\). Similarly, it is called almost minimal when \(\deg {V} = {{\,\mathrm{codim}\,}}{V}+2\). Considering the degrees and codimensions of the varieties previously listed, Table 1 summarizes their characterization as minimal, almost minimal, and not minimal, for the cases \(n = 1\), \(n = 2\), and \(n > 2\).
7 The Extreme Points of the SDP Relaxation
In this section, we investigate further the convex cone of nonnegative polynomials and that of SOS polynomials over the rotational varieties, \(\mathrm{SO}(3)\), \(\mathrm{SO}(2)\), and \({{\mathcal {Q}}}\). The goal is to find out when the SDP relaxation is tight and to characterize all possible extreme points for the relaxation.
The following general result is proved in [4].
Lemma 1
(Blekherman et al. [4]) Let V be a real irreducible nondegenerate variety such that its subset of real points is Zariski dense. Every real quadratic form that is nonnegative on V is a sum of squares of linear forms if and only if V is a variety of minimal degree.
An illustration of the result is given in Fig. 2. We will now apply it to our varieties of interest.^{Footnote 1}
7.1 Minimal Varieties
Among the varieties in Table 1, we know that only \(\mathrm{SO}(2)\) and \({{\mathcal {Q}}}\) are minimal. In the remaining cases, the convex cones P and \(\Sigma \) defined in (18) and (19) are therefore strictly different, i.e., \(\Sigma \subsetneq P\). The proof of Lemma 1 is constructive, and it allows us to generate objective functions that are nonnegative, but not sums of squares, and thereby the SDP relaxations will not be tight for these optimization problems. However, our example applications have objective functions of a special form, and it remains to see whether there are such objectives which are not sums of squares. We will return to this question in Sect. 8. We can conclude that any hand–eye calibration problem defined over \(\mathrm{SO}(2)\) or \({{\mathcal {Q}}}\) will always have tight SDP relaxations.
7.2 Almost Minimal Varieties
In the case when V is almost minimal, that is, when V is either \(\mathrm{SO}(3)\), \(\mathrm{SO}(2) \times \mathrm{SO}(2)\) or \({{\mathcal {Q}}}\times {{\mathcal {Q}}}\) (Table 1), we will still have \(\Sigma \subsetneq P\), but the gap between the cones will be smaller. Furthermore, for problems in \(P \setminus \Sigma \), the extreme points of the corresponding SDP relaxation can be characterized based on the theory in [4]. An immediate reformulation of Proposition 3.5 for our purposes gives the following corollary.
Corollary 1
Assume that the variety V is almost minimal and arithmetically Cohen–Macaulay. Then, the extreme points \(X^*\) of the SDP relaxation in (15) either have \(\text {rank} \left( X^*\right) =1\) or \(\text {rank} \left( X^*\right) ={{\,\mathrm{codim}\,}}{V}\).
All of the varieties we study are smooth and therefore arithmetically Cohen–Macaulay. Furthermore recall from Sect. 6 that \({{\,\mathrm{codim}\,}}{\mathrm{SO}(3)} = 6\), \({{\,\mathrm{codim}\,}}{\mathrm{SO}(2) \times \mathrm{SO}(2)} = 2\) and \({{\,\mathrm{codim}\,}}{{{\mathcal {Q}}}\times {{\mathcal {Q}}}} = 2\). In the \(\mathrm{SO}(3)\) case, if the computed optimal solution \(X^*\) of the relaxation has not rank 1 nor 6, but say, for instance, rank 2, then \(X^*\) can be decomposed into two rank1 matrices, \(X^*=\lambda X_1^*+(1\lambda )X_2^*\) for some \(\lambda \in [ 0,1 ]\) where \(X_1^*\) and \(X_2^*\) are optimal solutions and extreme points.
If \(\text {rank}\left( X^*\right) =1\), then the corresponding objective function \(r^TMr\eta ^*\) (where \(\eta ^*\) is the optimal objective value) is a sum of squares, and as shown previously can be retrieved by solving the SDP. If \(\text {rank}\left( X^*\right) >1\) and \(X^*\) cannot be decomposed into rank1 extreme points, then the corresponding objective function \(r^TMr\eta ^*\) is not a sum of squares. For almost minimal arithmetically Cohen–Macaulay varieties, such extreme points \(X^*\) must be of \(\text {rank}\left( X^*\right) ={{\,\mathrm{codim}\,}}{V}\) and there are no other possibilities.
To summarize, if when minimizing a given problem over an almost minimal variety V we obtain the extreme point \(X^*\) which has \(\text {rank}\left( X^*\right) =1\), then we have indeed computed the globally optimal solution, but if it turns out that \(\text {rank}\left( X^*\right) ={{\,\mathrm{codim}\,}}{V}\), then the relaxation is not tight, and we do not even have a feasible solution to the original problem, just a lower bound on the optimal value.
7.3 Prevalence of Nontight Problem Instances
In Fig. 1, we presented the results of two sets of synthetic experiments, illustrating the significance of almost minimal varieties.
In the first set of experiments, the domain is the almost minimal variety \(V=\mathrm{SO}(3)\) and the entries of the objective function, encoded by the \(10\times 10\) symmetric matrix M in (3), were randomly drawn from a uniform distribution from \([\,\text {}1,1\,]\). In all 1000 examples, we obtained a rank1, globally optimal solution for the SDP relaxation, even though the variety is not minimal. This shows that the rank6 extreme points predicted by Corollary 1 are rare in practice among the random objective functions considered. It is, however, possible to produce such nontight examples, and we shall return to this question later.
In the second set of experiments, the optimization took place over \(V=\mathrm{SO}(3) \times \mathrm{SO}(3)\), which is not almost minimal. The entries of the \(19 \times 19\) symmetric matrix M were generated in the same way via a uniform distribution. In this case, the relaxation works poorer, and various ranks are obtained for its solutions.
Remark
For neither minimal nor almost minimal varieties, the nonnegative cone P becomes significantly larger than the SOS cone \(\Sigma \). Nontight SDP relaxations will be more prevalent, and various ranks will be observed for the solutions to these nontight relaxations. A rank1 solution will, however, always provide a solution to the primal problem.
8 Tightness of Our Example Applications
The theoretical results in the previous section apply to general quadratic objective functions. For actual applications, the objective functions will be structured. For instance, consider the hand–eye calibration problem in Example 3. There are only purely quadratic terms of the rotation variables in the objective and no linear ones. Hence, the last row and the last column of the matrix M will be zero. In this section, we analyze structured objective functions corresponding to different problem classes. We also relate our new results to previous ones in the literature.
In Table 2, we present a complete classification of SDP tightness for our example applications. In accordance with Table 1, applications for the minimal varieties \(\mathrm{SO}(2)\) and \({{\mathcal {Q}}}\) are always tight—this is a known result, as there is a single quadratic constraint (see, for example, Boyd and Vandenberghe [6]). For the almost minimal varieties, we generate rare nontight problem instances, and for the nonminimal cases we conclude that tightness can only be guaranteed in the low noise regime, supported by previous works and empirically demonstrated by us.
Noisefree case All of our considered example applications have objective functions of the form \(r^TMr = r^TU^TUr = \Vert Ur \Vert ^2\) for some matrix U. If the optimal value \(\eta ^* = \min _{r\in V(I)} r^T M r\) is equal to zero (which corresponds to the noisefree case), then \(r^T(M\eta ^*ee^T)r = r^TMr \in P\), where P is the cone of nonnegative quadratic forms in (18). Further, since M has nonnegative eigenvalues, \(M\succeq 0\) and we can factor it into a sum of rank1 matrices \(M = \sum _j a_j a_j^T\). It follows that \(r^TMr = \sum _j (a_j^Tr)^2\) is SOS. This is a wellknown result, and it has further been studied in [15] where it is shown that for low noise levels (\(\eta ^*\) close to zero), the nonnegative polynomial \(r^T M r  \eta ^*\) is a sum of squares as well.
8.1 Registration and Resectioning
The formulations of these two applications over the domain \(\mathrm{SO}(3)\) are given in Examples 1 and 2, respectively. As the variety \(\mathrm{SO}(3)\) is almost minimal (Table 1), one may wonder if there are actual problem instances that lead to nontight relaxations and extreme points of rank6 (Corollary 1)? In [4], there is a procedure for finding polynomials that are nonnegative, but not sums of squares. However, this will in general not result in objective functions originating from registration or resectioning problems. The objective function for this type of problem is of the form \(r^TMr = r^TU^TUr = \Vert Ur \Vert ^2\) where \(M=U^TU\) with U of size \(m \times 10\) and m is the number of correspondences of type pointtopoint, pointtoline or pointtoplane. That in particular means that M has nonnegative eigenvalues and \(M\succeq 0\), and there are some additional requirements as well.
In Sect. A in appendix, we show how to modify the procedure of Blekherman et al. [4] in order to achieve such objective functions. For every nontight problem instance generated with Procedure 1 described in Sect. A, we get a rank6 solution \(X^*\) as predicted by Corollary 1, and consequently, no feasible solution is obtained. Hence, there exist indeed problem instances that are nontight, but they are rare in practice. See also the first column of Table 2.
Relation to the empirical results of Briales and Gonzalez–Jimenez [9]. Extensive experiments using the SDP relaxation in (15) for registration over \(\mathrm{SO}(3)\) are performed in [9], but not a single instance with a nontight relaxation among their real or synthetic experiments is found. This is consistent with our experiments, as presented in Fig. 1a, where we have done an empirical analysis of SDP tightness over \(\mathrm{SO}(3)\) for quadratic objective functions with random entries. The counterexamples are indeed rare in practice for this almost minimal variety.
8.2 Hand–Eye Calibration
As previously mentioned, the objective function for hand–eye calibration contains only purely quadratic terms of the rotation variables and no linear ones. Hence, the last row and the last column of the \(10\times 10\) symmetric matrix M will be zero. We tested the same procedure as for registration (see Procedure 1 in Sect. A in appendix) in order to generate nontight counterexamples with the structure of a hand–eye calibration objective. We succeeded in obtaining problem instances for averaging 8 rotation matrices that yielded objective functions with nontight relaxations, see the second column of Table 2 for a summary. Again, all of these optimization problems attain their minima at rank6 extreme points, in accordance with Corollary 1.
8.3 Rotation Averaging
In [18], Eriksson et al. proved that the SDP relaxation for problems involving three rotations with \(\mathrm{SO}(3)\)parametrization is always tight. This result trivially extends to \(\mathrm{SO}(2)\). Further, for instances with more than three cameras, it is shown that in the low noise regime the SDP relaxation is tight. Low noise results applicable to \(\mathrm{SO}(3)\) as well as \(\mathrm{SO}(2)\) have also been presented by Rosen et al. [36], although \(\mathrm{SO}(2)\) is parametrized by all matrix elements in their case. Fredriksson and Olsson [20] parametrize the rotation averaging problem with quaternions \({{\mathcal {Q}}}\) and in all the reported experiments, the SDP relaxation was always numerically found to be tight. For \(\mathrm{SO}(2)\), Zhong and Boumal [42] proved the existence of an upper bound on the noise level for which the SDP relaxation is tight; however, no explicit estimates were given.^{Footnote 2}
Here we present results for the case of four rotations in \(\mathrm{SO}(2)\) (a threerotation problem is always tight [18]). Figure 3 shows the average rank of the computed SDP solution \(X^*\). The \(M_0\) matrix in (11) was generated by sampling the relative rotation angles from \({\mathcal {N}}(0, \sigma ^2)\), \(\sigma \in [0, 1]\) radians. For each noise level \(\sigma \), we ran the problem 10, 000 times and plotted the average obtained rank of the lifted variables \(X^*\). The observed ranks were 1 or 2. Similar to our results, Fan et al. [19] find instances of the 2D SLAM problem with nontight relaxations\(^{2}\), and Carlone et al. [12] present analogous results for 3D posegraph optimization (PGO).
Relation to Mangelson et al. [32]. The planar posegraph problem with
is studied in [32]. Here, additional relative translation estimates \(t_{ij}\) are present, but \(\tau = 0\) reduces the problem to rotation averaging. A ‘proof’ of strong duality for the SparseBSOS relaxation [31, 39] is presented. This would imply that rotationaveraging can be solved exactly in polynomial time but that the SDP relaxation (15) still gives a duality gap. While such a weakness is entirely plausible, we note that the presented proof in [32] is in fact flawed as the domain, \(\mathrm{SO}(2)^n\) using unit norm constraints on the diagonals of the rotation matrices, is incorrectly claimed to be SOSconvex (see [3, 31, 39] for a definition). It is not even a convex domain.
While the lack of a proof does not exclude the possibility that SparseBSOS is exact, our counterexamples in Fig. 3 show that this is only possible if SparseBSOS is stronger than the SDP relaxation (15). A detailed comparison of these two formulations would reveal if this is the case. Such an indepth analysis is, however, beyond the scope of this paper.
8.4 Point Set Averaging
In previous work by Chaudhury et al. and Iglesias et al. [14, 28], it has been shown that SDP relaxations are tight in the low noise regime for registering multiple point clouds, while in the high noise regime nontight instances arise. Here we reproduce similar results, registering an artificial point set over four frames. 100 points are sampled from \({\mathcal {N}}(0, 1)\), after which one direction is squeezed with a factor 1/100, causing higher prevalence of nontight instances. Gaussian noise, sampled from \({\mathcal {N}}(0, \sigma ^{2})\), was added to each point. Figure 4 shows the average rank over 10.000 problem instances of the computed SDP solution \(X^*\), for each noise level \(\sigma \).
9 Conclusions
We have presented a framework for analyzing the power of SDP relaxations for optimization over rotational constraints. The key to our analysis has been to investigate the two convex cones of nonnegative and sumofsquares polynomials and to establish a connection between them and the tightness of an SDP relaxation. We have shown that certain parametrizations lead to tight SDP relaxations and others do not. For our applications which have structured objective functions, we have generated nontight counterexamples to settle the question of whether the relaxation is always tight or not.
An interesting avenue for future research is to develop algorithms that can recover a good solution from a nontight relaxation for practitioners of SDP relaxations. This was recently done for the rotation averaging problem [17]. Another interesting direction is to explore the existence of noise bounds for which the registration and hand–eye calibration problems over \(\mathrm{SO}(3)\) are guaranteed to be tight.
Notes
All of the varieties that we study are real, irreducible, nondegenerate and their corresponding subsets of real points are Zariski dense.
Their \({\mathbb {C}}\) parametrization is equivalent to our representation.
References
Arrigoni, F., Magri, L., Rossi, B., Fragneto, P., Fusiello, A.: Robust absolute rotation estimation via lowrank and sparse matrix decomposition. In: International Conference on 3D Vision (2014)
Besl, P., McKay, N.: A method for registration two 3d shapes. IEEE Trans. Pattern Anal. Mach. Intell. 14(2), 239–256 (1992)
Blekherman, G., Parrilo, P., Thomas, R.: Semidefinite optimization and convex algebraic geometry. SIAM J. Opt. 10, 673–696 (2012)
Blekherman, G., Smith, G., Velasco, M.: Sums of squares and varieties of minimal degree. J. Am. Math. Soc. 29(3), 893–913 (2016)
Boumal, N.: A riemannian lowrank method for optimization over semidefinite matrices with blockdiagonal constraints. arXiv preprint arXiv:1506.00575 (2015)
Boyd, S., Vandenberghe, L.: Convex Optimization. Cambridge University Press, Cambridge (2004)
Brandt, M., Bruce, J., Brysiewicz, T., Krone, R., Robeva, E.: The Degree of \(SO(n, \mathbb{C})\). Springer, New York (2017). https://doi.org/10.1007/9781493974863_11
Briales, J., GonzalezJimenez, J.: Fast global optimality verification in 3D SLAM. In: International Conference on Intelligent Robots and Systems (2016)
Briales, J., GonzalezJimenez, J.: Convex global 3D registration with Lagrangian duality. In: Computer Vision and Pattern Recognition (2017)
Brynte, L., Kahl, F.: bmvc. Pose Proposal Critic: Robust Pose Refinement by Learning Reprojection Errors. In: British Machine Vision Conference (2020)
Carlone, L., Calafiore, G.C., Tommolillo, C., Dellaert, F.: Planar pose graph optimization: duality, optimal solutions, and verification. IEEE Trans. Robot. 32(3), 545–565 (2016). https://doi.org/10.1109/TRO.2016.2544304
Carlone, L., Rosen, D.M., Calafiore, G., Leonard, J.J., Dellaert, F.: Lagrangian duality in 3D SLAM: verification techniques and optimal solutions. In: International Conference on Intelligent Robots and Systems (2015)
Chatterjee, A., Govindu, V.: Efficient and robust largescale rotation averaging. In: International Conference on Computer Vision (2013)
Chaudhury, K., Khoo, Y., Singer, A.: Global registration of multiple point clouds using semidefinite programming. SIAM J. Opt. 25(1), 468–501 (2015)
Cifuentes, D., Agarwal, S., Parrilo, P.A., Thomas, R.R.: On the local stability of semidefinite relaxations. arXiv preprint arXiv:1710.04287 (2017)
Cifuentes, D., Harris, C., Sturmfels, B.: The geometry of SDPexactness in quadratic optimization. Math. Program. 182(1/2), 399–428 (2020). https://doi.org/10.1007/s10107019013998
Dellaert, F., Rosen, D., Wu, J., Mahony, R., Carlone, L.: Shonan rotation averaging: global optimality by surfing \(SO(p)^n\). In: European Conference on computer vision (ECCV) (2020)
Eriksson, A., Olsson, C., Kahl, F., Chin, T.J.: Rotation averaging with the chordal distance: global minimizers and strong duality. IEEE Trans. Pattern Anal. Mach. Intell. 43, 256–268 (2019)
Fan, T., Wang, H., Rubenstein, M., Murphey, T.: Cplslam: efficient and certifiably correct planar graphbased slam using the complex number representation. IEEE Trans. Robot. 36(6), 1719–1737 (2020). https://doi.org/10.1109/TRO.2020.3006717
Fredriksson, J., Olsson, C.: Simultaneous multiple rotation averaging using Lagrangian duality. In: Asian Conference on Computer Vision (ACCV) (2012)
Giamou, M., Ma, Z., Peretroukhin, V., Kelly, J.: Certifiably globally optimal extrinsic calibration from persensor egomotion. IEEE Robot. Autom. Lett. 4(2), 367–374 (2019)
Govindu, V.: Combining twoview constraints for motion estimation. In: Computer Vision and Pattern Recognition (2001)
Hartley, R., Aftab, K., Trumpf, J.: \(L_1\) rotation averaging using the Weiszfeld algorithm. In: IEEE Conference on Computer Vision and Pattern Recognition (2011)
Hartley, R., Trumpf, J., Dai, Y., Li, H.: Rotation averaging. Int. J. Comput. Vis. IJCV 103(3), 267–305 (2013)
Henrion, D., Korda, M., Lasserre, J.B.: The MomentSOS Hierarchy. World Scientific, Singapore (2020). https://doi.org/10.1142/q0252
Horaud, R., Dornaika, F.: Handeye calibration. Int. J. Robot. Res. 14(3), 195–210 (1995)
Horn, B.K.P.: Closedform solution of absolute orientation using unit quaternions. J. Opt. Soc. Am. A 4(4), 629–642 (1987)
Iglesias, J., Olsson, C., Kahl, F.: Global optimality for point set registration using semidefinite programming. In: Computer Vision and Pattern Recognition (2020)
Kahl, F., Henrion, D.: Globally optimal estimates for geometric reconstruction problems. Int. J. Comput. Vis. IJCV 74(1), 3–15 (2007). https://doi.org/10.1007/s112630060015y
Lasserre, J.B.: Global optimization with polynomials and the problem of moments. SIAM J. Opt. 11(3), 796–817 (2001). https://doi.org/10.1137/S1052623400366802
Lasserre, J.B., Toh, K.C., Yang, S.: A bounded degree SOS hierarchy for polynomial optimization. EURO J. Comput. Opt. 5, 10 (2015). https://doi.org/10.1007/s136750150050y
Mangelson, J., Liu, J., Eustice, R., Vasudevan, R.: Guaranteed globally optimal planar pose graph and landmark slam via sparsebounded sumsofsquares programming. In: International Conference on Robotics and Automation (2019)
Nakano, G.: Globally optimal DLS method for PnP problem with Cayley parameterization. In: British Machine Vision Conference (2015)
Olsson, C., Kahl, F., Oskarsson, M.: The registration problem revisited: optimal solutions from points, lines and planes. In: Computer Vision and Pattern Recognition (2006)
Olsson, C., Kahl, F., Oskarsson, M.: Branchandbound methods for Euclidean registration problems. IEEE Trans. Pattern Anal. Mach. Intell. 31(5), 783–794 (2009)
Rosen, D., Carlone, L., Bandeira, A., Leonard, J.: SESync: a certifiably correct algorithm for synchronization over the special Euclidean group. Int. J. Robot. Res. 38(2–3), 95–125 (2018)
Sanyal, R., Sottile, F., Sturmfels, B.: Orbitopes. Mathematika 57(2), 275–314 (2011)
Sweeney, C., Fragoso, V., Höllerer, T., Turk, M.: gDLS: a scalable solution to the generalized pose and scale problem. In: European Conference on Computer Vision (ECCV) (2014)
Weisser, T., Lasserre, J.B., Toh, K.C.: A bounded degree SOS hierarchy for large scale polynomial optimization with sparsity. Math. Program. Comput. (MPC) (2016). https://doi.org/10.1007/s1253201701216
Wilson, K., Bindel, D., Snavely, N.: When is rotations averaging hard? In: European Conference on Computer Vision (ECCV) (2016)
Zheng, Y., Kuang, Y., Sugimoto, S., Astrom, K., Okutomi, M.: Revisiting the PnP problem: a fast, general and optimal solution. In: International Conference on Computer Vision (ICCV) (2013)
Zhong, Y., Boumal, N.: Nearoptimal bounds for phase synchronization. SIAM J. Opt. 28(2), 989–1016 (2018)
Acknowledgements
Open access funding provided by Chalmers University of Technology. This work was partially supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation, the Swedish Foundation for Strategic Research (Semantic Mapping and Visual Navigation for Smart Robots) and the Swedish Research Council (Grant No. 201805375). Special thanks for the insightful and invaluable comments of our reviewers.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Generating Nontight LeastSquares Problems
Generating Nontight LeastSquares Problems
Using Procedure 1, one can generate nonnegative polynomials that are not sums of squares, and consequently, such objective functions minimized over the variety will result in nontight relaxations. The procedure is adapted from Procedure 3.3 in [4] (for general nonminimal varieties) and presented here specifically for the \(\mathrm{SO}(3)\)case. We will generate explicit counterexamples for the hand–eye calibration problem and the registration problem that do not have tight SDP relaxations.
There are a few modifications of the procedure that are required so that the objective functions do originate from a specific example application. For leastsquares problems (such as hand–eye and registration), we know that \(M \succeq 0\). This is not automatically fulfilled via Procedure 1, but we can change Step 2 to account for this. More specifically, one can choose a quadratic form \(M_0\), which fulfills the two prescribed conditions (i) and (ii), and at the same time maximizes the minimum eigenvalue of \(M_0\). This can be cast as a convex optimization problem. If an \(M_0 \succeq 0\) is found, then \(M \succeq 0\) will follow. Further, for hand–eye calibration and registration, there are linear relationships between the elements of M that should be satisfied for any problem instance. This can also be accounted for when maximizing the minimum eigenvalue of \(M_0\). This will give us a matrix M which satisfies necessary conditions for our applications.
To show that such an M corresponds to an actual problem instance, we look for, in the case of hand–eye calibration, pairs of rotation matrices \((U_i,V_i)\), \(i=1,\ldots ,m\) such that the objective function in (6) gives the correct matrix M. Note that we do not need to find rotation matrices that map exactly to M, since the set P of quadratic forms that are nonnegative is closed and thereby quadratic forms close to \(r^TMr\) will also be in P, but not in \(\Sigma \) (cf. Fig. 2). In practice, we solve the following minimization problem with gradient descent:
where \(M(\{ (U_i,V_i) \}_{i=1}^m)\) is the matrix obtained from formula (7) using \((U_i,V_i)\), \(i=1,\ldots ,m\) and M is the matrix obtained from Procedure 1. Empirically, we have found that for \(m=8\), one can find such rotation matrices which result in a nontight objective function.
In the case of registration, we have empirically found nontight problem instances with 5 pointline correspondences where the residuals are of form (4). Similarly to the case of hand–eye calibration in (21), we explicitly optimize for pointline correspondences that produce an objective function which is close to the given matrix M.
Remark
Although Procedure 1 in general is guaranteed to find a polynomial \(p \in P \setminus \Sigma \), it is not evident a priori that there exist such polynomials with the \(M \succeq 0\) constraint. Nevertheless, in practice we are able to find many such counterexamples, with nontight relaxations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Brynte, L., Larsson, V., Iglesias, J.P. et al. On the Tightness of Semidefinite Relaxations for Rotation Estimation. J Math Imaging Vis 64, 57–67 (2022). https://doi.org/10.1007/s1085102101054y
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s1085102101054y