Rectification of planar targets using line segments

Abstract

This paper presents a method that performs the rectification of planar objects. Based on the 2D Manhattan world assumption (i.e., the majority of line segments are aligned with principal axes), we develop a cost function whose minimization yields a rectification transform. We parameterize the homography with camera parameters and design a cost function which encodes the measure of line segment alignment. Since there are outliers in the line segment detection, we also develop an iterative optimization scheme for the robust estimation. Experimental results on a range of images with planar objects show that our method performs rectification robustly and accurately.

Appendix: Jacobian matrix of the proposed cost function

The cost function (12) can be minimized via the Levenberg–Marquardt algorithm. For the efficient implementation of the algorithm, we need the derivatives of

$$\begin{aligned} d_\mu ( \mathrm {H}^{-1}\mathbf {u}, \mathrm {H}^{-1}\mathbf {v}) \end{aligned}$$

(24)

and

$$\begin{aligned} \frac{\text {max}\left( a,f \right) }{\text {min}\left( a,f \right) } \end{aligned}$$

(25)

with respect to four parameters(i.e., $t \in \{\theta _1, \theta _2, \theta _3, f\}$). However, the $\min (\cdot ,\cdot )$ function in (24) is not differentiable at some points; therefore, we use a simple approximation:

$$\begin{aligned} \frac{\partial }{\partial {t}} \left( \min ( f(\cdot ), g(\cdot ) ) \right) = \left\{ \begin{array}{ll} \frac{\partial f(\cdot )}{\partial {t}} &{} ~~~~\text{ if } ~~~f(\cdot ) \le g(\cdot )\\ \frac{\partial g(\cdot )}{\partial {t}} &{} \quad \text{ otherwise. } \end{array}\right. \end{aligned}$$

(26)

Although this approximation introduces ambiguities when $f(\cdot ) = g(\cdot )$, this case seldom happens and the approximation works well in practice. We handle $\min (\cdot ,\cdot )$ and $|\cdot |$ functions in a similar manner.

1.1 Derivatives of (24)

Let us denote $\hat{\varvec{p}}=\mathrm {H}^{-1}\mathbf {u} = \begin{bmatrix} \hat{p}_1&\hat{p}_2&\hat{p}_3 \end{bmatrix}^\top$ and $\hat{\varvec{q}}=\mathrm {H}^{-1}\mathbf {v} = \begin{bmatrix} \hat{q}_1&\hat{q}_2&\hat{q}_3 \end{bmatrix}^\top$ and denote their inhomogeneous representation as $\tilde{\varvec{p}} = \begin{bmatrix} x_1(\cdot )&y_1(\cdot ) \end{bmatrix}^\top$ and $\tilde{\varvec{q}} = \begin{bmatrix} x_2(\cdot )&y_2(\cdot ) \end{bmatrix}^\top$, respectively. Then, the derivative of (24) with respect to t

$$\begin{aligned} \frac{\partial }{\partial t} \left[ d_\mu ( \hat{\varvec{p}}, \varvec{\hat{q}} ) \right] = \frac{\partial }{\partial t} \left[ \min \left( \left| x_1 ( \cdot ) - x_2 ( \cdot ) \right| , \left| y_1 ( \cdot ) - y_2 (\cdot ) \right| \right) \right] \end{aligned}$$

(27)

can be decomposed into four cases:

$$ \begin{aligned} \left\{ \begin{array}{ll} \frac{\partial }{\partial t} \left( x_1(\cdot ) - x_2(\cdot ) \right) &{}\quad \text{ if }~~ x_1 (\cdot )> x_2 (\cdot )~ \& ~ |x_1(\cdot ) - x_2(\cdot ) | \le | y_1(\cdot ) - y_2(\cdot ) | \\ \frac{\partial }{\partial t} \left( x_2(\cdot ) - x_1(\cdot ) \right) &{}\quad \text{ if }~~ x_1 (\cdot ) \le x_2 (\cdot )~ \& ~ |x_1(\cdot ) - x_2(\cdot ) | \le | y_1(\cdot ) - y_2(\cdot ) | \\ \frac{\partial }{\partial t} \left( y_2(\cdot ) - y_1(\cdot ) \right) &{}\quad \text{ if }~~ y_1 (\cdot ) \le y_2 (\cdot )~ \& ~ |x_1(\cdot ) - x_2(\cdot ) | > | y_1(\cdot ) - y_2(\cdot ) | \\ \frac{\partial }{\partial t} \left( y_1(\cdot ) - y_2(\cdot ) \right) &{}\quad \text{ otherwise. } \end{array}\right. \end{aligned}$$

(28)

Since we can derive $\frac{\partial \tilde{\varvec{p}}}{\partial t} = \begin{bmatrix} \frac{\partial {x_1(\cdot )}}{\partial t}&\frac{\partial {y_1(\cdot )}}{\partial t} \end{bmatrix}^\top$ by using a chain rule:

$$\begin{aligned} \frac{\partial \tilde{\varvec{p}}}{\partial t} = \frac{\partial \tilde{\varvec{p}}}{\partial \hat{\varvec{p}}} \frac{\partial \hat{\varvec{p}}}{\partial t} \end{aligned}$$

(29)

where

$$\begin{aligned} \frac{\partial \tilde{\varvec{p}}}{\partial \hat{\varvec{p}}} =\frac{\partial \begin{bmatrix} \hat{p}_1/\hat{p}_3&\hat{p}_2/\hat{p}_3 \end{bmatrix}}{\partial \begin{bmatrix} \hat{p}_1&\hat{p}_2&\hat{p}_3 \end{bmatrix}}=\begin{bmatrix} 1/\hat{p}_3&\quad 0&\quad -\hat{p}_1/\hat{p}_3^{2}\\ 0&\quad 1/\hat{p}_3&\quad -\hat{p}_2/\hat{p}_3^{2} \end{bmatrix}, \end{aligned}$$

(30)

it is sufficient to compute $\frac{\partial \hat{\varvec{p}}}{\partial t}$ and $\frac{\partial \hat{\varvec{q}}}{\partial t}$ to get (27). By using (8), we can get $\frac{\partial \hat{\varvec{p}}}{\partial t}$ for each parameter:

$$\begin{aligned} \frac{\partial \hat{\varvec{p}}}{\partial \theta _{1}}= & {} \left\{ \mathbf {I}-\frac{\left( \mathbf {t}+\mathbf {e}_{3} \right) \mathbf {e}_{3}^\top }{\mathbf {e}_{3}^\top \mathbf {t}} \right\} \frac{\partial \mathbf {R}^\top }{\partial \theta _{1}}\mathbf {K}^{-1}\mathbf {u},\end{aligned}$$

(31)

$$\begin{aligned} \frac{\partial \hat{\varvec{p}}}{\partial \theta _{2}}= & {} \left\{ \mathbf {I}-\frac{\left( \mathbf {t}+\mathbf {e}_{3} \right) \mathbf {e}_{3}^\top }{\mathbf {e}_{3}^\top \mathbf {t}} \right\} \frac{\partial \mathbf {R}^\top }{\partial \theta _{2}}\mathbf {K}^{-1}\mathbf {u},\end{aligned}$$

(32)

$$\begin{aligned} \frac{\partial \hat{\varvec{p}}}{\partial \theta _{3}}= & {} \left\{ \mathbf {I}-\frac{\left( \mathbf {t}+\mathbf {e}_{3} \right) \mathbf {e}_{3}^\top }{\mathbf {e}_{3}^\top \mathbf {t}} \right\} \frac{\partial \mathbf {R}^\top }{\partial \theta _{3}}\mathbf {K}^{-1}\mathbf {u},\end{aligned}$$

(33)

$$\begin{aligned} \frac{\partial \hat{\varvec{p}}}{\partial f}= & {} \left\{ \mathbf {I}-\frac{\left( \mathbf {t}+\mathbf {e}_{3} \right) \mathbf {e}_{3}^\top }{\mathbf {e}_{3}^\top \mathbf {t}} \right\} \mathbf {R}^\top \frac{\partial \mathbf {K}^{-1}}{\partial f}\mathbf {u}. \end{aligned}$$

(34)

The derivatives of $\frac{\partial \hat{\varvec{q}}}{\partial t}$ can be derived in a similar way.

1.2 Derivatives of (25)

We can get the derivative of (25) with respect to f similarly:

$$\begin{aligned} \frac{\partial }{\partial f} \left[ \frac{\text {max}\left( a,f \right) }{\text {min}\left( a,f \right) } \right] =\left\{ \begin{array}{ll} \frac{1}{a}, &{}\quad \text{ if } a < f \\ -\frac{a}{f^{2}}, &{}\quad \text{ if } a > f \\ 0, &{}\quad \text{ otherwise }. \end{array}\right. \end{aligned}$$

(35)