Stereo Matching: Fundamentals, State-of-the-Art, and Existing Challenges

Abstract

Stereo matching is the process of generating dense correspondences in stereo images in order to create a disparity map for depth perception. Stereo matching is different from flow estimation task due to stereo rectification, which ensures that correspondences are always co-linear in a pair of stereo images. Stereo vision has become increasingly popular in mobile devices, such as autonomous cars and unmanned aerial vehicles, thanks to recent advances in full-feature embedded microcomputers. However, due to limited computing resources, there is a growing need for stereo matching algorithms that strike a balance between disparity estimation accuracy and efficiency. Challenges in this field include the lack of disparity ground truth, domain adaptation, and intractable areas such as occlusions. This chapter covers the fundamentals of stereopsis, including the perspective camera model and epipolar geometry, and reviews the most advanced stereo matching algorithms. It also explores disparity confidence measures, disparity estimation evaluation metrics, and publicly available datasets and benchmarks, before summarizing the outstanding challenges in this field.

Appendices

Appendix

Lie Group SO(3)

In the three-dimensional space, the coordinates of a 3D point in two coordinate systems are $\boldsymbol{x}_{1}=[x_{1},y_{1},z_{1}]^\top$ and $\boldsymbol{x}_{2}=[x_{2},y_{2},z_{2}]^\top \in \mathbb {R}^{3\times 1}$, respectively. $\boldsymbol{x}_{1}$ can be transformed into $\boldsymbol{x}_{2}$ using a rotation matrix $\boldsymbol{R}\in \mathbb {R}^{3\times 3}$ and a translation vector $\boldsymbol{t}\in \mathbb {R}^{3\times 1}$:

$$\begin{aligned} \boldsymbol{x_2}=\boldsymbol{R}\boldsymbol{x_1}+\boldsymbol{t}, \end{aligned}$$

(A.1)

where $\boldsymbol{R}$ satisfies orthogonality:

$$\begin{aligned} \boldsymbol{R}^\top =\boldsymbol{R}^{-1}\ \ \text {and} \ \ |\text {det}(\boldsymbol{R})|=1, \end{aligned}$$

(A.2)

where $\text {det}(\boldsymbol{R})$ represents the determinant of $\boldsymbol{R}$. The subgroup of orthogonal matrices with $\text {det}(\boldsymbol{R})=+1$ is referred to as a special orthogonal group and is denoted as SO(3).

Skew-Symmetric Matrix

In linear algebra, a skew-symmetric matrix $\boldsymbol{A}$ satisfies the following property:

$$\begin{aligned} \boldsymbol{A}\text { is a skew-symmetrix matrix}\Leftrightarrow \boldsymbol{A}^\top =-\boldsymbol{A}. \end{aligned}$$

(B.1)

In 3D computer vision, the skew-symmetric matrix $[\boldsymbol{a}]_{\times }$ of a vector $\boldsymbol{a}=[a_{1},a_{2},a_{3}]$ is defined as [104]:

$$\begin{aligned}{}[\boldsymbol{a}]_{\times }=\left[ \begin{matrix} 0 &{} -{{a}_{3}} &{} {{a}_{2}} \\ {{a}_{3}} &{} 0 &{} -{{a}_{1}} \\ -{{a}_{2}} &{} {{a}_{1}} &{} 0 \\ \end{matrix} \right] . \end{aligned}$$

(B.2)

The two properties of a skew-symmetric matrix are as follows:

$$\begin{aligned} \boldsymbol{a}^\top [\boldsymbol{a}]_{\times } = \boldsymbol{0}^\top , \text { } [\boldsymbol{a}]_{\times } \boldsymbol{a} = \boldsymbol{0}, \end{aligned}$$

(B.3)

where $\boldsymbol{0}=[0,0,0]^\top$ is a zero vector. Furthermore, a skew-symmetric matrix can also represent cross product as matrix multiplication. Specifically, for two vectors $\boldsymbol{a}$ and $\boldsymbol{b}$, their cross product can be expressed as [104]:

$$\begin{aligned} \boldsymbol{a} \times \boldsymbol{b} = [\boldsymbol{a}]_{\times } \boldsymbol{b} = -[\boldsymbol{b}]_{\times } \boldsymbol{a}. \end{aligned}$$

(B.4)