3D Scene Flow Estimation with a Piecewise Rigid Scene Model

Abstract

3D scene flow estimation aims to jointly recover dense geometry and 3D motion from stereoscopic image sequences, thus generalizes classical disparity and 2D optical flow estimation. To realize its conceptual benefits and overcome limitations of many existing methods, we propose to represent the dynamic scene as a collection of rigidly moving planes, into which the input images are segmented. Geometry and 3D motion are then jointly recovered alongside an over-segmentation of the scene. This piecewise rigid scene model is significantly more parsimonious than conventional pixel-based representations, yet retains the ability to represent real-world scenes with independent object motion. It, furthermore, enables us to define suitable scene priors, perform occlusion reasoning, and leverage discrete optimization schemes toward stable and accurate results. Assuming the rigid motion to persist approximately over time additionally enables us to incorporate multiple frames into the inference. To that end, each view holds its own representation, which is encouraged to be consistent across all other viewpoints and frames in a temporal window. We show that such a view-consistent multi-frame scheme significantly improves accuracy, especially in the presence of occlusions, and increases robustness against adverse imaging conditions. Our method currently achieves leading performance on the KITTI benchmark, for both flow and stereo.

Appendix: Higher-Order Reductions for Occlusion Handling with a Reference View

We here describe how to convert the occlusion-sensitive data term from Eq. (17) into a quadratic pseudo-Boolean function. Note that the only interesting case is $|\fancyscript{O}_\mathbf {p}^0| \!\ge \! 2$, that is there are two or more possibly occluding pixels. Otherwise, the problem is already in quadratic form $(|\fancyscript{O}_\mathbf {p}^0| \!=\! 1)$, or there is no occluding pixel and only the (unary) data term is required $(|\fancyscript{O}_\mathbf {p}^0| \!=\! 0)$.

Recall that Eq. (17) is defined as part of a single $\alpha$-expansion step, i.e. a pixel can only be assigned two possible labels ($\alpha$ or its previous label). For simplicity we restrict the analysis to the case $i \!=\! 0$. We thus consider the term

$$\begin{aligned} \hat{u}^0_{\mathbf {p}} [x_\mathbf {p}=0] \!\! \prod _{(\mathbf {q},j) \in \fancyscript{O}_\mathbf {p}^0} \!\! [x_\mathbf {q}\ne j]. \end{aligned}$$

(25)

The reduction for $i \!=\! 1$ is analogous.

First, let us consider the special case in which there is a pixel $\mathbf {q}$ that occludes pixel $\mathbf {p}$ in both possible assignments of $x_{\mathbf {q}}$, that is $(\mathbf {q},0) \in \fancyscript{O}_\mathbf {p}^0$ and $(\mathbf {q},1) \in \fancyscript{O}_\mathbf {p}^0$. In that case the pixel $\mathbf {p}$ is always occluded and Eq. (25) vanishes. For the remaining cases, we distinguish between $\hat{u}^0_{\mathbf {p}} \!<\! 0$ and $\hat{u}^0_{\mathbf {p}} \!>\! 0$.

Case $\hat{u}^0_{\mathbf {p}} <0$: We can substitute the whole term with the help of at most one non-submodular term with weight $\hat{u}^0_{\mathbf {p}}$. No non-submodular term is introduced if all Boolean variables in the term are inverted, i.e.  $j\equiv 1$. In that case Eq. (25) becomes

$$\begin{aligned} \hat{u}^0_{\mathbf {p}} (1-x_\mathbf {p}) \!\! \prod _{(\mathbf {q},1) \in \fancyscript{O}_\mathbf {p}^0} \!\!(1-x_\mathbf {q}). \end{aligned}$$

(26)

Introducing an additional variable $z$, the polynomial in Eq. (26) can be replaced by

$$\begin{aligned} \begin{aligned} \min _{z} \hat{u}^0_{\mathbf {p}} \Bigg ( 1-z - (1-z)x_\mathbf {p} - \!\!\!\! \sum _{(\mathbf {q},1)\in \fancyscript{O}_\mathbf {p}^0}\!\! (1-z)x_\mathbf {q} \Bigg ) \end{aligned} \end{aligned}$$

(27)

in quadratic form.

If $x_\mathbf {p} \!=\! 0$ and the other variables encode a constellation where $\mathbf {p}$ is not occluded, then the expression becomes equal to $\hat{u}^0_{\mathbf {p}}$ (by setting $z \!=\! 0$). Otherwise, the minimum is attained at $0$ (with $z \!=\! 1$).

In the case of there being a $(\mathbf {q},0)\in \fancyscript{O}_\mathbf {p}^0$, we follow the scheme introduced by Rother et al. (2009). With the introduction of two auxiliary variables $z_0,z_1$, we replace the product in Eq. (25) by

$$\begin{aligned} \begin{aligned} \min _{z_0,z_1}&- \hat{u}^0_{\mathbf {p}}( z_0 z_1 - z_1 +(1-z_0)x_\mathbf {p}) \\&- \hat{u}^0_{\mathbf {p}} \sum _{(\mathbf {q},j)\in \fancyscript{O}_\mathbf {p}^0} \Big (z_1(1-x_\mathbf {q}) + (1-z_0) x_\mathbf {q}\Big ). \end{aligned} \end{aligned}$$

(28)

Here, the term $-\hat{u}^0_{\mathbf {p}} z_0 z_1$ is not submodular. Like in the previous case, if the variables do not encode an occlusion, and if $x_\mathbf {p}\!=\!0$, the minimum is $\hat{u}^0_{\mathbf {p}}$ (setting $z_0\!=\!0$ and $z_1\!=\!1$). Otherwise the minimum is 0 (setting $z_0\!=\!1$ and $z_1\!=\!0$).

Case $\hat{u}^0_{\mathbf {p}} >0$: We approach this problem using a series of substitutions. Following Ali et al. (2008), we replace a product of two variables in Eq. (25), $x_{\mathbf {q}_1} x_{\mathbf {q}_2}$, with a new variable $z$, and add

$$\begin{aligned} \min _z \hat{u}^0_{\mathbf {p}} (x_{\mathbf {q}_1} x_{\mathbf {q}_2} - 2 x_{\mathbf {q}_1} z - 2 x_{\mathbf {q}_2} z + 3 z), \end{aligned}$$

(29)

such that after the substitution Eq. (25) becomes

$$\begin{aligned} \begin{aligned}&\hat{u}^0_{\mathbf {p}} (x_{\mathbf {q}_1} x_{\mathbf {q}_2} - 2 x_{\mathbf {q}_1} z - 2 x_{\mathbf {q}_2} z + 3 z ) + \\&\hat{u}^0_{\mathbf {p}} (1-x_\mathbf {p}) z \!\! \!\!\!\! \prod _{\begin{array}{c} (\mathbf {q},j) \in \fancyscript{O}_\mathbf {p}^0 \setminus \\ \{(\mathbf {q}_1,0), (\mathbf {q}_2,0)\} \end{array}} \!\!\!\! [x_\mathbf {q}\ne j]. \end{aligned} \end{aligned}$$

(30)

Two inverted Boolean variables can be replaced in the same manner. Note that we are not restricted to replacing only variables from $\fancyscript{O}_\mathbf {p}^0$, but can also substitute $1-x_\mathbf {p}$ itself.

The substitution introduces one non-submodular term with weight $\hat{u}^0_{\mathbf {p}}$. To arrive at a quadratic polynomial one needs to replace all but two literals of the product as described, leading to $n-1$ or $n-2$ non-submodular terms.