Integrating Geometrical Context for Semantic Labeling of Indoor Scenes using RGBD Images

Abstract

Inexpensive structured light sensors can capture rich information from indoor scenes, and scene labeling problems provide a compelling opportunity to make use of this information. In this paper we present a novel conditional random field (CRF) model to effectively utilize depth information for semantic labeling of indoor scenes. At the core of the model, we propose a novel and efficient plane detection algorithm which is robust to erroneous depth maps. Our CRF formulation defines local, pairwise and higher order interactions between image pixels. At the local level, we propose a novel scheme to combine energies derived from appearance, depth and geometry-based cues. The proposed local energy also encodes the location of each object class by considering the approximate geometry of a scene. For the pairwise interactions, we learn a boundary measure which defines the spatial discontinuity of object classes across an image. To model higher-order interactions, the proposed energy treats smooth surfaces as cliques and encourages all the pixels on a surface to take the same label. We show that the proposed higher-order energies can be decomposed into pairwise sub-modular energies and efficient inference can be made using the graph-cuts algorithm. We follow a systematic approach which uses structured learning to fine-tune the model parameters. We rigorously test our approach on SUN3D and both versions of the NYU-Depth database. Experimental results show that our work achieves superior performance to state-of-the-art scene labeling techniques.

Appendix: Disintegration of Higher-Order Energies

In this appendix, we will show how the higher-order energies can be minimized using graph cuts. Since, graph cuts can efficiently minimize submodular functions, we will transform our higher-order energy function (Eq. 9) to a submodular second-order energy function. For the case of both $\alpha \beta$-swap and $\alpha$-expansion move making algorithms, we will explain this transformation and the process of optimal moves computation³. All of the previously defined notations are used in the same context and all of the newly introduced symbols are defined in this section. The function that accounts for the number of disagreeing nodes in a clique is defined as:

$$\begin{aligned} n_{\ell }({y}_{\mathbf {c}}) = \sum \limits _{i \in {\mathbf {c}}} w_i^{\ell } \mathbf {1}_{y_i = \ell } \end{aligned}$$

The function $\mathbf {1}_{y_i = \ell }$ is a zero-one indicator function that returns a unit value when $y_i = \ell$. We suppose here that weights are symmetric for all labels $\ell \in {\mathcal {L}}$ i.e., $w_i^{\ell } = w_i$. Further, for our implementation we set $w_i=1 \;\; \forall i \in {\mathbf {c}}$. This setting satisfies the required constraints for these parameters, i.e.,

$$\begin{aligned} w_i^{\ell } \ge 0 \quad \text {and}\quad \sum \limits _{i \in {\mathbf {c}}} w_i^{\ell } = \# {\mathbf {c}} \;\; \forall \ell \in {\mathcal {L}}. \end{aligned}$$

We define a summation function that adds the weights for a subset $\mathbf {s}$ of ${\mathbf {c}}$,

$$\begin{aligned} W(\mathbf {s}) = \sum \limits _{i\in \mathbf {s}} w_i^{\ell } = \# \mathbf {s} \quad \forall \ell \in {\mathcal {L}}. \end{aligned}$$

1.1 Disintegration of Higher-Order Energies to Second-Order Sub-modular Energies for Swap Moves

Suppose, in a clique ‘${\mathbf {c}}$’, the locations of the active nodes is represented by a set of indices ${\mathbf {c}}_{a}$. The nodes which remain inactive during the move making process are termed the passive nodes. Their locations are denoted by $\bar{{\mathbf {c}}}_{a} = \{{\mathbf {c}}\setminus \forall c_i \in {\mathbf {c}}_{a}\}$. The corresponding set of available moves to the swap move making algorithm are encoded in the form of a vector $\mathbf {t}_{c_a}$. For the sake of a simple demonstration, let us focus on the two class labeling problem i.e., $\ell \in \{0,1\}$. The induced labeling is the combination of the old labeling for the inactive nodes and the new labeling for the active nodes i.e., ${y}^n_c = {y}^{\circ }_{\bar{c}_{a}} \cup T_{\alpha \beta }({y}^{\circ }_{{c}_{a}}, \mathbf {t}_{c_a})$. If ${y}^n_c$ denotes the new labeling induced by move $\mathbf {t}_{c_a}$ and ${y}^{\circ }_c$ denotes the old labeling, we can define the energy of move for an $\alpha \beta$ swap as:

$$\begin{aligned}&\psi ^m_{{\mathbf {c}}}(\mathbf {t}_{c_a}) = \psi _{{\mathbf {c}}}({y}^n_c) = \psi _{{\mathbf {c}}}({y}^{\circ }_{\bar{c}_{a}} \cup T_{\alpha \beta }({y}^{\circ }_{c_{a}}, \mathbf {t}_{c_a}))\\&\quad = \underset{\ell \in {\mathcal {L}}}{{\text {min}}} \left\{ \lambda _{max} - (\lambda _{max} - \lambda _{\ell })\right. \\&\left. \quad \text {exp} {\left( - \frac{W({\mathbf {c}}) - n_{\ell }({y}^{\circ }_{\bar{c}_{a}} \cup T_{\alpha \beta }({y}^{\circ }_{c_{a}}, \mathbf {t}_{c_a}))}{Q_{\ell }}\right) }\right\} \\&\quad = \underset{\ell \in {\mathcal {L}}}{{\text {min}}} \left\{ \lambda _{max} - (\lambda _{max} - \lambda _{\alpha })\text {exp} { \left( - \frac{W({\mathbf {c}}) - n_{0}^m(\mathbf {t}_{c_a})}{Q_{\alpha }}\right) }, \right. \\&\left. \quad \lambda _{max} - (\lambda _{max} - \lambda _{\beta })\text {exp} {\left( - \frac{W({\mathbf {c}} - {\mathbf {c}}_a) + n_{0}^m(\mathbf {t}_{c_a})}{Q_{\beta }}\right) } \right\} , \end{aligned}$$

where, $W({\mathbf {c}}_a) = n_0^m(\mathbf {t}_{c_a}) + n_1^m(\mathbf {t}_{c_a})$. The minimization operation in the above equation can be replaced by defining a piecewise function:

$$\begin{aligned} \psi _{{\mathbf {c}}}^m(\mathbf {t}_{c_a}) = \left\{ \begin{array}{l} \lambda _{max} - (\lambda _{max} - \lambda _{\alpha })\text {exp} { \left( - \frac{W({\mathbf {c}}) - n_{0}^m(\mathbf {t}_{c_a})}{Q_{\alpha }}\right) } \\ \qquad \text {if}\quad n_{0}^m(\mathbf {t}_{c_a}) > \varrho _{\alpha \beta }\left( \frac{W({\mathbf {c}})}{Q_{\alpha }} - \frac{W({\mathbf {c}} - {\mathbf {c}}_a)}{Q_{\beta }} \right. \\ \qquad \qquad \qquad \qquad \left. - \log \left( \frac{\lambda _{max} - \lambda _{\alpha }}{\lambda _{max} - \lambda _{\beta }}\right) \right) ,\\ \lambda _{max} - (\lambda _{max} - \lambda _{\beta })\text {exp} {\left( - \frac{W({\mathbf {c}} - {\mathbf {c}}_a) + n_{0}^m(\mathbf {t}_{c_a})}{Q_{\beta }}\right) }\\ \qquad \text {if}\quad n_{0}^m(\mathbf {t}_{c_a}) < \varrho _{\alpha \beta }\left( \frac{W({\mathbf {c}})}{Q_{\alpha }} - \frac{W({\mathbf {c}} - {\mathbf {c}}_a)}{Q_{\beta }} \right. \\ \qquad \qquad \qquad \qquad \left. - \log \left( \frac{\lambda _{max} - \lambda _{\alpha }}{\lambda _{max} - \lambda _{\beta }}\right) \right) , \end{array}\right. \end{aligned}$$

where, $\varrho _{\alpha \beta } = \frac{Q_{\alpha }Q_{\beta }}{Q_{\alpha }+Q_{\beta }}$. The function $n^m_{\ell }(\mathbf {t}_{c_a})$ is defined as:

$$\begin{aligned} n^m_{\ell }(\mathbf {t}_{c_a}) = \sum \limits _{i \in {\mathbf {c}}_{a}}w_i \delta _{\ell }(\mathbf {t}_i). \end{aligned}$$

From Theorem 1 in Kohli et al. (2009), the energy defined above can be transformed to the submodular quadratic pseudo-boolean function with two binary meta variables. In this form the $\alpha \beta$-swap algorithm can be used for minimizing the energy function.

1.2 Disintegration of Higher-Order Energies to Second-Order Sub-modular Energies for Expansion Moves

Suppose, in a clique ‘ c’, the location of the nodes with label $\ell$ is represented by a set of indices ${\mathbf {c}}_{\ell }$. The current labeling solution is denoted by ${y}_{{\mathbf {c}}}^{\circ }$.

If the dominant label is denoted by $d \in {\mathcal {L}}$ in the current labeling ${y}_{{\mathbf {c}}}^{\circ }$ is,

$$\begin{aligned} \text {s.t} \quad W({\mathbf {c}}_d) > W({\mathbf {c}}) - Q_d \quad \text {where} \;d \ne \alpha , \end{aligned}$$

there must be one dominant label:

$$\begin{aligned}&Q_a + Q_b < W({\mathbf {c}}) \qquad \forall a \ne b \in {\mathcal {L}},\\&\begin{array}{l} \psi _{{\mathbf {c}}}^{m}(t_c) = \psi _{{\mathbf {c}}} (T_{\alpha }({y}_c^{\circ }, t_c)) \\ = \underset{\ell \in {\mathcal {L}}}{{\text {min}}} \left\{ \lambda _{max} - (\lambda _{max} - \lambda _{\alpha }) {\text {exp}} \left( - \frac{\sum \limits _{i\in c} w_i t_i}{Q_{\alpha }}\right) , \right. \\ \left. \lambda _{max} - (\lambda _{max} - \lambda _{d}) \text {exp} \left( - \frac{W({\mathbf {c}}) - \sum \limits _{i\in c} w_i t_i}{Q_{d}}\right) \right\} . \end{array} \end{aligned}$$

The minimization operator in the above function can be replaced by a piecewise function:

$$\begin{aligned} \psi _{{\mathbf {c}}}^m(\mathbf {t}_{c}, \mathbf {t}_{c_d}) = \left\{ \begin{array}{l} \lambda _{max} - (\lambda _{max} - \lambda _{\alpha })\text {exp} {\left( - \frac{n_{0}^m(\mathbf {t}_{c})}{Q_{\alpha }}\right) } \\ \qquad \text {if}\quad n_{0}^m(\mathbf {t}_{c}) > \varrho _{\alpha d}\left( \frac{W({\mathbf {c}})}{Q_{\alpha }} \right. \\ \qquad \qquad \qquad \qquad \left. - \log \left( \frac{\lambda _{max} - \lambda _{\alpha }}{\lambda _{max} - \lambda _{d}}\right) \right) , \\ \lambda _{max} - (\lambda _{max} - \lambda _{d})\text {exp} {\left( - \frac{W({\mathbf {c}}) - n_{0}^m(\mathbf {t}_{c_d})}{Q_{d}}\right) }\\ \qquad \text {if}\quad n_{0}^m(\mathbf {t}_{c}) < \varrho _{\alpha d}\left( \frac{W({\mathbf {c}})}{Q_{\alpha }} \right. \\ \qquad \qquad \qquad \qquad \left. - \log \left( \frac{\lambda _{max} - \lambda _{\alpha }}{\lambda _{max} - \lambda _{d}}\right) \right) , \end{array}\right. \end{aligned}$$

where, $\varrho _{\alpha d} = \frac{Q_{\alpha }Q_{d}}{Q_{\alpha }+Q_{d}}$ and function $n^m_{\ell }(\mathbf {t}_c)$ is defined as:

$$\begin{aligned} n^m_{\ell }(\mathbf {t}_c) = \sum \limits _{i\in {\mathbf {c}}}w_i \delta _{\ell }(\mathbf {t}_i). \end{aligned}$$

From Theorem 2 in Kohli et al. (2009), the energy defined above can be transformed to the submodular quadratic pseudo-boolean function with two binary meta variables. In this form the $\alpha$-expansion algorithm can be used for minimizing the energy function.