Content-Aware Semantic Indoor Scene Modeling from a Single Image

Abstract

Digitalizing indoor scenes into a 3D virtual world enables people to visit and roam in their daily-life environments through remote devices. However, reconstructing indoor geometry with enriched semantics (e.g. the room layout, object category and support relationship) requires computers to parse and holistically understand the scene context, which is challenging considering the complexity and clutter of our living surroundings. However, with the rising development of deep learning techniques, modeling indoor scenes from single RGB images has been available. In this chapter, we introduce an automatic method for semantic indoor scene modeling based on deep convolutional features. Specifically, we decouple the task of indoor scene modeling into different hierarchies of scene understanding subtasks to parse semantic and geometric contents from scene images (i.e. object masks, scene depth map and room layout). Above these semantic and geometric contents, we deploy a data-driven support relation inference to estimate the physical contact between indoor objects. Under the support context, we adopt an image-CAD matching strategy to retrieve an indoor scene from global searching to local fine-tuning. The experiments show that this method can retrieve CAD models efficiently with enriched semantics, and demonstrate its feasibility in handling serious object occlusions.

Appendix: Parameter Decision

In image segmentation and room corner searching, the training configurations and parameters setup are followed with Li et al. (2016) and Nie et al. (2018). In object modeling, we set $\mathbf {d}_{1}=[0.5, 0.5, 0.5]^\mathbf{T }$ (in meters, the same below) for normal objects. For those supported by a wall, $\mathbf {d}_{1}$ is set as $[0.2, \infty , \infty ]^\mathbf{T }$ or $[\infty , 0.2, \infty ]^\mathbf{T }$ depending on the orientation of the wall. $\mathbf {d}_{2}$ is set as $[1.0, 1.0, 0.5]^{\text {T}}$ as the point cloud is noisier in horizontal plane than in the vertical direction (see Fig. 8.8). For model scales, we set $\rho _{1}^{\text {L}}=\rho _{2}^{\text {L}}=\rho _{3}^{\text {L}}=0.8$, $\rho _{1}^{\text {U}}=\rho _{2}^{\text {U}}=1.2$, and $\rho _{3}^{\text {U}}=1.0$. While for objects whose top part is occluded, the point cloud could underestimate the model height size. We hence change the lower bounds to $\rho _{1}^{\text {L}}=\rho _{2}^{\text {L}}=\rho _{3}^{\text {L}}=1.0$, and the upper bounds to $\rho _{1}^{\text {U}}=\rho _{2}^{\text {U}}=\rho _{3}^{\text {U}}=2.0$ or more. In global searching, the maximal iterations number is limited to 50, while in the local matching, generally we do not set the maximal iteration number to ensure convergence, the only stopping criteria is set as when the absolute tolerance reaches $10^{-3}$.