A Contour Completion Model for Augmenting Surface Reconstructions

  • Nathan Silberman
  • Lior Shapira
  • Ran Gal
  • Pushmeet Kohli
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8691)


The availability of commodity depth sensors such as Kinect has enabled development of methods which can densely reconstruct arbitrary scenes. While the results of these methods are accurate and visually appealing, they are quite often incomplete. This is either due to the fact that only part of the space was visible during the data capture process or due to the surfaces being occluded by other objects in the scene. In this paper, we address the problem of completing and refining such reconstructions. We propose a method for scene completion that can infer the layout of the complete room and the full extent of partially occluded objects. We propose a new probabilistic model, Contour Completion Random Fields, that allows us to complete the boundaries of occluded surfaces. We evaluate our method on synthetic and real world reconstructions of 3D scenes and show that it quantitatively and qualitatively outperforms standard methods. We created a large dataset of partial and complete reconstructions which we will make available to the community as a benchmark for the scene completion task. Finally, we demonstrate the practical utility of our algorithm via an augmented-reality application where objects interact with the completed reconstructions inferred by our method.


3D Reconstruction Scene Completion Surface Reconstruction Contour Completion 


  1. 1.
    Henry, P., Krainin, M., Herbst, E., Ren, X., Fox, D.: Rgb-d mapping: Using depth cameras for dense 3d modeling of indoor environments. In: ISER, vol. 20, pp. 22–25 (2010)Google Scholar
  2. 2.
    Newcombe, R.A., Davison, A.J., Izadi, S., Kohli, P., Hilliges, O., Shotton, J., Molyneaux, D., Hodges, S., Kim, D., Fitzgibbon, A.: Kinectfusion: Real-time dense surface mapping and tracking. In: ISMAR (2011)Google Scholar
  3. 3.
    Qualcomm: Vuforia. Website (2011),
  4. 4.
    Curless, B., Levoy, M.: A volumetric method for building complex models from range images. In: CGI, pp. 303–312 (1996)Google Scholar
  5. 5.
    Lorensen, W.E., Cline, H.E.: Marching cubes: A high resolution 3d surface construction algorithm. In: Proceedings of the 14th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH (1987)Google Scholar
  6. 6.
    Furukawa, Y., Curless, B., Seitz, S.M., Szeliski, R.: Reconstructing building interiors from images. In: ICCV (2009)Google Scholar
  7. 7.
    Furukawa, Y., Curless, B., Seitz, S., Szeliski, R.: Manhattan-world stereo. In: CVPR (2009)Google Scholar
  8. 8.
    Salas-Moreno, R.F., Newcombe, R.A., Strasdat, H., Kelly, P.H., Davison, A.J.: Slam++: Simultaneous localisation and mapping at the level of objects. In: CVPR (June 2013)Google Scholar
  9. 9.
    Barinova, O., Lempitsky, V., Tretiak, E., Kohli, P.: Geometric image parsing in man-made environments. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010, Part II. LNCS, vol. 6312, pp. 57–70. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  10. 10.
    Hoiem, D., Hedau, V., Forsyth, D.: Recovering free space of indoor scenes from a single image (2012)Google Scholar
  11. 11.
    Guo, R., Hoiem, D.: Beyond the line of sight: Labeling the underlying surfaces. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012, Part V. LNCS, vol. 7576, pp. 761–774. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  12. 12.
    Guo, R., Hoiem, D.: Support surface prediction in indoor scenes. In: ICCV (2013)Google Scholar
  13. 13.
    Kim, Y.M., Mitra, N.J., Yan, D.M., Guibas, L.: Acquiring 3d indoor environments with variability and repetition. ACM Transactions on Graphics 31(6) (2012)Google Scholar
  14. 14.
    Soo Kim, B., Kohli, P., Saverese, S.: 3d scene understanding by voxel-crf. In: ICCV (2013)Google Scholar
  15. 15.
    Zheng, B., Zhao, Y., Yu, J.C., Ikeuchi, K., Zhu, S.C.: Beyond point clouds: Scene understanding by reasoning geometry and physics. In: CVPR (2013)Google Scholar
  16. 16.
    Nemhauser, G.L., Wolsey, L.A., Fisher, M.L.: An analysis of approximations for maximizing submodular set functions–I. Mathematical Programming 14(1), 265–294 (1978)CrossRefzbMATHMathSciNetGoogle Scholar
  17. 17.
    Duda, R.O., Hart, P.E.: Use of the hough transformation to detect lines and curves in pictures. Communications of the ACM 15(1), 11–15 (1972)CrossRefGoogle Scholar
  18. 18.
    Silberman, N., Hoiem, D., Kohli, P., Fergus, R.: Indoor segmentation and support inference from RGBD images. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012, Part V. LNCS, vol. 7576, pp. 746–760. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  19. 19.
    Jegelka, S., Bilmes, J.: Submodularity beyond submodular energies: Coupling edges in graph cuts. In: CVPR (2011)Google Scholar
  20. 20.
    Kohli, P., Osokin, A., Jegelka, S.: A principled deep random field model for image segmentation. In: CVPR (2013)Google Scholar
  21. 21.
    Kohli, P., Torr, P.H., et al.: Robust higher order potentials for enforcing label consistency. IJCV 82(3) (2009)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Nathan Silberman
    • 1
  • Lior Shapira
    • 2
  • Ran Gal
    • 2
  • Pushmeet Kohli
    • 3
  1. 1.Courant InstituteNew York UniversityUSA
  2. 2.Microsoft ResearchUSA
  3. 3.Microsoft ResearchCambridgeUK

Personalised recommendations