Computational Visual Media

, Volume 3, Issue 2, pp 131–146 | Cite as

View suggestion for interactive segmentation of indoor scenes

  • Sheng YangEmail author
  • Jie Xu
  • Kang Chen
  • Hongbo Fu
Open Access
Research Article


Point cloud segmentation is a fundamental problem. Due to the complexity of real-world scenes and the limitations of 3D scanners, interactive segmentation is currently the only way to cope with all kinds of point clouds. However, interactively segmenting complex and large-scale scenes is very time-consuming. In this paper, we present a novel interactive system for segmenting point cloud scenes. Our system automatically suggests a series of camera views, in which users can conveniently specify segmentation guidance. In this way, users may focus on specifying segmentation hints instead of manually searching for desirable views of unsegmented objects, thus significantly reducing user effort. To achieve this, we introduce a novel view preference model, which is based on a set of dedicated view attributes, with weights learned from a user study. We also introduce support relations for both graph-cut-based segmentation and finding similar objects. Our experiments show that our segmentation technique helps users quickly segment various types of scenes, outperforming alternative methods.


point cloud segmentation view suggestion interactive segmentation 



This work was supported by the Joint NSFC–ISF Research Program (Project No. 61561146393), the National Natural Science Foundation of China (Project No. 61521002), the Research Grant of Beijing Higher Institution Engineering Research Center, and the Tsinghua–Tencent Joint Laboratory for Internet Innovation Technology. The work was partially supported by grants from the Research Grants Council of the Hong Kong Special Administrative Region, China (Project Nos. CityU113513 and CityU11300615).

Supplementary material

41095_2017_78_MOESM1_ESM.mp4 (51.9 mb)
Supplementary material, approximately 51.8 MB.
41095_2017_78_MOESM2_ESM.pdf (2.4 mb)
View suggestion for interactive segmentation of indoor scenes


  1. [1]
    Lai, K.; Bo, L.; Ren, X.; Fox, D. Detection-based object labeling in 3D scenes. In: Proceedings of the IEEE International Conference on Robotics and Automation, 1330–1337, 2012.Google Scholar
  2. [2]
    Johnson, A. E.; Hebert, M. Using spin images for efficient object recognition in cluttered 3D scenes. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 21, No. 5, 433–449, 1999.CrossRefGoogle Scholar
  3. [3]
    Zheng, B.; Zhao, Y.; Yu, J. C.; Ikeuchi, K.; Zhu, S. C. Beyond point clouds: Scene understanding by reasoning geometry and physics. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 3127–3134, 2013.Google Scholar
  4. [4]
    Holz, D.; Behnke, S. Fast range image segmentation and smoothing using approximate surface reconstruction and region growing. In: Intelligent Autonomous Systems 12. Lee, S.; Cho, H.; Yoon, K.-J.; Lee, J. Eds. Springer Berlin Heidelberg, 61–73, 2013.CrossRefGoogle Scholar
  5. [5]
    Rabbani, T.; van den Heuvel, F. A.; Vosselmann, G. Segmentation of point clouds using smoothness constraint. International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences Vol. 36, No. 5, 248–253, 2006.Google Scholar
  6. [6]
    Boykov, Y.; Funka-Lea, G. Graph cuts and efficient N-D image segmentation. International Journal of Computer Vision Vol. 70, No. 2, 109–131, 2006.CrossRefGoogle Scholar
  7. [7]
    Golovinskiy, A.; Funkhouser, T. Min-cut based segmentation of point clouds. In: Proceedings of the IEEE 12th International Conference on Computer Vision Workshops, 39–46, 2009.Google Scholar
  8. [8]
    Sedlacek, D.; Zara, J. Graph cut based pointcloud segmentation for polygonal reconstruction. In: Advances in Visual Computing. Bebis, G.; Boyle, R.; Parvin, B.; Koracin, D. et al. Eds. Springer Berlin Heidelberg, 218–227, 2009.CrossRefGoogle Scholar
  9. [9]
    Kim, Y. M.; Mitra, N. J.; Yan, D.-M.; Guibas, L. Acquiring 3D indoor environments with variability and repetition. ACM Transactions on Graphics Vol. 31, No. 6, Article No. 138, 2012.Google Scholar
  10. [10]
    Nan, L.; Xie, K.; Sharf, A. A search-classify approach for cluttered indoor scene understanding. ACM Transactions on Graphics Vol. 31, No. 6, Article No. 137, 2012.Google Scholar
  11. [11]
    Silberman, N.; Hoiem, D.; Kohli, P.; Fergus, R. Indoor segmentation and support inference from RGBD images. In: Computer Vision–ECCV 2012. Fitzgibbon, A.; Lazebnik, S.; Perona, P.; Sato, Y.; Schmid, C. Eds. Springer Berlin Heidelberg, 746–760, 2012.CrossRefGoogle Scholar
  12. [12]
    Nguyen, A.; Le, B. 3D point cloud segmentation: A survey. In: Proceedings of the 6th IEEE Conference on Robotics, Automation and Mechatronics, 225–230, 2013.Google Scholar
  13. [13]
    Shao, T.; Xu, W.; Zhou, K.; Wang, J.; Li, D.; Guo, B. An interactive approach to semantic modeling of indoor scenes with an RGBD camera. ACM Transactions on Graphics Vol. 31, No. 6, Article No. 136, 2012.Google Scholar
  14. [14]
    Yuan, X.; Xu, H.; Nguyen, M. X.; Shesh, A.; Chen, B. Sketch-based segmentation of scanned outdoor environment models. In: Proceedings of the Eurographics Workshop on Sketch-Based Interfaces and Modeling, 19–26, 2005.Google Scholar
  15. [15]
    Silberman, N.; Fergus, R. Indoor scene segmentation using a structured light sensor. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, 601–608, 2011.Google Scholar
  16. [16]
    Xiao, J.; Owens, A.; Torralba, A. SUN3D: A database of big spaces reconstructed using SfM and object labels. In: Proceedings of the IEEE International Conference on Computer Vision, 1625–1632, 2013.Google Scholar
  17. [17]
    Chang, A. X.; Funkhouser, T.; Guibas, L.; Hanrahan, P.; Huang, Q.; Li, Z.; Savarese, S.; Savva, M.; Song, S.; Su, H.; Xiao, J.; Yi, L.; Yu, F. ShapeNet: An information-rich 3D model repository. arXiv preprint arXiv:1512.03012, 2015.Google Scholar
  18. [18]
    Hinterstoisser, S.; Lepetit, V.; Ilic, S.; Holzer, S.; Bradski, G. R.; Konolige, K.; Navab, N. Model based training, detection and pose estimation of texture-less 3D objects in heavily cluttered scenes. In: Computer Vision–ACCV 2012. Lee, K. M.; Matsushita, Y.; Rehg, J. M.; Hu, Z. Eds. Springer Berlin Heidelberg, 548–562, 2012.Google Scholar
  19. [19]
    Chen, K.; Lai, Y.-K.; Wu, Y.-X.; Martin, R.; Hu, S.-M. Automatic semantic modeling of indoor scenes from low-quality RGB-D data using contextual information. ACM Transactions on Graphics Vol. 33, No. 6, Article No. 208, 2014.Google Scholar
  20. [20]
    Silberman, N.; Sontag, D.; Fergus, R. Instance segmentation of indoor scenes using a coverage loss. In: Computer Vision–ECCV 2014. Fleet, D.; Pajdla, T.; Schiele, B.; Tuytelaars, T. Eds. Springer International Publishing, 616–631, 2014.Google Scholar
  21. [21]
    Chen, K.; Lai, Y. K.; Hu, S.-M. 3D indoor scene modeling from RGB-D data: a survey. Computational Visual Media Vol. 1, No. 4, 267–278, 2015.CrossRefGoogle Scholar
  22. [22]
    Shen, C.-H.; Huang, S.-S.; Fu, H.; Hu, S.-M. Adaptive partitioning of urban facades. ACM Transactions on Graphics Vol. 30, No. 6, Article No. 184, 2011.Google Scholar
  23. [23]
    Zhang, H.; Xu, K.; Jiang, W.; Lin, J.; Cohen-Or, D.; Chen, B. Layered analysis of irregular facades via symmetry maximization. ACM Transactions on Graphics Vol. 32, No. 4, Article No. 121, 2013.Google Scholar
  24. [24]
    Mattausch, O.; Panozzo, D.; Mura, C.; Sorkine-Hornung, O.; Pajarola, R. Object detection and classification from large-scale cluttered indoor scans. Computer Graphics Forum Vol. 33, No. 2, 11–21, 2014.CrossRefGoogle Scholar
  25. [25]
    Valentin, J.; Vineet, V.; Cheng, M.-M.; Kim, D.; Shotton, J.; Kohli, P.; Nießner, M.; Criminisi, A.; Izadi, S.; Torr, P. SemanticPaint: Interactive 3D labeling and learning at your fingertips. ACM Transactions on Graphics Vol. 34, No. 5, Article No. 154, 2015.Google Scholar
  26. [26]
    Wong, Y.-S.; Chu, H.-K.; Mitra, N. J. SmartAnnotator an interactive tool for annotating indoor RGBD images. Computer Graphics Forum Vol. 34, No. 2, 447–457, 2015.CrossRefGoogle Scholar
  27. [27]
    Christie, M.; Olivier, P. Camera control in computer graphics: Models, techniques and applications. In: Proceedings of the ACM SIGGRAPH ASIA 2009 Courses, Article No. 3, 2009.Google Scholar
  28. [28]
    Scott, W. R.; Roth, G.; Rivest, J.-F. View planning for automated three-dimensional object reconstruction and inspection. ACM Computing Surveys Vol. 35, No. 1, 64–96, 2003.CrossRefGoogle Scholar
  29. [29]
    Secord, A.; Lu, J.; Finkelstein, A.; Singh, M.; Nealen, A. Perceptual models of viewpoint preference. ACM Transactions on Graphics Vol. 30, No. 5, Article No. 109, 2011.Google Scholar
  30. [30]
    Vázquez, P.-P.; Feixas, M.; Sbert, M.; Heidrich, W. Viewpoint selection using viewpoint entropy. In: Proceedings of the Vision Modeling and Visualization Conference, 273–280, 2001.Google Scholar
  31. [31]
    Andújar, C.; Vázquez, P.; Fairén, M. Way-Finder: Guided tours through complex walkthrough models. Computer Graphics Forum Vol. 23, No. 3, 499–508, 2004.CrossRefGoogle Scholar
  32. [32]
    Li, T.-Y.; Lien, J.-M.; Chiu, S.-Y.; Yu, T.-H. Automatically generating virtual guided tours. In: Proceedings of the Computer Animation, 99–106, 1999.Google Scholar
  33. [33]
    Christie, M.; Languénou, E. A constraint-based approach to camera path planning. In: Smart Graphics. Butz, A.; Krüger, A.; Olivier, P. Eds. Springer Berlin Heidelberg, 172–181, 2003.CrossRefGoogle Scholar
  34. [34]
    Salomon, B.; Garber, M.; Lin, M. C.; Manocha, D. Interactive navigation in complex environments using path planning. In: Proceedings of the Symposium on Interactive 3D Graphics, 41–50, 2003.Google Scholar
  35. [35]
    Choi, S.; Zhou, Q.-Y.; Koltun, V. Robust reconstruction of indoor scenes. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 5556–5565, 2015.Google Scholar
  36. [36]
    Newcombe, R. A.; Izadi, S.; Hilliges, O.; Molyneaux, D.; Kim, D.; Davison, A. J.; Kohli, P.; Shotton, J.; Hodges, S.; Fitzgibbon, A. KinectFusion: Real-time dense surface mapping and tracking. In: Proceedings of the 10th IEEE International Symposium on Mixed and Augmented Reality, 127–136, 2011.Google Scholar
  37. [37]
    Ikehata, S.; Yang, H.; Furukawa, Y. Structured indoor modeling. In: Proceedings of the IEEE International Conference on Computer Vision, 1323–1331, 2015.Google Scholar
  38. [38]
    Furukawa, Y.; Curless, B.; Seitz, S. M.; Szeliski, R. Manhattan-world stereo. In: Proceeding of the IEEE Conference on Computer Vision and Pattern Recognition, 1422–1429, 2009.Google Scholar
  39. [39]
    Asha, V.; Bhajantri, N. U.; Nagabhushan, P. GLCM-based chi-square histogram distance for automatic detection of defects on patterned textures. International Journal of Computational Vision and Robotics Vol. 2, No. 4, 302–313, 2011.CrossRefGoogle Scholar
  40. [40]
    Früh, C.; Zakhor, A. Constructing 3D city models by merging aerial and ground views. IEEE Computer Graphics and Applications Vol. 23, No. 6, 52–61, 2003.CrossRefGoogle Scholar
  41. [41]
    Fisher, M.; Savva, M.; Li, Y.; Hanrahan, P.; Nießner, M. Activity-centric scene synthesis for functional 3D scene modeling. ACM Transactions on Graphics Vol. 34, No. 6, Article No. 179, 2015.Google Scholar
  42. [42]
    Marton, Z. C.; Rusu, R. B.; Beet, M. On fast surface reconstruction methods for large and noisy point clouds. In: Proceedings of the IEEE International Conference on Robotics and Automation, 3218–3223, 2009.Google Scholar
  43. [43]
    Tibshirani, R. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Vol. 58, No. 1, 267–288, 1996.MathSciNetzbMATHGoogle Scholar
  44. [44]
    Boykov, Y.; Veksler, O.; Zabih, R. Fast approximate energy minimization via graph cuts. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 23, No. 11, 1222–1239, 2001.CrossRefGoogle Scholar
  45. [45]
    Handa, A.; Whelan, T.; McDonald, J.; Davison, A. J. A benchmark for RGB-D visual odometry, 3D reconstruction and SLAM. In: Proceedings of the IEEE International Conference on Robotics and Automation, 1524–1531, 2014.Google Scholar
  46. [46]
    Gosset, W. S. The probable error of a mean. Biometrika Vol. 6, No. 1, 1–25, 1908.MathSciNetCrossRefGoogle Scholar

Copyright information

© The Author(s) 2017

Open Access The articles published in this journal are distributed under the terms of the Creative Commons Attribution 4.0 International License (, which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Authors and Affiliations

  1. 1.Tsinghua UniversityBeijingChina
  2. 2.Massachusetts Institute of TechnologyCambridgeUSA
  3. 3.City University of Hong KongHong KongChina

Personalised recommendations