View suggestion for interactive segmentation of indoor scenes

Abstract

Point cloud segmentation is a fundamental problem. Due to the complexity of real-world scenes and the limitations of 3D scanners, interactive segmentation is currently the only way to cope with all kinds of point clouds. However, interactively segmenting complex and large-scale scenes is very time-consuming. In this paper, we present a novel interactive system for segmenting point cloud scenes. Our system automatically suggests a series of camera views, in which users can conveniently specify segmentation guidance. In this way, users may focus on specifying segmentation hints instead of manually searching for desirable views of unsegmented objects, thus significantly reducing user effort. To achieve this, we introduce a novel view preference model, which is based on a set of dedicated view attributes, with weights learned from a user study. We also introduce support relations for both graph-cut-based segmentation and finding similar objects. Our experiments show that our segmentation technique helps users quickly segment various types of scenes, outperforming alternative methods.

References

  1. [1]

    Lai, K.; Bo, L.; Ren, X.; Fox, D. Detection-based object labeling in 3D scenes. In: Proceedings of the IEEE International Conference on Robotics and Automation, 1330–1337, 2012.

    Google Scholar 

  2. [2]

    Johnson, A. E.; Hebert, M. Using spin images for efficient object recognition in cluttered 3D scenes. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 21, No. 5, 433–449, 1999.

    Article  Google Scholar 

  3. [3]

    Zheng, B.; Zhao, Y.; Yu, J. C.; Ikeuchi, K.; Zhu, S. C. Beyond point clouds: Scene understanding by reasoning geometry and physics. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 3127–3134, 2013.

    Google Scholar 

  4. [4]

    Holz, D.; Behnke, S. Fast range image segmentation and smoothing using approximate surface reconstruction and region growing. In: Intelligent Autonomous Systems 12. Lee, S.; Cho, H.; Yoon, K.-J.; Lee, J. Eds. Springer Berlin Heidelberg, 61–73, 2013.

    Google Scholar 

  5. [5]

    Rabbani, T.; van den Heuvel, F. A.; Vosselmann, G. Segmentation of point clouds using smoothness constraint. International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences Vol. 36, No. 5, 248–253, 2006.

    Google Scholar 

  6. [6]

    Boykov, Y.; Funka-Lea, G. Graph cuts and efficient N-D image segmentation. International Journal of Computer Vision Vol. 70, No. 2, 109–131, 2006.

    Article  Google Scholar 

  7. [7]

    Golovinskiy, A.; Funkhouser, T. Min-cut based segmentation of point clouds. In: Proceedings of the IEEE 12th International Conference on Computer Vision Workshops, 39–46, 2009.

    Google Scholar 

  8. [8]

    Sedlacek, D.; Zara, J. Graph cut based pointcloud segmentation for polygonal reconstruction. In: Advances in Visual Computing. Bebis, G.; Boyle, R.; Parvin, B.; Koracin, D. et al. Eds. Springer Berlin Heidelberg, 218–227, 2009.

    Google Scholar 

  9. [9]

    Kim, Y. M.; Mitra, N. J.; Yan, D.-M.; Guibas, L. Acquiring 3D indoor environments with variability and repetition. ACM Transactions on Graphics Vol. 31, No. 6, Article No. 138, 2012.

    Google Scholar 

  10. [10]

    Nan, L.; Xie, K.; Sharf, A. A search-classify approach for cluttered indoor scene understanding. ACM Transactions on Graphics Vol. 31, No. 6, Article No. 137, 2012.

    Google Scholar 

  11. [11]

    Silberman, N.; Hoiem, D.; Kohli, P.; Fergus, R. Indoor segmentation and support inference from RGBD images. In: Computer Vision–ECCV 2012. Fitzgibbon, A.; Lazebnik, S.; Perona, P.; Sato, Y.; Schmid, C. Eds. Springer Berlin Heidelberg, 746–760, 2012.

    Google Scholar 

  12. [12]

    Nguyen, A.; Le, B. 3D point cloud segmentation: A survey. In: Proceedings of the 6th IEEE Conference on Robotics, Automation and Mechatronics, 225–230, 2013.

    Google Scholar 

  13. [13]

    Shao, T.; Xu, W.; Zhou, K.; Wang, J.; Li, D.; Guo, B. An interactive approach to semantic modeling of indoor scenes with an RGBD camera. ACM Transactions on Graphics Vol. 31, No. 6, Article No. 136, 2012.

    Google Scholar 

  14. [14]

    Yuan, X.; Xu, H.; Nguyen, M. X.; Shesh, A.; Chen, B. Sketch-based segmentation of scanned outdoor environment models. In: Proceedings of the Eurographics Workshop on Sketch-Based Interfaces and Modeling, 19–26, 2005.

    Google Scholar 

  15. [15]

    Silberman, N.; Fergus, R. Indoor scene segmentation using a structured light sensor. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, 601–608, 2011.

    Google Scholar 

  16. [16]

    Xiao, J.; Owens, A.; Torralba, A. SUN3D: A database of big spaces reconstructed using SfM and object labels. In: Proceedings of the IEEE International Conference on Computer Vision, 1625–1632, 2013.

    Google Scholar 

  17. [17]

    Chang, A. X.; Funkhouser, T.; Guibas, L.; Hanrahan, P.; Huang, Q.; Li, Z.; Savarese, S.; Savva, M.; Song, S.; Su, H.; Xiao, J.; Yi, L.; Yu, F. ShapeNet: An information-rich 3D model repository. arXiv preprint arXiv:1512.03012, 2015.

    Google Scholar 

  18. [18]

    Hinterstoisser, S.; Lepetit, V.; Ilic, S.; Holzer, S.; Bradski, G. R.; Konolige, K.; Navab, N. Model based training, detection and pose estimation of texture-less 3D objects in heavily cluttered scenes. In: Computer Vision–ACCV 2012. Lee, K. M.; Matsushita, Y.; Rehg, J. M.; Hu, Z. Eds. Springer Berlin Heidelberg, 548–562, 2012.

    Google Scholar 

  19. [19]

    Chen, K.; Lai, Y.-K.; Wu, Y.-X.; Martin, R.; Hu, S.-M. Automatic semantic modeling of indoor scenes from low-quality RGB-D data using contextual information. ACM Transactions on Graphics Vol. 33, No. 6, Article No. 208, 2014.

    Google Scholar 

  20. [20]

    Silberman, N.; Sontag, D.; Fergus, R. Instance segmentation of indoor scenes using a coverage loss. In: Computer Vision–ECCV 2014. Fleet, D.; Pajdla, T.; Schiele, B.; Tuytelaars, T. Eds. Springer International Publishing, 616–631, 2014.

    Google Scholar 

  21. [21]

    Chen, K.; Lai, Y. K.; Hu, S.-M. 3D indoor scene modeling from RGB-D data: a survey. Computational Visual Media Vol. 1, No. 4, 267–278, 2015.

    Article  Google Scholar 

  22. [22]

    Shen, C.-H.; Huang, S.-S.; Fu, H.; Hu, S.-M. Adaptive partitioning of urban facades. ACM Transactions on Graphics Vol. 30, No. 6, Article No. 184, 2011.

    Google Scholar 

  23. [23]

    Zhang, H.; Xu, K.; Jiang, W.; Lin, J.; Cohen-Or, D.; Chen, B. Layered analysis of irregular facades via symmetry maximization. ACM Transactions on Graphics Vol. 32, No. 4, Article No. 121, 2013.

    Google Scholar 

  24. [24]

    Mattausch, O.; Panozzo, D.; Mura, C.; Sorkine-Hornung, O.; Pajarola, R. Object detection and classification from large-scale cluttered indoor scans. Computer Graphics Forum Vol. 33, No. 2, 11–21, 2014.

    Article  Google Scholar 

  25. [25]

    Valentin, J.; Vineet, V.; Cheng, M.-M.; Kim, D.; Shotton, J.; Kohli, P.; Nießner, M.; Criminisi, A.; Izadi, S.; Torr, P. SemanticPaint: Interactive 3D labeling and learning at your fingertips. ACM Transactions on Graphics Vol. 34, No. 5, Article No. 154, 2015.

    Google Scholar 

  26. [26]

    Wong, Y.-S.; Chu, H.-K.; Mitra, N. J. SmartAnnotator an interactive tool for annotating indoor RGBD images. Computer Graphics Forum Vol. 34, No. 2, 447–457, 2015.

    Article  Google Scholar 

  27. [27]

    Christie, M.; Olivier, P. Camera control in computer graphics: Models, techniques and applications. In: Proceedings of the ACM SIGGRAPH ASIA 2009 Courses, Article No. 3, 2009.

    Google Scholar 

  28. [28]

    Scott, W. R.; Roth, G.; Rivest, J.-F. View planning for automated three-dimensional object reconstruction and inspection. ACM Computing Surveys Vol. 35, No. 1, 64–96, 2003.

    Article  Google Scholar 

  29. [29]

    Secord, A.; Lu, J.; Finkelstein, A.; Singh, M.; Nealen, A. Perceptual models of viewpoint preference. ACM Transactions on Graphics Vol. 30, No. 5, Article No. 109, 2011.

    Google Scholar 

  30. [30]

    Vázquez, P.-P.; Feixas, M.; Sbert, M.; Heidrich, W. Viewpoint selection using viewpoint entropy. In: Proceedings of the Vision Modeling and Visualization Conference, 273–280, 2001.

    Google Scholar 

  31. [31]

    Andújar, C.; Vázquez, P.; Fairén, M. Way-Finder: Guided tours through complex walkthrough models. Computer Graphics Forum Vol. 23, No. 3, 499–508, 2004.

    Article  Google Scholar 

  32. [32]

    Li, T.-Y.; Lien, J.-M.; Chiu, S.-Y.; Yu, T.-H. Automatically generating virtual guided tours. In: Proceedings of the Computer Animation, 99–106, 1999.

    Google Scholar 

  33. [33]

    Christie, M.; Languénou, E. A constraint-based approach to camera path planning. In: Smart Graphics. Butz, A.; Krüger, A.; Olivier, P. Eds. Springer Berlin Heidelberg, 172–181, 2003.

    Google Scholar 

  34. [34]

    Salomon, B.; Garber, M.; Lin, M. C.; Manocha, D. Interactive navigation in complex environments using path planning. In: Proceedings of the Symposium on Interactive 3D Graphics, 41–50, 2003.

    Google Scholar 

  35. [35]

    Choi, S.; Zhou, Q.-Y.; Koltun, V. Robust reconstruction of indoor scenes. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 5556–5565, 2015.

    Google Scholar 

  36. [36]

    Newcombe, R. A.; Izadi, S.; Hilliges, O.; Molyneaux, D.; Kim, D.; Davison, A. J.; Kohli, P.; Shotton, J.; Hodges, S.; Fitzgibbon, A. KinectFusion: Real-time dense surface mapping and tracking. In: Proceedings of the 10th IEEE International Symposium on Mixed and Augmented Reality, 127–136, 2011.

    Google Scholar 

  37. [37]

    Ikehata, S.; Yang, H.; Furukawa, Y. Structured indoor modeling. In: Proceedings of the IEEE International Conference on Computer Vision, 1323–1331, 2015.

    Google Scholar 

  38. [38]

    Furukawa, Y.; Curless, B.; Seitz, S. M.; Szeliski, R. Manhattan-world stereo. In: Proceeding of the IEEE Conference on Computer Vision and Pattern Recognition, 1422–1429, 2009.

    Google Scholar 

  39. [39]

    Asha, V.; Bhajantri, N. U.; Nagabhushan, P. GLCM-based chi-square histogram distance for automatic detection of defects on patterned textures. International Journal of Computational Vision and Robotics Vol. 2, No. 4, 302–313, 2011.

    Article  Google Scholar 

  40. [40]

    Früh, C.; Zakhor, A. Constructing 3D city models by merging aerial and ground views. IEEE Computer Graphics and Applications Vol. 23, No. 6, 52–61, 2003.

    Article  Google Scholar 

  41. [41]

    Fisher, M.; Savva, M.; Li, Y.; Hanrahan, P.; Nießner, M. Activity-centric scene synthesis for functional 3D scene modeling. ACM Transactions on Graphics Vol. 34, No. 6, Article No. 179, 2015.

    Google Scholar 

  42. [42]

    Marton, Z. C.; Rusu, R. B.; Beet, M. On fast surface reconstruction methods for large and noisy point clouds. In: Proceedings of the IEEE International Conference on Robotics and Automation, 3218–3223, 2009.

    Google Scholar 

  43. [43]

    Tibshirani, R. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Vol. 58, No. 1, 267–288, 1996.

    MathSciNet  MATH  Google Scholar 

  44. [44]

    Boykov, Y.; Veksler, O.; Zabih, R. Fast approximate energy minimization via graph cuts. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 23, No. 11, 1222–1239, 2001.

    Article  Google Scholar 

  45. [45]

    Handa, A.; Whelan, T.; McDonald, J.; Davison, A. J. A benchmark for RGB-D visual odometry, 3D reconstruction and SLAM. In: Proceedings of the IEEE International Conference on Robotics and Automation, 1524–1531, 2014.

    Google Scholar 

  46. [46]

    Gosset, W. S. The probable error of a mean. Biometrika Vol. 6, No. 1, 1–25, 1908.

    MathSciNet  Article  Google Scholar 

Download references

Acknowledgements

This work was supported by the Joint NSFC–ISF Research Program (Project No. 61561146393), the National Natural Science Foundation of China (Project No. 61521002), the Research Grant of Beijing Higher Institution Engineering Research Center, and the Tsinghua–Tencent Joint Laboratory for Internet Innovation Technology. The work was partially supported by grants from the Research Grants Council of the Hong Kong Special Administrative Region, China (Project Nos. CityU113513 and CityU11300615).

Author information

Affiliations

Authors

Corresponding author

Correspondence to Sheng Yang.

Additional information

This article is published with open access at Springerlink.com

Sheng Yang received his B.S. degree in computer science from Wuhan University in 2014. He is currently a Ph.D. candidate in computer science in Tsinghua University. His research interests include computer graphics and point cloud processing.

Jie Xu is a Ph.D. student at the Computer Science and Artificial Intelligence Laboratory in Massachusetts Institute of Technology. His research interests include computer graphics and geometric processing.

Kang Chen received his B.S. degree in computer science from Nanjing University in 2012. He is currently a Ph.D. candidate in the Institute for Interdisciplinary Information Sciences, Tsinghua University. His research interests include computer graphics, geometric modeling and processing.

Hongbo Fu is an associate professor in the School of Creative Media, City University of Hong Kong. He received his Ph.D. degree in computer science from the Hong Kong University of Science and Technology in 2007 and B.S. degree in information sciences from Peking University in 2002. His primary research interests fall in the fields of computer graphics and human computer interaction. He has served as an associate editor of The Visual Computer, Computers & Graphics, and Computer Graphics Forum.

Other papers from this open access journal are available free of charge from http://www.springer.com/journal/41095. To submit a manuscript, please go to https://www.editorialmanager.com/cvmj.

Electronic supplementary material

Rights and permissions

Open Access The articles published in this journal are distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Yang, S., Xu, J., Chen, K. et al. View suggestion for interactive segmentation of indoor scenes. Comp. Visual Media 3, 131–146 (2017). https://doi.org/10.1007/s41095-017-0078-4

Download citation

Keywords

  • point cloud segmentation
  • view suggestion
  • interactive segmentation