Skip to main content
Download PDF

3D floor plan recovery from overlapping spherical images

Computational Visual Media volume 4, pages 367–383 (2018)Cite this article

Abstract

We present a novel approach to automatically recover, from a small set of partially overlapping spherical images, an indoor structure representation in terms of a 3D floor plan registered with a set of 3D environment maps. We introduce several improvements over previous approaches based on color and spatial reasoning exploiting Manhattan world priors. In particular, we introduce a new method for geometric context extraction based on a 3D facet representation, which combines color distribution analysis of individual images with sparse multi-view clues. We also introduce an efficient method to combine the facets from different viewpoints in a single consistent model, taking into the reliability of the facet information. The resulting capture and reconstruction pipeline automatically generates 3D multi-room environments in cases where most previous approaches fail, e.g., in the presence of hidden corners and large clutter, without the need for additional dense 3D data or tools. We demonstrate the effectiveness and performance of our approach on different real-world indoor scenes. Our test data is available to allow further studies and comparisons.

References

  1. Kopf, J. 360 video stabilization. ACM Transactions on Graphics Vol. 35, No. 6, Article No. 195, 2016.

    Google Scholar 

  2. Matzen, K.; Cohen, M. F.; Evans, B.; Kopf, J.; Szeliski, R. Low-cost 360 stereo photography and video capture. ACM Transactions on Graphics Vol. 36, No. 4, Article No. 148, 2017.

    Google Scholar 

  3. Brown, M.; Lowe, D. G. Automatic panoramic image stitching using invariant features. International Journal of Computer Vision Vol. 74, No. 1, 59–73, 2007.

    Article  Google Scholar 

  4. Pintore, G.; Garro, V.; Ganovelli, F.; Gobbetti, E.; Agus, M. Omnidirectional image capture on mobile devices for fast automatic generation of 2.5D indoor maps. In: Proceedings of the IEEE Winter Conference on Applications of Computer Vision, 1–9, 2016.

    Google Scholar 

  5. Pintore, G.; Gobbetti, E. Effective mobile mapping of multi-room indoor structures. The Visual Computer Vol. 30, Nos. 6–8, 707–716, 2014.

    Article  Google Scholar 

  6. Pintore, G.; Ganovelli, F.; Gobbetti, E.; Scopigno, R. Mobile mapping and visualization of indoor structures to simplify scene understanding and location awareness. In: Computer Vision—ECCV 2016 Workshops. Lecture Notes in Computer Science, Vol. 9914. Hua, G.; Jégou, H. Eds. Springer Cham, 130–145, 2016.

    Google Scholar 

  7. Yang, H.; Zhang, H. Efficient 3D room shape recovery from a single panorama. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 5422–5430, 2016.

    Google Scholar 

  8. Cabral, R.; Furukawa, Y. Piecewise planar and compact floorplan reconstruction from images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 628–635, 2014.

    Google Scholar 

  9. Pintore, G.; Ganovelli, F.; Pintus, R.; Scopigno, R.; Gobbetti, E. Recovering 3D indoor floor plans by exploiting low-cost spherical photography. In: Proceedings of the Pacific Graphics, 2018. Available at http://publications.crs4.it/pubdocs/2018/PGPSG18/pg2018s-indoorplan.pdf.

    Google Scholar 

  10. Xiong, X.; Adan, A.; Akinci, B.; Huber, D. Automatic creation of semantically rich 3D building models from laser scanner data. Automation in Construction Vol. 31, 325–337, 2013.

    Article  Google Scholar 

  11. Mura, C.; Mattausch, O.; Villanueva, A. J.; Gobbetti, E.; Pajarola, R. Automatic room detection and reconstruction in cluttered indoor environments with complex room layouts. Computers & Graphics Vol. 44, 20–32, 2014.

    Article  Google Scholar 

  12. Mura, C.; Mattausch, O.; Pajarola, R. Piecewise-planar reconstruction of multi-room interiors with arbitrary wall arrangements. Computer Graphics Forum Vol. 35, No. 7, 179–188, 2016.

    Article  Google Scholar 

  13. Guo, R.; Hoiem, D. Support surface prediction in indoor scenes. In: Proceedings of the IEEE International Conference on Computer Vision, 2144–2151, 2013.

    Google Scholar 

  14. Jia, Z.; Gallagher, A.; Saxena, A.; Chen, T. 3Dbased reasoning with blocks, support, and stability. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1–8, 2013.

    Google Scholar 

  15. Google. Tango. 2014. Available at www.google.com/atap/projecttango/.

  16. Ikehata, S.; Yang, H.; Furukawa, Y. Structured indoor modeling. In: Proceedings of the IEEE International Conference on Computer Vision, 1323–1331, 2015.

    Google Scholar 

  17. Kim, Y. M.; Mitra, N. J.; Yan, D.-M.; Guibas, L. Acquiring 3D indoor environments with variability and repetition. ACM Transactions on Graphics Vol. 31, No. 6, Article No. 138, 2012.

    Google Scholar 

  18. Nan, L.; Xie, K.; Sharf, A. A search-classify approach for cluttered indoor scene understanding. ACM Transactions on Graphics Vol. 31, No. 6, Article No. 137, 2012.

    Google Scholar 

  19. Autodesk. 123D Catch. Available at www.123dapp.com/catch.

  20. Microsoft. Photosynth. Available at photosynth.net/.

  21. Seitz, S. M.; Curless, B.; Diebel, J.; Scharstein, D.; Szeliski, R. A comparison and evaluation of multi-view stereo reconstruction algorithms. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 519–528, 2006.

    Google Scholar 

  22. Furukawa, Y.; Curless, B.; Seitz, S. M.; Szeliski, R. Reconstructing building interiors from images. In: Proceedings of the IEEE 12th International Conference on Computer Vision, 80–87, 2009.

    Google Scholar 

  23. Flint, A.; Murray, D.; Reid, I. Manhattan scene understanding using monocular, stereo, and 3D features. In: Proceedings of the International Conference on Computer Vision, 2228–2235, 2011.

    Google Scholar 

  24. Tsai, G.; Xu, C.; Liu, J.; Kuipers, B. Real-time indoor scene understanding using Bayesian filtering with motion cues. In: Proceedings of the International Conference on Computer Vision, 121–128, 2011.

    Google Scholar 

  25. Coughlan, J. M.; Yuille, A. L. Manhattan world: Compass direction from a single image by Bayesian inference. In: Proceedings of the 7th IEEE International Conference on Computer Vision, Vol. 2, 941–947, 1999.

    Google Scholar 

  26. Bao, S. Y.; Furlan, A.; Fei-Fei, L.; Savarese, S. Understanding the 3D layout of a cluttered room from multiple images. In: Proceedings of the IEEE Winter Conference on Applications of Computer Vision, 690–697, 2014.

    Chapter  Google Scholar 

  27. H¨ane, C.; Heng, L.; Lee, G. H.; Sizov, A.; Pollefeys, M. Real-time direct dense matching on fisheye images using plane-sweeping stereo. In: Proceedings of the 2nd International Conference on 3D Vision, 57–64, 2014.

    Google Scholar 

  28. Chang, P.; Hebert, M. Omni-directional structure from motion. In: Proceedings of the IEEE Workshop on Omnidirectional Vision, 127–133, 2000.

    Chapter  Google Scholar 

  29. Sch¨onbein, M.; Geiger, A. Omnidirectional 3D reconstruction in augmented Manhattan worlds. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, 716–723, 2014.

    Google Scholar 

  30. Micusik, B.; Pajdla, T. Structure from motion with wide circular field of view cameras. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 28, No. 7, 1135–1149, 2006.

    Article  Google Scholar 

  31. Micusik, B.; Pajdla, T. Autocalibration & 3D reconstruction with non-central catadioptric cameras. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, I-58–I-65, 2004.

    Google Scholar 

  32. Bunschoten, R.; Krose, B. Robust scene reconstruction from an omnidirectional vision system. IEEE Transactions on Robotics and Automation Vol. 19, No. 2, 351–357, 2003.

    Article  Google Scholar 

  33. Zingg, S.; Scaramuzza, D.; Weiss, S.; Siegwart, R. MAV navigation through indoor corridors using optical flow. In: Proceedings of the IEEE International Conference on Robotics and Automation, 3361–3368, 2010.

    Google Scholar 

  34. Li, S. Binocular spherical stereo. IEEE Transactions on Intelligent Transportation Systems Vol. 9, No. 4, 589–600, 2008.

    MathSciNet  Article  Google Scholar 

  35. Geyer, C.; Daniilidis, K. A unifying theory for central panoramic systems and practical implications. In: Computer Vision—ECCV 2000. Lecture Notes in Computer Science, Vol. 1843. Vernon, D. Ed. Springer Berlin Heidelberg, 445–461, 2000.

    Google Scholar 

  36. Kim, H.; Hilton, A. 3D scene reconstruction from multiple spherical stereo pairs. International Journal of Computer Vision Vol. 104, No. 1, 94–116, 2013.

    MathSciNet  MATH  Article  Google Scholar 

  37. Im, S.; Ha, H.; Rameau, F.; Jeon, H.-G.; Choe, G.; Kweon, I. S. All-around depth from small motion with a spherical panoramic camera. In: Computer Vision— ECCV 2016. Lecture Notes in Computer Science, Vol. 9907. Leibe, B.; Matas, J.; Sebe, N.; Welling, M. Eds. Springer Cham, 156–172, 2016.

    Chapter  Google Scholar 

  38. Caruso, D.; Engel, J.; Cremers, D. Large-scale direct SLAM for omnidirectional cameras. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, 141–148, 2015.

    Google Scholar 

  39. Pintore, G.; Pintus, R.; Ganovelli, F.; Scopigno, R.; Gobbetti, E. Recovering 3D existing-conditions of indoor structures from spherical images. Computers & Graphics Vol. 77, 16–29, 2018.

    Article  Google Scholar 

  40. Kangni, F.; Laganiere, R. Orientation and pose recovery from spherical panoramas. In: Proceedings of the IEEE 11th International Conference on Computer Vision, 1–8, 2007.

    Google Scholar 

  41. Achanta, R.; Shaji, A.; Smith, K.; Lucchi, A.; Fua, P.; S¨usstrunk, S. SLIC superpixels compared to stateof- the-art superpixel methods. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 34, No. 11, 2274–2282, 2012.

    Article  Google Scholar 

  42. Marroquim, R.; Kraus, M.; Cavalcanti, P. R. Efficient image reconstruction for point-based and line-based rendering. Computers & Graphics Vol. 32, No. 2, 189–203, 2008.

    Article  Google Scholar 

  43. Grompone von Gioi, R.; Jakubowicz, J.; Morel, J.- M.; Randall, G. LSD: A line segment detector. Image Processing On Line No. 2, 35–55, 2012.

    Article  Google Scholar 

  44. Douglas, D. H.; Peucker, T. K. Algorithms for the reduction of the number of points required to represent a digitized line or its caricature. Cartographica: The International Journal for Geographic Information and Geovisualization Vol. 10, No. 2, 112–122, 1973.

    Article  Google Scholar 

  45. Lee, D. C.; Hebert, M.; Kanade, T. Geometric reasoning for single image structure recovery. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2136–2143, 2009.

    Google Scholar 

  46. Zhang, Y.; Song, S.; Tan, P.; Xiao, J. PanoContext: A whole-room 3D context model for panoramic scene understanding. In: Computer Vision–ECCV 2014. Lecture Notes in Computer Science, Vol. 8694. Fleet, D.; Pajdla, T.; Schiele, B.; Tuytelaars, T. Eds. Springer Cham, 668–686, 2014.

    Google Scholar 

  47. Schindler, G.; Dellaert, F. Atlanta world: An expectation maximization framework for simultaneous low-level edge grouping and camera calibration in complex man-made environments. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, I-203–I-209, 2004.

    Google Scholar 

  48. Schwing, A. G.; Urtasun, R. Efficient exact inference for 3D indoor scene understanding. In: Computer Vision–ECCV 2012. Lecture Notes in Computer Science, Vol. 7577. Fitzgibbon, A.; Lazebnik, S.; Perona, P.; Sato, Y.; Schmid, C. Eds. Springer Berlin Heidelberg, 299–313, 2012.

    Google Scholar 

Download references

Acknowledgements

This work was partially supported by projects VIGEC and 3DCLOUDPRO. The authors also acknowledge the contribution of the Sardinian Regional Authorities.

Author information

Affiliations

  1. CRS4, Visual Computing Group, Cagliari, Italy

    Giovanni Pintore, Fabio Ganovelli, Ruggero Pintus & Roberto Scopigno

  2. CNR-ISTI, Visual Computing Group, Pisa, Italy

    Enrico Gobbetti

Authors
  1. Giovanni Pintore
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Fabio Ganovelli
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ruggero Pintus
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Roberto Scopigno
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Enrico Gobbetti
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Giovanni Pintore.

Additional information

Giovanni Pintore is a researcher in the Visual Computing (ViC) group at the Center for Advanced Studies, Research, and Development in Sardinia (CRS4). He holds a Laurea (M.Sc.) degree (2002) in electronic engineering from the University of Cagliari. His research interests include multiresolution representations of large and complex 3D models, lightfield displays, reconstruction and rendering of architectural scenes exploiting mobile devices, and the new generation of mobile spherical cameras.

Fabio Ganovelli is a researcher in the Visual Computing Lab at ISTI-CNR in Pisa. He holds a Laurea (1995) and a Ph.D. degree (2001) in computer science from the University of Pisa. His research spans many areas of computer graphics and computer vision and has widely published in major journals and conferences.

Ruggero Pintus is a researcher in the Visual Computing (ViC) group at the Center for Advanced Studies, Research, and Development in Sardinia (CRS4). He holds a Laurea (M.Sc., 2003) and a Ph.D. degree (2007) in electronic engineering from the University of Cagliari, Italy. His research currently focuses on acquisition, processing, and rendering of complex 3D models.

Roberto Scopigno graduated in computer science at the University of Pisa in 1984. He is a research director with CNR-ISTI and leads the Visual Computing Lab. He has been engaged in research projects concerned with scientific visualization, multi-resolution technologies, 3D range digitization, and CH applications. He has published more than 200 papers in international journals and conferences.

Enrico Gobbetti is the director of Visual Computing (ViC) group at the Center for Advanced Studies, Research, and Development in Sardinia (CRS4), Italy. He holds an engineering degree (1989) and a Ph.D. degree (1993) in computer science from the Swiss Federal Institute of Technology in Lausanne (EPFL). Prior to joining CRS4, he held research and teaching positions at EPFL, UMBC, and NASA. Enrico’s research spans many areas of visual computing and has widely published in major journals and conferences.

Rights and permissions

Open Access The articles published in this journal are distributed under the terms of the Creative Commons Attribution 4.0 International License (https://doi.org/creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Other papers from this open access journal are available free of charge from https://doi.org/www.springer.com/journal/41095. To submit a manuscript, please go to https://doi.org/www.editorialmanager.com/cvmj.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Pintore, G., Ganovelli, F., Pintus, R. et al. 3D floor plan recovery from overlapping spherical images. Comp. Visual Media 4, 367–383 (2018). https://doi.org/10.1007/s41095-018-0125-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s41095-018-0125-9

Keywords

  • indoor reconstruction
  • spherical panoramic cameras
  • 360 degree photography
  • multi-room environments