HMD-Guided Image-Based Modeling and Rendering of Indoor Scenes

  • Daniel AndersenEmail author
  • Voicu Popescu
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11162)


We present a system that enables a novice user to acquire a large indoor scene in minutes as a collection of images sufficient for five degrees-of-freedom virtual navigation by image morphing. The user walks through the scene wearing an augmented reality head-mounted display (AR HMD) enhanced with a panoramic video camera. The AR HMD shows a 2D grid of a dynamically generated floor plan, which guides the user to acquire a panorama from each grid cell. After acquisition, panoramas are preliminarily registered using the AR HMD tracking data, corresponding features are detected in pairs of neighboring panoramas, and the correspondences are used to refine panorama registration. The registered panoramas and their correspondences support rendering the scene interactively with any view direction and from any viewpoint on the acquisition plane. An HMD VR interface guides the user who optimizes visualization fidelity interactively, by aligning the viewpoint with one of the hundreds of acquisition locations evenly sampling the floor plane.


Augmented reality 3D acquisition Image-based rendering 



We thank the ART research group at Purdue University for their feedback during development. This work was supported in part by the United States National Science Foundation under Grant DGE-1333468.


  1. 1.
    Ahn, J., Wohn, K.: Interactive scan planning for heritage recording. Multimed. Tools Appl. 75(7), 3655–3675 (2016)CrossRefGoogle Scholar
  2. 2.
    Aliaga, D.G., Carlbom, I.: Plenoptic stitching: a scalable method for reconstructing 3D interactive walk throughs. In: Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, pp. 443–450. ACM (2001)Google Scholar
  3. 3.
    Bradley, D., Brunton, A., Fiala, M., Roth, G.: Image-based navigation in real environments using panoramas. In: IEEE International Workshop on Haptic Audio Visual Environments and their Applications, p. 3, October 2005Google Scholar
  4. 4.
    Chiang, C.C., Way, D.L., Shieh, J.W., Shen, L.S.: A new image morphing technique for smooth vista transitions in panoramic image-based virtual environment. In: Proceedings of the ACM Symposium on Virtual Reality Software and Technology, VRST 1998, pp. 81–90 (1998)Google Scholar
  5. 5.
    Dai, A., Nießner, M., Zollhöfer, M., Izadi, S., Theobalt, C.: BundleFusion: real-time globally consistent 3D reconstruction using on-the-fly surface reintegration. ACM Trans. Graph. (TOG) 36(3), 24 (2017)CrossRefGoogle Scholar
  6. 6.
    Davis, A., Levoy, M., Durand, F.: Unstructured light fields. In: Computer Graphics Forum. vol. 31, pp. 305–314. Wiley Online Library (2012)Google Scholar
  7. 7.
    DiVerdi, S., Wither, J., Höllerer, T.: All around the map: online spherical panorama construction. Comput. Graph. 33(1), 73–84 (2009)CrossRefGoogle Scholar
  8. 8.
    Dong, S., Höllerer, T.: Real-time re-textured geometry modeling using microsoft HoloLens (2018)Google Scholar
  9. 9.
    Fan, X., Zhang, L., Brown, B., Rusinkiewicz, S.: Automated view and path planning for scalable multi-object 3D scanning. ACM Trans. Graph. 35(6), 239:1–239:13 (2016)CrossRefGoogle Scholar
  10. 10.
    Fischler, M.A., Bolles, R.C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. In: Readings in Computer Vision, pp. 726–740. Elsevier (1987)Google Scholar
  11. 11.
    Hedman, P., Ritschel, T., Drettakis, G., Brostow, G.: Scalable inside-out image-based rendering. ACM Trans. Graph. (TOG) 35(6), 231 (2016)CrossRefGoogle Scholar
  12. 12.
    HTC: VIVE (2017).
  13. 13.
    Huang, J., Chen, Z., Ceylan, D., Jin, H.: 6-DOF VR videos with a single 360-camera. In: 2017 IEEE Virtual Reality (VR), pp. 37–44. IEEE (2017)Google Scholar
  14. 14.
    Jung, J.-H., Kang, H.-B.: An efficient arbitrary view generation method using panoramic-based image morphing. In: Huang, D.-S., Li, K., Irwin, G.W. (eds.) ICIC 2006. LNCS, vol. 4113, pp. 1207–1212. Springer, Heidelberg (2006). Scholar
  15. 15.
    Kawai, N.: A simple method for light field resampling. In: ACM SIGGRAPH 2017 Posters, SIGGRAPH 2017, pp. 15:1–15:2 (2017)Google Scholar
  16. 16.
    Kawai, N., Audras, C., Tabata, S., Matsubara, T.: Panorama image interpolation for real-time walkthrough. In: ACM SIGGRAPH Posters, pp. 33:1–33:2 (2016)Google Scholar
  17. 17.
    Kolhatkar, S., Laganire, R.: Real-time virtual viewpoint generation on the GPU for scene navigation. In: 2010 Canadian Conference on Computer and Robot Vision, pp. 55–62, May 2010Google Scholar
  18. 18.
    Microsoft: Microsoft HoloLens (2017).
  19. 19.
    Pagani, A., Gava, C.C., Cui, Y., Krolla, B., Hengen, J.M., Stricker, D.: Dense 3D point cloud generation from multiple high-resolution spherical images. In: VAST, pp. 17–24 (2011)Google Scholar
  20. 20.
    Pagani, A., Stricker, D.: Structure from motion using full spherical panoramic cameras. In: 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops), pp. 375–382. IEEE (2011)Google Scholar
  21. 21.
    Pan, Q., Reitmayr, G., Drummond, T.W.: Interactive model reconstruction with user guidance. In: 2009 8th IEEE International Symposium on Mixed and Augmented Reality, pp. 209–210, October 2009Google Scholar
  22. 22.
    Pankratz, F., Klinker, G.: [POSTER] AR4AR: using augmented reality for guidance in augmented reality systems setup. In: 2015 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 140–143. IEEE (2015)Google Scholar
  23. 23.
    Rusinkiewicz, S., Hall-Holt, O., Levoy, M.: Real-time 3D model acquisition. ACM Trans. Graph. (TOG) 21(3), 438–446 (2002)CrossRefGoogle Scholar
  24. 24.
    Samsung: Gear 360 Camera (2017).
  25. 25.
    Shi, F.: Panorama interpolation for image-based navigation. Master’s thesis, University of Ottawa (2007)Google Scholar
  26. 26.
    Tuite, K., Snavely, N., Hsiao, D.Y., Smith, A.M., Popović, Z.: Reconstructing the world in 3D: bringing games with a purpose outdoors. In: Proceedings of the Fifth International Conference on the Foundations of Digital Games, pp. 232–239. ACM (2010)Google Scholar
  27. 27.
    Tuite, K., Snavely, N., Hsiao, D.Y., Tabing, N., Popovic, Z.: PhotoCity: training experts at large-scale image acquisition through a competitive game. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 1383–1392. ACM (2011)Google Scholar
  28. 28.
    Xiao, J., Shah, M.: Tri-view morphing. Comput. Vis. Image Underst. 96(3), 345–366 (2004)CrossRefGoogle Scholar
  29. 29.
    Xu, K.: 3D attention-driven depth acquisition for object identification. ACM Trans. Graph. 35(6), 238:1–238:14 (2016)Google Scholar
  30. 30.
    Zhang, Y., Zhu, Z.: Walk-able and stereo virtual tour based on spherical panorama matrix. In: De Paolis, L.T., Bourdot, P., Mongelli, A. (eds.) AVR 2017. LNCS, vol. 10324, pp. 50–58. Springer, Cham (2017). Scholar
  31. 31.
    Zhao, Q., Wan, L., Feng, W., Zhang, J., Wong, T.T.: Cube2Video: navigate between cubic panoramas in real-time. IEEE Trans. Multimed. 15(8), 1745–1754 (2013)CrossRefGoogle Scholar
  32. 32.
    Zhao, Q., Feng, W., Wan, L., Zhang, J.: SPHORB: a fast and robust binary feature on the sphere. Int. J. Comput. Vis. 113(2), 143–159 (2015)MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.Purdue UniversityWest LafayetteUSA

Personalised recommendations