Real-Time Multi-view Grid Map-Based Spatial Representation for Mixed Reality Applications

  • Pedro GirãoEmail author
  • João Paulo
  • Luís Garrote
  • Paulo Peixoto
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10850)


In this paper it is presented a markerless multi-view vision-based system to create a spatial representation of an indoor environment. Mixed reality integrates real-world elements into a virtual world. The modeling of such elements should be accurate to avoid interaction inconsistencies. The goal of this work is to achieve the virtualization of a room and its elements, providing seamless user navigation and interaction. This spatial representation based on grid maps provides a clear distinction between occupied and free cells. Scene objects are segmented using clustering techniques. The proposed approach has several potential applications, such as interactive virtual visits to heritage museums, real estate, and training scenarios. Preliminary qualitative results evidence a visually accurate modeling of the considered scenes, identifying elements and free space, encouraging future work towards the identification and tracking of static and dynamic elements on the scene.


Mixed reality Spatial representation Grid maps DB-SCAN Kinect 



This work was supported by the project POCI-01-0247-FEDER-017644 HTPDIR - “Human Tracking and Perception in Dynamic Immersive Rooms” financed by the Portugal2020 program and European Union’s structural funds.


  1. 1.
    Ricci, A., Piunti, M., Tummolini, L., Castelfranchi, C.: The mirror world: preparing for mixed-reality living. IEEE Pervasive Comput. 14(2), 60–63 (2015)CrossRefGoogle Scholar
  2. 2.
    Elfes, A.: Using occupancy grids for mobile robot perception and navigation. Computer 22(6), 46–57 (1989)CrossRefGoogle Scholar
  3. 3.
    Newcombe, R.A., Izadi, S., Hilliges, O., Molyneaux, D., Kim, D., Davison, A.J., Kohi, P., Shotton, J., Hodges, S., Fitzgibbon, A.: KinectFusion: real-time dense surface mapping and tracking. In: 2011 10th IEEE International Symposium on Mixed and Augmented Reality, pp. 127–136, October 2011Google Scholar
  4. 4.
    Urmson, C., Anhalt, J., Bagnell, D., Baker, C., Bittner, R., Clark, M., Dolan, J., Duggins, D., Galatali, T., Geyer, C., et al.: Autonomous driving in urban environments: boss and the urban challenge. J. Field Robot. 25(8), 425–466 (2008)CrossRefGoogle Scholar
  5. 5.
    Montemerlo, M., Becker, J., Bhat, S., Dahlkamp, H., Dolgov, D., Ettinger, S., Haehnel, D., Hilden, T., Hoffmann, G., Huhnke, B., et al.: Junior: the stanford entry in the urban challenge. J. Field Robot. 25(9), 569–597 (2008)CrossRefGoogle Scholar
  6. 6.
    Laugier, C., Paromtchik, I.E., Perrollaz, M., Yong, M., Yoder, J.D., Tay, C., Mekhnacha, K., Nègre, A.: Probabilistic analysis of dynamic scenes and collision risks assessment to improve driving safety. IEEE Intell. Transp. Syst. Mag. 3(4), 4–19 (2011)CrossRefGoogle Scholar
  7. 7.
    Pfeiffer, D., Franke, U.: Efficient representation of traffic scenes by means of dynamic stixels. In: Intelligent Vehicles Symposium (IV), 2010 IEEE, pp. 217–224. IEEE (2010)Google Scholar
  8. 8.
    Lu, F., Milios, E.E.: Robot pose estimation in unknown environments by matching 2D range scans. In: 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 935–938, June 1994Google Scholar
  9. 9.
    Cole, D.M., Newman, P.M.: Using laser range data for 3D slam in outdoor environments. In: Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006, pp. 1556–1563. IEEE (2006)Google Scholar
  10. 10.
    Leonard, J.J., Durrant-Whyte, H.F.: Simultaneous map building and localization for an autonomous mobile robot. In: IEEE/RSJ International Workshop on Intelligent Robots and Systems 1991. Intelligence for Mechanical Systems, Proceedings IROS 1991, pp. 1442–1447. IEEE (1991)Google Scholar
  11. 11.
    Dissanayake, M.G., Newman, P., Clark, S., Durrant-Whyte, H.F., Csorba, M.: A solution to the simultaneous localization and map building (slam) problem. IEEE Trans. Robot. Autom. 17(3), 229–241 (2001)CrossRefGoogle Scholar
  12. 12.
    Choset, H., Nagatani, K.: Topological simultaneous localization and mapping (slam): toward exact localization without explicit localization. IEEE Trans. Robot. Autom. 17(2), 125–137 (2001)CrossRefGoogle Scholar
  13. 13.
    Nuchter, A., Lingemann, K., Hertzberg, J., Surmann, H.: 6D slam with approximate data association. In: 12th International Conference on Advanced Robotics, 2005. ICAR 2005. Proceedings, pp. 242–249. IEEE (2005)Google Scholar
  14. 14.
    Montemerlo, M., Thrun, S., Koller, D., Wegbreit, B., et al.: FastSLAM: a factored solution to the simultaneous localization and mapping problem. In: AAAI/IAAI, pp. 593–598 (2002)Google Scholar
  15. 15.
    Siadat, A., Kaske, A., Klausmann, S., Dufaut, M., Husson, R.: An optimized segmentation method for a 2D laser-scanner applied to mobile robot navigation. IFAC Proc. Vol. 30(7), 149–154 (1997)CrossRefGoogle Scholar
  16. 16.
    Sack, D., Burgard, W.: A comparison of methods for line extraction from range data. IFAC Proc. Vol. 37(8), 728–733 (2004)CrossRefGoogle Scholar
  17. 17.
    Petrovskaya, A., Thrun, S.: Model based vehicle detection and tracking for autonomous urban driving. Auton. Robots 26(2–3), 123–139 (2009)CrossRefGoogle Scholar
  18. 18.
    Fayad, F., Cherfaoui, V.: Tracking objects using a laser scanner in driving situation based on modeling target shape. In: Intelligent Vehicles Symposium, 2007 IEEE, pp. 44–49. IEEE (2007)Google Scholar
  19. 19.
    Zhang, S., Adams, M., Tang, F., Xie, L.: Geometrical feature extraction using 2D range scanner. In: 4th International Conference on Control and Automation, 2003. ICCA 2003. Proceedings, pp. 901–905. IEEE (2003)Google Scholar
  20. 20.
    Pascoal, R., Santos, V., Premebida, C., Nunes, U.: Simultaneous segmentation and superquadrics fitting in laser-range data. IEEE Trans. Veh. Technol. 64(2), 441–452 (2015)CrossRefGoogle Scholar
  21. 21.
    Wang, C.C., Thorpe, C.: Simultaneous localization and mapping with detection and tracking of moving objects. In: IEEE International Conference on Robotics and Automation, 2002. Proceedings. ICRA 2002, vol. 3, pp. 2918–2924. IEEE (2002)Google Scholar
  22. 22.
    Vu, T.D.: Vehicle perception: localization, mapping with detection, classification and tracking of moving objects. Ph.D thesis, Institut National Polytechnique de Grenoble-INPG (2009)Google Scholar
  23. 23.
    Suma, E.A., Lange, B., Rizzo, A.S., Krum, D.M., Bolas, M.: FAAST: the flexible action and articulated skeleton toolkit. In: 2011 IEEE Virtual Reality Conference, pp. 247–248, March 2011Google Scholar
  24. 24.
    Maimone, A., Fuchs, H.: Reducing interference between multiple structured light depth sensors using motion. In: 2012 IEEE Virtual Reality Workshops (VRW), pp. 51–54, March 2012Google Scholar
  25. 25.
    Butler, D., Izadi, S., Hilliges, O., Molyneaux, D., Hodges, S., Kim, D., Butler, A., Molyneaux, D., Hilliges, O., Izadi, S., Hodges, S.: Shake’n’sense: reducing interference for overlapping structured light depth cameras. In: Proceedings of the 2012 ACM annual conference on Human Factors in Computing Systems, pp. 1933–1936, January 2012Google Scholar
  26. 26.
    Sarbolandi, H., Lefloch, D., Kolb, A.: Kinect range sensing: structured-light versus Time-of-Flight kinect. Comput. Vis. Image Underst. 139, 1–20 (2015)CrossRefGoogle Scholar
  27. 27.
    Martínez-Zarzuela, M., Pedraza-Hueso, M., Díaz-Pernas, F., González-Ortega, D., Antón-Rodríguez, M.: Indoor 3D Video Monitoring Using Multiple Kinect Depth-Cameras. arXiv preprint arXiv:1403.2895 (2014)
  28. 28.
    Leoncini, P., Sikorski, B., Baraniello, V., Martone, F., Luongo, C., Guida, M.: Multiple NUI device approach to full body tracking for collaborative virtual environments. In: De Paolis, L.T., Bourdot, P., Mongelli, A. (eds.) AVR 2017. LNCS, vol. 10324, pp. 131–147. Springer, Cham (2017). Scholar
  29. 29.
    Kowalski, M., Naruniec, J., Daniluk, M.: Livescan3D: a fast and inexpensive 3D data acquisition system for multiple kinect v2 sensors. In: 2015 International Conference on 3D Vision, pp. 318–325, October 2015Google Scholar
  30. 30.
    Blake, J., Echtler, F., Kerl, C.: libfreenect2 Project (2015).
  31. 31.
    Zhang, Z.: A flexible new technique for camera calibration. IEEE Trans. Pattern Anal. Mach. Intell. 22(11), 1330–1334 (2000)CrossRefGoogle Scholar
  32. 32.
    Bouguet, J.Y.: Camera calibration tool-box for matlab (2002).
  33. 33.
    Rodehorst, V., Heinrichs, M., Hellwich, O.: Evaluation of relative pose estimation methods for multi-camera setups. In: International Archives of Photogrammetry and Remote Sensing (ISPRS 2008), pp. 135–140 (2008)Google Scholar
  34. 34.
    Garrote, L., Rosa, J., Paulo, J., Premebida, C., Peixoto, P., Nunes, U.J.: 3D point cloud Downsampling for 2D indoor scene modelling in mobile robotics. In: 2017 IEEE International Conference on Autonomous Robot Systems and Competitions (ICARSC), pp. 228–233, April 2017Google Scholar
  35. 35.
    Kaufman, L., Rousseeuw, P.J.: Finding Groups in Data: An Introduction to Cluster Analysis, vol. 344. John Wiley & Sons, Hoboken (2009)zbMATHGoogle Scholar
  36. 36.
    Ester, M., Kriegel, H.P., Sander, J., Xu, X., et al.: A density-based algorithm for discovering clusters in large spatial databases with noise. In: KDD, vol. 96, pp. 226–231 (1996)Google Scholar
  37. 37.
    Schubert, E., Sander, J., Ester, M., Kriegel, H.P., Xu, X.: Dbscan revisited, revisited: Why and how you should (still) use dbscan. ACM Trans. Database Syst. (TODS) 42(3), 19 (2017)MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Pedro Girão
    • 1
    Email author
  • João Paulo
    • 1
  • Luís Garrote
    • 1
  • Paulo Peixoto
    • 1
  1. 1.Institute of Systems and Robotics, Department of Electrical and Computer EngineeringUniversity of CoimbraCoimbraPortugal

Personalised recommendations