Journal of Intelligent & Robotic Systems

, Volume 83, Issue 3–4, pp 635–648 | Cite as

SAGE: Semantic Annotation of Georeferenced Environments

  • Peyman Moghadam
  • Benjamin Evans
  • Elliot Duff


The emergence of portable 3D mapping systems are revolutionizing the way we generate digital 3D models of environments. These systems are human-centric and require the user to hold or carry the device while continuously walking and mapping an environment. In this paper, we adapt this unique coexistence of man and machines to propose SAGE (Semantic Annotation of Georeferenced Environments). SAGE consists of a portable 3D mobile mapping system and a smartphone that enables the user to assign semantic content to georeferenced 3D point clouds while scanning a scene. The proposed system contains several components including touchless speech acquisition, background noise adaptation, real time audio and vibrotactile feedback, automatic speech recognition, distributed clock synchronization, 3D annotation localization, user interaction, and interactive visualization. The most crucial advantage of SAGE technology is that it can be used to infer dynamic activities within an environment. Such activities are difficult to be identified with existing post-processing semantic annotation techniques. The capability of SAGE leads to many promising applications such as intelligent scene classification, place recognition and navigational aid tasks. We conduct several experiments to demonstrate the effectiveness of the proposed system.


3D mobile mapping Semantic annotation Human-in-the-loop Semantic sensor 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Anguelov, D., Taskarf, B., Chatalbashev, V., Koller, D., Gupta, D., Heitz, G., Ng, A.: Discriminative learning of markov random fields for segmentation of 3D scan data. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition volume 2, pages 169–176. IEEE (2005)Google Scholar
  2. 2.
    Ben Hmida, H., Cruz, C., Boochs, F., Nicolle, C.: From 3D Point Clouds To Semantic Objects An Ontology-Based Detection Approach. ArXiv e-prints (2013)Google Scholar
  3. 3.
    Bok, Y., Jeong, Y., Choi, D.G., bok Kweon, I.S.: Capturing village-level heritages with a hand-held camera-laser fusion sensor. Int. J. Comput. Vis. 94(1), 36–53 (2011)CrossRefGoogle Scholar
  4. 4.
    Bosse, M., Zlot, R.: Place recognition using keypoint voting in large 3D lidar datasets. In: IEEE International Conference on Robotics and Automation (2013)Google Scholar
  5. 5.
    Bosse, M., Zlot, R., Zebedee, P.F.: Design of a spring-mounted 3D range sensor with application to mobile mapping. IEEE Trans. Robot. 28(5), 1104–1119 (2012)CrossRefGoogle Scholar
  6. 6.
    Dorninger, P., Nothegger, C.: 3D segmentation of unstructured point clouds for building modelling. Int. Arch. Photogramm. Remote. Sens. Spat. Inf. Sci. 35(3/W49A), 191–196 (2007)Google Scholar
  7. 7.
    Du, H., Henry, P., Ren, X., Cheng, M., Goldman, D.B., Seitz, S.M., Fox, D.: Interactive 3D modeling of indoor environments with a consumer depth camera. In: Proceedings of the 13th international conference on Ubiquitous computing pages 75–84. ACM (2011)Google Scholar
  8. 8.
    Henry, P., Krainin, M, Herbst, E., Ren, X., Fox, D.: RGB-D mapping: Using kinect-style depth cameras for dense 3D modeling of indoor environments. Int. J. Robot. Res. 31(5), 647–663 (2012)CrossRefGoogle Scholar
  9. 9.
    Lai, K., Bo, L., Ren, X., Fox, D.: Sparse distance learning for object recognition combining rgb and depth information. In: IEEE International Conference on Robotics and Automation, pages 4007–4013. IEEE (2011)Google Scholar
  10. 10.
    Lai, K., Bo, L., Ren, X., Fox, D.: Detection-based object labeling in 3D scenes. In: IEEE International Conference on Robotics and Automation, pp. 1330–1337 (2012)Google Scholar
  11. 11.
    Lai, K., Fox, D.: Object recognition in 3D point clouds using web data and domain adaptation. Int. J. Robot. Res. 29(8), 1019–1037 (2010)CrossRefGoogle Scholar
  12. 12.
    Liu, T., Carlberg, M., Chen, G., Chen, J., Kua, J., Zakhor, A.: Indoor localization and visualization using a human-operated backpack system. In: International Conference on Indoor Positioning and Indoor Navigation (2010)Google Scholar
  13. 13.
    Moghadam, P., Bosse, M., Zlot, R.: Line-based extrinsic calibration of range and image sensors. In: IEEE International Conference on Robotics and Automation, pp. 3685–3691 (2013)Google Scholar
  14. 14.
    Nüchter, A., Hertzberg, J.: Towards semantic maps for mobile robots. Robot. Auton. Syst. 56 (11), 915–926 (2008)CrossRefGoogle Scholar
  15. 15.
    Teichman, A., Levinson, J., Thrun, S.: Towards 3D object recognition via classification of arbitrary object tracks. In: IEEE International Conference on Robotics and Automation, pages 4034–4041. IEEE (2011)Google Scholar
  16. 16.
    Vidas, S., Moghadam, P.: HeatWave: a handheld 3D thermography system for energy auditing. Energy Build. 66, 445–460 (2013)CrossRefGoogle Scholar
  17. 17.
    Vidas, S., Moghadam, P., Bosse, M.: 3D thermal mapping of building interiors using an RGB-D and thermal camera. In: IEEE International Conference on Robotics and Automation, pp. 3670–3676 (2013)Google Scholar
  18. 18.
    Wolf, D.F., Sukhatme, G.S.: Semantic mapping using mobile robots. IEEE Trans. Robot. 24(2), 245–258 (2008)CrossRefGoogle Scholar
  19. 19.
    Xiong, X., Huber, D.: Using context to create semantic 3D models of indoor environments. In Proc. BMVC (2010)Google Scholar
  20. 20.
    Li, Z., Liu, Z., Xia, C.H.: Clock synchronization algorithms for network measurements. In: INFOCOM 2002. Twenty-First Annual Joint Conference of the IEEE Computer and Communications Societies., volume 1, pages 160–169. IEEE (2002)Google Scholar
  21. 21.
    Zhang, Y., Luo, C., Liu, J.: Walk&sketch: Create floor plans with an RGB-D camera. In: UbiComp ’12 Proceedings of the 2012 ACM Conference on Ubiquitous Computing, pages 461–470. ACM (2012)Google Scholar

Copyright information

© Springer Science+Business Media Dordrecht 2015

Authors and Affiliations

  1. 1.Autonomous SystemsCSIROBrisbaneAustralia

Personalised recommendations