Making Sense of Indoor Spaces Using Semantic Web Mining and Situated Robot Perception

  • Jay YoungEmail author
  • Valerio Basile
  • Markus Suchi
  • Lars Kunze
  • Nick Hawes
  • Markus Vincze
  • Barbara Caputo
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10577)


Intelligent Autonomous Robots deployed in human environments must have understanding of the wide range of possible semantic identities associated with the spaces they inhabit – kitchens, living rooms, bathrooms, offices, garages, etc. We believe robots should learn this information through their own exploration and situated perception in order to uncover and exploit structure in their environments – structure that may not be apparent to human engineers, or that may emerge over time during a deployment. In this work, we combine semantic web-mining and situated robot perception to develop a system capable of assigning semantic categories to regions of space. This is accomplished by looking at web-mined relationships between room categories and objects identified by a Convolutional Neural Network trained on 1000 categories. Evaluated on real-world data, we show that our system exhibits several conceptual and technical advantages over similar systems, and uncovers semantic structure in the environment overlooked by ground-truth annotators.


Robotics Artificial intelligence Semantic web-mining Deep vision Service robots Machine learning Space classification Semantic mapping Imagenet Convolutional Neural Networks 



The research leading to these results has received funding from EU FP7 grant agreement No. 600623, STRANDS, and CHIST-ERA Project ALOOF.

The authors also wish to thank Daniel Hayes for his careful and attentive contributions upon which this work depends.


  1. 1.
    Faeulhammer, T., et al.: Autonomous learning of object models on a mobile robot. IEEE RAL PP(99), 1 (2016)Google Scholar
  2. 2.
    Kilgarriff, A., Fellbaum, C.: Wordnet: An electronic lexical database (2000)CrossRefGoogle Scholar
  3. 3.
    Russakovsky, O., et al.: ImageNet large scale visual recognition challenge. Int. J. Comput. Vis. (IJCV) 115(3), 211–252 (2015)MathSciNetCrossRefGoogle Scholar
  4. 4.
    Krizhevsky, A., et al.: Imagenet classification with deep convolutional neural networks. In: Pereira, F., Burges, C.J.C., Bottou, L., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems 25, pp. 1097–1105. Curran Associates Inc. (2012)Google Scholar
  5. 5.
    Wu, Z., Palmer, M.: Verbs semantics and lexical selection. In: ACL. ser. ACL 1994, pp. 133–138. Association for Computational Linguistics, Stroudsburg (1994)Google Scholar
  6. 6.
    Basile, V., Jebbara, S., Cabrio, E., Cimiano, P.: Populating a knowledge base with object-location relations using distributional semantics. In: Blomqvist, E., Ciancarini, P., Poggi, F., Vitali, F. (eds.) EKAW 2016. LNCS (LNAI), vol. 10024, pp. 34–50. Springer, Cham (2016). Scholar
  7. 7.
    Camacho-Collados, J., et al.: Nasari: a novel approach to a semantically-aware representation of items. In: Mihalcea, R., Chai, J.Y., Sarkar, A. (eds.) HLT-NAACL, pp. 567–577. The Association for Computational Linguistics (2015)Google Scholar
  8. 8.
    Ambruş, R., et al.: Meta-rooms: building and maintaining long term spatial models in a dynamic world. In: 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 1854–1861. IEEE (2014)Google Scholar
  9. 9.
    Young, J., et al.: Towards lifelong object learning by integrating situated robot perception and semantic web mining. In: Proceedings of the European Conference on Artificial Intelligence (ECAI) (2016)Google Scholar
  10. 10.
    Young, J., et al.: Semantic web-mining and deep vision for lifelong object discovery. In: Proceedings of the IEEE International Conference on Robotics and Automation (ICRA) (2017)Google Scholar
  11. 11.
    Navigli, R., Ponzetto, S.P.: Babelnet: the automatic construction, evaluation and application of a wide-coverage multilingual semantic network. Artif. Intell. 193, 217–250 (2012)MathSciNetCrossRefGoogle Scholar
  12. 12.
    Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 (2013)
  13. 13.
    Vasudevan, S., Siegwart, R.: Bayesian space conceptualization and place classification for semantic maps in mobile robotics. Robot. Auton. Syst. 56(6), 522–537 (2008)CrossRefGoogle Scholar
  14. 14.
    Vasudevan, S., et al.: Cognitive maps for mobile robotsan object based approach. Robot. Auton. Syst. 55(5), 359–371 (2007)CrossRefGoogle Scholar
  15. 15.
    Pronobis, A., Jensfelt, P.: Large-scale semantic mapping and reasoning with heterogeneous modalities. In: IEEE International Conference on Robotics and Automation (2012)Google Scholar
  16. 16.
    Kostavelis, I., Gasteratos, A.: Semantic mapping for mobile robotics tasks: a survey. Robot. Auton. Syst. 66, 86–103 (2015)CrossRefGoogle Scholar
  17. 17.
    Zender, H., et al.: Conceptual spatial representations for indoor mobile robots. Robot. Auton. Syst. 56(6), 493–502 (2008)CrossRefGoogle Scholar
  18. 18.
    Hanheide, M., et al.: Dora, a robot exploiting probabilistic knowledge under uncertain sensing for efficient object search. In: Proceedings of Systems Demonstration of the 21st International Conference on Automated Planning and Scheduling (ICAPS), Freiburg, Germany (2011)Google Scholar
  19. 19.
    Potapova, E., et al.: Attention-driven object detection and segmentation of cluttered table scenes using 2.5 d symmetry. In: 2014 IEEE International Conference on Robotics and Automation (ICRA), pp. 4946–4952. IEEE (2014)Google Scholar
  20. 20.
    He, K., et al.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Jay Young
    • 1
    Email author
  • Valerio Basile
    • 2
  • Markus Suchi
    • 3
  • Lars Kunze
    • 4
  • Nick Hawes
    • 1
  • Markus Vincze
    • 3
  • Barbara Caputo
    • 5
  1. 1.The University of BirminghamBirminghamUK
  2. 2.Université Côte d’Azur, Inria, CNRS, I3SSophia AntipolisFrance
  3. 3.Technische Universität WienViennaAustria
  4. 4.Oxford Robotics Institute, Department of Engineering ScienceUniversity of OxfordOxfordUK
  5. 5.Università di Roma - SapienzaRomeItaly

Personalised recommendations