A Cross-Situational Learning Based Framework for Grounding of Synonyms in Human-Robot Interactions

  • Oliver RoeslerEmail author
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 1093)


Natural human-robot interaction requires robots to link words to objects and actions through grounding. Although grounding has been investigated in previous studies, not many considered grounding of synonyms and the majority of employed models only worked offline. In this paper, we try to fill this gap by introducing an online learning framework for grounding synonymous object and action names using cross-situational learning. Words are grounded through geometric characteristics of objects and kinematic features of the robot joints during action execution. An interaction experiment between a human tutor and HSR robot is used to evaluate the proposed framework. The results show that the employed framework is able to successfully ground all used words.


Language grounding Cross-situational learning Human-robot interaction 


  1. 1.
    Aly, A., Taniguchi, A., Taniguchi, T.: A generative framework for multimodal learning of spatial concepts and object categories: an unsupervised part-of-speech tagging and 3D visual perception based approach. In: IEEE International Conference on Development and Learning and the International Conference on Epigenetic Robotics (ICDL-EpiRob), Lisbon, Portugal, September 2017Google Scholar
  2. 2.
    Blythe, R.A., Smith, K., Smith, A.D.M.: Learning times for large lexicons through cross-situational learning. Cogn. Sci. 34, 620–642 (2010)CrossRefGoogle Scholar
  3. 3.
    Clark, E.V.: The principle of contrast: a constraint on language acquisition. In: Mechanisms of Language Acquisition, pp. 1–33. Lawrence Erlbaum Associates (1987)Google Scholar
  4. 4.
    Craye, C., Filliat, D., Goudou, J.F.: Environment exploration for object-based visual saliency learning. In: IEEE International Conference on Robotics and Automation (ICRA), Stockholm, Sweden, May 2016Google Scholar
  5. 5.
    Dawson, C.R., Wright, J., Rebguns, A., Escárcega, M.V., Fried, D., Cohen, P.R.: A generative probabilistic framework for learning spatial language. In: IEEE Third Joint International Conference on Development and Learning and Epigenetic Robotics (ICDL), Osaka, Japan, August 2013Google Scholar
  6. 6.
    Ester, M., Kriegel, H.P., Sander, J., Xu, X.: A density-based algorithm for discovering clusters in large spatial databases with noise. In: Proceedings of the 2nd International Conference on Knowledge Discovery and Data Mining (KDD), Portland, Oregon, USA, pp. 226–231, August 1996Google Scholar
  7. 7.
    Filin, S., Pfeifer, N.: Segmentation of airborne laser scanning data using a slope adaptive neighborhood. ISPRS J. Photogram. Remote Sens. (P&RS) 60, 71–80 (2006)CrossRefGoogle Scholar
  8. 8.
    Fischler, M.A., Bolles, R.C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM (CACM) 24(6), 381–395 (1981)MathSciNetCrossRefGoogle Scholar
  9. 9.
    Fisher, C., Hall, D.G., Rakowitz, S., Gleitman, L.: When it is better to receive than to give: syntactic and conceptual constraints on vocabulary growth. Lingua 92, 333–375 (1994)CrossRefGoogle Scholar
  10. 10.
    Fontanari, J.F., Tikhanoff, V., Cangelosi, A., Ilin, R., Perlovsky, L.I.: Cross-situational learning of object-word mapping using neural modeling fields. Neural Netw. 22(5–6), 579–585 (2009)CrossRefGoogle Scholar
  11. 11.
    Fontanari, J.F., Tikhanoff, V., Cangelosi, A., Perlovsky, L.I.: A cross-situational algorithm for learning a lexicon using neural modeling fields. In: International Joint Conference on Neural Networks (IJCNN), Atlanta, GA, USA, June 2009Google Scholar
  12. 12.
    Harnad, S.: The symbol grounding problem. Physica D 42, 335–346 (1990)CrossRefGoogle Scholar
  13. 13.
    Hubert, L., Arabie, P.: Comparing partitions. J. Classif. 2(1), 193–218 (1985)CrossRefGoogle Scholar
  14. 14.
    International Federation of Robotics: World robotics 2017 - service robots (2017)Google Scholar
  15. 15.
    Kemp, C.C., Edsinger, A., Torres-Jara, E.: Challenges for robot manipulation in human environments. IEEE Robot. Autom. Mag. 14(1), 20–29 (2007)CrossRefGoogle Scholar
  16. 16.
    Koster, K., Spann, M.: MIR: an approach to robust clustering-application to range image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 22(5), 430–444 (2000)CrossRefGoogle Scholar
  17. 17.
    Nguyen, A., Le, B.: 3D point cloud segmentation: a survey. In: 6th IEEE Conference on Robotics, Automation and Mechatronics (RAM). IEEE, Manila, November 2013Google Scholar
  18. 18.
    Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., Duchesnay, E.: Scikit-learn: machine learning in python. J. Mach. Learn. Res. 12, 2825–2830 (2011)MathSciNetzbMATHGoogle Scholar
  19. 19.
    Pinker, S.: Learnability and Cognition. MIT Press, Cambridge (1989)Google Scholar
  20. 20.
    Roesler, O., Aly, A., Taniguchi, T., Hayashi, Y.: A probabilistic framework for comparing syntactic and semantic grounding of synonyms through cross-situational learning. In: ICRA-18 Workshop on Representing a Complex World: Perception, Inference, and Learning for Joint Semantic, Geometric, and Physical Understanding, Brisbane, Australia, May 2018Google Scholar
  21. 21.
    Roesler, O., Aly, A., Taniguchi, T., Hayashi, Y.: Evaluation of word representations in grounding natural language instructions through computational human-robot interaction. In: Proceedings of the 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI), Daegu, South Korea, March 2019Google Scholar
  22. 22.
    Rusu, R.B., Bradski, G., Thibaux, R., Hsu, J.: Fast 3D recognition and pose using the viewpoint feature histogram. In: Proceedings of the 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Taipei, Taiwan, pp. 2155–2162, October 2010Google Scholar
  23. 23.
    Sappa, A.D., Devy, M.: Fast range image segmentation by an edge detection strategy. In: Proceedings of the Third International Conference on 3-D Digital Imaging and Modeling (3DIM), Quebec City, Quebec, Canada, August 2002Google Scholar
  24. 24.
    Schnabel, R., Wahl, R., Klein, R.: Efficient ransac for point-cloud shape detection. Comput. Graphics Forum 26(2), 214–226 (2007)CrossRefGoogle Scholar
  25. 25.
    Schubert, E., Sander, J., Ester, M., Kriegel, H.P., Xu, X.: DBSCAN revisited, revisited: why and how you should (still) use DBSCAN. ACM Trans. Database Syst. (TODS) 42(3), 19 (2017)MathSciNetCrossRefGoogle Scholar
  26. 26.
    She, L., Yang, S., Cheng, Y., Jia, Y., Chai, J.Y., Xi, N.: Back to the blocks world: learning new actions through situated human-robot dialogue. In: Proceedings of the SIGDIAL 2014 Conference, Philadelphia, U.S.A., pp. 89–97, June 2014Google Scholar
  27. 27.
    Siskind, J.M.: A computational study of cross-situational techniques for learning word-to-meaning mappings. Cognition 61, 39–91 (1996)CrossRefGoogle Scholar
  28. 28.
    Smith, A.D.M., Smith, K.: Cross-Situational Learning, pp. 864–866. Springer, Boston (2012). Scholar
  29. 29.
    Smith, K., Smith, A.D.M., Blythe, R.A.: Cross-situational learning: an experimental study of word-learning mechanisms. Cogn. Sci. 35(3), 480–498 (2011)CrossRefGoogle Scholar
  30. 30.
    Steels, L., Loetzsch, M.: The grounded naming game. In: Steels, L. (ed.) Experiments in Cultural Language Evolution, pp. 41–59. John Benjamins, Amsterdam (2012)CrossRefGoogle Scholar
  31. 31.
    Strom, J., Richardson, A., Olson, E.: Graph-based segmentation for colored 3D laser point clouds. In: International Conference on Intelligent Robots and Systems (IROS), Taipei, Taiwan (2010)Google Scholar
  32. 32.
    Taniguchi, A., Taniguchi, T., Cangelosi, A.: Cross-situational learning with Bayesian generative models for multimodal category and word learning in robots. Front. Neurorobot. 11, 66 (2017)CrossRefGoogle Scholar
  33. 33.
    Tellex, S., Kollar, T., Dickerson, S., Walter, M.R., Banerjee, A.G., Teller, S., Roy, N.: Approaching the symbol grounding problem with probabilistic graphical models. AI Mag. 32(4), 64–76 (2011)CrossRefGoogle Scholar
  34. 34.
    Toyota Motor Corporation: HSR Manual, 2017.4.17 edn., April 2017Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Artificial Intelligence LabVrije Universiteit BrusselBrusselsBelgium

Personalised recommendations