International Journal of Social Robotics

, Volume 7, Issue 2, pp 165–181 | Cite as

Building a Model of the Environment from a Route Perspective for Human–Robot Interaction

  • Yoichi Morales
  • Satoru Satake
  • Takayuki Kanda
  • Norihiro Hagita


We built a model of the environment for human–robot interaction by learning from humans cognitive processes. Our method, which differs from previous map building techniques in terms of perspectives, is based on a route perspective that is a mental tour of the environment. The main contribution of this work is the theory and computational implementation of the concept of route with its respective visual memory. The concept of route is modeled as a three layered model composed of memory layer, survey layer and route layer. By imitating the human concept of a route, the route layer is modeled as a directional path segmented by action taking associated with visual memory taken while traveling the path. We developed a system that generates human understandable route directions which was evaluated towards two methods: one copied from the explanation of a human expert and one generated with a model without the route perspective layer. Finally, experimental results demonstrate the usefulness of the route perspective layer, since it performed better than the model without the route layer and similarly to the human expert.


Cognitive human–robot interaction  Service robots Social human–robot interaction  Environment modeling Cognitive map 


  1. 1.
    Morales Y, Satake S, Kanda T, Hagita N (2011) Modeling environments from a route perspective. Presented at the ACM/IEEE international conference on human robot interaction (HRI2011), Lausanne SwitzerlandGoogle Scholar
  2. 2.
    Burgard W, Cremers AB, Fox D, Hahnel D, Lakemeyer G, Schulz D et al (1998) The interactive museum tour-guide robot. Presented at the national conference on artificial intelligence (AAAI1998)Google Scholar
  3. 3.
    Siegwart R, Arras KO, Bouabdallah S, Burnier D, Froidevaux G, Greppin X et al (2003) Robox at expo. 02: a large scale installation of personal robots. Robot Auton Syst 42:203–222CrossRefMATHGoogle Scholar
  4. 4.
    Kulyukin V, Gharpure C, Nicholson J (2005) RoboCart: toward robot-assisted navigation of grocery stores by the visually impaired. Presented at the IEEE/RSJ international conference on intelligent robots and systems (IROS2005)Google Scholar
  5. 5.
    Gross H-M, Boehme H-J, Schroeter C, Mueller S, Koenig A, Martin C, et al. (2008) ShopBot: progress in developing an interactive mobile shopping assistant for everyday use. Presented at the IEEE international conference on systems, man, and cybernetics (SMC2008)Google Scholar
  6. 6.
    Mühlbauer Q, Sosnowski S, Xu T, Zhang T, Kühnlenz K, Buss M (2009) Navigation through Urban environments by visual perception and interaction. Presented at the IEEE international conference on robotics and automation (ICRA2009)Google Scholar
  7. 7.
    Alami R, Warnier M, Guitton J, Lemaignan S, Sisbot E (2011) When the robot considers the human. In: Proceedings of the 15th international symposium on robotics researchGoogle Scholar
  8. 8.
    Trafton JG, Cassimatis NL, Bugajska MD, Brock DP, Mintz FE, Schultz AC (2005) Enabling effective human–Robot interaction using perspective-taking in robots. IEEE Trans Syst Man Cybern. Part A 35:460–470CrossRefGoogle Scholar
  9. 9.
    Berlin M, Gray J, Thomaz AL, Breazeal C (2006) Perspective taking: an organizing principle for learning in human–Robot interaction. Presented at the national conference on artificial intelligence (AAAI2006)Google Scholar
  10. 10.
    Pandey AK, Alami R (2010) Visuo-spatial reasoning for human–robot interaction. Presented at the 12th international symposium on experimental robotics (ISER-2010)Google Scholar
  11. 11.
    Okuno Y, Kanda T, Imai M, Ishiguro H, Hagita N (2009) Providing route directions: design of robot’s utterance, gesture, and timing. Presented at the ACM/IEEE international conference on human–robot interaction (HRI2009)Google Scholar
  12. 12.
    Ono T, Imai M, Ishiguro H (2001) A model of embodied communications with gestures between humans and robots. Presented at the annual meeting of the cognitive science society (CogSci2001)Google Scholar
  13. 13.
    Beeson P, Jong NK, Kuipers B (2005) Towards autonomous topological place detection using the extended voronoi graph. Presented at the IEEE international conference on robotics and automation (ICRA2005)Google Scholar
  14. 14.
    Topp EA, Christensen HI (2006) Topological modelling for human augmented mapping. Presented at the IEEE/RSJ international conference on intelligent robots and systems (IROS2006)Google Scholar
  15. 15.
    Werner S, Krieg-Brückner B, Herrmann T (2000) Modelling navigational knowledge by route graphs in spatial cognition II (Number 1849 in LNAI), Freksa C, Habel C, Wender K (eds) ed: Springer, pp. 295–317Google Scholar
  16. 16.
    Lynch K (1960) The image of the city. The MIT Press, CambridgeGoogle Scholar
  17. 17.
    Golledge RG, Stimson RJ (1996) Spatial behavior: a geographic perspective. The Guilford Press, New YorkGoogle Scholar
  18. 18.
    Tversky B (1993) Cognitive maps, cognitive collages, and spatial mental models. Spatial information theory a theoretical basis for GIS, pp 14–24Google Scholar
  19. 19.
    Fox D, Burgard W, Thrun S (1999) Markov localization for mobile robots in dynamic environments. J Artif Intell Res 11:391–427MATHGoogle Scholar
  20. 20.
    Brunskill E, Kollar T, Roy N (2007) Topological mapping using spectral clustering and classification. Presented at the IEEE/RSJ international conference on intelligent robots and systems (IROS2007)Google Scholar
  21. 21.
    Striegnitz K, Tepper P, Lovett A, Cassell J (2005) Knowledge representation for generating locating gestures in route directions. Presented at the Workshop on spatial language and dialogue (5th workshop on language and space)Google Scholar
  22. 22.
    Kopp S, Tepper PA, Ferriman K, Striegnitz K, Cassell J (2008) Trading spaces: how humans and humanoids use speech and gesture to give directions. In: T Nishida (ed) Conversational informatics: an engineering approach, pp. 133–160Google Scholar
  23. 23.
    Durrant-Whyte H, Bailey T (2006) Simultaneous localization and mapping: part i. IEEE Robot Autom Mag 13:99–110CrossRefGoogle Scholar
  24. 24.
    Bailey T, Durrant-Whyte H (2006) Simultaneous localization and mapping (slam): part ii. IEEE Robot Autom Mag 13:108–117CrossRefGoogle Scholar
  25. 25.
    Thrun S, Burgard W, Fox D (2005) Probabilistic robotics (intelligent robotics and autonomous agents). The MIT Press, CambridgeGoogle Scholar
  26. 26.
    Montemerlo M, Thrun S, Koller D, Wegbreit B (2002) FastSLAM: a factored solution to the simultaneous localization and mapping problem. Presented at the proceedings of the AAAI national conference on artificial intelligence, EdmontonGoogle Scholar
  27. 27.
    Olson E, Leonard J, Teller S (2006) Fast iterative alignment of pose graphs with poor initial estimates. Presented at the IEEE international conference on robotics and automation (ICRA 2006)Google Scholar
  28. 28.
    Bosse M, Newman PM, Leonard JJ, Teller S (2004) Slam in large scale cyclic environments using the atlas framework. Int J Robot Res 23:1113–1139CrossRefGoogle Scholar
  29. 29.
    Konolige K (2004) Large-scale map-making. Presented at the national conference on artificial intelligence (AAAI’04)Google Scholar
  30. 30.
    Paul R, Newman P (2010) Fab-map 3d: topological mapping with spatial and visual appearance. Presented at the proceedings IEEE international conference on robotics and automation (ICRA’10), Anchorage, AlaskaGoogle Scholar
  31. 31.
    Elfes A (1989) Using occupancy grids for mobile robot perception and navigation. Computer 22:46–57CrossRefGoogle Scholar
  32. 32.
    Grisetti G, Stachniss C, Burgard W (2007) Improved techniques for grid mapping with rao-blackwellized particle filters. IEEE Trans Robot 23:34–46CrossRefGoogle Scholar
  33. 33.
    Konolige K, Bowman J, Chen J, Mihelich P, Calonder M, Lepetit V et al (2010) View-based maps. Int J Robot Res 29:941–957CrossRefGoogle Scholar
  34. 34.
    Kuipers B (2000) The spatial semantic hierarchy. Artif Intell 119:191–233CrossRefMATHMathSciNetGoogle Scholar
  35. 35.
    Beeson P, Modayil J, Kuipers B (2010) Factoring the mapping problem: mobile robot map-building in the hybrid spatial semantic hierarchy. Int J Robot Res 29:428–459CrossRefGoogle Scholar
  36. 36.
    Zender H, Mozos ÓM, Jensfelt P, Kruijff G-JM, Burgard W (2008) Conceptual spatial representations for indoor mobile robots. Robot Auton Syst 56:493–502CrossRefGoogle Scholar
  37. 37.
    Kollar T, Tellex S, Roy D, Roy N (2010) Toward understanding natural language directions. Presented at the ACM/IEEE international conference on human–robot interaction (HRI2010)Google Scholar
  38. 38.
    Matuszek C, Fox D, Koscher K (2010) Following directions using statistical machine translation. Presented at the ACM/IEEE international conference on human–robot interaction (HRI2010)Google Scholar
  39. 39.
    Yeap WK (1988) Towards a computational theory of cognitive maps. Artif Intell 34:297–360CrossRefGoogle Scholar
  40. 40.
    Mozos ÓM, Triebel R, Jensfelt P, Rottmann A, Burgard W (2007) Supervised semantic labeling of places using information extracted from sensor data. Robot Auton Syst 55:391–402CrossRefGoogle Scholar
  41. 41.
    Vasudevan S, Gachter S, Nguyen V, Siegwart R (2007) Cognitive maps for mobile robots-an object based approach. Robot Auton Syst 55:359–371CrossRefGoogle Scholar
  42. 42.
    Yeap WK, Jefferies ME (1999) Computing a representation of the local environment. Artif Intell 107:265–301CrossRefMATHGoogle Scholar
  43. 43.
    Yeap WK, Jefferies ME (2000) On early cognitive mapping. Spat Cogn Comput 2:85–116Google Scholar
  44. 44.
    Peltason J, Siepmann FHK, Spexard TP, Wrede B, Hanheide M, Topp EA (2009) Mixed-initiative in human augmented mapping. Presented at the IEEE international conference on robotics and automation (ICRA2009)Google Scholar
  45. 45.
    Hato Y, Satake S, Kanda T, Imai M, Hagita N (2010) Pointing to space: modeling of deictic interaction referring to regions. Presented at the ACM/IEEE international conference on human–robot interaction (HRI2010)Google Scholar
  46. 46.
    Kendon A (2004) Gesture: visible action as utterance. Cambridge University Press, New YorkCrossRefGoogle Scholar
  47. 47.
    Kita S (2003) Interplay of gaze, hand, torso orientation and language in pointing. In: Kita S (ed) Pointing: where language, culture, and cognition meet, pp. 307–328Google Scholar
  48. 48.
    Daniel M-P, Tom A, Manghi E, Denis M (2003) Testing the value of route directions through navigational performance. Spat Cogn Comput 3:269–289Google Scholar
  49. 49.
    Vanetti EJ, Allen GL (1988) Communicating environmental knowledge: the impact of verbal and spatial abilities on the production and comprehension of route directions. Environ Behav 20:667–682CrossRefGoogle Scholar
  50. 50.
    Levit M, Roy D (2007) Interpretation of spatial language in a map navigation task. IEEE Trans Syst Man Cybern Part B 37:667–679CrossRefGoogle Scholar
  51. 51.
    Shimizu N, Haas A (2009) Learning to follow navigational route instructions. Presented at the international Joint conference on artificial intelligence (IJCAI2009)Google Scholar
  52. 52.
    Vogel A, Jurafsky D (2010) Learning to follow navigational directions, presented at the the 48th annual meeting of the association for computational linguistics stroudsburg, PAGoogle Scholar
  53. 53.
    Look G (2008) Cognitively-inspired direction giving, Ph.D. ThesisGoogle Scholar
  54. 54.
    Ross RJ, Mandel C, Bateman JA, Hui S, Frese U, Towards stratified spatial modeling for communication & navigation, from sensors to human spatial concepts, p. 41Google Scholar
  55. 55.
    Shi H, Tenbrink T (2005) Telling rolland where to go: HRI dialogues on route navigation. In: Proceedings of WoSLaD workshop on spatial language and dialogueGoogle Scholar
  56. 56.
    Mandel C, Frese U, Rofer T (2006) Robot navigation based on the mapping of coarse qualitative route descriptions to route graphs, in IEEE/RSJ international conference on intelligent robots and systems (IROS), pp. 205–210Google Scholar
  57. 57.
    Nothegger C, Winter S, Raubal M (2004) Selection of salient features for route directions. Spat Cogn Comput 4:113–136Google Scholar
  58. 58.
    Montello DR (1997) The perception and cognition of environmental distance: direct sources of information. Presented at the proceedings of the international conference on spatial information theory: a theoretical basis for GISGoogle Scholar
  59. 59.
    Richter K-F, Klippel A (2004) A model for context-specific route directions. Lecture notes in computer science: SPATIAL COGNITION IV. REASONING, ACTION. INTERACTION 3343/2005, 58–78Google Scholar
  60. 60.
    Klippel A, Hansen S, Davies J, Winter S (2005) A high-level cognitive framework for route directions. Presented at the the national biennial conference of the spatial science instituteGoogle Scholar
  61. 61.
    Denis M (1997) The description of routes: a cognitive approach to the production of spatial discourse. Cah de Psychol Cogn 16:409–458Google Scholar
  62. 62.
    Nuechter A (2009) 3D robotic mapping. Springer Verlag, HeidelbergMATHGoogle Scholar
  63. 63.
    Moravec HP (1989) Certainty grids for sensor fusion in mobile robots, Sensor devices and systems for robotics, pp. 243–276Google Scholar
  64. 64.
    Besl P, McKay N (1992) A method for registration of 3-D shapes. IEEE Trans Pattern Anal Mach Intell 14:239–256CrossRefGoogle Scholar
  65. 65.
    Nuchter A, Lingemann K, Hertzberg J, Surmann H (2007) 6d slam-3d mapping outdoor environments. J Field Robot 24:699–722CrossRefGoogle Scholar
  66. 66.
    Remolina E, Kuipers B (2004) Towards a general theory of topological maps. Artif Intell 152:47–104CrossRefMATHMathSciNetGoogle Scholar
  67. 67.
    Nagatani K, Choset H (1999) Toward robust sensor based exploration by constructing reduced generalized voronoi graph. Presented at the IEEE/RSJ international conference on intelligent robots and systemsGoogle Scholar
  68. 68.
    Morris AC, Silver D, Ferguson D, Thayer S (2005) Towards topological exploration of abandoned mines. Presented at the the IEEE international conference on robotics and automation (ICRA2005), BarcelonaGoogle Scholar
  69. 69.
    Dellaert F, Fox D, Burgard W, Thrun S (1999) Monte carlo localization for mobile robots, in robotics and automation, 1999. Proceedings. 1999 IEEE international conference on, pp. 1322–1328.Google Scholar
  70. 70.
    Kuipers B (1978) Modeling spatial knowledge. Cogn Sci 2:129–153Google Scholar
  71. 71.
    Smith R (2007) An overview of the tesseract OCR engine. Presented at the Ninth international conference on document analysis and recognition (ICDAR 2007)Google Scholar
  72. 72.
    Levenshtein VI (1966) Binary codes capable of correcting deletions, insertions, and reversals. Sov Phys Dokl 10:707–710MathSciNetGoogle Scholar

Copyright information

© Springer Science+Business Media Dordrecht 2014

Authors and Affiliations

  • Yoichi Morales
    • 1
  • Satoru Satake
    • 1
  • Takayuki Kanda
    • 1
  • Norihiro Hagita
    • 1
  1. 1.Intelligent Robotics and Communication LaboratoriesAdvanced Telecommunications Research Institute InternationalKyotoJapan

Personalised recommendations