International Journal of Social Robotics

, Volume 4, Issue 3, pp 263–272 | Cite as

Telerobotic Pointing Gestures Shape Human Spatial Cognition

  • John-John CabibihanEmail author
  • Wing-Chee So
  • Sujin Saj
  • Zhengchen Zhang


This paper aimed to explore whether human beings can understand gestures produced by telepresence robots. If it were the case, they can derive meaning conveyed in telerobotic gestures when processing spatial information. We conducted two experiments over Skype in the present study. Participants were presented with a robotic interface that had arms, which were teleoperated by an experimenter. The robot can point to virtual locations that represented certain entities. In Experiment 1, the experimenter described spatial locations of fictitious objects sequentially in two conditions: speech only condition (SO, verbal descriptions clearly indicated the spatial layout) and speech and robotic gesture condition (SR, verbal descriptions were ambiguous but accompanied by robotic pointing gestures). Participants were then asked to recall the objects’ spatial locations. We found that the number of spatial locations recalled in the SR condition was on par with that in the SO condition, suggesting that telerobotic pointing gestures compensated ambiguous speech during the process of spatial information. In Experiment 2, the experimenter described spatial locations non-sequentially in the SR and SO conditions. Surprisingly, the number of spatial locations recalled in the SR condition was even higher than that in the SO condition, suggesting that telerobotic pointing gestures were more powerful than speech in conveying spatial information when information was presented in an unpredictable order. The findings provide evidence that human beings are able to comprehend telerobotic gestures, and importantly, integrate these gestures with co-occurring speech. This work promotes engaging remote collaboration among humans through a robot intermediary.


Pointing gesture Spatial memory Telepresence robots Human-robot interaction Social robotics 



This work was supported by the National University of Singapore Academic Research Funding Grant No. R-263-000-576-112. Special thanks to Lwin Htay Thet and Nagasubramaniam Kumarappan for their assistance in the experiments and pilot studies.

Supplementary material

Sequential Order, Speech Only Condition (WMV 3.6 MB)

Sequential Order, Speech with Robot Pointing Condition (WMV 3.5 MB)

Non-Sequential Order, Speech Only Condition (WMV 3.8 MB)

Non-Sequential Order, Speech with Robot Pointing Condition (WMV 3.8 MB)


  1. 1.
    Minsky M (1980) Telepresence. Omni magazine, June Google Scholar
  2. 2.
    Minsky M (2010) Telepresence: a manifesto. IEEE Spectrum, September Google Scholar
  3. 3.
    McNeill D, Cassell J, Levy ET (1993) Abstract deixis. Semiotica 95:5–19 CrossRefGoogle Scholar
  4. 4.
    Alibali MW, Kita S, Young AJ (2000) Gesture and the process of speech production: we think, therefore we gesture. Lang Cogn Process 15:593–613 CrossRefGoogle Scholar
  5. 5.
    Woodall WG, Folger JP (1985) Nonverbal cue context and episodic memory: on the availability and endurance of nonverbal behaviors as retrieval cues. Commun Monogr 52:319–333 CrossRefGoogle Scholar
  6. 6.
    Guizzo E (2010) When my avatar went to work. IEEE Spectrum 47 Google Scholar
  7. 7.
    Petelin JB, Nelson ME, Goodman J (2007) Deployment and early experience with remote-presence patient care in a community hospital. Surg Endosc Other Interv Tech 21:53–56 CrossRefGoogle Scholar
  8. 8.
    Ellison LM, Pinto PA, Kim F, Ong AM, Patriciu A, Stoianovici D, Rubin H, Jarrett T, Kavoussi LR (2004) Telerounding and patient satisfaction after surgery. J Am Coll Surg 199:523–530 CrossRefGoogle Scholar
  9. 9.
    Smith CD, Skandalakis JE (2005) Remote presence proctoring by using a wireless remote-control videoconferencing system. Surg Innov 12:139–143 CrossRefGoogle Scholar
  10. 10.
    Chung KK, Grathwohl KW, Poropatich RK, Wolf SE, Holcomb JB (2007) Robotic telepresence: past, present, and future. J Cardiothorac Vasc Anesth 21:593–596 CrossRefGoogle Scholar
  11. 11.
    Gandsas A, Parekh M, Bleech MM, Tong DA (2007) Robotic telepresence: profit analysis in reducing length of stay after laparoscopic gastric bypass. J Am Coll Surg 205:72–77 CrossRefGoogle Scholar
  12. 12.
    Vespa PM, Miller C, Hu X, Nenov V, Buxey F, Martin NA (2007) Intensive care unit robotic telepresence facilitates rapid physician response to unstable patients and decreased cost in neurointensive care. Surg Neurol 67:331–337 CrossRefGoogle Scholar
  13. 13.
    Sampsel D, Bharwani G, Mehling D, Smith S (2010) Robots as faculty: student and faculty perceptions. Clin Simul Nurs Google Scholar
  14. 14.
    Yorita A, Kubota N (2010) Essential technology for team teaching using robot partners. In: Lecture notes in artificial intelligence, vol 6425. Springer, Berlin, pp 517–528 Google Scholar
  15. 15.
    Minamide A, Takemataand K, Hoon PS (2009) Design of engineering education system using long-distance-controlled robots. In: Proc IEEE intl conf on advanced learning technologies, pp 256–257 CrossRefGoogle Scholar
  16. 16.
    Cabibihan JJ, So WC, Nazar M, Ge SS (2009) Pointing gestures for a robot mediated communication interface. In: Xie M, Xiong Y, Xiong C, Liu H, Hu Z (eds) Intelligent robotics and applications, vol 5928. Springer, Berlin, pp 67–77 CrossRefGoogle Scholar
  17. 17.
    Adalgeirsson SO, Breazeal C (2010) MeBot: a robotic platform for socially embodied presence. In: Proc intl conf human-robot interaction, pp 15–22 Google Scholar
  18. 18.
    Paulos E, Canny J (2001) Social tele-embodiment: understanding presence. Auton Robots 11:87–95 CrossRefzbMATHGoogle Scholar
  19. 19.
    Ogawa K, Bartneck C, Sakamoto D, Kanda T, Ono T, Ishiguro H (2009) Can an android persuade you? In: Proc IEEE intl workshop on robot and human interactive communication, pp 516–521 Google Scholar
  20. 20.
    Nishio S, Ishiguro H, Hagita N (2007) Can a teleoperated android represent personal presence?—A case study with children. Psychologia 50:330–342 CrossRefGoogle Scholar
  21. 21.
    Ishiguro H, Nishio S (2007) Building artificial humans to understand humans. J Artif Organs 10:133–142 CrossRefGoogle Scholar
  22. 22.
    Giannoulis G, Coliou M, Kamarinos GS, Roussou M, Trahanias P, Argyros A, Cremers A, Schulz D, Burgard W, Haehnel D, Savvaides V, Freiss P, Konstantios D, Katselaki A (2001) Enhancing museum visitor access through robotic avatars connected to the web. In: Proc of museums and the web, Seattle, USA Google Scholar
  23. 23.
    Burgard W, Cremers AB, Fox D, Hahnel D, Lakemeyer G, Schulz D, Steiner W, Thrun S (1999) Experiences with an interactive museum tour-guide robot. Artif Intell 114:3–55 CrossRefzbMATHGoogle Scholar
  24. 24.
    Cubano M, Poulose BK, Talamini MA, Stewart R, Antosek LE, Lentz R, Nibe R, Kutka MF, Mendoza-Sagaon M (1999) Long distance telementoring: a novel tool for laparoscopy aboard the USS Abraham Lincoln. Surg Endosc 13:673–678 CrossRefGoogle Scholar
  25. 25.
    Latifi R, Peck K, Satava R, Anvari M (2004) Telepresence and telementoring in surgery. Stud Health Technol Inform 104:200–206 Google Scholar
  26. 26.
    Lee BR, Moore R (2000) International telementoring: a feasible method of instruction. World J Urol 18:296–298 CrossRefGoogle Scholar
  27. 27.
    Sebajang H, Trudeau P, Dougall A, Hegge S, McKinley C, Anvari M (2006) The role of telementoring and telerobotic assistance in the provision of laparoscopic colorectal surgery in rural areas. Surg Endosc Other Interv Tech 20:1389–1393 CrossRefGoogle Scholar
  28. 28.
    Agarwal R, Levinson AW, Allaf M, Markov D, Nason A, Su LM (2007) The RoboConsultant: telementoring and remote presence in the operating room during minimally invasive urologic surgeries using a novel mobile robotic interface. Urology 70:970–974 CrossRefGoogle Scholar
  29. 29.
    Sereno S, Mutter D, Dallemagne B, Smith CD, Marescaux J (2007) Telementoring for minimally invasive surgical training by wireless robot. Surg Innov 14:184–191 CrossRefGoogle Scholar
  30. 30.
    Rothenberg SS, Yoder S, Kay S, Ponsky T (2009) Initial experience with surgical telementoring in pediatric laparoscopic surgery using remote presence technology. J Laparoendosc Adv Surg Tech 19:S219–S222 CrossRefGoogle Scholar
  31. 31.
    Hintzman DL, O’Dell CS, Arndt DR (1981) Orientation in cognitive maps. Cogn Psychol 13:149–206 CrossRefGoogle Scholar
  32. 32.
    Sohn MH, Carlson RA (2003) Viewpoint alignment and response conflict during spatial judgment. Psychon Bull Rev 10:907–916 CrossRefGoogle Scholar
  33. 33.
    Avraamides MN, Ioannidou LM, Kyranidou MN (2007) Locating targets from imagined perspectives: comparing labelling with pointing responses. Q J Exp Psychol 60:1660–1679 CrossRefGoogle Scholar
  34. 34.
    Newcombe N, Huttenlocher J (2000) Making space: the development of spatial representation and reasoning. MIT Press, Cambridge Google Scholar
  35. 35.
    Kita S (2003) Pointing: where language, culture and cognition meet. Erlbaum Publishers Hillsdale Google Scholar
  36. 36.
    Bangerter A (2004) Using pointing and describing to achieve joint focus of attention in dialogue. Psychol Sci 15:415–419 CrossRefGoogle Scholar
  37. 37.
    Kendon A (2004) Gesture: visible action as utterance. Cambridge Univ Press, Cambridge Google Scholar
  38. 38.
    de Ruiter JP (2006) Can gesticulation help aphasic people speak, or rather, communicate? Adv Speech-Lang Pathol 8:124–127 CrossRefGoogle Scholar
  39. 39.
    Kanda T, Kamasima M, Imai M, Ono T, Sakamoto D, Ishiguro H, Anzai Y (2007) A humanoid robot that pretends to listen to route guidance from a human. Auton Robots 22:87–100 CrossRefGoogle Scholar
  40. 40.
    Oberman LM, McCleery JP, Ramachandran VS, Pineda JA (2007) EEG evidence for mirror neuron activity during the observation of human and robot actions: toward an analysis of the human qualities of interactive robots. Neurocomputing 70:2194–2203 CrossRefGoogle Scholar
  41. 41.
    McNeill D (1992) Hand and mind: what gestures reveal about thought. Univ of Chicago Press, Chicago Google Scholar
  42. 42.
    So WC, Kita S, Goldin-Meadow S (2009) Using the hands to identify who does what to whom: gesture and speech go hand-in-hand. Cogn Sci 33:115–125 CrossRefGoogle Scholar
  43. 43.
    Alibali MW (2005) Gesture in spatial cognition: expressing, communicating, and thinking about spatial information. Spat Cogn Comput 5:307–331 CrossRefGoogle Scholar
  44. 44.
    Lavergne J, Kimura D (1987) Hand movement asymmetry during speech: no effect of speaking topic. Neuropsychologia 25:689–693 CrossRefGoogle Scholar
  45. 45.
    Kita S (2000) How representational gestures help speaking. In: McNeill D (ed) Language and gesture. Cambridge University Press, Cambridge, pp 162–185 CrossRefGoogle Scholar
  46. 46.
    Paivio A (1971) Imagery and verbal processes. Holt, Rinehart & Winston, New York Google Scholar
  47. 47.
    Paivio A (1986) Mental representations: a dual coding approach. Oxford Univ Press, New York Google Scholar
  48. 48.
    Kulhavy RW, Lee JB, Caterino LC (1985) Conjoint retention of maps and related discourse. Contemp Educ Psychol 10:28–37 CrossRefGoogle Scholar
  49. 49.
    Kulhavy RW, Stock WA, Kealy WA (1993) How geographic maps increase recall of instructional text. Educ Technol Res Dev 41:47–62 CrossRefGoogle Scholar
  50. 50.
    Kulhavy RW, Stock WA, Caterino LC (1994) Reference maps as a framework for remembering text. Adv Psychol 108:153–162 CrossRefGoogle Scholar
  51. 51.
    Clark HH (2003) Pointing and placing. In: Kita S (ed) Pointing: where language, culture and cognition meet. Erlbaum Publishers, Hillsdale, pp 243–268 Google Scholar
  52. 52.
    Kraut RE, Fussell SR, Siegel J (2003) Visual information as a conversational resource in collaborative physical tasks. Hum-Comput Interact 18:13–49 CrossRefGoogle Scholar
  53. 53.
    Clark HH (1996) Using language. Cambridge Univ Press, New York CrossRefGoogle Scholar
  54. 54.
    Olson GM, Olson JS (2000) Distance matters. Hum-Comput Interact 15:139–178 CrossRefGoogle Scholar
  55. 55.
    Fussell SR, Setlock LD, Yang J, Ou J, Mauer E, Kramer ADI (2004) Gestures over video streams to support remote collaboration on physical tasks. Hum-Comput Interact 19:273–309 CrossRefGoogle Scholar

Copyright information

© Springer Science & Business Media BV 2012

Authors and Affiliations

  • John-John Cabibihan
    • 1
    • 2
    Email author
  • Wing-Chee So
    • 3
  • Sujin Saj
    • 1
    • 2
  • Zhengchen Zhang
    • 1
    • 2
  1. 1.Social Robotics Laboratory, Interactive and Digital Media InstituteNational University of SingaporeSingaporeSingapore
  2. 2.Department of Electrical and Computer EngineeringNational University of SingaporeSingaporeSingapore
  3. 3.Department of Educational PsychologyThe Chinese University of Hong KongNew TerritoriesHong Kong S.A.R.

Personalised recommendations