Skip to main content
SpringerLink
Log in

Telerobotic Pointing Gestures Shape Human Spatial Cognition

  • Published:
International Journal of Social Robotics Aims and scope Submit manuscript

Abstract

This paper aimed to explore whether human beings can understand gestures produced by telepresence robots. If it were the case, they can derive meaning conveyed in telerobotic gestures when processing spatial information. We conducted two experiments over Skype in the present study. Participants were presented with a robotic interface that had arms, which were teleoperated by an experimenter. The robot can point to virtual locations that represented certain entities. In Experiment 1, the experimenter described spatial locations of fictitious objects sequentially in two conditions: speech only condition (SO, verbal descriptions clearly indicated the spatial layout) and speech and robotic gesture condition (SR, verbal descriptions were ambiguous but accompanied by robotic pointing gestures). Participants were then asked to recall the objects’ spatial locations. We found that the number of spatial locations recalled in the SR condition was on par with that in the SO condition, suggesting that telerobotic pointing gestures compensated ambiguous speech during the process of spatial information. In Experiment 2, the experimenter described spatial locations non-sequentially in the SR and SO conditions. Surprisingly, the number of spatial locations recalled in the SR condition was even higher than that in the SO condition, suggesting that telerobotic pointing gestures were more powerful than speech in conveying spatial information when information was presented in an unpredictable order. The findings provide evidence that human beings are able to comprehend telerobotic gestures, and importantly, integrate these gestures with co-occurring speech. This work promotes engaging remote collaboration among humans through a robot intermediary.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  1. Minsky M (1980) Telepresence. Omni magazine, June

  2. Minsky M (2010) Telepresence: a manifesto. IEEE Spectrum, September

  3. McNeill D, Cassell J, Levy ET (1993) Abstract deixis. Semiotica 95:5–19

    Article  Google Scholar 

  4. Alibali MW, Kita S, Young AJ (2000) Gesture and the process of speech production: we think, therefore we gesture. Lang Cogn Process 15:593–613

    Article  Google Scholar 

  5. Woodall WG, Folger JP (1985) Nonverbal cue context and episodic memory: on the availability and endurance of nonverbal behaviors as retrieval cues. Commun Monogr 52:319–333

    Article  Google Scholar 

  6. Guizzo E (2010) When my avatar went to work. IEEE Spectrum 47

  7. Petelin JB, Nelson ME, Goodman J (2007) Deployment and early experience with remote-presence patient care in a community hospital. Surg Endosc Other Interv Tech 21:53–56

    Article  Google Scholar 

  8. Ellison LM, Pinto PA, Kim F, Ong AM, Patriciu A, Stoianovici D, Rubin H, Jarrett T, Kavoussi LR (2004) Telerounding and patient satisfaction after surgery. J Am Coll Surg 199:523–530

    Article  Google Scholar 

  9. Smith CD, Skandalakis JE (2005) Remote presence proctoring by using a wireless remote-control videoconferencing system. Surg Innov 12:139–143

    Article  Google Scholar 

  10. Chung KK, Grathwohl KW, Poropatich RK, Wolf SE, Holcomb JB (2007) Robotic telepresence: past, present, and future. J Cardiothorac Vasc Anesth 21:593–596

    Article  Google Scholar 

  11. Gandsas A, Parekh M, Bleech MM, Tong DA (2007) Robotic telepresence: profit analysis in reducing length of stay after laparoscopic gastric bypass. J Am Coll Surg 205:72–77

    Article  Google Scholar 

  12. Vespa PM, Miller C, Hu X, Nenov V, Buxey F, Martin NA (2007) Intensive care unit robotic telepresence facilitates rapid physician response to unstable patients and decreased cost in neurointensive care. Surg Neurol 67:331–337

    Article  Google Scholar 

  13. Sampsel D, Bharwani G, Mehling D, Smith S (2010) Robots as faculty: student and faculty perceptions. Clin Simul Nurs

  14. Yorita A, Kubota N (2010) Essential technology for team teaching using robot partners. In: Lecture notes in artificial intelligence, vol 6425. Springer, Berlin, pp 517–528

    Google Scholar 

  15. Minamide A, Takemataand K, Hoon PS (2009) Design of engineering education system using long-distance-controlled robots. In: Proc IEEE intl conf on advanced learning technologies, pp 256–257

    Chapter  Google Scholar 

  16. Cabibihan JJ, So WC, Nazar M, Ge SS (2009) Pointing gestures for a robot mediated communication interface. In: Xie M, Xiong Y, Xiong C, Liu H, Hu Z (eds) Intelligent robotics and applications, vol 5928. Springer, Berlin, pp 67–77

    Chapter  Google Scholar 

  17. Adalgeirsson SO, Breazeal C (2010) MeBot: a robotic platform for socially embodied presence. In: Proc intl conf human-robot interaction, pp 15–22

    Google Scholar 

  18. Paulos E, Canny J (2001) Social tele-embodiment: understanding presence. Auton Robots 11:87–95

    Article  MATH  Google Scholar 

  19. Ogawa K, Bartneck C, Sakamoto D, Kanda T, Ono T, Ishiguro H (2009) Can an android persuade you? In: Proc IEEE intl workshop on robot and human interactive communication, pp 516–521

    Google Scholar 

  20. Nishio S, Ishiguro H, Hagita N (2007) Can a teleoperated android represent personal presence?—A case study with children. Psychologia 50:330–342

    Article  Google Scholar 

  21. Ishiguro H, Nishio S (2007) Building artificial humans to understand humans. J Artif Organs 10:133–142

    Article  Google Scholar 

  22. Giannoulis G, Coliou M, Kamarinos GS, Roussou M, Trahanias P, Argyros A, Cremers A, Schulz D, Burgard W, Haehnel D, Savvaides V, Freiss P, Konstantios D, Katselaki A (2001) Enhancing museum visitor access through robotic avatars connected to the web. In: Proc of museums and the web, Seattle, USA

    Google Scholar 

  23. Burgard W, Cremers AB, Fox D, Hahnel D, Lakemeyer G, Schulz D, Steiner W, Thrun S (1999) Experiences with an interactive museum tour-guide robot. Artif Intell 114:3–55

    Article  MATH  Google Scholar 

  24. Cubano M, Poulose BK, Talamini MA, Stewart R, Antosek LE, Lentz R, Nibe R, Kutka MF, Mendoza-Sagaon M (1999) Long distance telementoring: a novel tool for laparoscopy aboard the USS Abraham Lincoln. Surg Endosc 13:673–678

    Article  Google Scholar 

  25. Latifi R, Peck K, Satava R, Anvari M (2004) Telepresence and telementoring in surgery. Stud Health Technol Inform 104:200–206

    Google Scholar 

  26. Lee BR, Moore R (2000) International telementoring: a feasible method of instruction. World J Urol 18:296–298

    Article  Google Scholar 

  27. Sebajang H, Trudeau P, Dougall A, Hegge S, McKinley C, Anvari M (2006) The role of telementoring and telerobotic assistance in the provision of laparoscopic colorectal surgery in rural areas. Surg Endosc Other Interv Tech 20:1389–1393

    Article  Google Scholar 

  28. Agarwal R, Levinson AW, Allaf M, Markov D, Nason A, Su LM (2007) The RoboConsultant: telementoring and remote presence in the operating room during minimally invasive urologic surgeries using a novel mobile robotic interface. Urology 70:970–974

    Article  Google Scholar 

  29. Sereno S, Mutter D, Dallemagne B, Smith CD, Marescaux J (2007) Telementoring for minimally invasive surgical training by wireless robot. Surg Innov 14:184–191

    Article  Google Scholar 

  30. Rothenberg SS, Yoder S, Kay S, Ponsky T (2009) Initial experience with surgical telementoring in pediatric laparoscopic surgery using remote presence technology. J Laparoendosc Adv Surg Tech 19:S219–S222

    Article  Google Scholar 

  31. Hintzman DL, O’Dell CS, Arndt DR (1981) Orientation in cognitive maps. Cogn Psychol 13:149–206

    Article  Google Scholar 

  32. Sohn MH, Carlson RA (2003) Viewpoint alignment and response conflict during spatial judgment. Psychon Bull Rev 10:907–916

    Article  Google Scholar 

  33. Avraamides MN, Ioannidou LM, Kyranidou MN (2007) Locating targets from imagined perspectives: comparing labelling with pointing responses. Q J Exp Psychol 60:1660–1679

    Article  Google Scholar 

  34. Newcombe N, Huttenlocher J (2000) Making space: the development of spatial representation and reasoning. MIT Press, Cambridge

    Google Scholar 

  35. Kita S (2003) Pointing: where language, culture and cognition meet. Erlbaum Publishers Hillsdale

    Google Scholar 

  36. Bangerter A (2004) Using pointing and describing to achieve joint focus of attention in dialogue. Psychol Sci 15:415–419

    Article  Google Scholar 

  37. Kendon A (2004) Gesture: visible action as utterance. Cambridge Univ Press, Cambridge

    Google Scholar 

  38. de Ruiter JP (2006) Can gesticulation help aphasic people speak, or rather, communicate? Adv Speech-Lang Pathol 8:124–127

    Article  Google Scholar 

  39. Kanda T, Kamasima M, Imai M, Ono T, Sakamoto D, Ishiguro H, Anzai Y (2007) A humanoid robot that pretends to listen to route guidance from a human. Auton Robots 22:87–100

    Article  Google Scholar 

  40. Oberman LM, McCleery JP, Ramachandran VS, Pineda JA (2007) EEG evidence for mirror neuron activity during the observation of human and robot actions: toward an analysis of the human qualities of interactive robots. Neurocomputing 70:2194–2203

    Article  Google Scholar 

  41. McNeill D (1992) Hand and mind: what gestures reveal about thought. Univ of Chicago Press, Chicago

    Google Scholar 

  42. So WC, Kita S, Goldin-Meadow S (2009) Using the hands to identify who does what to whom: gesture and speech go hand-in-hand. Cogn Sci 33:115–125

    Article  Google Scholar 

  43. Alibali MW (2005) Gesture in spatial cognition: expressing, communicating, and thinking about spatial information. Spat Cogn Comput 5:307–331

    Article  Google Scholar 

  44. Lavergne J, Kimura D (1987) Hand movement asymmetry during speech: no effect of speaking topic. Neuropsychologia 25:689–693

    Article  Google Scholar 

  45. Kita S (2000) How representational gestures help speaking. In: McNeill D (ed) Language and gesture. Cambridge University Press, Cambridge, pp 162–185

    Chapter  Google Scholar 

  46. Paivio A (1971) Imagery and verbal processes. Holt, Rinehart & Winston, New York

    Google Scholar 

  47. Paivio A (1986) Mental representations: a dual coding approach. Oxford Univ Press, New York

    Google Scholar 

  48. Kulhavy RW, Lee JB, Caterino LC (1985) Conjoint retention of maps and related discourse. Contemp Educ Psychol 10:28–37

    Article  Google Scholar 

  49. Kulhavy RW, Stock WA, Kealy WA (1993) How geographic maps increase recall of instructional text. Educ Technol Res Dev 41:47–62

    Article  Google Scholar 

  50. Kulhavy RW, Stock WA, Caterino LC (1994) Reference maps as a framework for remembering text. Adv Psychol 108:153–162

    Article  Google Scholar 

  51. Clark HH (2003) Pointing and placing. In: Kita S (ed) Pointing: where language, culture and cognition meet. Erlbaum Publishers, Hillsdale, pp 243–268

    Google Scholar 

  52. Kraut RE, Fussell SR, Siegel J (2003) Visual information as a conversational resource in collaborative physical tasks. Hum-Comput Interact 18:13–49

    Article  Google Scholar 

  53. Clark HH (1996) Using language. Cambridge Univ Press, New York

    Book  Google Scholar 

  54. Olson GM, Olson JS (2000) Distance matters. Hum-Comput Interact 15:139–178

    Article  Google Scholar 

  55. Fussell SR, Setlock LD, Yang J, Ou J, Mauer E, Kramer ADI (2004) Gestures over video streams to support remote collaboration on physical tasks. Hum-Comput Interact 19:273–309

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by the National University of Singapore Academic Research Funding Grant No. R-263-000-576-112. Special thanks to Lwin Htay Thet and Nagasubramaniam Kumarappan for their assistance in the experiments and pilot studies.

Author information

Authors and Affiliations

  1. Social Robotics Laboratory, Interactive and Digital Media Institute, National University of Singapore, Singapore, 117576, Singapore

    John-John Cabibihan, Sujin Saj & Zhengchen Zhang

  2. Department of Electrical and Computer Engineering, National University of Singapore, Singapore, 117576, Singapore

    John-John Cabibihan, Sujin Saj & Zhengchen Zhang

  3. Department of Educational Psychology, The Chinese University of Hong Kong, New Territories, Hong Kong S.A.R.

    Wing-Chee So

Authors
  1. John-John Cabibihan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wing-Chee So
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sujin Saj
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zhengchen Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to John-John Cabibihan.

Electronic Supplementary Material

Below are the links to the electronic supplementary material.

Sequential Order, Speech Only Condition (WMV 3.6 MB)

Sequential Order, Speech with Robot Pointing Condition (WMV 3.5 MB)

Non-Sequential Order, Speech Only Condition (WMV 3.8 MB)

Non-Sequential Order, Speech with Robot Pointing Condition (WMV 3.8 MB)

Appendix

Appendix

figure a

Rights and permissions

Reprints and permissions

About this article

Cite this article

Cabibihan, JJ., So, WC., Saj, S. et al. Telerobotic Pointing Gestures Shape Human Spatial Cognition. Int J of Soc Robotics 4, 263–272 (2012). https://doi.org/10.1007/s12369-012-0148-9

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12369-012-0148-9

Keywords

Search

Navigation