International Journal of Social Robotics

, Volume 1, Issue 1, pp 83–93 | Cite as

Where Robots and Virtual Agents Meet

A Survey of Social Interaction Research across Milgram’s Reality-Virtuality Continuum
  • Thomas Holz
  • Mauro DragoneEmail author
  • G. M. P. O’Hare
Original Paper


Traditionally, social interaction research has concentrated on either fully virtually embodied agents (e.g. embodied conversational agents) or fully physically embodied agents (e.g. robots). For some time, however, both areas have started augmenting their agents’ capabilities for social interaction using ubiquitous and intelligent environments.

We are placing different agent systems for social interaction along Milgram’s Reality-Virtuality Continuum—according to the degree they are embodied in a physical, virtual or mixed reality environment—and show systems that follow the next logical step in this progression, namely social interaction in the middle of Milgram’s continuum, that is, agents richly embodied in the physical and virtual world.

This paper surveys the field of social interaction research with embodied agents with a particular view towards their embodiment forms and highlights some of the advantages and issues associated with the very recent field of social interaction with mixed reality agents.


Social interaction Mixed reality agents Human-robot interaction Human-computer interaction 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Ambrose RO, Aldridge H, Askew RS, Burridge RR, Bluethmann W, Diftler M, Lovchik C, Magruder D, Rehnmark F (2000) Robonaut: NASA’s space humanoid. IEEE Int Syst 15(4):57–63 CrossRefGoogle Scholar
  2. 2.
    Anabuki M, Kakuta H, Yamamoto H, Tamura H (2000) Welbo: an embodied conversational agent living in mixed reality space. In: CHI ’00: extended abstracts on human factors in computing systems. ACM, New York, pp 10–11 CrossRefGoogle Scholar
  3. 3.
    Barakonyi I, Psik T, Schmalstieg D (2004) Agents that talk and hit back: animated agents in augmented reality. In: Proceedings of the 3rd IEEE and ACM international symposium on mixed and augmented reality (ISMAR ’04), pp 141–150 Google Scholar
  4. 4.
    Bartneck C (2001) How convincing is Mr Data’s smile: affective expressions of machines. User Mod User-Adapt Interact 11(4):279–295 zbMATHCrossRefGoogle Scholar
  5. 5.
    Bartneck C (2002) eMuu—an embodied emotional character for the ambient intelligent home. PhD thesis, Eindhoven University of Technology, Eindhoven, Netherlands, unpublished Google Scholar
  6. 6.
    Bartneck C (2003) Interacting with an embodied emotional character. In: DPPI ’03: proceedings of the 2003 international conference on designing pleasurable products and interfaces. ACM, New York, pp 55–60 CrossRefGoogle Scholar
  7. 7.
    Baylor AL, Rosenberg-Kima RB, Plant EA (2006) Interface agents as social models: the impact of appearance on females’ attitude toward engineering. In: CHI ’06: extended abstracts on human factors in computing systems. ACM, New York, pp 526–531 CrossRefGoogle Scholar
  8. 8.
    Breazeal C (2002) Designing sociable robots. MIT Press, Cambridge Google Scholar
  9. 9.
    Breazeal C (2003) Toward sociable robots. Robot Auton Syst 42(3):167–175 zbMATHCrossRefGoogle Scholar
  10. 10.
    van Breemen A, Yan X, Meerbeek B (2005) iCat: an animated user-interface robot with personality. In: Proceedings of the 4th international joint conference on autonomous agents and multiagent systems (AAMAS ’05). ACM, New York, pp 143–144 CrossRefGoogle Scholar
  11. 11.
    Chittaro L, Ieronutti L, Rigutti S (2005) Supporting presentation techniques based on virtual humans in educational virtual worlds. In: CW ’05: proceedings of the 2005 international conference on cyberworlds. IEEE Computer Society, Washington, pp 245–252 CrossRefGoogle Scholar
  12. 12.
    Collett T, MacDonald B (2006) Augmented reality visualisation for player. In: Proceedings of the 2006 IEEE international conference on robotics and automation (ICRA ’06), pp 3954–3959 Google Scholar
  13. 13.
    Collier RW (2001) Agent factory: a framework for the engineering of agent-oriented applications. PhD thesis, University College Dublin, Dublin, Ireland Google Scholar
  14. 14.
    Dautenhahn K (1997) I could be you: the phenomenological dimension of social understanding. Cybern Syst 28:417–453 (37) CrossRefGoogle Scholar
  15. 15.
    Dautenhahn K (1998) The art of designing socially intelligent agents: science, fiction, and the human in the loop. Appl Artif Intell 12:573–617 (45) CrossRefGoogle Scholar
  16. 16.
    Dautenhahn K (2004) Robots we like to live with?!—a developmental perspective on a personalized, life-long robot companion. In: Proceedings of the 13th IEEE international workshop on robot and human interaction communication (RO-MAN ’04), pp 17–22 Google Scholar
  17. 17.
    Dautenhahn K (2007) Socially intelligent robots: dimensions of human-robot interaction. Philos Trans R Soc B: Biol Sci 362(1480):679–804 CrossRefGoogle Scholar
  18. 18.
    Doswell JT (2005) It’s virtually pedagogical: pedagogical agents in mixed reality learning environments. In: ACM SIGGRAPH 2005 educators program. ACM, New York, p 25 CrossRefGoogle Scholar
  19. 19.
    Dragone M, Holz T, O’Hare GMP (2006) Mixing robotic realities. In: Proceedings of the 11th international conference on intelligent user interfaces (IUI ’06). ACM, New York, pp 261–263 CrossRefGoogle Scholar
  20. 20.
    Dragone M, Holz T, O’Hare GMP (2007) Using mixed reality agents as social interfaces for robots. In: Proceedings of the 16th IEEE international workshop on robot and human interaction communication (RO-MAN ’07). IEEE Press, Jeju Island Google Scholar
  21. 21.
    Duffy BR, O’Hare GMP, O’Donoghue RP, Rooney CF, Collier RW (1999) Reality and virtual reality in mobile robotics. In: Proceedings of the 1st international workshop on managing interactions in smart environments (MANSE ’99), Dublin, Ireland, 13–14 December 1999 Google Scholar
  22. 22.
    Duffy BR, O’Hare GMP, Martin A, Bradley JF, Schön B (2003) Agent chameleons: agent minds and bodies. In: Proceedings of the 16th international conference on computer animation and social agents (CASA ’03), pp 118–125 Google Scholar
  23. 23.
    Fong T, Nourbakhsh I, Dautenhahn K (2003) A survey of socially interactive robots. Robot Auton Syst 42(3–4):143–166 zbMATHCrossRefGoogle Scholar
  24. 24.
    Fujita Y (2002) Personal robot PaPeRo. J Robot Mechatron 14(1):60–63 Google Scholar
  25. 25.
    Gockley R, Bruce A, Forlizzi J, Michalowski M, Mundell A, Rosenthal S, Sellner B, Simmons R, Snipes K, Schultz AC, Wang J (2005) Designing robots for long-term social interaction. In: Proceedings of the 2005 IEEE/RSJ international conference on intelligent robots and systems (IROS ’05), pp 1338–1343 Google Scholar
  26. 26.
    Ha YG, Sohn JC, Cho YJ, Yoon H (2005) Towards a ubiquitous robotic companion: design and implementation of ubiquitous robotic service framework. ETRI J 27(6):666–676 Google Scholar
  27. 27.
    Hanson D, Olney A, Prilliman S, Mathews E, Zielke M, Hammons D, Fernandez R, Stephanou H (2005) Upending the uncanny valley. In: Proceedings of the national conference on artificial intelligence (AAAI ’05), Pittsburgh, PA, USA Google Scholar
  28. 28.
    Holz T, Dragone M, O’Hare GMP, Martin A, Duffy BR (2006) Mixed reality agents as museum guides. In: ABSHL ’06: agent-based systems for human learning workshop. ACM, New York Google Scholar
  29. 29.
    Ishiguro H (2007) Android science: toward a new cross-disciplinary framework. In: Springer tracts in advanced robotics. Springer, Berlin, pp 118–127 Google Scholar
  30. 30.
    Johnson WL, Rickel J (1997) Steve: an animated pedagogical agent for procedural training in virtual environments. SIGART Bull 8(1–4):16–21 CrossRefGoogle Scholar
  31. 31.
    Kidd CD (2003) Sociable robots: the role of presence and task in human-robot interaction. Master’s thesis, MIT Media Lab, Cambridge, MA, USA Google Scholar
  32. 32.
    Kidd CD, Breazeal C (2005) Sociable robot systems for real-world problems. In: Proceedings of the IEEE international workshop on robot and human interaction communication (ROMAN ’05), pp 353–358 Google Scholar
  33. 33.
    Kim BK, Ohara K, Ohba K, Tanikawa T, Hirai S (2005) Design of ubiquitous functions for networked robots in the informative spaces. In: Proceedings of the 2nd international conference on ubiquitous robots and ambient intelligence, pp 71–76 Google Scholar
  34. 34.
    Kim JH (2005) Ubiquitous robot. In: Computer intelligence, theory and applications, vol 33. Springer, Berlin CrossRefGoogle Scholar
  35. 35.
    Klein J, Moon Y, Picard RW (1999) This computer responds to user frustration. In: CHI ’99: extended abstracts on human factors in computing systems. ACM, New York, pp 242–243 CrossRefGoogle Scholar
  36. 36.
    Kleweno CP, Seibel EJ, Viirre ES, Kelly JP, Furness III TA (2001) The virtual retinal display as a low-vision computer interface: a pilot study. J Rehab Res Dev 38(4):431–442 Google Scholar
  37. 37.
    Kopp S, Gesellensetter L, Krämer NC, Wachsmuth I (2005) A conversational agent as museum guide: design and evaluation of a real-world application. In: Proceedings of the 5th international working conference on intelligent virtual agents (IVA ’05). Springer, London, pp 329–343 Google Scholar
  38. 38.
    Lee KM, Jung Y, Kim J, Kim SR (2006) Are physically embodied social agents better than disembodied social agents? The effects of physical embodiment, tactile interaction, and people’s loneliness in human-robot interaction. Int J Hum-Comput Stud 64(10):962–973 CrossRefGoogle Scholar
  39. 39.
    MacDorman KF (2005) Androids as an experimental apparatus: why is there an uncanny valley and can we exploit it? In: CogSci ’05 workshop: toward social mechanisms of android science, pp 106–118 Google Scholar
  40. 40.
    Maeyama S, Yuta S, Harada A (1998) Remote museum system using a networked robot. Adv Robot 13:315–316 (2) CrossRefGoogle Scholar
  41. 41.
    Martin A, O’Hare GMP, Duffy BR, Schön B, Bradley JF (2005) Maintaining the identity of dynamically embodied agents. In: Proceedings of the 5th international working conference on intelligent virtual agents (IVA ’05). Springer, London, pp 454–465 Google Scholar
  42. 42.
    Milgram P, Rastogi A, Grodski JJ (1995) Telerobotic control using augmented reality. In: Proceedings of the 4th IEEE international workshop on robot and human communication (RO-MAN 1995), pp 21–29 Google Scholar
  43. 43.
    Milgram P, Takemura H, Utsumi A, Kishino F (1995) Augmented reality: a class of displays on the reality-virtuality continuum. In: Proceedings of the SPIE conference on telemanipulator and telepresence technologies, Boston, MA, USA. Proceedings of SPIE, vol 2351, pp 282–292 Google Scholar
  44. 44.
    Mori M (1982) The Buddha in the Robot. C.E. Tuttle Google Scholar
  45. 45.
    Muldoon C, O’Hare GMP, O’Grady MJ (2006) Collaborative agent tuning: performance enhancement on mobile devices. In: Proceedings of the 6th international workshop on engineering societies in the agents’ world (ESAW ’05), revised selected and invited papers, Kusadasi, Turkey, 26–28 October 2005. Lecture notes in computer science, vol 3963. Springer, Berlin, pp 241–258 Google Scholar
  46. 46.
    Nourbakhsh IR, Kunz C, Willeke T (2003) The mobot museum robot installations: a five year experiment. In: Proceedings of the 2003 IEEE/RSJ international conference on intelligent robots and systems (IROS ’03), vol 4, pp 3636–3641 Google Scholar
  47. 47.
    Ogden B, Dautenhahn K (2000) Robotic etiquette: Structured interaction in humans and robots. In: Proceedings of the symposium on intelligent robots and systems (SIRS ’00), pp 353–361 Google Scholar
  48. 48.
    O’Hare GMP (1996) The agent factory: An environment for the fabrication of distributed artificial systems. In: O’Hare GMP, Jennings NR (eds) Foundations of distributed artificial intelligence, sixth generation computer series. Wiley Interscience, New York, pp 449–484 Google Scholar
  49. 49.
    O’Hare GMP, Duffy BR (2002) Agent chameleons: migration and mutation within and between real and virtual spaces. In: Proceedings of the AISB ’02 symposium on animated expressive characters for social interactions, 3–5 April 2002. Imperial College of Science & Technology, London, pp 63–68 Google Scholar
  50. 50.
    O’Hare GMP, Duffy BR, Bradley JF, Martin AN (2003) Moving minds from robots to digital information spaces. In: Proceedings of the 2nd international symposium on autonomous minirobots for research and edutainment, Brisbane, Australia, 18–21 February 2003 Google Scholar
  51. 51.
    O’Hare GMP, Duffy BR, Schön B, Martin A, Bradley JF (2003) Agent chameleons: virtual agents real intelligence. In: Proceedings of the 4th international working conference on intelligent virtual agents (IVA ’03), Irsee, Germany, 15–17 September 2003. Lecture notes in computer science, vol 2792. Springer, Berlin, pp 218–225 Google Scholar
  52. 52.
    O’Hare GMP, Campbell AG, Stafford JW, Aiken R (2005) NeXuS: behavioural realism in mixed reality scenarios through virtual sensing. In: Proceedings of the 18th international conference on computed animation and social agents (CASA ’05) Google Scholar
  53. 53.
    Pacchierotti E, Christensen H, Jensfelt P (2006) Evaluation of passing distance for social robots. In: Proceedings of the 15th IEEE international symposium on robot and human interaction communication (RO-MAN ’06), pp 315–320 Google Scholar
  54. 54.
    Prendingera H, Dohi H, Wang H, Mayer S, Ishizuka M (2004) Empathic embodied interfaces: addressing users’ affective state. In: Proceedings of the tutorial and research workshop on affective dialogue systems. Springer, Berlin, pp 53–64 Google Scholar
  55. 55.
    Rao AS, Georgeff MP (1991) Modeling rational agents within a BDI-architecture. In: Allen J, Fikes R, Sandewall E (eds) Proceedings of the 2nd international conference on principles of knowledge representation and reasoning (KR ’91). Morgan Kaufmann, San Mateo Google Scholar
  56. 56.
    Reeves B, Nass C (1998) The media equation: how people treat computers, television, and new media like real people and places. Cambridge University Press, New York Google Scholar
  57. 57.
    Rickenberg R, Reeves B (2000) The effects of animated characters on anxiety, task performance, and evaluations of user interfaces. In: CHI ’00: Proceedings of the SIGCHI conference on human factors in computing systems. ACM, New York, pp 49–56 CrossRefGoogle Scholar
  58. 58.
    Shoji M, Miura K, Konno A (2006) U-Tsu-Shi-O-Mi: the virtual humanoid you can reach. In: ACM SIGGRAPH 2006 emergence technologies. ACM, New York, p 34 CrossRefGoogle Scholar
  59. 59.
    Simmons R, Fernandez J, Goodwin R, Koenig S, O’Sullivan J (2000) Lessons learned from Xavier. IEEE Robot Autom Mag 7(2):33–39 CrossRefGoogle Scholar
  60. 60.
    Simmons R, Goldberg D, Goode A, Montemerlo M, Roy N, Sellner B, Urmson C, Schultz A, Abramson M, Adams W, Atrash A, Bugajska M, Coblenz M, MacMahon M, Perzanowski D, Horswill I, Zubek R, Kortenkamp D, Wolfe B, Milam T, Maxwell B (2003) GRACE: an autonomous robot for the AAAI robot challenge. AI Mag 24(2):51–72 Google Scholar
  61. 61.
    Slater M, Pertaub DP, Steed A (1999) Public speaking in virtual reality: facing an audience of avatars. IEEE Comput Graph Appl 19(2):6–9 CrossRefGoogle Scholar
  62. 62.
    Thrun S, Bennewitz M, Burgard W, Cremers AB, Dellaert F, Fox D, Hähnel D, Rosenberg C, Roy N, Schulte J, Schulz D (1999) MINERVA: a second-generation museum tour-guide robot. In: Proceedings of the 1999 IEEE international conference on robotics and automation, vol 3, pp 1999–2005 Google Scholar
  63. 63.
    Wagner D, Billinghurst M, Schmalstieg D (2006) How real should virtual characters be? In: ACE ’06: proceedings of the 2006 ACM SIGCHI international conference on advances in computer entertainment technology. ACM, New York, p 57 CrossRefGoogle Scholar
  64. 64.
    Yamato J, Shinozawa K, Naya F, Kogure K (2001) Evaluation of communication with robot and agent: are robots better social actors than agents? In: Proceedings of the 8th IFIP TC 13th international conference on human-computer interaction (INTERACT ’01), Tokyo, Japan, pp 690–691 Google Scholar
  65. 65.
    Yamato J, Shinozawa K, Brooks R, Naya F (2003) Human-robot dynamic social interaction. NTT Tech Rev 1(6):37–43 Google Scholar
  66. 66.
    Young JE, Xin M, Sharlin E (2007) Robot expressionism through cartooning. In: Proceedings of the ACM/IEEE international conference on human-robot interaction (HRI ’07). ACM, New York, pp 309–316 CrossRefGoogle Scholar
  67. 67.
    Zanbaka CA, Ulinski AC, Goolkasian P, Hodges LF (2007) Social responses to virtual humans: implications for future interface design. In: CHI ’07: proceedings of the SIGCHI conference on human factors in computer systems. ACM, New York, pp 1561–1570 CrossRefGoogle Scholar
  68. 68.
    Ziemke T (2003) What’s that thing called embodiment? In: Proceedings of the 25th annual meeting of the cognitive sciences society Google Scholar

Copyright information

© Springer 2008

Authors and Affiliations

  • Thomas Holz
    • 1
  • Mauro Dragone
    • 1
    Email author
  • G. M. P. O’Hare
    • 1
  1. 1.CLARITY Centre for Sensor Web Technologies, School of Computer Science & InformaticsUniversity College DublinDublinIreland

Personalised recommendations