The Visual Computer

, 25:1109 | Cite as

Multimodal augmented reality tangible gaming

  • Fotis Liarokapis
  • Louis Macan
  • Gary Malone
  • Genaro Rebolledo-Mendez
  • Sara de Freitas
Original Article


This paper presents tangible augmented reality gaming environment that can be used to enhance entertainment using a multimodal tracking interface. Players can interact using different combinations between a pinch glove, a Wiimote, a six-degrees-of-freedom tracker, through tangible ways as well as through I/O controls. Two tabletop augmented reality games have been designed and implemented including a racing game and a pile game. The goal of the augmented reality racing game is to start the car and move around the track without colliding with either the wall or the objects that exist in the gaming arena. Initial evaluation results showed that multimodal-based interaction games can be beneficial in gaming. Based on these results, an augmented reality pile game was implemented with goal of completing a circuit of pipes (from a starting point to an end point on a grid). Initial evaluation showed that tangible interaction is preferred to keyboard interaction and that tangible games are much more enjoyable.


Serious games Pervasive computing Augmented reality Multimodal interfaces 


  1. 1.
    Tudge, J.R.H.: Processes and consequences of peer collaboration: a Vygotskian analysis. Child Dev. 63, 1364–1379 (1992) CrossRefGoogle Scholar
  2. 2.
    Csikszentmihalyi, M.: Flow: The Psychology of Optimal Experience. Harper and Row, New York (1990) Google Scholar
  3. 3.
    Craig, S., Graesser, A., et al.: Affect and learning: an exploratory look into the role of affect in learning. J. Educ. Media 29, 241–250 (2004) Google Scholar
  4. 4.
    de Freitas, S.: Serious Virtual Worlds: A Scoping Study JISC. Serious Games Institute Coventry University, London (2008) Google Scholar
  5. 5.
    Malone, T., Lepper, M.: Making learning fun. In: Snow, R., Farr, M. (eds.) Aptitude, Learning and Instruction: Co-native and Affective Process Analyses, pp. 223–253. Erlbaum, Lawrence (1987) Google Scholar
  6. 6.
    Lester, J.C., Towns, S.G., et al.: Deictic and emotive communication in animated pedagogical agents. In: Cassell, J., Prevost, S., Sullivant, J., Churchill, E. (eds.) Embodied Conversational Agents, pp. 123–154. MIT Press, Boston (2000) Google Scholar
  7. 7.
    Lester, J.C., Converse, S.A., et al.: Animated pedagogical agents and problem-solving effectiveness: a large-scale empirical evaluation. In: Proc. of the 8th Int’l Conference on Artificial Intelligence in Education, pp. 23–30. IOS Press, Kobe (1997) Google Scholar
  8. 8.
    Yoon, S.-Y., Blumberg, B.M., et al.: Motivation driven learning for interactive synthetic characters. In: Fourth International Conference on Autonomous Agents, Barcelona (2000) Google Scholar
  9. 9.
    Laurel, B.: Interface agents: Metaphors with characters. In: Bradshaw, J.M. (ed.) Software Agents, pp. 67–78. AAAI Press/MIT Press, London (1997) Google Scholar
  10. 10.
    Iuppa, N., Weltman, G., et al.: Bringing Hollywood storytelling techniques to branching storylines for training applications. In: 3rd Narrative and Interactive Learning Environments. Edinburgh, Scotland, pp. 1–8 (2004) Google Scholar
  11. 11.
    Walther, B.K.: Reflections on the methodology of pervasive gaming. In: Proc. of the 2005 ACM SIGCHI Int’l Conference on Advances in Computer Entertainment Technology, pp. 176–179. ACM Press, Valencia (2005), CrossRefGoogle Scholar
  12. 12.
    Oda, O., Lister, L., J., White, S., Feiner, S.: Developing an augmented reality racing game. In: Proc. of the 2nd Int’l Conference on INtelligent TEchnologies for Interactive Entertainment, January 8–10. Cancun, Mexico, Article No. 2 (2008) Google Scholar
  13. 13.
    Beigl, M., Krohn, A., Zimmer, T., Decker, C.: Typical sensors needed in ubiquitous and pervasive computing. In: Proc. of the 1st Int’l Workshop on Networked Sensing Systems (INSS), Tokyo, Japan, June, pp. 153–158 (2004) Google Scholar
  14. 14.
    Dombroviak, K.M., Ramnath, R.: A taxonomy of mobile and pervasive applications. In: Proc. of the 2007 ACM Symposium on Applied Computing, pp. 1609–1615. ACM Press, Seoul (2007) CrossRefGoogle Scholar
  15. 15.
    Billinghurst, M., Kato, H., Poupyrev, I.: The MagicBook: a transitional AR interface. Comput. Graph. 25(5), 745–753 (2001) CrossRefGoogle Scholar
  16. 16.
    Rekimoto, J., Nagao, K.: The world through the computer: computer augmented interaction with real-world environments. In: Proc. of the 8th Annual Symposium on User Interface Software and Technology (UIST’95), Pittsburgh, Pennsylvania, USA, November, pp. 29–36. ACM Press, New York (1995) CrossRefGoogle Scholar
  17. 17.
    Starner, T., Mann, S., et al.: Augmented reality through wearable computing. Presence: Teleoper. Virtual Environ. 6(4), 386–398 (1997) Google Scholar
  18. 18.
    Butz, A., Höllerer, T., et al.: Enveloping users and computers in a collaborative 3D augmented reality. In: Proc. of the 2nd IEEE and ACM International Workshop on Augmented Reality, October, pp. 35–44. IEEE Computer Society, San Francisco (1999), CrossRefGoogle Scholar
  19. 19.
    Sandor, C., Klinker, G.: A rapid prototyping software infrastructure for user interfaces in ubiquitous augmented reality. Personal Ubiquitous Comput. 9(3):169–185 (2005) Google Scholar
  20. 20.
    Reitmayr, G., Schmalstieg, D.: Semantic world models for ubiquitous augmented reality. In: Proc. of Workshop Towards Semantic Virtual Environments’ (SVE 2005), March (2005) Google Scholar
  21. 21.
    Cheok, A.D., Fong, S.W., et al.: Human pacman: a sensing-based mobile entertainment system with ubiquitous computing and tangible interaction. In: Proc. of the 2nd Workshop on Network and System Support for Games, pp. 106–117. ACM Press, California (2003) CrossRefGoogle Scholar
  22. 22.
    Henrysson, A., Billinghurst, M., Ollila, M.: AR tennis. In: International Conference on Computer Graphics and Interactive Techniques Archive ACM SIGGRAPH 2006 Sketches, Article No. 13. ACM Press, New York (2006) Google Scholar
  23. 23.
    Magerkurth, C., Engelke, T., Memisoglu, M.: Augmenting the virtual domain with physical and social elements: towards a paradigm shift in computer entertainment technology. Comput. Entertainment 2(4), 12 (2004) CrossRefGoogle Scholar
  24. 24.
    Stapleton, C.B., Hughes, C.E., Moshell, J.M.: MIXED FANTASY: exhibition of entertainment research for mixed reality. In: Proc. of the 2nd Int’l Symposium on Mixed and Augmented Reality, pp. 354–355. IEEE Computer Society, Tokyo (2003) Google Scholar
  25. 25.
    Korris, J.: Full spectrum warrior: How the institute for creative technologies built a cognitive training tool for the xbox. In: 24th Army Science Conference. Florida, Orlando, December (2004) Google Scholar
  26. 26.
    Hughes, C.E., et al.: Mixed reality in education, entertainment, and training. IEEE Comput. Graph. Appl., November/December: 24–30 (2005) Google Scholar
  27. 27.
    Benford, S., Magerkurth, C., Ljungstrand, P.: Bridging the physical and digital in pervasive gaming. Commun. ACM 48(3), 54–57 (2005) CrossRefGoogle Scholar
  28. 28.
    Magerkurth, C., Engelke, T., Grollman, D.: A component-based architecture for distributed, pervasive gaming applications. In: Proc. of the 2006 ACM SIGCHI International Conference on Advances in Computer Entertainment Technology. ACM Press, California (2006). Article No 15 Google Scholar
  29. 29.
    Goldsmith, D., Liarokapis, F., et al.: Augmented reality environmental monitoring using wireless sensor networks. In: Proc. of the 12th Int’l Conference on Information Visualisation, pp. 539–544. IEEE Computer Society, Los Alamitos (2008) CrossRefGoogle Scholar
  30. 30.
    ARToolKit. Available at: Accessed at: 21/09/2008
  31. 31.
    ARTag. Available at: Accessed at: 21/09/2008
  32. 32.
    Cogent Computing Applied Research Centre. Available at: Accessed at: 11/10/2008
  33. 33.
    Serious Games Institute. Available at: Accessed at: 11/10/2008
  34. 34.
    Dix, A., Finlay, J., Abowd, G., Beale, R.: Human–Computer Interaction. Prentice Hall, Harlow (2004) Google Scholar
  35. 35.
    Empire Interactive Entertainment: Pipemania 2008 New _2_ With Consoles. Available at: Accessed at: 19/05/2009

Copyright information

© Springer-Verlag 2009

Authors and Affiliations

  • Fotis Liarokapis
    • 1
  • Louis Macan
    • 1
  • Gary Malone
    • 1
  • Genaro Rebolledo-Mendez
    • 2
  • Sara de Freitas
    • 2
  1. 1.Interactive Worlds Applied Research GroupCoventry UniversityCoventryUK
  2. 2.Serious Games InstituteCoventry UniversityCoventryUK

Personalised recommendations