Multimodal Interaction Concepts for Mobile Augmented Reality Applications

  • Wolfgang Hürst
  • Casper van Wezel
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6524)


Augmented reality on mobile phones – i.e. applications where users look at the live image of the device’s video camera and the scene that they see is enriched by 3D virtual objects – provides great potential in areas such as cultural heritage, entertainment, and tourism. However, current interaction concepts are often limited to pure 2D pointing and clicking on the device’s screen. This paper explores different interaction approaches that rely on multimodal sensor input and aim at providing a richer, more complex, and engaging interaction experience. We present a user study that investigates the usefulness of our approaches, verify their usability, and identify limitations as well as possibilities for interaction development for mobile augmented reality.


Mobile augmented reality multi-sensor input interaction design 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Xbox Kinect (formerly known as Microsoft’s project Natal), (last accessed 10/15/10)
  2. 2.
    Mistry, P., Maes, P.: SixthSense: a wearable gestural interface. In: ACM SIGGRAPH ASIA 2009 Sketches (2009)Google Scholar
  3. 3.
    EyeSight’s Touch Free Interface Technology Software, (last accessed 10/15/10)
  4. 4.
    Niikura, T., Hirobe, Y., Cassinelli, A., Watanabe, Y., Komuro, T., Ishikawa, M.: In-air typing interface for mobile devices with vibration feedback. In: ACM SIGGRAPH 2010 Emerging Technologies (2010)Google Scholar
  5. 5.
    Hagbi, N., Bergig, O., El-Sana, J., Billinghurst, M.: Shape recognition and pose estimation for mobile augmented reality. In: Proceedings of the 2009 8th IEEE International Symposium on Mixed and Augmented Reality (2009)Google Scholar
  6. 6.
    Bergig, O., Hagbi, N., El-Sana, J., Billinghurst, M.: In-place 3D sketching for authoring and augmenting mechanical systems. In: Proceedings of the 2009 8th IEEE International Symposium on Mixed and Augmented Reality (2009)Google Scholar
  7. 7.
    Billinghurst, M., Kato, H., Myojin, S.: Advanced Interaction Techniques for Augmented Reality Applications. In: Proceedings of the 3rd international Conference on Virtual and Mixed Realit, Part of HCI International 2009 (2009)Google Scholar
  8. 8.
    Lee, M., Green, R., Billinghurst, M.: 3D natural hand interaction for AR applications. In: Proceedings of Image and Vision Computing, New Zealand (2008)Google Scholar
  9. 9.
    Callahan, J., Hopkins, D., Weiser, M., Shneiderman, B.: An empirical comparison of pie vs. linear menus. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (1988)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Wolfgang Hürst
    • 1
  • Casper van Wezel
    • 1
  1. 1.Utrecht UniversityUtrechtThe Netherlands

Personalised recommendations