Hand gesture-based tangible interactions for manipulating virtual objects in a mixed reality environment

  • Jae Yeol LeeEmail author
  • Gue Won Rhee
  • Dong Woo Seo


In the environment of mixed reality (MR) or augmented reality (AR), there have been several previous works dealing with user interfaces for manipulating and interacting with virtual objects aimed at improving immersive feeling and natural interaction. However, it is still considered that there must be much progress in supporting human behavior-like interactions for providing control efficiency and natural feeling in MR/AR environments. This paper proposes a tangible interaction method by combining the advantages of soft interactions such as hand gesture and MR and hard interactions such as vibro-tactile feedback. One of the main goals is to provide more natural interaction interfaces similar to the manipulation task in the real world by utilizing hand gesture-based tangible interactions. It also provides multimodal interfaces by adopting the vibro-tactile feedback and tangible interaction for the virtual object manipulation. Thus, it can make users get more immersive and natural feeling in the manipulation and interaction with virtual objects. Furthermore, it provides an alternative instruction guideline based on the analysis of the previous interaction while manipulating virtual objects, which makes it possible for the user to minimize manipulation errors during the interaction phase and the learning process, which guides the user to the right direction. We will show the effectiveness and advantage of the proposed approach by demonstrating several implementation results.


Augmented reality Hand gesture Human–computer interface Mixed reality Pinch glove Tangible interface Vibro-tactile feedback 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Weiser M (1999) The computer of the 21st century. ACM SIGMOBILE Mobile Computing and Communications Review 3:3–11Google Scholar
  2. 2.
    Lee JY, Rhee GW, Kim H, Lee K-W, Suh Y-H, Kim K (2006) Convergence of context-awareness and augmented reality for ubiquitous services and immersive interactions. LNCS 3983:466–474Google Scholar
  3. 3.
    Schmalstieg D, Reitmayr G (2005) The world as a user interface: augmented reality for ubiquitous computing. Proc. of Central European Multimedia and Virtual Reality Conf., Prague, Czech RepublicGoogle Scholar
  4. 4.
    Piekarski W (2004) Interactive 3D modeling in outdoor augmented reality worlds. PhD Thesis, The University of South AustraliaGoogle Scholar
  5. 5.
    Ong SK, Yuan ML, Nee AYC (2008) Augmented reality applications in manufacturing: a survey. Int J Prod Res 46:2707–2742zbMATHCrossRefGoogle Scholar
  6. 6.
    Seichter H (2007) Augmented reality and tangible user interfaces in collaborative urban design. Proc. of the 12th International CAAD Futures Conf., Sydney, AustraliaGoogle Scholar
  7. 7.
    Sidharta R, Oliver J, Sannier A (2006) Augmented reality tangible interface for distributed design review. Proc. of the International Conference on Computer Graphics, Imaging and Visualization 464–470, Sydney, AustraliaGoogle Scholar
  8. 8.
    Lee JY, Rhee GW (2008) Context aware adaptable ubiquitous car services using augmented reality. Int Jo Advan Manuf Technol 37:431–442CrossRefGoogle Scholar
  9. 9.
    Lee JY, Seo DW, Rhee GW (2008) Visualization and interaction of pervasive services using context-aware augmented reality. Expert Syst Appl 35:1873–1882CrossRefGoogle Scholar
  10. 10.
    Kato H, Billinghurst M, Poupyrev I, Imamoto K, Tachibana K (2000) Virtual object manipulation on a table-top AR environment. Proc. of ISAR 2000 111-119, Munich, GermanyGoogle Scholar
  11. 11.
    Park H, Moon H-C, Lee JY (2009) Tangible augmented prototyping of digital handheld products. Comput Ind 60:114–125CrossRefGoogle Scholar
  12. 12.
    Lee JY, Rhee G, Park H (2009) AR/RP-based tangible interactions for collaborative design evaluation of digital products. Int J AdvanManuf Technol 45:649–665CrossRefGoogle Scholar
  13. 13.
    Terry M, Cheung J, Lee J, Park T, Williams N (2007) Jump: a system for interactive, tangible queries of paper. Proc. of Graphics Interface 2007 127–134, Montreal, CanadaGoogle Scholar
  14. 14.
    Hong D, Looser J, Seichter H, Billinghurst M, Woo W (2008) A sensor-based interaction for ubiquitous virtual reality systems. Proc. of International Symp. on Ubiquitous Virtual Reality 2008 10–13, Gwangju, South KoreaGoogle Scholar
  15. 15.
    Buchmann V, Violich S, Billinghurst M, Cockburn A (2004) FingARtips: gesture based direct manipulation in augmented reality. Proc. of the 2nd International Conf. on Computer Graphics and Interactive Techniques in Australasia and South East Asia 212–221, SingaporeGoogle Scholar
  16. 16.
    Ott R, Thalmann D, Vexo F (2007) Haptic feedback in mixed-reality environment. Vis Comput 23:843–849CrossRefGoogle Scholar
  17. 17.
    White M, Petridis P, Liarokapis F, Plecinckx D (2007) Multimodal mixed reality interfaces for visualizing digital heritage. Int J Architect Comput 5:322–337CrossRefGoogle Scholar
  18. 18.
    Thomas BH, Piekarski W (2002) Glove based user interaction techniques for augmented reality in an outdoor environment. Virtual Real 6:167–180CrossRefGoogle Scholar
  19. 19.
    Robert W, Linderman JN, Sibert JL, Cutler JR (2002) Handling of virtual contact in immersive virtual environments: beyond visuals. Virtual Real 6:130–139CrossRefGoogle Scholar
  20. 20.
    Veigl S, Kaltenbach A, Ledermann F, Reitmayr G, Schmalstieg D (2002) Two-handed direct interaction with ARToolkit. Proc. of IEEE First International Augmented Reality Toolkit Workshop, Darmstadt, GermanyGoogle Scholar
  21. 21.
    Yi X, Qin S, Kang J (2009) Generating 3D architectural models based on hand motion and gesture. Comput Ind 60:677–685CrossRefGoogle Scholar
  22. 22.
  23. 23.
    Bowman DA, Wingrave CA (2001) Design and evaluation of menu systems for immersive virtual environments. Proc. of IEEE Virtual Reality 2001 149-156, Yokohama, Japan Google Scholar
  24. 24.
    David H (1987) Statecharts: a visual formalism for complex systems. Sci Comput Programming archive 8:231–274zbMATHCrossRefGoogle Scholar

Copyright information

© Springer-Verlag London Limited 2010

Authors and Affiliations

  1. 1.Department of Industrial EngineeringChonnam National UniversityGwangjuSouth Korea
  2. 2.Asiana IDTGwangjuSouth Korea

Personalised recommendations