Freehand Gesture-Based 3D Manipulation Methods for Interaction with Large Displays

  • Paulo Dias
  • João Cardoso
  • Beatriz Quintino Ferreira
  • Carlos Ferreira
  • Beatriz Sousa Santos
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10291)


Gesture-based 3D interaction is a research topic with application in numerous scenarios which gained relevance with the recent advances in low-cost tracking systems. Yet, it poses many challenges due to its novelty and consequent lack of systematic development methodologies. Developing easy to use and learn gesture-based 3D interfaces is particularly difficult since the most adequate and intuitive gestures are not always obvious and there is often a variety of different gestures used to perform similar actions. This paper presents the development and evaluation of interaction methods to manipulate 3D virtual objects in a large display set-up using freehand gestures detected by a Kinect depth sensor. We describe the implementation of these methods and the user studies conducted to improve them and assess their usability as manipulation methods. Based on the results of these studies we also propose a method that overcomes the lack of roll movement detection by the Kinect and makes simpler the scaling and rotation in all degrees-of-freedom using hand gestures.


3D user interfaces Free-hand gesture based interfaces 3D object manipulation Large displays User studies 


  1. 1.
    Tuukka, M.T., Rauhamaa, P., Takala, T.: Survey of 3DUI applications and development challenges. In: 3DUI, pp. 89–96 (2012)Google Scholar
  2. 2.
    Ni, T.: A Framework of Freehand Gesture Interaction: Techniques, Guidelines, and Applications. Ph.D. thesis, Virginia Tech (2011)Google Scholar
  3. 3.
    Billinghurst, M., Piumsomboon, T., Huidong, B.: Hands in space-gesture interaction with augmented reality interfaces. IEEE Comput. Graph. Appl. 34(1), 77–80 (2014)CrossRefGoogle Scholar
  4. 4.
    Garber, L.: Gestural technology: moving interfaces in a new direction. Computer 46(10), 22–25 (2013)CrossRefGoogle Scholar
  5. 5.
    Wachs, J., Kölsch, M., Stern, H., Edan, Y.: Vision-based hand gesture applications. Commun. ACM 54(2), 60–71 (2011)CrossRefGoogle Scholar
  6. 6.
    Norman, D.A., Nielsen, J.: Gestural interfaces: a step backward in usability. Interactions 17, 46–49 (2010)CrossRefGoogle Scholar
  7. 7.
    Bowman, D.A.: 3D user interfaces. In: Soegaard, M., Dam, R.F. (eds.) The Encyclopedia of Human-Computer Interaction, 2nd edn. The Interaction Design Foundation, Aarhus (2014). Google Scholar
  8. 8.
    Dias, P., Sousa, T., Parracho, J., Cardoso, J., Monteiro, A., Sousa, Santos B.: Student Projects Involving Novel Interaction with Large Displays. IEEE Comput. Graphics Appl. 34(2), 80–86 (2014)CrossRefGoogle Scholar
  9. 9.
    Jankowski, J., Hachet, M.: Advances in interaction with 3D environments. Comput. Graph. Forum 34(1), 152–190 (2015)CrossRefGoogle Scholar
  10. 10.
    Bowman, D.A., Hodges, L.: An evaluation of techniques for grabbing and manipulating remote objects in immersive virtual environments. In: Proceedings of the 1997 Symposium on Interactive 3D Graphics, pp. 35–38 (1997)Google Scholar
  11. 11.
    Fisher, S., McGreevy, M., Humphries, J., Robinett, W.: Virtual environment display system. In: I3D 1986, Proceedings of the 1986 Workshop on Interactive 3D Graphics, pp. 77–87 (1986)Google Scholar
  12. 12.
    Freeman, W.T., Weissman, C.: Television control by hand gestures. In: Proceedings of International Workshop on Automatic Face and Gesture Recognition, pp. 179–183 (1995)Google Scholar
  13. 13.
    Boussemart, Y., Rioux, F., Rudzicz, F., Wozniewski, M., Cooperstock, J.R.: A frame-work for 3D visualization and manipulation in an immersive space using an untethered bi-manual gestural interface. In: Proceedings of the ACM Symposium on Virtual Reality Software and Technology - VRST 2004, pp. 162–165 (2004)Google Scholar
  14. 14.
    Karam, M.: A framework for research and design of gesture-based human computer interactions. Ph.D. thesis, University of Southampton (2006)Google Scholar
  15. 15.
    Kim, T., Park, J.: 3D Object manipulation using virtual handles with a grabbing metaphor. IEEE Comput. Graph. Appl. 34(3), 30–38 (2014)CrossRefGoogle Scholar
  16. 16.
    Cabral, M.C., Morimoto, C.H., Zuffo, M.K.: On the usability of gesture inter-faces in virtual reality environments. In: Proceedings of the 2005 Latin American Conference on Human-Computer Interaction - CLIHC 2005, pp. 100–108 (2005)Google Scholar
  17. 17.
    Vogel, D., Balakrishnan, R.: Distant freehand pointing and clicking on very large, high resolution displays. In: Proceedings of the 18th Annual ACM Symposium on User Interface Software and Technology, pp. 33–42 (2005)Google Scholar
  18. 18.
    Song, P., Goh, W.B., Hutama, W., Fu, C.W., Liu, X.: A handle bar metaphor for virtual object manipulation with mid-air interaction. In: Proceedings 2012 ACM Annual Conference on Human Factors in Computing Systems - CHI 2012, pp. 1297–1306 (2012)Google Scholar
  19. 19.
    Hespanhol, L., Tomitsch, M., Grace, K., Collins, A., Kay, J.: Investigating intuitiveness and effectiveness of gestures for free spatial interaction with large displays. In: Proceedings of the International Symposium on Pervasive Displays (PerDis 2012), pp. 1–6. ACM, New York (2012)Google Scholar
  20. 20.
    Bowman, D.A., Coquillart, S., Froehlich, B., Hirose, M., Kitamura, Y., Kiyokawa, K., Stuerzlinger, W.: 3D user interfaces: new directions and perspectives. IEEE Comput. Graph. Appl. 28(6), 20–36 (2008)CrossRefGoogle Scholar
  21. 21.
    Bowman, D.A., Kruij, E., LaViola, J.J., Poupyrev, I.: 3D User Interfaces: Theory and Practice. Addison Wesley Longman Publishing Co. Inc., Redwood City (2004)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Paulo Dias
    • 1
    • 2
  • João Cardoso
    • 1
  • Beatriz Quintino Ferreira
    • 2
  • Carlos Ferreira
    • 2
    • 3
  • Beatriz Sousa Santos
    • 1
    • 2
  1. 1.DETI/UA- Department of Electronics, Telecommunications and InformaticsUniversity of AveiroAveiroPortugal
  2. 2.IEETA- Institute of Electronics and Informatics Engineering of AveiroUniversity of AveiroAveiroPortugal
  3. 3.DEGEI/UA- Department of Economics, Management and Industrial EngineeringUniversity of AveiroAveiroPortugal

Personalised recommendations