Skip to main content
SpringerLink
Log in

Investigating the role of gesture modalities and screen size in an AR 3D game

Multimedia Tools and Applications Aims and scope Submit manuscript

Cite this article

Abstract

In pursuit of immersive Augmented Reality (AR) Games, gesture interaction is considered a promising mode. On the other hand, despite the considerable effect of screen size on user experience and usability in game contexts, the effect is still under-explored in AR game contexts. This, in turn, sparks a specific research interest in the interaction between gesture modalities and screen sizes in AR games. This work contributes to a controlled study investigating the effect of two different gesture modalities (touch and tilt) on varying screen sizes in a custom-made AR game. Competence, engagement, fatigue, and user preference were evaluated using the combined effect of gesture modalities and screen sizes. The results revealed that gesture modalities affect game competence and fatigue while they had no significant impact on user engagement. Further analysis has revealed that touch outperforms tilt for target-selection tasks like destroying enemies while tilt outperforms touch for path-following tasks like turning a character. However, no significant effect was found on screen size, contradicting past studies that suggested that screen size has an effect on engagement, fatigue, and performance. The findings of the study could be useful for AR game designers to further develop usable and engaging AR games.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Data availability

The datasets generated during and/or analyzed during the current study are available from the corresponding author upon reasonable request.

References

  1. Aliprantis J, Konstantakis M, Nikopoulou R, Mylonas P, Caridakis G (2019) Natural interaction in augmented reality context. In: VIPERC@ IRCDL, pp 50–61

  2. Bai H (2016) Mobile augmented reality: Free-hand gesture-based interaction

  3. Bhattacharyya P, Nath R, Jo Y, Jadhav K, Hammer J (2019) Brick: Toward a model for designing synchronous colocated augmented reality games, pp 1–9. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3290605.3300553

  4. Cairns P, Li J, Wang W, Nordin AI (2014) The influence of controllers on immersion in mobile games. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. CHI ’14, pp 371–380. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/2556288.2557345

  5. Cuaresma J, MacKenzie IS (2014) A comparison between tilt-input and facial tracking as input methods for mobile games. In: 2014 IEEE Games Media Entertainment, pp 1–7. https://doi.org/10.1109/GEM.2014.7048080

  6. Daiber F, Li L, Krüger A (2012) Designing gestures for mobile 3d gaming. In: Proceedings of the 11th International Conference on Mobile and Ubiquitous Multimedia, pp 1–4

  7. Dong Z, Zhang J, Bai X, Clark A, Lindeman RW, He W, Piumsomboon T (2022) Touch-move-release: Studies of surface and motion gestures for mobile augmented reality. Front Virtual Reality 3:927258

    Article  Google Scholar 

  8. Dong Z, Piumsomboon T, Zhang J, Clark A, Bai H, Lindeman R (2020) A comparison of surface and motion user-defined gestures for mobile augmented reality. In: Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems. CHI EA ’20, pp. 1–8. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3334480.3382883

  9. Ferrer V, Perdomo A, Rashed-Ali H, Fies C, Quarles J (2013) How does usability impact motivation in augmented reality serious games for education? In: 2013 5th International Conference on Games and Virtual Worlds for Serious Applications (VS-GAMES), pp 1–8. https://doi.org/10.1109/VS-GAMES.2013.6624233

  10. Furió D, González-Gancedo S, Juan M-C, Seguí I, Costa M (2013) The effects of the size and weight of a mobile device on an educational game. Comput Educ 64:24–41. https://doi.org/10.1016/j.compedu.2012.12.015

    Article  Google Scholar 

  11. Furió D, Juan M-C, Seguí I, Vivó R (2015) Mobile learning vs. traditional classroom lessons: a comparative study. J Comput Assist Learn 31(3):189–201

    Article  Google Scholar 

  12. Gardeli A, Vosinakis S (2019) Arquest: a tangible augmented reality approach to developing computational thinking skills. In: 2019 11th International Conference on Virtual Worlds and Games for Serious Applications (VS-Games), pp 1–8. https://doi.org/10.1109/VS-Games.2019.8864603

  13. Goh ES, Sunar MS, Ismail AW (2019) 3D object manipulation techniques in handheld mobile augmented reality interface: a review. 7:40581–40601. https://doi.org/10.1109/ACCESS.2019.2906394

  14. Guarino A, Lettieri N, Malandrino D, Zaccagnino R, Capo C (2022) Adam or eve? automatic users’ gender classification via gestures analysis on touch devices. Neural Comput Appl 34(21):18473–18495

    Article  Google Scholar 

  15. Guarino A, Malandrino D, Zaccagnino R, Capo C, Lettieri N (2023) Touchscreen gestures as images. A transfer learning approach for soft biometric traits recognition. Exp Syst Appl 219:119614

    Article  Google Scholar 

  16. Hinckley K, Heo S, Pahud M, Holz C, Benko H, Sellen A, Banks R, O’Hara K, Smyth G, Buxton W (2016) Pre-touch sensing for mobile interaction. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, pp 2869–2881

  17. Hou J, Nam Y, Peng W, Lee KM (2012) Effects of screen size, viewing angle, and players’ immersion tendencies on game experience. Comput Human Behav 28(2):617–623. https://doi.org/10.1016/j.chb.2011.11.007

    Article  Google Scholar 

  18. Hu W, Wei N, Li Z-M, Li K (2018) Effects of muscle fatigue on directional coordination of fingertip forces during precision grip. PLoS One 13(12):0208740

    Article  Google Scholar 

  19. Hürst W, Van Wezel C (2013) Gesture-based interaction via finger tracking for mobile augmented reality. Multimed Tools Appl 62(1):233–258

    Article  Google Scholar 

  20. Hynninen T (2012) First-person shooter controls on touchscreen devices: a heuristic evaluation of three games on the ipod touch. Master’s thesis

  21. Jain A, Kanhangad V (2019) Gender recognition in smartphones using touchscreen gestures. Patt Recog Lett 125:604–611

    Article  Google Scholar 

  22. Karambakhsh A, Kamel A, Sheng B, Li P, Yang P, Feng DD (2019) Deep gesture interaction for augmented anatomy learning. Int J Inf Manag 45:328–336. https://doi.org/10.1016/j.ijinfomgt.2018.03.004

    Article  Google Scholar 

  23. Laine TH, Suk HJ (2016) Designing mobile augmented reality exergames. Games Culture 11(5):548–580. https://doi.org/10.1177/1555412015572006

    Article  Google Scholar 

  24. Lee C-C, Chiang H-S, Hsiao M-H (2021) Effects of screen size and visual presentation on visual fatigue based on regional brain wave activity. J Supercomput 77(5):4831–4851

    Article  Google Scholar 

  25. Marzo A, Bossavit B, Hachet M (2014) Combining multi-touch input and device movement for 3d manipulations in mobile augmented reality environments. In: Proceedings of the 2nd ACM Symposium on Spatial User Interaction. SUI ’14, pp. 13–16. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/2659766.2659775

  26. Medryk S, MacKenzie IS (2013) A comparison of accelerometer and touch-based input for mobile gaming. In: International Conference on Multimedia and Human-Computer Interaction-MHCI 2013, pp 117–1

  27. Mine MR, Brooks Jr FP, Sequin CH (1997) Moving objects in space: Exploiting proprioception in virtual-environment interaction. In: Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques, pp 19–26

  28. O’Brien HL, Cairns P, Hall M (2018) A practical approach to measuring user engagement with the refined user engagement scale (ues) and new ues short form. Int J Human Comput Stud 112:28–39

    Article  Google Scholar 

  29. Oshita M, Ishikawa H (2012) Gamepad vs. touchscreen: a comparison of action selection interfaces in computer games. In: Proceedings of the Workshop at SIGGRAPH Asia, pp 27–31

  30. Raptis D, Tselios N, Kjeldskov J, Skov MB (2013) Does size matter? Investigating the impact of mobile phone screen size on users’ perceived usability, effectiveness and efficiency. In: Proceedings of the 15th International Conference on Human-Computer Interaction with Mobile Devices and Services. MobileHCI ’13, pp 127–136. Association for Computing Machinery, New York, NY, USA. 10.1145/2493190.2493204

  31. Rigby JM, Brumby DP, Cox AL, Gould SJJ (2016) Watching movies on netflix: Investigating the effect of screen size on viewer immersion. In: Proceedings of the 18th International Conference on Human-Computer Interaction with Mobile Devices and Services Adjunct. MobileHCI ’16, pp 714–721. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/2957265.2961843

  32. Rocha F, Machado Santa P, Cardoso JCS, Pereira LL, Roque L (2019) Mapping controls on a 2D user drawn racetracks driving game - an usability assessment. In: Extended Abstracts of the Annual Symposium on Computer-Human Interaction in Play Companion Extended Abstracts. CHI PLAY ’19 Extended Abstracts, pp 653–660. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3341215.3356302

  33. Ruiz J, Li Y, Lank E (2011) User-defined motion gestures for mobile interaction. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp 197–206

  34. Soares LP, Musse SR, Pinho MS, Boussu JB (2018) Evaluation of selection techniques on a mobile augmented reality game. In: 2018 17th Brazilian Symposium on Computer Games and Digital Entertainment (SBGames), pp 127–12709. https://doi.org/10.1109/SBGAMES.2018.00024

  35. Teather RJ, MacKenzie IS (2014) Comparing order of control for tilt and touch games. In: Proceedings of the 2014 Conference on Interactive Entertainment. IE2014, pp 1–10. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/2677758.2677766

  36. Teather R, Roth A, MacKenzie I (2017) Tilt-touch synergy: Input control for dual-analog style mobile games 21:33–43. https://doi.org/10.1016/j.entcom.2017.04.005

    Article  Google Scholar 

  37. Terui H, Hosobe H (2022) A hand gesture-based authentication method that makes forgery difficult. In: HCI for Cybersecurity, Privacy and Trust: 4th International Conference, HCI-CPT 2022, Held as Part of the 24th HCI International Conference, HCII 2022, Virtual Event, June 26–July 1, 2022, Proceedings, pp 268–279. Springer

  38. Wang L, Hossain MS, Pulfrey J, Lancor L (2021) The effectiveness of zoom touchscreen gestures for authentication and identification and its changes over time. Comput Secur 111:102462

    Article  Google Scholar 

  39. Wibirama S, Nugroho HA (2017) Towards understanding addiction factors of mobile devices: an eye tracking study on effect of screen size. In: 2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp 2454–2457. https://doi.org/10.1109/EMBC.2017.8037353

  40. Xi W, Gong H, Wang Q (2019) How hand gestures influence the enjoyment in gamified mobile marketing. Int J Human Comput Stud 127:169–180. https://doi.org/10.1016/j.ijhcs.2018.09.010. Strengthening gamification studies: critical challenges and new opportunities

  41. Xi W, Jin M, Gong H, Wang Q (2018) Touch or shake? The interaction effect between hand gesture and reward setting on the enjoyment of gamified marketing. In: GamiFIN, pp 100–107

  42. Yi X, Zhang S, Pan Z, Shi L, Han F, Kong Y, Li H, Shi Y (2023) Squeez’in: Private authentication on smartphones based on squeezing gestures. In: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pp 1–15

  43. Young JC, Kristanda MB, Hansun S (2016) Armatika: 3D game for arithmetic learning with augmented reality technology. In: 2016 International Conference on Informatics and Computing (ICIC), pp 355–360. https://doi.org/10.1109/IAC.2016.7905744

  44. Yu X, Zhou Z, Xu M, You X, Li X-Y (2020) Thumbup: Identification and authentication by smartwatch using simple hand gestures. In: 2020 IEEE International Conference on Pervasive Computing and Communications (PerCom), pp 1–10. IEEE Computer Society

  45. Zhang Y, Zhang Z (2020) A comparative study of the influence of the screen size of mobile devices on the experience effect of 3D content in the form of ar/vr technology. In: De Paolis LT, Bourdot P (eds) Augmented reality, virtual reality, and computer graphics. Springer, Cham, pp 363–374

    Chapter  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Information and Communications Technologies, Asian Institute of Technology, Pathum Thani, 12120, Thailand

    Abdul Raheem Fathima Shafana & Chaklam Silpasuwanchai

  2. Department of Information and Communication Technology, South Eastern University of Sri Lanka, Oluvil, 32360, Sri Lanka

    Abdul Raheem Fathima Shafana

Authors
  1. Abdul Raheem Fathima Shafana
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chaklam Silpasuwanchai
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chaklam Silpasuwanchai.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and Permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Shafana, A.R.F., Silpasuwanchai, C. Investigating the role of gesture modalities and screen size in an AR 3D game. Multimed Tools Appl (2023). https://doi.org/10.1007/s11042-023-16052-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11042-023-16052-y

Keywords

search

Navigation