Skip to main content
SpringerLink
Log in

Gesture-Based Interaction

  • Living reference work entry
  • First Online:
Handbook of Human Computer Interaction

Abstract

Many interactive devices and systems, from smartphones and tablets to smart wearables, video game consoles, ambient displays and interactive surfaces to systems rendering virtual and augmented reality environments, leverage users’ capabilities to communicate and interact with gestures of many kinds: taps, touches, grasps, pinches, head nods, pointing, hand poses, signs and emblems, flicks and swipes, and mid-air movements of the fingers, wrists, arms, legs, and the whole body. These gestures are sensed, modeled, recognized, and interpreted by interactive computer systems to leverage the rich expressiveness of human communicative skills and, consequently, enable natural, intuitive, and fluent means of communication for users in relation to a computer interlocutor. This chapter presents multiple aspects relevant to the design and engineering of gesture-based interaction, including desirable quality properties of gesture input, gesture representation and recognition techniques, gesture analysis methods, and corresponding software tools to assist the design of gesture sets for interactive systems and accessibility aspects of gesture interaction for users with various sensory or motor abilities. Also, this chapter is meant to provide an overview of the large scientific literature available on the topic of gesture-based interaction, pointing the reader to both theoretical and practical aspects and highlighting design challenges, technical solutions, and opportunities for designing and engineering gesture-based interaction with computer systems.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Institutional subscriptions

References

  • Ali AX, Morris MR, Wobbrock JO (2018) Crowdsourcing similarity judgments for agreement analysis in end-user elicitation studies. In: Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology, UIST’18. ACM, New York, pp 177–188. https://doi.org/10.1145/3242587.3242621

    Chapter  Google Scholar 

  • Ali AX, Morris MR, Wobbrock JO (2019) Crowdlicit: a system for conducting distributed end-user elicitation and identification studies. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, New York, pp 1–12. https://doi.org/10.1145/3290605.3300485

    Google Scholar 

  • Ali A, Ringel Morris M, O Wobbrock J (2021) I am iron man: priming improves the learnability and memorability of user-elicited gestures. In: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. ACM, New York. https://doi.org/10.1145/3411764.3445758

  • Aloba A, Anthony L (2021) Characterizing children’s motion qualities: implications for the design of motion applications for children. In: Proceedings of the 2021 International Conference on Multimodal Interaction, ICMI’21. ACM, New York, pp 229–238. https://doi.org/10.1145/3462244.3479941

    Google Scholar 

  • Aloba A, Woodward J, Anthony L (2020) FilterJoint: toward an understanding of whole-body gesture articulation. In: Proceedings of the 2020 International Conference on Multimodal Interaction, ICMI’20. ACM, New York, pp 213–221. https://doi.org/10.1145/3382507.3418822

    Google Scholar 

  • Anderson F, Bischof WF (2013) Learning and performance with gesture guides. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI’13. ACM, New York, pp 1109–1118. https://doi.org/10.1145/2470654.2466143

    Google Scholar 

  • Annett M, Grossman T, Wigdor D, Fitzmaurice G (2011) Medusa: a proximity-aware multi-touch tabletop. In: Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, UIST’11. ACM, New York, pp 337–346. https://doi.org/10.1145/2047196.2047240

    Chapter  Google Scholar 

  • Anthony L, Wobbrock JO (2010) A lightweight multistroke recognizer for user interface prototypes. In: Proceedings of Graphics Interface 2010, GI’10. Canadian Information Processing Society, pp 245–252. https://doi.org/10.5555/1839214.1839258

  • Anthony L, Wobbrock JO (2012) $N-Protractor: a fast and accurate multistroke recognizer. In: Proceedings of Graphics Interface 2012, GI’12. Canadian Information Processing Society. https://doi.org/10.5555/2305276.2305296

  • Anthony L, Kim Y, Findlater L (2013a) Analyzing user-generated youtube videos to understand touchscreen use by people with motor impairments. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI’13. ACM, New York, pp 1223–1232. https://doi.org/10.1145/2470654.2466158

    Google Scholar 

  • Anthony L, Vatavu RD, Wobbrock JO (2013b) Understanding the consistency of users’ pen and finger stroke gesture articulation. In: Proceedings of Graphics Interface 2013, GI’13. Canadian Information Processing Society, Canada, pp 87–94. https://doi.org/10.5555/2532129.2532145

    Google Scholar 

  • Appert C, Zhai S (2009) Using strokes as command shortcuts: cognitive benefits and toolkit support. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI’09. ACM, New York, pp 2289–2298. https://doi.org/10.1145/1518701.1519052

    Google Scholar 

  • Arefin Shimon SS, Lutton C, Xu Z, Morrison-Smith S, Boucher C, Ruiz J (2016) Exploring non-touchscreen gestures for smartwatches. In: Proceedings of the CHI Conference on Human Factors in Computing Systems, CHI’16. ACM, New York, pp 3822–3833. https://doi.org/10.1145/2858036.2858385

    Google Scholar 

  • Asadi-Aghbolaghi M, Clapés A, Bellantonio M, Escalante HJ, Ponce-López V, Baró X, Guyon I, Kasaei S, Escalera S (2017) A survey on deep learning based approaches for action and gesture recognition in image sequences. In: Proceedings of the 12th IEEE International Conference on Automatic Face Gesture Recognition, FG’17, pp 476–483. https://doi.org/10.1109/FG.2017.150

  • Ashbrook D, Starner T (2010) MAGIC: a motion gesture design tool. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI’10. ACM, New York, pp 2159–2168. https://doi.org/10.1145/1753326.1753653

    Google Scholar 

  • Azai T, Otsuki M, Shibata F, Kimura A (2018) Open palm menu: a virtual menu placed in front of the palm. In: Proceedings of the 9th Augmented Human International Conference, AH’18. ACM, New York. https://doi.org/10.1145/3174910.3174929

  • Bai H, Sasikumar P, Yang J, Billinghurst M (2020) A user study on mixed reality remote collaboration with eye gaze and hand gesture sharing. In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, CHI’20. ACM, New York, pp 1–13. https://doi.org/10.1145/3313831.3376550

    Google Scholar 

  • Bailey SW, Bodenheimer B (2012) A comparison of motion capture data recorded from a vicon system and a microsoft kinect sensor. In: Proceedings of the ACM Symposium on Applied Perception, SAP’12. ACM, New York, p 121. https://doi.org/10.1145/2338676.2338703

  • Bailly G, Müller J, Lecolinet E (2012) Design and evaluation of finger-count interaction: combining multitouch gestures and menus. Int J Hum Comput Stud 70(10):673–689. https://doi.org/10.1016/j.ijhcs.2012.05.006

    Article  Google Scholar 

  • Bardot S, Rawat S, Nguyen DT, Rempel S, Zheng H, Rey B, Li J, Fan K, Huang DY, Li W, Irani P (2021) ARO: exploring the design of smart-ring interactions for encumbered hands. In: Proceedings of the 23rd International Conference on Mobile Human-Computer Interaction, MobileHCI’21. ACM, New York. https://doi.org/10.1145/3447526.3472037

  • Bau O, Mackay WE (2008) OctoPocus: a dynamic guide for learning gesture-based command sets. In: Proceedings of the 21st Annual ACM Symposium on User Interface Software and Technology, UIST’08. ACM, New York, pp 37–46. https://doi.org/10.1145/1449715.1449724

    Chapter  Google Scholar 

  • Baudel T, Beaudouin-Lafon M (1993) Charade: remote control of objects using free-hand gestures. Commun ACM 36(7):28–35. https://doi.org/10.1145/159544.159562

    Article  Google Scholar 

  • Bergström J, Hornbæk K (2019) Human–computer interaction on the skin. ACM Comput Surv 52(4). https://doi.org/10.1145/3332166

  • Bi X, Li Y, Zhai S (2013) FFitts law: modeling finger touch with fitts’ law. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI’13. ACM, New York, pp 1363–1372. https://doi.org/10.1145/2470654.2466180

    Google Scholar 

  • Bilius LB, Vatavu RD (2021) A multistudy investigation of drivers and passengers’ gesture and voice input preferences for in-vehicle interactions. J Intell Transp Syst 25(2):197–220. https://doi.org/10.1080/15472450.2020.1846127

    Article  Google Scholar 

  • Bilius LB, Vatavu RD, Marquardt N (2021) Smart vehicle proxemics: a conceptual framework operationalizing proxemics in the context of outside-the-vehicle interactions. In: Proceedings of the 18th IFIP TC 13 International Conference, INTERACT’21. Springer, Berlin/Heidelberg, pp 150–171. https://doi.org/10.1007/978-3-030-85616-8_11

    Google Scholar 

  • Bobick A, Davis J (2001) The recognition of human movement using temporal templates. IEEE Trans Pattern Anal Mach Intell 23(3):257–267. https://doi.org/10.1109/34.910878

    Article  Google Scholar 

  • Bobick AF, Intille SS, Davis JW, Baird F, Pinhanez CS, Campbell LW, Ivanov YA, Schütte A, Wilson A (1999) The kidsroom: a perceptually-based interactive and immersive story environment. Presence: Teleoper Virtual Environ 8(4):369–393. https://doi.org/10.1162/105474699566297

    Article  Google Scholar 

  • Bolt RA (1980) Put-that-there: voice and gesture at the graphics interface. SIGGRAPH Comput Graph 14(3):262–270. https://doi.org/10.1145/965105.807503

    Article  MathSciNet  Google Scholar 

  • Borah PP, Sorathia K (2019) Natural and intuitive deformation gestures for one-handed landscape mode interaction. In: Proceedings of the Thirteenth International Conference on Tangible, Embedded, and Embodied Interaction, TEI’19. ACM, New York, pp 229–236. https://doi.org/10.1145/3294109.3300996

    Google Scholar 

  • Cadoz C (1994) Le geste canal de communication homme/machine. Tech Sci Inform 13(1):31–61

    Google Scholar 

  • Cai Z, Han J, Liu L, Shao L (2017) RGB-D datasets using microsoft kinect or similar sensors: a survey. Multimed Tools Appl 76(3):4313–4355. https://doi.org/10.1007/s11042-016-3374-6

    Article  Google Scholar 

  • Cao X, Zhai S (2007) Modeling human performance of pen stroke gestures. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI’07. ACM, New York, pp 1495–1504. https://doi.org/10.1145/1240624.1240850

    Google Scholar 

  • Caramiaux B, Donnarumma M, Tanaka A (2015) Understanding gesture expressivity through muscle sensing. ACM Trans Comput-Hum Interact 21(6). https://doi.org/10.1145/2687922

  • Carrington P, Hurst A, Kane SK (2014) The gest-rest: a pressure-sensitive chairable input pad for power wheelchair armrests. In: Proceedings of the 16th International ACM SIGACCESS Conference on Computers & Accessibility, ASSETS’14. ACM, New York, pp 201–208. https://doi.org/10.1145/2661334.2661374

    Chapter  Google Scholar 

  • Cauchard JR, E JL, Zhai KY, Landay JA (2015) Drone & me: an exploration into natural human-drone interaction. In: Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing, UbiComp’15. ACM, New York, pp 361–365. https://doi.org/10.1145/2750858.2805823

  • Chan L, Hsieh CH, Chen YL, Yang S, Huang DY, Liang RH, Chen BY (2015) Cyclops: wearable and single-piece full-body gesture input devices. In: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, CHI’15. ACM, New York, pp 3001–3009. https://doi.org/10.1145/2702123.2702464

    Google Scholar 

  • Chen L, Wang F, Deng H, Ji K (2013) A survey on hand gesture recognition. In: Proceedings of the 2013 International Conference on Computer Sciences and Applications, CSA’13. IEEE Computer Society, pp 313–316. https://doi.org/10.1109/CSA.2013.79

  • Dang H, Buschek D (2021) GestureMap: supporting visual analytics and quantitative analysis of motion elicitation data by learning 2D embeddings. In: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. ACM, New York. https://doi.org/10.1145/3411764.3445765

  • Delamare W, Janssoone T, Coutrix C, Nigay L (2016) Designing 3D gesture guidance: visual feedback and feedforward design options. In: Proceedings of the International Working Conference on Advanced Visual Interfaces, AVI’16. ACM, New York, pp 152–159. https://doi.org/10.1145/2909132.2909260

    Chapter  Google Scholar 

  • Dessì S, Spano LD (2020) DG3: exploiting gesture declarative models for sample generation and online recognition. Proc ACM Hum-Comput Interact 4(EICS). https://doi.org/10.1145/3397870

  • Dingler T, Rzayev R, Shirazi AS, Henze N (2018) Designing consistent gestures across device types: eliciting rsvp controls for phone, watch, and glasses. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, New York, pp 1–12. https://doi.org/10.1145/3173574.3173993

    Google Scholar 

  • Dong Z, Zhang J, Lindeman R, Piumsomboon T (2020) Surface vs motion gestures for mobile augmented reality. In: Symposium on Spatial User Interaction, SUI’20. ACM, New York. https://doi.org/10.1145/3385959.3422694

  • El Ali A, Kildal J, Lantz V (2012) Fishing or a Z? Investigating the effects of error on mimetic and alphabet device-based gesture interaction. In: Proceedings of the 14th ACM International Conference on Multimodal Interaction, ICMI’12. ACM, New York, pp 93–100. https://doi.org/10.1145/2388676.2388701

    Google Scholar 

  • Engelbart DC, English WK (1968) A research center for augmenting human intellect. In: Proceedings of the Fall Joint Computer Conference, Part I, 9–11 Dec, 1968, AFIPS’68 (Fall, part I). ACM, New York, pp 395–410. https://doi.org/10.1145/1476589.1476645

  • Evans AC, Wobbrock JO, Davis K (2016) Modeling collaboration patterns on an interactive tabletop in a classroom setting. In: Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing, CSCW’16. ACM, New York, pp 860–871. https://doi.org/10.1145/2818048.2819972

    Google Scholar 

  • Felberbaum Y, Lanir J (2018) Better understanding of foot gestures: an elicitation study. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, New York, pp 1–12. https://doi.org/10.1145/3173574.3173908

    Google Scholar 

  • Ferstl Y, Neff M, McDonnell R (2021) It’s a match! Gesture generation using expressive parameter matching. In: Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS’21. Richland, pp 1495–1497

    Google Scholar 

  • Fu LP, Landay J, Nebeling M, Xu Y, Zhao C (2018) Redefining natural user interface. In: Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems, CHI EA’18. ACM, New York, pp 1–3. https://doi.org/10.1145/3170427.3190649

    Google Scholar 

  • Gheran BF, Vatavu RD (2020) From controls on the steering wheel to controls on the finger: using smart rings for in-vehicle interactions. In: Companion Publication of the 2020 ACM Designing Interactive Systems Conference, DIS’20 Companion. ACM, New York, pp 299–304. https://doi.org/10.1145/3393914.3395851

    Google Scholar 

  • Gheran BF, Vanderdonckt J, Vatavu RD (2018a) Gestures for smart rings: empirical results, insights, and design implications. In: Proceedings of the 2018 Designing Interactive Systems Conference, DIS’18. ACM, New York, pp 623–635. https://doi.org/10.1145/3196709.3196741

  • Gheran BF, Vatavu RD, Vanderdonckt J (2018b) Ring x2: designing gestures for smart rings using temporal calculus. In: Proceedings of the ACM Conference Companion Publication on Designing Interactive Systems, DIS’18 Companion. ACM, New York, pp 117–122. https://doi.org/10.1145/3197391.3205422

    Google Scholar 

  • Gil H, Kim H, Oakley I (2018) Fingers and angles: exploring the comfort of touch input on smartwatches. Proc ACM Interact Mob Wearable Ubiquitous Technol 2(4). https://doi.org/10.1145/3287042

  • Greenberg S, Marquardt N, Ballendat T, Diaz-Marino R, Wang M (2011) Proxemic interactions: the new ubicomp? Interactions 18(1):42–50. https://doi.org/10.1145/1897239.1897250

    Article  Google Scholar 

  • Gupta A, Irudayaraj A, Chandran V, Palaniappan G, Truong KN, Balakrishnan R (2016) Haptic learning of semaphoric finger gestures. In: Proceedings of the 29th Annual Symposium on User Interface Software and Technology, UIST’16. ACM, New York, pp 219–226. https://doi.org/10.1145/2984511.2984558

    Chapter  Google Scholar 

  • Hayashi E, Lien J, Gillian N, Giusti L, Weber D, Yamanaka J, Bedal L, Poupyrev I (2021) RadarNet: efficient gesture recognition technique utilizing a miniature radar sensor. In: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. ACM, New York. https://doi.org/10.1145/3411764.3445367

  • Hoffman M, Varcholik P, LaViola JJ (2010) Breaking the status quo: improving 3D gesture recognition with spatially convenient input devices. In: Proceedings of the 2010 IEEE Virtual Reality Conference, VR’10. IEEE Computer Society, pp 59–66. https://doi.org/10.1109/VR.2010.5444813

  • Holz C, Baudisch P (2011) Understanding touch. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI’11. ACM, New York, pp 2501–2510. https://doi.org/10.1145/1978942.1979308

    Google Scholar 

  • Hoober S (2015) Fingers, thumbs, and people. Interactions 22(3):48–51. https://doi.org/10.1145/2745957

    Article  Google Scholar 

  • Hu B, Rakthanmanon T, Hao Y, Evans S, Lonardi S, Keogh E (2015) Using the minimum description length to discover the intrinsic cardinality and dimensionality of time series. Data Min Knowl Discov 29(2):358–399. https://doi.org/10.1007/s10618-014-0345-2

    Article  MathSciNet  MATH  Google Scholar 

  • Hu F, He P, Xu S, Li Y, Zhang C (2020) FingerTrak: continuous 3D hand pose tracking by deep learning hand silhouettes captured by miniature thermal cameras on wrist. Proc ACM Interact Mob Wearable Ubiquitous Technol 4(2). https://doi.org/10.1145/3397306

  • Hu X, Wang J, Gao W, Yu C, Shi Y (2021) FootUI: assisting people with upper body motor impairments to use smartphones with foot gestures on the bed. In: Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems. ACM, New York. https://doi.org/10.1145/3411763.3451782

  • Ishii H, Ullmer B (1997) Tangible bits: towards seamless interfaces between people, bits and atoms. In: Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems, CHI’97. ACM, New York, pp 234–241. https://doi.org/10.1145/258549.258715

    Google Scholar 

  • Ishii H, Lakatos D, Bonanni L, Labrune JB (2012) Radical atoms: beyond tangible bits, toward transformable materials. Interactions 19(1):38–51. https://doi.org/10.1145/2065327.2065337

    Article  Google Scholar 

  • Jang S, Stuerzlinger W, Ambike S, Ramani K (2017) Modeling cumulative arm fatigue in mid-air interaction based on perceived exertion and kinetics of arm motion. In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, CHI’17. ACM, New York, pp 3328–3339. https://doi.org/10.1145/3025453.3025523

    Google Scholar 

  • Jones B, Sodhi R, Murdock M, Mehra R, Benko H, Wilson A, Ofek E, MacIntyre B, Raghuvanshi N, Shapira L (2014) RoomAlive: magical experiences enabled by scalable, adaptive projector-camera units. In: Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology, UIST’14. ACM, New York, pp 637–644. https://doi.org/10.1145/2642918.2647383

    Chapter  Google Scholar 

  • Kane SK, Wobbrock JO, Ladner RE (2011) Usable gestures for blind people: understanding preference and performance. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI’11. ACM, New York, pp 413–422. https://doi.org/10.1145/1978942.1979001

    Google Scholar 

  • Karam M, schraefel MC (2005) A taxonomy of gestures in human computer interactions. http://eprints.soton.ac.uk/id/eprint/261149

  • Khurana R, Goel M, Lyons K (2019) Detachable smartwatch: more than a wearable. Proc ACM Interact Mob Wearable Ubiquitous Technol 3(2). https://doi.org/10.1145/3328921

  • Kin K, Miller T, Bollensdorff B, DeRose T, Hartmann B, Agrawala M (2011) Eden: a professional multitouch tool for constructing virtual organic environments. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI’11. ACM, New York, pp 1343–1352. https://doi.org/10.1145/1978942.1979141

    Google Scholar 

  • Kin K, Hartmann B, DeRose T, Agrawala M (2012) Proton: multitouch gestures as regular expressions. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI’12. ACM, New York, pp 2885–2894. https://doi.org/10.1145/2207676.2208694

    Google Scholar 

  • Koelle M, Ananthanarayan S, Boll S (2020) Social acceptability in HCI: a survey of methods, measures, and design strategies. In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, CHI’20. ACM, New York, pp 1–19. https://oi.org/10.1145/3313831.3376162

    Google Scholar 

  • Kohli L, Whitton M (2005) The haptic hand: providing user interface feedback with the non-dominant hand in virtual environments. In: Proceedings of Graphics Interface 2005, GI’05. Canadian Human-Computer Communications Society, Waterloo, pp 1–8. https://doi.org/10.5555/1089508.1089510

    Google Scholar 

  • Köpsel A, Bubalo N (2015) Benefiting from legacy bias. Interactions 22(5):44–47. https://doi.org/10.1145/2803169

    Article  Google Scholar 

  • Kramer J, Burrus N, C DH, Echtler F, Parker M (2012) Hacking the Kinect, 1st edn. Apress. https://doi.org/10.5555/2207813

  • Kristensson PO, Zhai S (2004) SHARK2: a large vocabulary shorthand writing system for pen-based computers. In: Proceedings of the 17th Annual ACM Symposium on User Interface Software and Technology, UIST’04. ACM, New York, pp 43–52. https://doi.org/10.1145/1029632.1029640

    Chapter  Google Scholar 

  • Krueger MW, Gionfriddo T, Hinrichsen K (1985) VIDEOPLACE-an artificial reality. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI’85. ACM, New York, pp 35–40. https://doi.org/10.1145/317456.317463

    Chapter  Google Scholar 

  • Kurtenbach GP (1993) The design and evaluation of marking menus. PhD thesis, University of Toronto, Canada. https://doi.org/10.5555/193157, uMI Order No. GAXNN-82896

  • Kurtenbach G, Hulteen E (1990) Gestures in human-computer communication. In: Laurel B (ed) The art of human-computer interface design. Addison-Wesley Publishing Company, Inc., New York

    Google Scholar 

  • Lafreniere B, Jonker RT, Santosa S, Parent M, Glueck M, Grossman T, Benko H, Wigdor D (2021) False positives vs. false negatives: the effects of recovery time and cognitive costs on input error preference. In: Proceedings of the 34th Annual ACM Symposium on User Interface Software and Technology, UIST’21. ACM, New York. pp 54–68. https://doi.org/10.1145/3472749.3474735

  • LaViola JJ (1999) A survey of hand posture and gesture recognition techniques and technology. Technical report, Brown University. https://doi.org/10.5555/864649

    Google Scholar 

  • Lawson JYL, Vanderdonckt J, Vatavu RD (2018) Mass-computer interaction for thousands of users and beyond. In: Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems, CHI EA’18. ACM, New York, pp 1–6. https://doi.org/10.1145/3170427.3188465

    Google Scholar 

  • Le HV, Kosch T, Bader P, Mayer S, Henze N (2018) PalmTouch: using the palm as an additional input modality on commodity smartphones. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, New York, pp 1–13. https://doi.org/10.1145/3173574.3173934

    Google Scholar 

  • Lee JC (2008) Hacking the nintendo wii remote. IEEE Pervasive Comput 7(3):39–45. https://doi.org/10.1109/MPRV.2008.53

    Article  Google Scholar 

  • Lee JC (2010) In search of a natural gesture. XRDS 16(4):9–12. https://doi.org/10.1145/1764848.1764853

    Article  Google Scholar 

  • Lee D, Lee Y, Shin Y, Oakley I (2018) Designing socially acceptable hand-to-face input. In: Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology, UIST’18. ACM, New York, pp 711–723. https://doi.org/10.1145/3242587.3242642

    Chapter  Google Scholar 

  • Lefeuvre K, Totzauer S, Storz M, Kurze A, Bischof A, Berger A (2018) Bricks, blocks, boxes, cubes, and dice: on the role of cubic shapes for the design of tangible interactive devices. In: Proceedings of the 2018 Designing Interactive Systems Conference, DIS’18. ACM, New York, pp 485–496. https://doi.org/10.1145/3196709.3196768

    Google Scholar 

  • Leiva LA, Martín-Albo D, Plamondon R (2015) Gestures à go: authoring synthetic human-like stroke gestures using the kinematic theory of rapid movements. ACM Trans Intell Syst Technol 7(2). https://doi.org/10.1145/2799648

  • Leiva LA, Martín-Albo D, Plamondon R, Vatavu RD (2018) KeyTime: super-accurate prediction of stroke gesture production times. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, New York, pp 1–12. https://doi.org/10.1145/3173574.3173813

    Google Scholar 

  • Leiva LA, Kljun M, Sandor C, Copic Pucihar K (2020a) The wearable radar: sensing gestures through fabrics. In: Proceedings of the 22nd International Conference on Human-Computer Interaction with Mobile Devices and Services, MobileHCI’20. ACM, New York. https://doi.org/10.1145/3406324.3410720

  • Leiva LA, Vatavu RD, Martin-Albo D, Plamondon R (2020b) Omnis prædictio: estimating the full spectrum of human performance with stroke gestures. Int J Hum-Comput Stud 142:102466. https://doi.org/10.1016/j.ijhcs.2020.102466

    Article  Google Scholar 

  • Li Y (2010a) Gesture search: a tool for fast mobile data access. In: Proceedings of the 23nd Annual ACM Symposium on User Interface Software and Technology, UIST’10. ACM, New York, pp 87–96. https://doi.org/10.1145/1866029.1866044

    Chapter  Google Scholar 

  • Li Y (2010b) Protractor: a fast and accurate gesture recognizer. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI’10. ACM, New York, pp 2169–2172. https://doi.org/10.1145/1753326.1753654

    Google Scholar 

  • Li Y, Li T, Patel RA, Yang XD, Zhou X (2018) Self-powered gesture recognition with ambient light. In: Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology, UIST’18. ACM, New York, pp 595–608. https://doi.org/10.1145/3242587.3242635

    Chapter  Google Scholar 

  • Li Z, Chan J, Walton J, Benko H, Wigdor D, Glueck M (2021) Armstrong: an empirical examination of pointing at non-dominant arm-anchored UIs in virtual reality. In: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. ACM, New York. https://doi.org/10.1145/3411764.3445064

  • Lien J, Gillian N, Karagozler ME, Amihood P, Schwesig C, Olson E, Raja H, Poupyrev I (2016) Soli: ubiquitous gesture sensing with millimeter wave radar. ACM Trans Graph 35(4). https://doi.org/10.1145/2897824.2925953

  • Liu C, Chapuis O, Beaudouin-Lafon M, Lecolinet E (2016) Shared interaction on a wall-sized display in a data manipulation task. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, CHI’16. ACM, New York, pp 2075–2086. https://doi.org/10.1145/2858036.2858039

    Google Scholar 

  • Liu C, Chapuis O, Beaudouin-Lafon M, Lecolinet E (2017) CoReach: cooperative gestures for data manipulation on wall-sized displays. In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, CHI’17. ACM, New York, pp 6730–6741. https://doi.org/10.1145/3025453.3025594

    Google Scholar 

  • Liu Z, Vogel D, Wallace JR (2018) Applying the cumulative fatigue model to interaction on large, multi-touch displays. In: Proceedings of the 7th ACM International Symposium on Pervasive Displays, PerDis’18. ACM, New York. https://doi.org/10.1145/3205873.3205890

  • Long AC, Landay JA, Rowe LA, Michiels J (2000) Visual similarity of pen gestures. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI’00. ACM, New York, pp 360–367. https://doi.org/10.1145/332040.332458

    Chapter  Google Scholar 

  • Lü H, Li Y (2012) Gesture coder: a tool for programming multi-touch gestures by demonstration. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI’12. ACM, New York, pp 2875–2884. https://doi.org/10.1145/2207676.2208693

    Google Scholar 

  • Lü H, Li Y (2013) Gesture studio: authoring multi-touch interactions through demonstration and declaration. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, New York, pp 257–266. https://doi.org/10.1145/2470654.2470690

    Chapter  Google Scholar 

  • Luo Y, Wu K, Palacios T, Matusik W (2021) KnitUI: fabricating interactive and sensing textiles with machine knitting. In: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. ACM, New York. https://doi.org/10.1145/3411764.3445780

  • Magrofuoco N, Vanderdonckt J (2019) Gelicit: a cloud platform for distributed gesture elicitation studies. Proc ACM Hum-Comput Interact 3(EICS). https://doi.org/10.1145/3331148

  • Magrofuoco N, Roselli P, Vanderdonckt J (2021) Two-dimensional stroke gesture recognition: a survey. ACM Comput Surv 54(7). https://doi.org/10.1145/3465400

  • Malu M, Chundury P, Findlater L (2018) Exploring accessible smartwatch interactions for people with upper body motor impairments. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, New York, pp 1–12. https://doi.org/10.1145/3173574.3174062

    Google Scholar 

  • Marquardt N, Diaz-Marino R, Boring S, Greenberg S (2011) The proximity toolkit: prototyping proxemic interactions in ubiquitous computing ecologies. In: Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, UIST’11. ACM, New York, pp 315–326. https://doi.org/10.1145/2047196.2047238

    Chapter  Google Scholar 

  • Masson D, Goguey A, Malacria S, Casiez G (2017) WhichFingers: identifying fingers on touch surfaces and keyboards using vibration sensors. In: Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology, UIST’17. ACM, New York, pp 41–48. https://doi.org/10.1145/3126594.3126619

    Chapter  Google Scholar 

  • Matthies DJC, Perrault ST, Urban B, Zhao S (2015) Botential: localizing on-body gestures by measuring electrical signatures on the human skin. In: Proceedings of the 17th International Conference on Human-Computer Interaction with Mobile Devices and Services, MobileHCI’15. ACM, New York, pp 207–216. https://doi.org/10.1145/2785830.2785859

    Google Scholar 

  • May KR, Gable TM, Walker BN (2017) Designing an in-vehicle air gesture set using elicitation methods. In: Proceedings of the 9th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, AutomotiveUI’17. ACM, New York, pp 74–83. https://doi.org/10.1145/3122986.3123015

    Google Scholar 

  • McNeill D (1992) Hand and mind: what gestures reveal about thought. University of Chicago Press, USA

    Google Scholar 

  • Microsoft Corporation (2013) Human interface guidelines v1.6. https://download.microsoft.com/ download/B/0/7/B070724E-52B4-4B1A-BD1B-05CC28D07899/Human_Interface_Guidelines _v1.7.0.pdf

  • Mitra S, Acharya T (2007) Gesture recognition: a survey. IEEE Trans Syst Man Cybernet Part C (Appl Rev) 37(3):311–324. https://doi.org/10.1109/TSMCC.2007.893280

    Article  Google Scholar 

  • Morris MR (2012) Web on the wall: insights from a multimodal interaction elicitation study. In: Proceedings of the 2012 ACM International Conference on Interactive Tabletops and Surfaces, ITS’12. ACM, New York, pp 95–104. https://doi.org/10.1145/2396636.2396651

    Google Scholar 

  • Morris MR, Huang A, Paepcke A, Winograd T (2006) Cooperative gestures: multi-user gestural interactions for co-located groupware. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI’06. ACM, New York, pp 1201–1210. https://doi.org/10.1145/1124772.1124952

    Chapter  Google Scholar 

  • Morris MR, Wobbrock JO, Wilson AD (2010) Understanding users’ preferences for surface gestures. In: Proceedings of Graphics Interface 2010, GI’10. Canadian Information Processing Society, Canada, pp 261–268. https://doi.org/10.5555/1839214.1839260

    Google Scholar 

  • Morris MR, Danielescu A, Drucker S, Fisher D, Lee B, schraefel MC, Wobbrock JO (2014) Reducing legacy bias in gesture elicitation studies. Interactions 21(3):40–45. https://doi.org/10.1145/2591689

  • Mott ME, Vatavu RD, Kane SK, Wobbrock JO (2016) Smart touch: improving touch accuracy for people with motor impairments with template matching. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, CHI’16. ACM, New York, pp 1934–1946. https://doi.org/10.1145/2858036.2858390

    Google Scholar 

  • Nacenta MA, Kamber Y, Qiang Y, Kristensson PO (2013) Memorability of pre-designed and user-defined gesture sets. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI’13. ACM, New York, pp 1099–1108. https://doi.org/10.1145/2470654.2466142

    Google Scholar 

  • Nebeling M, Ott D, Norrie MC (2015) Kinect analysis: a system for recording, analysing and sharing multimodal interaction elicitation studies. In: Proceedings of the 7th ACM SIGCHI Symposium on Engineering Interactive Computing Systems, EICS’15. ACM, New York, pp 142–151. https://doi.org/10.1145/2774225.2774846

    Google Scholar 

  • Norman DA (2010) Natural user interfaces are not natural. Interactions 17(3):6–10. https://doi.org/10.1145/1744161.1744163

    Article  Google Scholar 

  • Norman DA, Nielsen J (2010) Gestural Interfaces: a step backward in usability. Interactions 17(5):46–49. https://doi.org/10.1145/1836216.1836228

    Article  Google Scholar 

  • Olwal A, Starner T, Mainini G (2020) E-textile microinteractions: augmenting twist with flick, slide and grasp gestures for soft electronics. In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, CHI’20. ACM, New York, pp 1–13. https://doi.org/10.1145/3313831.3376236

    Google Scholar 

  • Ousmer M, Vanderdonckt J, Buraga S (2019) An ontology for reasoning on body-based gestures. In: Proceedings of the ACM SIGCHI Symposium on Engineering Interactive Computing Systems, EICS’19. ACM, New York. https://doi.org/10.1145/3319499.3328238

  • Palipana S, Salami D, Leiva LA, Sigg S (2021) Pantomime: mid-air gesture recognition with sparse millimeter-wave radar point clouds. Proc ACM Interact Mob Wearable Ubiquitous Technol 5(1). https://doi.org/10.1145/3448110

  • Perrault ST, Lecolinet E, Eagan J, Guiard Y (2013) Watchit: simple gestures and eyes-free interaction for wristwatches and bracelets. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI’13. ACM, New York, pp 1451–1460. https://doi.org/10.1145/2470654.2466192

    Google Scholar 

  • Piumsomboon T, Clark A, Billinghurst M, Cockburn A (2013) User-defined gestures for augmented reality. In: CHI’13 Extended Abstracts on Human Factors in Computing Systems, CHI EA’13. ACM, New York, pp 955–960. https://doi.org/10.1145/2468356.2468527

    Google Scholar 

  • Plamondon R (1995) A kinematic theory of rapid human movements. Part I: movement representation and control. Biol Cybern 72(4):295–307. https://doi.org/doi.org/10.1007/bf00202785

    Article  MATH  Google Scholar 

  • Plaumann K, Lehr D, Rukzio E (2016) Who has the force? Solving conflicts for multi user mid-air gestures for TVs. In: Proceedings of the ACM International Conference on Interactive Experiences for TV and Online Video, TVX’16. ACM, New York, pp 25–29. https://doi.org/10.1145/2932206.2932208

    Chapter  Google Scholar 

  • Rakthanmanon T, Campana B, Mueen A, Batista G, Westover B, Zhu Q, Zakaria J, Keogh E (2012) Searching and mining trillions of time series subsequences under dynamic time warping. In: Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD’12. ACM, New York, pp 262–270. https://doi.org/10.1145/2339530.2339576

    Google Scholar 

  • Rateau H, Lank E, Liu Z (2022) Leveraging smartwatch and earbuds gesture capture to support wearable interaction. Proc ACM Hum-Comput Interact 6(ISS). https://doi.org/10.1145/3567710

  • Rekik Y, Vatavu RD, Grisoni L (2014a) Match-up & conquer: a two-step technique for recognizing unconstrained bimanual and multi-finger touch input. In: Proceedings of the 2014 International Working Conference on Advanced Visual Interfaces, AVI’14. ACM, New York, pp 201–208. https://doi.org/10.1145/2598153.2598167

    Chapter  Google Scholar 

  • Rekik Y, Vatavu RD, Grisoni L (2014b) Understanding users’ perceived difficulty of multi-touch gesture articulation. In: Proceedings of the 16th International Conference on Multimodal Interaction, ICMI’14. ACM, New York, pp 232–239. https://doi.org/10.1145/2663204.2663273

    Google Scholar 

  • Rendl C, Greindl P, Probst K, Behrens M, Haller M (2014) Presstures: exploring pressure-sensitive multi-touch gestures on trackpads. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI’14. ACM, New York, pp 431–434. https://doi.org/10.1145/2556288.2557146

    Google Scholar 

  • Rico J, Brewster S (2010) Usable gestures for mobile interfaces: evaluating social acceptability. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI’10. ACM, New York, pp 887–896. https://doi.org/10.1145/1753326.1753458

    Google Scholar 

  • Roy Q, Malacria S, Guiard Y, Lecolinet E, Eagan J (2013) Augmented letters: mnemonic gesture-based shortcuts. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI’13. ACM, New York, pp 2325–2328. https://doi.org/10.1145/2470654.2481321

    Google Scholar 

  • Rubine D (1991) Specifying gestures by example. In: Proceedings of the 18th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH’91. ACM, New York, pp 329–337. https://doi.org/10.1145/122718.122753

    Chapter  Google Scholar 

  • Ruiz J, Vogel D (2015) Soft-constraints to reduce legacy and performance bias to elicit whole-body gestures with low arm fatigue. In: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, CHI’15. ACM, New York, pp 3347–3350. https://doi.org/10.1145/2702123.2702583

    Google Scholar 

  • Ruiz J, Li Y, Lank E (2011) User-defined motion gestures for mobile interaction. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI’11. ACM, New York, pp 197–206. https://doi.org/10.1145/1978942.1978971

    Google Scholar 

  • Saffer D (2009) Designing gestural interfaces. O’Reilly Media, Inc., Sebastopol

    Google Scholar 

  • Saponas TS, Harrison C, Benko H (2011) Pockettouch: through-fabric capacitive touch input. In: Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, UIST’11. ACM, New York, pp 303–308. https://doi.org/10.1145/2047196.2047235

    Chapter  Google Scholar 

  • Schiettecatte B, Vanderdonckt J (2008) AudioCubes: a distributed cube tangible interface based on interaction range for sound design. In: Proceedings of the 2nd International Conference on Tangible and Embedded Interaction, TEI’08. ACM, New York, pp 3–10. https://doi.org/10.1145/1347390.1347394

    Chapter  Google Scholar 

  • Schipor OA, Vatavu RD, Wu W (2019) SAPIENS: towards software architecture to support peripheral interaction in smart environments. Proc ACM Hum-Comput Interact 3(EICS). https://doi.org/10.1145/3331153

  • Schlömer T, Poppinga B, Henze N, Boll S (2008) Gesture recognition with a wii controller. In: Proceedings of the 2nd International Conference on Tangible and Embedded Interaction, TEI’08. ACM, New York, pp 11–14. https://doi.org/10.1145/1347390.1347395

    Chapter  Google Scholar 

  • Schönauer C, Mossel A, Zaiundefinedi IA, Vatavu RD (2022) Touch, movement and vibration: user perception of vibrotactile feedback for touch and mid-air gestures. In: Proceedings of Human-Computer Interaction – INTERACT 2015. Springer, Berlin/Heidelberg, pp 165–172. https://doi.org/10.1007/978-3-319-22723-8_14

    Google Scholar 

  • Schweigert R, Leusmann J, Hagenmayer S, Weiß M, Le HV, Mayer S, Bulling A (2019) KnuckleTouch: enabling knuckle gestures on capacitive touchscreens using deep learning. In: Proceedings of Mensch Und Computer 2019, MuC’19. ACM, New York, pp 387–397. https://doi.org/10.1145/3340764.3340767

  • Sharma A, Roo JS, Steimle J (2019) Grasping microgestures: eliciting single-hand microgestures for handheld objects. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, New York, pp 1–13. https://doi.org/10.1145/3290605.3300632

    Google Scholar 

  • Shilkrot R, Huber J, Steimle J, Nanayakkara S, Maes P (2015) Digital digits: a comprehensive survey of finger augmentation devices. ACM Comput Surv 48(2). https://doi.org/10.1145/2828993

  • Sluÿters A, Lambot S, Vanderdonckt J (2022) Hand gesture recognition for an off-the-shelf radar by electromagnetic modeling and inversion. In: Proceedings of the 27th International Conference on Intelligent User Interfaces, IUI’22. ACM, New York, pp 506–522. https://doi.org/10.1145/3490099.3511107

    Google Scholar 

  • Soliman M, Mueller F, Hegemann L, Roo JS, Theobalt C, Steimle J (2018) FingerInput: capturing expressive single-hand thumb-to-finger microgestures. In: Proceedings of the 2018 ACM International Conference on Interactive Surfaces and Spaces, ISS’18. ACM, New York, pp 177–187. https://doi.org/10.1145/3279778.3279799

    Chapter  Google Scholar 

  • Stößel C (2009) Familiarity as a factor in designing finger gestures for elderly users. In: Proceedings of the 11th International Conference on Human-Computer Interaction with Mobile Devices and Services, MobileHCI’09. ACM, New York. https://doi.org/10.1145/1613858.1613950

  • Suarez J, Murphy RR (2012) Hand gesture recognition with depth images: a review. In: Proceedings of the 21st IEEE International Symposium on Robot and Human Interactive Communication, pp 411–417. https://doi.org/10.1109/ROMAN.2012.6343787

  • Sutherland IE (1963) Sketchpad: a man-machine graphical communication system. In: Proceedings of the Spring Joint Computer Conference, AFIPS’63 (Spring), 21–23 May 1963. ACM, New York, pp 329–346. https://doi.org/10.1145/1461551.1461591

    Chapter  Google Scholar 

  • Tanase CA, Vatavu RD, Ştefan Gheorghe Pentiuc, Graur A (2008) Detecting and tracking multiple users in the proximity of interactive tabletops. Adv Electr Comput Eng 8(2):61–64

    Article  Google Scholar 

  • Taranta EM, LaViola JJ (2015) Penny pincher: a blazing fast, highly accurate $-family recognizer. In: Proceedings of the 41st Graphics Interface Conference, GI’15. Canadian Information Processing Society, pp 195–202

    Google Scholar 

  • Taranta II EM, Samiei A, Maghoumi M, Khaloo P, Pittman CR, LaViola JJ Jr (2017) Jackknife: a reliable recognizer with few samples and many modalities. In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, CHI’17. ACM, New York, pp 5850–5861. https://doi.org/10.1145/3025453.3026002

    Google Scholar 

  • Tsandilas T (2018) Fallacies of agreement: a critical review of consensus assessment methods for gesture elicitation. ACM Trans Comput-Hum Interact 25(3). https://doi.org/10.1145/3182168

  • Tu H, Ren X, Zhai S (2012) A comparative evaluation of finger and pen stroke gestures. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI’12. ACM, New York, pp 1287–1296. https://doi.org/10.1145/2207676.2208584

    Google Scholar 

  • Ungurean OC, Vatavu RD (2022) I gave up wearing rings: insights on the perceptions and preferences of wheelchair users for interactions with wearables. IEEE Pervasive Comput 21(3):92–101. https://doi.org/10.1109/MPRV.2022.3155952

    Article  Google Scholar 

  • Ungurean OC, Vatavu RD, Leiva LA, Martín-Albo D (2018a) Predicting stroke gesture input performance for users with motor impairments. In: Proceedings of the 20th International Conference on Human-Computer Interaction with Mobile Devices and Services Adjunct, MobileHCI’18. ACM, New York, pp 23–30. https://doi.org/10.1145/3236112.3236116

    Google Scholar 

  • Ungurean OC, Vatavu RD, Leiva LA, Plamondon R (2018b) Gesture input for users with motor impairments on touchscreens: empirical results based on the kinematic theory. In: Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems, CHI EA’18. ACM, New York, pp 1–6. https://doi.org/10.1145/3170427.3188619

  • Vanderdonckt J, Roselli P, Pérez-Medina JL (2018) !FTL, an articulation-invariant stroke gesture recognizer with controllable position, scale, and rotation invariances. In: Proceedings of the 20th ACM International Conference on Multimodal Interaction, ICMI’18. ACM, New York, pp 125–134. https://doi.org/10.1145/3242969.3243032

    Google Scholar 

  • Vanderdonckt J, Magrofuoco N, Kieffer S, Pérez J, Rase Y, Roselli P, Villarreal S (2019) Head and shoulders gestures: exploring user-defined gestures with upper body. In: Proceedings of the International Conference on Human-Computer Interaction, HCII’19. Lecture Notes in Computer Science, vol 11584, pp 192–213. https://doi.org/10.1007/978-3-030-23541-3_15

  • Vatavu RD, Anthony L, Wobbrock JO (2018a) $Q: a super-quick, articulation-invariant stroke-gesture recognizer for low-resource devices. In: Proceedings of the 20th International Conference on Human-Computer Interaction with Mobile Devices and Services, MobileHCI’18. ACM, New York. https://doi.org/10.1145/3229434.3229465

  • Vatavu RD, Gheran BF, Schipor MD (2018b) The impact of low vision on touch-gesture articulation on mobile devices. IEEE Pervasive Comput 17(1):27–37. https://doi.org/10.1109/MPRV.2018.011591059

    Article  Google Scholar 

  • Velloso E, Schmidt D, Alexander J, Gellersen H, Bulling A (2015) The feet in human–computer interaction: a survey of foot-based interaction. ACM Comput Surv 48(2). https://doi.org/10.1145/2816455

  • Villarreal-Narvaez S, Vanderdonckt J, Vatavu RD, Wobbrock JO (2020) A systematic review of gesture elicitation studies: what can we learn from 216 studies? In: Proceedings of the 2020 ACM Designing Interactive Systems Conference, DIS’20. Association for Computing Machinery, New York, pp 855–872. https://doi.org/10.1145/3357236.3395511

    Chapter  Google Scholar 

  • Vatavu RD (2012) User-defined gestures for free-hand tv control. In: Proceedings of the 10th European Conference on Interactive TV and Video, EuroITV’12. ACM, New York, pp 45–48. https://doi.org/10.1145/2325616.2325626

    Chapter  Google Scholar 

  • Vatavu RD (2013) The impact of motion dimensionality and bit cardinality on the design of 3D gesture recognizers. Int J Hum-Comput Stud 71(4):387–409. https://doi.org/10.1016/j.ijhcs.2012.11.005

    Article  Google Scholar 

  • Vatavu RD (2017a) Beyond features for recognition: human-readable measures to understand users’ whole-body gesture performance. Int J Hum-Comput Interact 33(9):713–730. https://doi.org/10.1080/10447318.2017.1278897

    Article  Google Scholar 

  • Vatavu RD (2017b) Improving gesture recognition accuracy on touch screens for users with low vision. In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, CHI’17. ACM, New York, pp 4667–4679. https://doi.org/10.1145/3025453.3025941

  • Vatavu RD (2017c) Smart-pockets: body-deictic gestures for fast access to personal data during ambient interactions. Int J Hum-Comput Stud 103:1–21. https://doi.org/10.1016/j.ijhcs.2017.01.005

    Article  Google Scholar 

  • Vatavu RD (2019) The dissimilarity-consensus approach to agreement analysis in gesture elicitation studies. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, New York, pp 1–13. https://doi.org/10.1145/3290605.3300454

    Google Scholar 

  • Vatavu RD, Bilius LB (2021) GestuRING: a web-based tool for designing gesture input with rings, ring-like, and ring-ready devices. In: Proceedings of the 34th Annual ACM Symposium on User Interface Software and Technology, UIST’21. ACM, New York, pp 710–723. https://doi.org/10.1145/3472749.3474780

    Google Scholar 

  • Vatavu RD, Ungurean OC (2019) Stroke-gesture input for people with motor impairments: empirical results & research roadmap. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, New York, pp 1–14. https://doi.org/10.1145/3290605.3300445

    Google Scholar 

  • Vatavu RD, Ungurean OC (2022) Understanding gesture input articulation with upper-body wearables for users with upper-body motor impairments. In: Proceedings of the CHI Conference on Human Factors in Computing Systems, CHI’22. ACM, New York, pp 2:1–2:16. https://doi.org/10.1145/3491102.3501964

  • Vatavu RD, Wobbrock JO (2015) Formalizing agreement analysis for elicitation studies: new measures, significance test, and toolkit. In: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, CHI’15. ACM, New York, pp 1325–1334. https://doi.org/10.1145/2702123.2702223

    Google Scholar 

  • Vatavu RD, Wobbrock JO (2016) Between-subjects elicitation studies: formalization and tool support. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, CHI’16. ACM, New York, pp 3390–3402. https://doi.org/10.1145/2858036.2858228

    Google Scholar 

  • Vatavu RD, Wobbrock JO (2022) Clarifying agreement calculations and analysis for end-user elicitation studies. ACM Trans Comput-Hum Interact 29(1). https://doi.org/10.1145/3476101

  • Vatavu RD, Zaiti IA (2013) Automatic recognition of object size and shape via user-dependent measurements of the grasping hand. Int J Hum-Comput Stud 71(5):590–607. https://doi.org/10.1016/j.ijhcs.2013.01.002

    Article  Google Scholar 

  • Vatavu RD, Vogel D, Casiez G, Grisoni L (2011) Estimating the perceived difficulty of pen gestures. In: Proceedings of the 13th IFIP TC 13 International Conference on Human-Computer Interaction – Volume Part II, INTERACT’11. Springer, Berlin/Heidelberg, pp 89–106. https://doi.org/10.1007/978-3-642-23771-3_9

    Google Scholar 

  • Vatavu RD, Anthony L, Wobbrock JO (2012) Gestures as point clouds: a $P recognizer for user interface prototypes. In: Proceedings of the 14th ACM International Conference on Multimodal Interaction, ICMI’12. ACM, New York, pp 273–280. https://doi.org/10.1145/2388676.2388732

    Google Scholar 

  • Vatavu RD, Anthony L, Wobbrock JO (2013) Relative accuracy measures for stroke gestures. In: Proceedings of the 15th ACM on International Conference on Multimodal Interaction, ICMI’13. ACM, New York, pp 279–286. https://doi.org/10.1145/2522848.2522875

    Google Scholar 

  • Vatavu RD, Anthony L, Wobbrock JO (2014) Gesture heatmaps: understanding gesture performance with colorful visualizations. In: Proceedings of the 16th International Conference on Multimodal Interaction, ICMI’14. ACM, New York, pp 172–179. https://doi.org/10.1145/2663204.2663256

    Google Scholar 

  • Vatavu RD, Cramariuc G, Schipor DM (2015) Touch interaction for children aged 3 to 6 years: experimental findings and relationship to motor skills. Int J Hum-Comput Stud 74:54–76. https://doi.org/10.1016/j.ijhcs.2014.10.007

    Article  Google Scholar 

  • Vogel D, Balakrishnan R (2004) Interactive public ambient displays: transitioning from implicit to explicit, public to personal, interaction with multiple users. In: Proceedings of the 17th Annual ACM Symposium on User Interface Software and Technology, UIST’04. ACM, New York, pp 137–146. https://doi.org/10.1145/1029632.1029656

    Chapter  Google Scholar 

  • Vogiatzidakis P, Koutsabasis P (2018) Gesture elicitation studies for mid-air interaction: a review. Multimodal Technol Interact 2(4). https://doi.org/10.3390/mti2040065

  • Walter R, Bailly G, Müller J (2013) StrikeAPose: revealing mid-air gestures on public displays. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI’13. ACM, New York, pp 841–850. https://doi.org/10.1145/2470654.2470774

    Google Scholar 

  • Wang Y, Neff M (2013) Data-driven glove calibration for hand motion capture. In: Proceedings of the 12th ACM SIGGRAPH/Eurographics Symposium on Computer Animation, SCA’13. ACM, New York, pp 15–24. https://doi.org/10.1145/2485895.2485901

    Chapter  Google Scholar 

  • Wang S, Song J, Lien J, Poupyrev I, Hilliges O (2016) Interacting with soli: exploring fine-grained dynamic gesture recognition in the radio-frequency spectrum. In: Proceedings of the 29th Annual Symposium on User Interface Software and Technology, UIST’16. ACM, New York, pp 851–860. https://doi.org/10.1145/2984511.2984565

    Chapter  Google Scholar 

  • Wang R, Yu C, Yang XD, He W, Shi Y (2019) EarTouch: facilitating smartphone use for visually impaired people in mobile and public scenarios. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, New York, pp 1–13. https://doi.org/10.1145/3290605.3300254

    Google Scholar 

  • Wang T, Qian X, He F, Hu X, Cao Y, Ramani K (2021) GesturAR: an authoring system for creating freehand interactive augmented reality applications. In: The 34th Annual ACM Symposium on User Interface Software and Technology, UIST’21. ACM, New York, pp 552–567. https://doi.org/10.1145/3472749.3474769

    Chapter  Google Scholar 

  • Webb A (2002) Statistical pattern recognition, 2nd edn. John Wiley & Sons, West Sussex

    Book  MATH  Google Scholar 

  • Wellner P (1993) Interacting with paper on the digitaldesk. Commun ACM 36(7):87–96. https://doi.org/10.1145/159544.159630

    Article  Google Scholar 

  • Wexelblat A (1995) An approach to natural gesture in virtual environments. ACM Trans Comput-Hum Interact 2(3):179–200. https://doi.org/10.1145/210079.210080

    Article  Google Scholar 

  • Wigdor D, Wixon D (2011) Brave NUI World: designing natural user interfaces for touch and gesture, 1st edn. Morgan Kaufmann Publishers Inc., San Francisco

    Google Scholar 

  • Wilson A, Bobick A (1999) Parametric hidden Markov models for gesture recognition. IEEE Trans Pattern Anal Mach Intell 21(9):884–900. https://doi.org/10.1109/34.790429

    Article  Google Scholar 

  • Wobbrock JO, Aung HH, Rothrock B, Myers BA (2005) Maximizing the guessability of symbolic input. In: CHI’05 Extended Abstracts on Human Factors in Computing Systems, CHI EA’05. ACM, New York, Pp 1869–1872. https://doi.org/10.1145/1056808.1057043

    Chapter  Google Scholar 

  • Wobbrock JO, Wilson AD, Li Y (2007) Gestures without Libraries, Toolkits or Training: A $1 Recognizer for user interface prototypes. In: Proceedings of the 20th Annual ACM Symposium on User Interface Software and Technology, UIST’07. ACM, New York, pp 159–168. https://doi.org/10.1145/1294211.1294238

    Chapter  Google Scholar 

  • Wobbrock JO, Morris MR, Wilson AD (2009) User-defined gestures for surface computing. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI’09. ACM, New York, pp 1083–1092. https://doi.org/10.1145/1518701.1518866

    Google Scholar 

  • Wobbrock JO, Gajos KZ, Kane SK, Vanderheiden GC (2018) Ability-based design. Commun ACM 61(6):62–71. https://doi.org/10.1145/3148051

    Article  Google Scholar 

  • Wolf K, Schleicher R, Kratz S, Rohs M (2013) Tickle: a surface-independent interaction technique for grasp interfaces. In: Proceedings of the 7th International Conference on Tangible, Embedded and Embodied Interaction, TEI’13. ACM, New York, pp 185–192. https://doi.org/10.1145/2460625.2460654

    Chapter  Google Scholar 

  • Wu M, Balakrishnan R (2003) Multi-finger and whole hand gestural interaction techniques for multi-user tabletop displays. In: Proceedings of the 16th Annual ACM Symposium on User Interface Software and Technology, UIST’03. ACM, New York, pp 193–202. https://doi.org/10.1145/964696.964718

    Chapter  Google Scholar 

  • Wu E, Yuan Y, Yeo HS, Quigley A, Koike H, Kitani KM (2020a) Back-hand-pose: 3D hand pose estimation for a wrist-worn camera via dorsum deformation network. In: Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology, UIST’20. ACM, New York, pp 1147–1160. https://doi.org/10.1145/3379337.3415897

    Chapter  Google Scholar 

  • Wu TY, Qi S, Chen J, Shang M, Gong J, Seyed T, Yang XD (2020b) Fabriccio: touchless gestural input on interactive fabrics. In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, CHI’20. ACM, New York, pp 1–14. https://doi.org/10.1145/3313831.3376681

  • Xia H, Glueck M, Annett M, Wang M, Wigdor D (2022) Iteratively designing gesture vocabularies: a survey and analysis of best practices in the hci literature. ACM Trans Comput-Hum Interact 29(4). https://doi.org/10.1145/3503537

  • Xiao R, Laput G, Harrison C (2014) Expanding the input expressivity of smartwatches with mechanical pan, twist, tilt and click. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI’14. ACM, New York, pp 193–196. https://doi.org/10.1145/2556288.2557017

    Google Scholar 

  • Yan Y, Yu C, Ma X, Yi X, Sun K, Shi Y (2018a) VirtualGrasp: leveraging experience of interacting with physical objects to facilitate digital object retrieval. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, New York, pp 1–13. https://doi.org/10.1145/3173574.3173652

  • Yan Y, Yu C, Yi X, Shi Y (2018b) HeadGesture: hands-free input approach leveraging head movements for HMD devices. Proc ACM Interact Mob Wearable Ubiquitous Technol 2(4). https://doi.org/10.1145/3287076

  • Yasen M, Jusoh S (2019) A systematic review on hand gesture recognition techniques, challenges and applications. Peer J Comput Sci 5(e218). https://doi.org/10.7717/peerj-cs.218

  • You CW, Lin YF, Luo E, Lin HY, Kao HLC (2019) Understanding social perceptions towards interacting with on-skin interfaces in public. In: Proceedings of the 23rd International Symposium on Wearable Computers, ISWC’19. ACM, New York, pp 244–253. https://doi.org/10.1145/3341163.3347751

    Chapter  Google Scholar 

  • Zaiti IA, Pentiuc SG, Vatavu RD (2015) On free-hand TV control: experimental results on user-elicited gestures with leap motion. Personal and Ubiquitous Computing 19(5-6):821–838. https://doi.org/10.1007/s00779-015-0863-y

    Article  Google Scholar 

  • Zhai S, Kristensson P, Appert C, Andersen T, Cao X (2012) Foundational issues in touch-surface stroke gesture design: an integrative review. Found Trends Hum-Comput Interact 5(2):97–205. https://doi.org/10.1561/1100000012

    Article  Google Scholar 

  • Zhang Z (2012) Microsoft kinect sensor and its effect. IEEE Multimed 19(2):4–10. https://doi.org/10.1109/MMUL.2012.24

    Article  Google Scholar 

  • Zheng J, Vogel D (2016) Finger-aware shortcuts. In: Proceedings of the CHI Conference on Human Factors in Computing Systems, CHI’16. ACM, New York, pp 4274–4285. https://doi.org/10.1145/2858036.2858355

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. MintViz Lab, MANSiD Research Center, Ştefan cel Mare University of Suceava, Suceava, Romania

    Radu-Daniel Vatavu

Authors
  1. Radu-Daniel Vatavu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Radu-Daniel Vatavu .

Editor information

Editors and Affiliations

  1. Universite catholique de Louvain, Louvain-la-Neuve, Belgium

    Jean Vanderdonckt

  2. IRIT - Interactive Critical Sys Group, Paul Sabatier University, Toulouse, France

    Philippe Palanque

  3. I3S, INRIA wimmics/SPARKS team, Université Nice Sophia Antipolis, Sophia Antipolis Cedex, France

    Marco Winckler

Rights and permissions

Reprints and permissions

Copyright information

© 2023 Springer Nature Switzerland AG

About this entry

Check for updates. Verify currency and authenticity via CrossMark

Cite this entry

Vatavu, RD. (2023). Gesture-Based Interaction. In: Vanderdonckt, J., Palanque, P., Winckler, M. (eds) Handbook of Human Computer Interaction. Springer, Cham. https://doi.org/10.1007/978-3-319-27648-9_20-1

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-27648-9_20-1

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-27648-9

  • Online ISBN: 978-3-319-27648-9

  • eBook Packages: Springer Reference Computer SciencesReference Module Computer Science and Engineering

Publish with us

Policies and ethics

Search

Navigation