Skip to main content
SpringerLink
Log in

User Experience and Mid-Air Haptics: Applications, Methods, and Challenges

  • Chapter
  • First Online:
Ultrasound Mid-Air Haptics for Touchless Interfaces

Part of the book series: Human–Computer Interaction Series ((HCIS))

Abstract

Mid-air haptic feedback presents exciting new opportunities for useful and delightful interactive systems. However, with these opportunities come several design challenges that vary greatly depending on the application at hand. In this chapter, we reveal these challenges from a user experience perspective. To that end, we first provide a comprehensive literature review covering many of the different applications of the technology. Then, we present 12 design guidelines and make recommendations for effective mid-air haptic interaction designs and implementations. Finally, we suggest an iterative haptic design framework that can be followed to create a quality mid-air haptic experience.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://github.com/ultraleap/UnityExamples.

  2. 2.

    https://developer.ultrahaptics.com/kb/unity.

References

  • Abdouni A, Clark R, Georgiou O (2019) Seeing is believing but feeling is the truth: Visualising mid-air haptics in oil baths and lightboxes. In: International conference on multimodal interaction, pp 504–505

    Google Scholar 

  • Abeele VV, Spiel K, Nacke L, Johnson D, Gerling K (2020) Development and validation of the player experience inventory: a scale to measure player experiences at the level of functional and psychosocial consequences. Int J Human-Comput Stud 135:102370

    Article  Google Scholar 

  • Ablart D, Frier W, Limerick H, Georgiou O, Obrist M (2019) Using ultrasonic mid-air haptic patterns in multi-modal user experiences. In: IEEE international symposium on haptic, audio and visual environments and games (HAVE). IEEE, pp 1–6

    Google Scholar 

  • Ablart D, Velasco C, Obrist M (2017) Integrating mid-air haptics into movie experiences. In: Proceedings of the 2017 ACM international conference on interactive experiences for TV and online video, pp 77–84

    Google Scholar 

  • Ahmad BI, Murphy JK, Langdon PM, Godsill SJ, Hardy R, Skrypchuk L (2015) Intent inference for hand pointing gesture-based interactions in vehicles. IEEE Trans Cybernet 46(4):878–889

    Article  Google Scholar 

  • Alpern M, Minardo K (2003) Developing a car gesture interface for use as a secondary task. In: CHI’03 extended abstracts on Human factors in computing systems, pp 932–933

    Google Scholar 

  • Balint P, Althoefer K (2018) Medical virtual reality palpation training using ultrasound based haptics and image processing. Proc Jt Work New Technol Comput Assist Surg

    Google Scholar 

  • Beattie D, Frier W, Georgiou O, Long B, Ablart D (2020) Incorporating the perception of visual roughness into the design of mid-air haptic textures. ACM Sympos Appl Percept 2020:1–10

    Google Scholar 

  • Boehm B, Port D (2001) Risk-based strategic software design: how much cots evaluation is enough? In: Proceedings of the third international workshop on economics-driven software engineering research. Toronto, Canada. Citeseer

    Google Scholar 

  • Breitschaft SJ, Clarke S, Carbon C-C (2019) A theoretical framework of haptic processing in automotive user interfaces and its implications on design and engineering. Front Psychol 10:1470

    Article  Google Scholar 

  • Brice D, Gibson Z, Mcguinness F, Rafferty K (2021) Using ultrasonic haptics within an immersive spider exposure environment to provide a multi-sensorial experience. Front Virtual Real 2:707731

    Article  Google Scholar 

  • Brice D, McRoberts T, Rafferty K (2019) A proof of concept integrated multi-systems approach for large scale tactile feedback in vr. In: International conference on augmented reality, virtual reality and computer graphics. Springer, pp 120–137

    Google Scholar 

  • Brown E, Large DR, Limerick H, Burnett G (2020) Ultrahapticons:“haptifying” drivers’ mental models to transform automotive mid-air haptic gesture infotainment interfaces. In: 12th international conference on automotive user interfaces and interactive vehicular applications, pp 54–57

    Google Scholar 

  • Burnett G, Irune A (2009) Drivers’ quality ratings for switches in cars: assessing the role of the vision, hearing and touch senses. In: Proceedings of the 1st international conference on automotive user interfaces and interactive vehicular applications, pp 107–114

    Google Scholar 

  • Carcagno S, Di Battista A, Plack CJ (2019) Effects of high-intensity airborne ultrasound exposure on behavioural and electrophysiological measures of auditory function. Acta Acust United Acust 105(6):1183–1197

    Article  Google Scholar 

  • Carter T, Seah SA, Long B, Drinkwater B, Subramanian S (2013) Ultrahaptics: multi-point mid-air haptic feedback for touch surfaces. In: Proceedings of the 26th annual ACM symposium on user interface software and technology, pp 505–514

    Google Scholar 

  • Corenthy L, Giordano M, Hayden R, Griffiths D, Jeffrey C, Limerick H, Georgiou O, Carter T, Müller J, Subramanian S (2018) Touchless tactile displays for digital signage: mid-air haptics meets large screens. In: Extended abstracts of the. CHI conference on human factors in computing systems, pp 1–4

    Google Scholar 

  • Cornelio Martinez PI, De Pirro S, Vi CT, Subramanian S (2017) Agency in mid-air interfaces. In: Proceedings of the 2017 CHI conference on human factors in computing systems, pp 2426–2439

    Google Scholar 

  • Cowan N (2001) The magical number 4 in short-term memory: a reconsideration of mental storage capacity. Behav Brain Sci 24(1):87–114

    Article  Google Scholar 

  • Cronin S, Doherty G (2019) Touchless computer interfaces in hospitals: a review. Health Informat J 25(4):1325–1342

    Article  Google Scholar 

  • Daniel H, Damien A, Schneider O, Obrist M (2020) I can feel it moving: science communicators talking about the potential of mid-air haptics. Front Comput Sci

    Google Scholar 

  • Davies N, Clinch S, Alt F (2014) Pervasive displays: understanding the future of digital signage. Synthesis Lect Mob Pervas Comput 8(1):1–128

    Article  Google Scholar 

  • Detjen H, Faltaous S, Pfleging B, Geisler S, Schneegass S (2021) How to increase automated vehicles’ acceptance through in-vehicle interaction design: a review. Int J Human-Comput Int 37(4):308–330

    Article  Google Scholar 

  • Di Battista A (2019) The effect of 40 kHz ultrasonic noise exposure on human hearing. Universitätsbibliothek der RWTH Aachen

    Google Scholar 

  • Driller KK, Frier W, Pont SC, Hartcher-O’Brien J (2019) Mid-air ultrasonic stimulations of the palm-the influence of frequency and stimulus duration on perceived intensity. In: WHC 2019: IEEE world haptics conference

    Google Scholar 

  • Dzidek B, Frier W, Harwood A, Hayden R (2018) Design and evaluation of mid-air haptic interactions in an augmented reality environment. In: International conference on human haptic sensing and touch enabled computer applications. Springer, pp 489–499

    Google Scholar 

  • Etherington D (2017) Bmw’s holoactive touch in-car interface offers tactile feedback on a floating display

    Google Scholar 

  • Evangelou G, Limerick H, Moore J (2021) I feel it in my fingers! sense of agency with mid-air haptics. In: IEEE world haptics conference (WHC). IEEE, pp 727–732

    Google Scholar 

  • Freeman E (2021) Enhancing ultrasound haptics with parametric audio effects. In: Proceedings of the 23rd ACM international conference on multimodal interaction-ICMI, vol 21

    Google Scholar 

  • Freeman E, Vo D-B, Brewster S (2019) Haptiglow: helping users position their hands for better mid-air gestures and ultrasound haptic feedback. In: IEEE world haptics conference (WHC). IEEE, pp 289–294

    Google Scholar 

  • Freeman E, Wilson G, Vo D-B, Ng A, Politis I, Brewster S (2017) Multimodal feedback in HCI: haptics, non-speech audio, and their applications. In: The handbook of multimodal-multisensor interfaces: foundations, user modeling and common modality combinations. Morgan & Claypool

    Google Scholar 

  • Frier W, Ablart D, Chilles J, Long B, Giordano M, Obrist M, Subramanian S (2018) Using spatiotemporal modulation to draw tactile patterns in mid-air. In: International conference on human haptic sensing and touch enabled computer applications. Springer, pp 270–281

    Google Scholar 

  • Frier W, Pittera D, Ablart D, Obrist M, Subramanian S (2019) Sampling strategy for ultrasonic mid-air haptics. In: Proceedings of the 2019 CHI conference on human factors in computing systems, pp 1–11

    Google Scholar 

  • Frutos-Pascual M, Harrison JM, Creed C, Williams I (2019) Evaluation of ultrasound haptics as a supplementary feedback cue for grasping in virtual environments. In: International conference on multimodal interaction, pp 310–318

    Google Scholar 

  • Gaffary Y, Le Gouis B, Marchal M, Argelaguet F, Arnaldi B, Lécuyer A (2017) AR feels “softer" than VR: haptic perception of stiffness in augmented versus virtual reality. IEEE Trans Visual Comput Graph 23(11):2372–2377

    Article  Google Scholar 

  • Garber L (2013) Gestural technology: moving interfaces in a new direction [technology news]. Computer 46(10):22–25

    Article  Google Scholar 

  • Georgiou O, Biscione V, Harwood A, Griffiths D, Giordano M, Long B, Carter T (2017) Haptic in-vehicle gesture controls. In: Proceedings of the 9th international conference on automotive user interfaces and interactive vehicular applications adjunct, pp 233–238

    Google Scholar 

  • Georgiou O, Jeffrey C, Chen Z, Tong BX, Chan SH, Yang B, Harwood A, Carter T (2018) Touchless haptic feedback for VR rhythm games. In: 2018 IEEE conference on virtual reality and 3D user interfaces (VR). IEEE, pp 553–554

    Google Scholar 

  • Gescheider GA, Bolanowski SJ, Pope JV, Verrillo RT (2002) A four-channel analysis of the tactile sensitivity of the fingertip: frequency selectivity, spatial summation, and temporal summation. Somatosens Motor Res 19(2):114–124

    Article  Google Scholar 

  • Girdler A, Georgiou O (2020) Mid-air haptics in aviation–creating the sensation of touch where there is nothing but thin air, arXiv preprint arXiv:2001.01445

  • Green P (2000) Crashes induced by driver information systems and what can be done to reduce them. Tech, Rep, SAE Technical Paper

    Google Scholar 

  • Greenberg S, Carpendale S, Marquardt N, Buxton B (2011) Sketching user experiences: the workbook. Elsevier

    Google Scholar 

  • Hajas D, Pittera D, Nasce A, Georgiou O, Obrist M (2020) Mid-air haptic rendering of 2d geometric shapes with a dynamic tactile pointer. IEEE Trans Hapt 13(4):806–817

    Article  Google Scholar 

  • Hajas D, Ablart D, Schneider O, Obrist M (2020) I can feel it moving: science communicators talking about the potential of mid-air haptics. Front Comput Sci 2

    Google Scholar 

  • Harrington K, Large DR, Burnett G, Georgiou O (2018) Exploring the use of mid-air ultrasonic feedback to enhance automotive user interfaces. In: Proceedings of the 10th international conference on automotive user interfaces and interactive vehicular applications, pp 11–20

    Google Scholar 

  • Hasegawa K, Shinoda H (2018) Aerial vibrotactile display based on multiunit ultrasound phased array. IEEE Trans Hapt 11(3):367–377

    Article  Google Scholar 

  • Hassenzahl M, Burmester M, Koller F (2003) Attrakdiff: ein fragebogen zur messung wahrgenommener hedonischer und pragmatischer qualität. In: Mensch Computer, Springer, pp 187–196

    Google Scholar 

  • Hayward N, Lewis E, Perra E, Jousmäki V, Saarinen V-M, McGlone F, Sams M, Nieminen H (2020) A novel ultrasonic haptic device induces touch sensations with potential applications in neuroscience research. In: IEEE international ultrasonics symposium (IUS). IEEE, pp 1–4

    Google Scholar 

  • Hermann DS (2018) Automotive displays-trends, opportunities and challenges. In: 25th International workshop on active-matrix flatpanel displays and devices (AM-FPD). IEEE, pp 1–6

    Google Scholar 

  • Horrey WJ, Wickens CD (2007) In-vehicle glance duration: distributions, tails, and model of crash risk. Transp Res Rec 1:22–28

    Article  Google Scholar 

  • Hoshi T, Takahashi M, Iwamoto T, Shinoda H (2010) Noncontact tactile display based on radiation pressure of airborne ultrasound. IEEE Trans Hapt 3(3):155–165

    Article  Google Scholar 

  • Howard T, Marchal M, Lécuyer A, Pacchierotti C (2019) Pumah: pan-tilt ultrasound mid-air haptics for larger interaction workspace in virtual reality. IEEE Trans Hapt 13(1):38–44

    Article  Google Scholar 

  • Howard T, Gallagher G, Lécuyer A, Pacchierotti C, Marchal M (2019) Investigating the recognition of local shapes using mid-air ultrasound haptics. In: IEEE world haptics conference (WHC). IEEE, pp 503–508

    Google Scholar 

  • Huang S, Ranganathan SP, Parsons I (2020) To touch or not to touch? comparing touch, mid-air gesture, mid-air haptics for public display in post covid-19 society. SIGGRAPH Asia Posters 1–2

    Google Scholar 

  • Hung GM, John NW, Hancock C, Gould DA, Hoshi T (2013) Ultrapulse-simulating a human arterial pulse with focussed airborne ultrasound. In: 35th annual international conference of the IEEE engineering in medicine and biology society (EMBC). IEEE, pp 2511–2514

    Google Scholar 

  • Hung GM, John NW, Hancock C, Hoshi T (2014) Using and validating airborne ultrasound as a tactile interface within medical training simulators. In: International symposium on biomedical simulation. Springer, pp 30–39

    Google Scholar 

  • Hwang I, Son H, Kim JR (2017) Airpiano: enhancing music playing experience in virtual reality with mid-air haptic feedback. In: IEEE world haptics conference (WHC). IEEE, pp 213–218

    Google Scholar 

  • Inoue S, Kobayashi-Kirschvink KJ, Monnai Y, Hasegawa K, Makino Y, Shinoda H (2014) Horn: the hapt-optic reconstruction. In: ACM SIGGRAPH. Emerging Technologies, pp 1–1

    Google Scholar 

  • Jang J, Frier W, Georgiou O, Park J (2021) Using mid-air tactile patterns in interactive volume exploration. In: 2021 IEEE world haptics conference—demo paper. IEEE

    Google Scholar 

  • Johnson J, Henderson A (2002) Conceptual models: begin by designing what to design. Interactions 9(1):25–32

    Article  Google Scholar 

  • Johnson J, Henderson A (2011) Conceptual models: core to good design. Synthesis Lect Human-Centered Inf 4(2):1–110

    Article  Google Scholar 

  • Kaaresoja T, Anttila E, Hoggan E (2011) The effect of tactile feedback latency in touchscreen interaction. In: IEEE world haptics conference. IEEE, pp 65–70

    Google Scholar 

  • Kappus B, Long B (2018) Spatiotemporal modulation for mid-air haptic feedback from an ultrasonic phased array. J Acoust Soc Am 143(3):1836–1836

    Article  Google Scholar 

  • Karafotias G, Korres G, Teranishi A, Park W, Eid M (2017) Mid-air tactile stimulation for pain distraction. IEEE Trans Hapt 11(2):185–191

    Article  Google Scholar 

  • Kervegant C, Raymond F, Graeff D, Castet J (2017) Touch hologram in mid-air. In ACM SIGGRAPH. Emerging technologies, pp 1–2

    Google Scholar 

  • Kim JR, Chan S, Huang X, Ng K, Fu LP, Zhao C (2019) Demonstration of refinity: an interactive holographic signage for new retail shopping experience. In: Extended abstracts of the. CHI conference on human factors in computing systems, pp 1–4

    Google Scholar 

  • Kim E, Schneider O (2020) Defining haptic experience: foundations for understanding, communicating, and evaluating HX

    Google Scholar 

  • Kim E, Schneider O (2020) Defining haptic experience: foundations for understanding, communicating, and evaluating hx. In: Proceedings of the 2020 CHI conference on human factors in computing systems, pp 1–13

    Google Scholar 

  • Klauer SG, Dingus TA, Neale VL, Sudweeks JD, Ramsey DJ et al (2006) The impact of driver inattention on near-crash/crash risk: an analysis using the 100-car naturalistic driving study data

    Google Scholar 

  • Korres G, Eid M (2016) Haptogram: ultrasonic point-cloud tactile stimulation. IEEE Access 4:7758–7769

    Article  Google Scholar 

  • Korres G, Chehabeddine S, Eid M (2020) Mid-air tactile feedback co-located with virtual touchscreen improves dual-task performance. IEEE Trans Hapt 13(4):825–830

    Article  Google Scholar 

  • Kulkarni C, Dow SP, Klemmer SR (2014) Early and repeated exposure to examples improves creative work. Design Thinking Research. Springer International Publishing, Cham, pp 49–62

    Google Scholar 

  • Large DR, Harrington K, Burnett G, Georgiou O (2019) Feel the noise: mid-air ultrasound haptics as a novel human-vehicle interaction paradigm. Appl Ergonom 81:102909

    Article  Google Scholar 

  • Lederman SJ, Jones LA (2011) Tactile and haptic illusions. IEEE Trans Hapt 4(4):273–294

    Article  Google Scholar 

  • Lee B, Srivastava S, Kumar R, Brafman R, Klemmer SR (2010) Designing with interactive example galleries. In: Proceedings of the 28th international conference on Human factors in computing systems - CHI ’10. ACM Press, New York, New York, USA, p 2257

    Google Scholar 

  • Lehser C, Wagner E, Strauss DJ (2018) Somatosensory evoked responses elicited by haptic sensations in midair. IEEE Trans Neural Syst Rehabil Eng 26(10):2070–2077

    Article  Google Scholar 

  • Lehser C, Strauss DJ (2019) Attentional correlates in somatosensory potentials evoked by ultrasound induced virtual objects in mid-air. In: 2019 9th international IEEE/EMBS conference on neural engineering (NER). IEEE, pp 933–936

    Google Scholar 

  • Li W-J, Hsieh C-Y, Lin L-F, Chu W-C (2017) Hand gesture recognition for post-stroke rehabilitation using leap motion. In: International conference on applied system innovation (ICASI). IEEE, pp 386–388

    Google Scholar 

  • Lim Y-K, Stolterman E, Tenenberg J (2008) The anatomy of prototypes. ACM Trans Comput-Human Interact 15(2):1–27

    Article  Google Scholar 

  • Limerick H (2020) Call to interact: communicating interactivity and affordances for contactless gesture controlled public displays. In: Proceedings of the 9TH ACM international symposium on pervasive displays, pp 63–70

    Google Scholar 

  • Limerick H, Coyle D, Moore JW (2014) The experience of agency in human-computer interactions: a review. Front Human Neurosci 8:643

    Article  Google Scholar 

  • Limerick H, Hayden R, Beattie D, Georgiou O, Müller J (2019) User engagement for mid-air haptic interactions with digital signage. In: Proceedings of the 8th ACM international symposium on pervasive displays, pp 1–7

    Google Scholar 

  • Loehmann S, Knobel M, Lamara M, Butz A (2013) Culturally independent gestures for in-car interactions. In: IFIP conference on human-computer interaction. Springer, pp 538–545

    Google Scholar 

  • Long B, Seah SA, Carter T, Subramanian S (2014) Rendering volumetric haptic shapes in mid-air using ultrasound. ACM Trans Graph (TOG) 33(6):1–10

    Article  Google Scholar 

  • MacLean KE, Schneider OS, Seifi H (2017) Multisensory haptic interactions: understanding the sense and designing for it. In: The handbook of multimodal-multisensor interfaces: foundations. User modeling, and common modality combinations, vol 1, pp 97–142

    Google Scholar 

  • Maggioni E, Agostinelli E, Obrist M (2017) Measuring the added value of haptic feedback. In: 2017 ninth international conference on quality of multimedia experience (QoMEX). IEEE, pp 1–6

    Google Scholar 

  • Makino Y, Furuyama Y, Inoue S, Shinoda H (2016) Haptoclone (haptic-optical clone) for mutual tele-environment by real-time 3d image transfer with midair force feedback. CHI 1980–1990

    Google Scholar 

  • Marchal M, Gallagher G, Lécuyer A, Pacchierotti C (2020) Can stiffness sensations be rendered in virtual reality using mid-air ultrasound haptic technologies? In: International conference on human haptic sensing and touch enabled computer applications. Springer, pp 297–306

    Google Scholar 

  • Martinez J, Griffiths D, Biscione V, Georgiou O, Carter T (2018) Touchless haptic feedback for supernatural vr experiences. In: 2018 IEEE conference on virtual reality and 3D user interfaces (VR). IEEE, pp 629–630

    Google Scholar 

  • Martinez J, Harwood A, Limerick H, Clark R, Georgiou O (2019) Mid-air haptic algorithms for rendering 3d shapes. In: IEEE international symposium on haptic, audio and visual environments and games (HAVE). IEEE, pp 1–6

    Google Scholar 

  • Marti P, Parlangeli O, Recupero A, Sirizzotti M, Guidi S (2021) Touching virtual objects in mid-air: a study on shape recognition. In: European conference on cognitive ergonomics, pp 1–6

    Google Scholar 

  • Matsubayashi A, Makino Y, Shinoda H (2019) Direct finger manipulation of 3d object image with ultrasound haptic feedback. In: Proceedings of the 2019 CHI conference on human factors in computing systems, pp 1–11

    Google Scholar 

  • Matsubayashi A, Makino Y, Shinoda H (2020) Rendering ultrasound pressure distribution on hand surface in real-time. In: International conference on human haptic sensing and touch enabled computer applications. Springer, pp 407–415

    Google Scholar 

  • Matsubayashi A, Oikawa H, Mizutani S, Makino Y, Shinoda H (2019) Display of haptic shape using ultrasound pressure distribution forming cross-sectional shape. In: IEEE world haptics conference (WHC). IEEE, pp 419–424

    Google Scholar 

  • May KR, Gable TM, Walker BN (2014) A multimodal air gesture interface for in vehicle menu navigation. In: Adjunct proceedings of the 6th international conference on automotive user interfaces and interactive vehicular applications, pp 1–6

    Google Scholar 

  • Mc Gee MR, Gray P, Brewster S (2000) The effective combination of haptic and auditory textural information. In: International workshop on haptic human-computer interaction. Springer, pp 118–126

    Google Scholar 

  • McGlone F, Wessberg J, Olausson H (2014) Discriminative and affective touch: sensing and feeling. Neuron 82(4):737–755

    Article  Google Scholar 

  • Mizutani S, Fujiwara M, Makino Y, Shinoda H (2019) Thresholds of haptic and auditory perception in midair facial stimulation. In: IEEE international symposium on haptic, audio and visual environments and games (HAVE). IEEE, pp 1–6

    Google Scholar 

  • Monnai Y, Hasegawa K, Fujiwara M, Yoshino K, Inoue S, Shinoda H (2014) Haptomime: mid-air haptic interaction with a floating virtual screen. In: Proceedings of the 27th annual ACM symposium on user interface software and technology, pp 663–667

    Google Scholar 

  • Moore W, Makdani A, Frier W, McGlone F (2021) Virtual touch: sensing and feeling with ultrasound. bioRxiv

    Google Scholar 

  • Morales R, Ezcurdia I, Irisarri J, Andrade MA, Marzo A (2021) Generating airborne ultrasonic amplitude patterns using an open hardware phased array. Appl Sci 11(7):2981

    Article  Google Scholar 

  • Moussette C (2012) Simple haptics: sketching perspectives for the design of haptic interactions. Ph.D. dissertation, UmeåUniversitet

    Google Scholar 

  • Mulot L, Gicquel G, Zanini Q, Frier W, Marchal M, Pacchierotti C, Howard T (2021) Dolphin: a framework for the design and perceptual evaluation of ultrasound mid-air haptic stimuli. ACM Sympos Appl Percept 2021:1–10

    Google Scholar 

  • Nakajima M, Makino Y, Shinoda H (2019) Displaying pain sensation in midair by thermal grill illusion. In: IEEE international symposium on haptic, audio and visual environments and games (HAVE). IEEE, pp 1–5

    Google Scholar 

  • Nielsen J (2005) Ten usability heuristics

    Google Scholar 

  • Nielsen M, Störring M, Moeslund TB, Granum E (2003) A procedure for developing intuitive and ergonomic gesture interfaces for HCI. In: International gesture workshop. Springer, pp 409–420

    Google Scholar 

  • Obrist M, Seah SA, Subramanian S (2013) Talking about tactile experiences. In: Proceedings of the SIGCHI conference on human factors in computing systems, pp 1659–1668

    Google Scholar 

  • Obrist M, Subramanian S, Gatti E, Long B, Carter T (2015) Emotions mediated through mid-air haptics. In: Proceedings of the 33rd annual ACM conference on human factors in computing systems, pp 2053–2062

    Google Scholar 

  • O’Conaill B, Provan J, Schubel J, Hajas D, Obrist M, Corenthy L (2020) Improving immersive experiences for visitors with sensory impairments to the aquarium of the pacific. In: Extended abstracts of the. CHI conference on human factors in computing systems, pp 1–8

    Google Scholar 

  • O’Hara K et al (2014) Touchless interaction in surgery. Commun ACM 57(1):70–77

    Article  Google Scholar 

  • Olson RL, Hanowski RJ, Hickman JS, Bocanegra J et al (2009) Driver distraction in commercial vehicle operations. United States. Department of Transportation. Federal Motor Carrier Safety\(\ldots \), Technical Report

    Google Scholar 

  • Ozkul C, Geerts D, Rutten I (2020) Combining auditory and mid-air haptic feedback for a light switch button. In: Proceedings of the 2020 international conference on multimodal interaction, pp 60–69

    Google Scholar 

  • Paneva V, Seinfeld S, Kraiczi M, Müller J (2020) Haptiread: reading braille as mid-air haptic information. In: Proceedings of the 2020 ACM designing interactive systems conference, pp 13–20

    Google Scholar 

  • Pauchot J, Di Tommaso L, Lounis A, Benassarou M, Mathieu P, Bernot D, Aubry S (2015) Leap motion gesture control with carestream software in the operating room to control imaging: installation guide and discussion. Surg Innovat 22(6):615–620

    Article  Google Scholar 

  • Pauna H, Léger P-M, Sénécal S, Fredette M, Courtemanche F, Chen S-L, É. Labonté-Lemoyne, Ménard J-F (2018) The psychophysiological effect of a vibro-kinetic movie experience: the case of the d-box movie seat. In: Information systems and neuroscience. Springer, pp 1–7

    Google Scholar 

  • Perquin MN, Taylor M, Lorusso J, Kolasinski J (2021) Directional biases in whole hand motion perception revealed by mid-air tactile stimulation. Cortex

    Google Scholar 

  • Pickering CA, Burnham KJ, Richardson MJ (2007) A research study of hand gesture recognition technologies and applications for human vehicle interaction. In: 3rd Institution of engineering and technology conference on automotive electronics. IET, pp 1–15

    Google Scholar 

  • Pickering C (2005) The search for a safer driver interface: a review of gesture recognition human machine interface. Comput Cont Eng J 16(1):34–40

    Article  Google Scholar 

  • Pinto AR, Kildal J, Lazkano E (2020) Multimodal mixed reality impact on a hand guiding task with a holographic cobot. Multimodal Tech Interact 4(4):78

    Article  Google Scholar 

  • Pitts MJ, Burnett GE, Williams MA, Wellings T (2010) Does haptic feedback change the way we view touchscreens in cars?. In: International conference on multimodal interfaces and the workshop on machine learning for multimodal interaction, pp 1–4

    Google Scholar 

  • Rakkolainen I, Sand A, Palovuori K (2015) Midair user interfaces employing particle screens. IEEE Comput Graph Appl 35(2):96–102

    Article  Google Scholar 

  • Rakkolainen I, Freeman E, Sand A, Raisamo R, Brewster S (2020) A survey of mid-air ultrasound haptics and its applications. IEEE Trans Hapt 14(1):2–19

    Article  Google Scholar 

  • Riener A, Ferscha A, Bachmair F, Hagmüller P, Lemme A, Muttenthaler D, Pühringer D, Rogner H, Tappe A, Weger F (2013) Standardization of the in-car gesture interaction space. In: Proceedings of the 5th international conference on automotive user interfaces and interactive vehicular applications, pp 14–21

    Google Scholar 

  • Rocchesso D, Cannizzaro FS, Capizzi G, Landolina F (2019) Accessing and selecting menu items by in-air touch. In: Proceedings of the 13th biannual conference of the Italian SIGCHI chapter: designing the next interaction, pp 1–9

    Google Scholar 

  • Romanus T, Frish S, Maksymenko M, Frier W, Corenthy L, Georgiou O (2019) Mid-air haptic bio-holograms in mixed reality. In: 2019 IEEE international symposium on mixed and augmented reality adjunct (ISMAR-Adjunct). IEEE, pp 348–352

    Google Scholar 

  • Rümelin S, Butz A (2013) How to make large touch screens usable while driving. In: Proceedings of the 5th international conference on automotive user interfaces and interactive vehicular applications, pp 48–55

    Google Scholar 

  • Rümelin S, Gabler T, Bellenbaum J (2017) Clicks are in the air: how to support the interaction with floating objects through ultrasonic feedback. In: Proceedings of the 9th international conference on automotive user interfaces and interactive vehicular applications, pp 103–108

    Google Scholar 

  • Rutten I, Frier W, Van den Bogaert L, Geerts D (2019) Invisible touch: how identifiable are mid-air haptic shapes? In: Extended abstracts of the. CHI conference on human factors in computing systems, pp 1–6

    Google Scholar 

  • Rutten E, Van Den Bogaert L, Geerts D (2020) From initial encounter with mid-air haptic feedback to repeated use: the role of the novelty effect in user experience. IEEE Trans Hapt

    Google Scholar 

  • Sand A, Rakkolainen I, Isokoski P, Kangas J, Raisamo R, Palovuori K (2015) Head-mounted display with mid-air tactile feedback. In: Proceedings of the 21st ACM symposium on virtual reality software and technology, pp 51–58

    Google Scholar 

  • Sand A, Rakkolainen I, Surakka V, Raisamo R, Brewster S (2020) Evaluating ultrasonic tactile feedback stimuli. In: International conference on human haptic sensing and touch enabled computer applications. Springer, pp 253–261

    Google Scholar 

  • Sathiyamurthy S, Liu M, Kim E, Schneider O (2021) Measuring haptic experience: elaborating the HX model with scale development. World Haptics ’21, p 6

    Google Scholar 

  • Schneider OS, MacLean KE (2016) Studying design process and example use with macaron, a web-based vibrotactile effect editor. In: 2016 IEEE haptics symposium (HAPTICS). IEEE, pp 52–58

    Google Scholar 

  • Schneider O, MacLean K, Swindells C, Booth K (2017) Haptic experience design: what hapticians do and where they need help. Int J Human Comput Stud 107:5–21

    Article  Google Scholar 

  • Schneider D, Otte A, Kublin AS, Martschenko A, Kristensson PO, Ofek E, Pahud M, Grubert J (2020) Accuracy of commodity finger tracking systems for virtual reality head-mounted displays. In: 2020 IEEE conference on virtual reality and 3D user interfaces abstracts and workshops (VRW). IEEE, pp 804–805

    Google Scholar 

  • Seifi H, Chun M, Gallacher C, Schneider O, MacLean KE (2020) How do novice hapticians design? a case study in creating haptic learning environments. IEEE transactions on haptics 13(4):791–805

    Article  Google Scholar 

  • Shakeri G, Williamson JH, Brewster S (2017) Novel multimodal feedback techniques for in-car mid-air gesture interaction. In: Proceedings of the 9th international conference on automotive user interfaces and interactive vehicular applications, pp 84–93

    Google Scholar 

  • Shakeri G, Williamson JH, Brewster S (2018) May the force be with you: Ultrasound haptic feedback for mid-air gesture interaction in cars. In: Proceedings of the 10th international conference on automotive user interfaces and interactive vehicular applications, pp 1–10

    Google Scholar 

  • Sharp H, Preece J, Rogers Y (2002) Interaction design: beyond human-computer interaction

    Google Scholar 

  • Shneiderman B, Plaisant C, Cohen MS, Jacobs S, Elmqvist N, Diakopoulos N (2016) Designing the user interface: strategies for effective human-computer interaction. Pearson

    Google Scholar 

  • Singhal T, Phutane M (2021) Elevating haptics: an accessible and contactless elevator concept with tactile mid-air controls. In: Extended abstracts of the. CHI conference on human factors in computing systems, pp 1–4

    Google Scholar 

  • Singhal T, Schneider O (2021) Juicy haptic design: vibrotactile embellishments can improve player experience in games. In: CHI ’21, p 10

    Google Scholar 

  • Trotta R, Hajas D, Camargo-Molina JE, Cobden R, Maggioni E, Obrist M (2020) Communicating cosmology with multisensory metaphorical experiences. J Sci Commun 19(2)

    Google Scholar 

  • Van den Bogaert L, Geerts D, Rutten I (2019) Grasping the future: Identifying potential applications for mid-air haptics in the home. In: Extended abstracts of the. CHI Conference on human factors in computing systems, pp 1–6

    Google Scholar 

  • Vaquero-Melchor D, Bernardos AM (2019) Enhancing interaction with augmented reality through mid-air haptic feedback: architecture design and user feedback. Appl Sci 9(23):5123

    Article  Google Scholar 

  • Vi CT, Ablart D, Gatti E, Velasco C, Obrist M (2017) Not just seeing, but also feeling art: mid-air haptic experiences integrated in a multisensory art exhibition. Int J Human-Comput Stud 108:1–14

    Article  Google Scholar 

  • Villarreal-Narvaez S, Vanderdonckt J, Vatavu R-D, Wobbrock JO (2020) A systematic review of gesture elicitation studies: what can we learn from 216 studies? In: Proceedings of the 2020 ACM designing interactive systems conference, pp 855–872

    Google Scholar 

  • Vo D-B, Brewster SA (2015) Touching the invisible: localizing ultrasonic haptic cues. In: IEEE world haptics conference (WHC). IEEE, pp 368–373

    Google Scholar 

  • Young G, Milne H, Griffiths D, Padfield E, Blenkinsopp R, Georgiou O (2020) Designing mid-air haptic gesture controlled user interfaces for cars. In: Proceedings of the ACM on human-computer interaction, vol 4, no EICS, pp 1–23

    Google Scholar 

Download references

Acknowledgements

This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreements No 801413 (H-Reality), and 101017746 (Touchless).

Author information

Authors and Affiliations

  1. Ultraleap Ltd., Bristol, UK

    Orestis Georgiou & William Frier

  2. Faculty of Engineering, University of Waterloo, Waterloo, Canada

    Oliver Schneider

Authors
  1. Orestis Georgiou
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. William Frier
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Oliver Schneider
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Orestis Georgiou .

Editor information

Editors and Affiliations

  1. Ultraleap, Bristol, UK

    Orestis Georgiou

  2. Ultraleap, Bristol, UK

    William Frier

  3. School of Computing Science, University of Glasgow, Glasgow, UK

    Euan Freeman

  4. Institut de Recherche en Informatique et Systèmes Aléatoires (IRISA), Centre national de la recherche scientifique (CNRS), Rennes, France

    Claudio Pacchierotti

  5. Pixie Dust Technologies, Inc., Tokyo, Japan

    Takayuki Hoshi

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Georgiou, O., Frier, W., Schneider, O. (2022). User Experience and Mid-Air Haptics: Applications, Methods, and Challenges. In: Georgiou, O., Frier, W., Freeman, E., Pacchierotti, C., Hoshi, T. (eds) Ultrasound Mid-Air Haptics for Touchless Interfaces. Human–Computer Interaction Series. Springer, Cham. https://doi.org/10.1007/978-3-031-04043-6_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-04043-6_2

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-04042-9

  • Online ISBN: 978-3-031-04043-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation