Abstract
Touchless user interfaces enable people to interact with digital services and information without physically touching an input device. There are numerous benefits to touchless interaction (including convenience, hygiene and the potential for more expressive input), and sensing technologies have advanced significantly in recent years. As a result, touchless user interfaces have been adopted on a wider scale across a variety of application areas, e.g. automotive, digital signage and gaming. However, usability remains a key concern; touchless gesture input poses several interaction challenges, many related to uncertainty and the inherent loss of tactile cues. Ultrasound haptic feedback has shown promise in helping users overcome such interaction challenges, restoring the missing sense of touch and closing the feedback loop for effective haptic interaction. This chapter explores how ultrasound haptic feedback has been used in touchless user interface design and presents design patterns used by industry and academia alike.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Ablart D, Velasco C, Obrist M (2017) Integrating mid-air haptics into movie experiences. In: Proceedings of the 2017 ACM international conference on interactive experiences for TV and online video—TVX ’17, ACM Press, pp 77–84, https://doi.org/10.1145/3077548.3077551, https://dl.acm.org/citation.cfm?id=3077548.3077551
Bellotti V, Back M, Edwards WK, Grinter RE, Henderson A, Lopes C (2002) Making sense of sensing systems: five questions for designers and researchers. In: Proceedings of the SIGCHI conference on human factors in computing systems—CHI ’02, ACM Press, pp 415–422. https://doi.org/10.1145/503376.503450, http://dl.acm.org/citation.cfm?id=503450
Carter T, Seah SA, Long B, Drinkwater B, Subramanian S (2013) UltraHaptics: multi-point mid-air haptic feedback for touch surfaces. In: Proceedings of the 26th symposium on user interface software and technology—UIST ’13. ACM Press, pp 505–514, https://doi.org/10.1145/2501988.2502018, http://dl.acm.org/citation.cfm?id=2502018
Corenthy L, Giordano M, Hayden R, Griffiths D, Jeffrey C, Limerick H, Georgiou O, Carter T, Müller J, Subramanian S (2018) Touchless tactile displays for digital signage. In: Extended abstracts of the 2018 CHI conference on human factors in computing systems—CHI ’18. ACM Press, p Demo 103. https://doi.org/10.1145/3170427.3186533, http://dl.acm.org/citation.cfm?doid=3170427.3186533
Cornelio-Martinez PI, Pirro SD, Vi CT, Subramanian S (2017) Agency in mid-air Interfaces. In: Proceedings of the 2017 CHI conference on human factors in computing systems—CHI ’17, ACM Press, pp 2426–2439. https://doi.org/10.1145/3025453.3025457, http://dl.acm.org/citation.cfm?id=3025457
Cronin S, Doherty G (2019) Touchless computer interfaces in hospitals: a review. Health Informatics J 25(4):1325–1342. https://doi.org/10.1177/1460458217748342, https://journals.sagepub.com/doi/full/10.1177/1460458217748342
Dzidek B, Frier W, Harwood A, Hayden R (2018) Design and evaluation of mid-air haptic interactions in an augmented reality environment. In: Proceedings of EuroHaptics 2018 in LNCS 10894—EuroHaptics ’18, Springer International Publishing, pp 489–499. https://doi.org/10.1007/978-3-319-93399-3_42, https://link.springer.com/chapter/10.1007/978-3-319-93399-3_42
Freeman E (2021) Enhancing ultrasound haptics with parametric audio effects. In: Proceedings of the 2021 international conference on multimodal interaction. Association for Computing Machinery, New York, NY, USA, ICMI ’21, pp 692–696. https://doi.org/10.1145/3462244.3479951, https://doi.org/10.1145/3462244.3479951
Freeman E, Wilson G (2021) Perception of ultrasound haptic focal point motion. In: Proceedings of the 2021 international conference on multimodal interaction. Association for Computing Machinery, New York, NY, USA, ICMI ’21, pp 697–701. https://doi.org/10.1145/3462244.3479950, https://doi.org/10.1145/3462244.3479950
Freeman E, Brewster S, Lantz V (2014) Tactile feedback for above-device gesture interfaces: adding touch to touchless interactions. In: Proceedings of the 16th international conference on multimodal interaction—ICMI ’14. ACM Press, pp 419–426. https://doi.org/10.1145/2663204.2663280, http://dl.acm.org/citation.cfm?id=2663280 http://dl.acm.org/citation.cfm?doid=2663204.2663280
Freeman E, Brewster S, Lantz V (2015) Towards In-Air Gesture Control of Household Appliances with Limited Displays. In: Proceedings of INTERACT ’15 in LNCS 9299, IFIP, pp 611–615, https://doi.org/10.1007/978-3-319-22723-8_73, http://link.springer.com/10.1007/978-3-319-22723-8_73
Freeman E, Brewster S, Lantz V (2016) Do that, there: an interaction technique for addressing in-air gesture systems. In: Proceedings of the 2016 CHI conference on human factors in computing systems—CHI ’16. ACM Press, pp 2319–2331, https://doi.org/10.1145/2858036.2858308, http://dl.acm.org/citation.cfm?doid=2858036.2858308
Freeman E, Vo DB, Brewster S (2019) HaptiGlow: helping users position their hands for better mid-air gestures and ultrasound haptic feedback. In: Proceedings of the IEEE world haptics conference 2019, the 8th joint Eurohaptics conference and the IEEE haptics symposium, IEEE, pp 289–294, https://doi.org/10.1109/WHC.2019.8816092, https://ieeexplore.ieee.org/document/8816092
Frier W, Ablart D, Chilles J, Long B, Giordano M, Obrist M, Subramanian S (2018) Using spatiotemporal modulation to draw tactile patterns in mid-air. In: Proceedings of EuroHaptics 2018. Springer, Berlin
Frier W, Pittera D, Ablart D, Obrist M, Subramanian S (2019) Sampling strategy for ultrasonic mid-air haptics. In: Proceedings of the 2019 CHI conference on human factors in computing systems proceedings—CHI ’19. ACM Press, p Paper 121, https://doi.org/10.1145/3290605.3300351, https://dl.acm.org/citation.cfm?id=3290605.3300351
Georgiou O, Griffiths D (2017) Haptic in-vehicle gesture controls. In: Adjunct proceedings of the 9th international ACM conference on automotive user interfaces and interactive vehicular applications—AutomotiveUI ’17, ACM Press, pp 233–238. https://doi.org/10.1145/3131726.3132045
Georgiou O, Jeffrey C, Chen Z, Tong BX, Chan SH, Yang B, Harwood A, Carter T (2018) Touchless haptic feedback for VR rhythm games. In: Proceedings of 25th IEEE conference on virtual reality and 3D user interfaces—IEEE VR ’18. IEEE, pp 2–3, https://doi.org/10.1109/VR.2018.8446619, https://ieeexplore.ieee.org/document/8446619
Hajas D, Pittera D, Nasce A, Georgiou O, Obrist M (2020) Mid-air haptic rendering of 2D geometric shapes with a dynamic tactile pointer. IEEE Trans Haptics 13(1):1–12. https://doi.org/10.1109/TOH.2020.2966445
Hirayama R, Martinez Plasencia D, Masuda N, Subramanian S (2019) A volumetric display for visual, tactile and audio presentation using acoustic trapping. Nature 575
Hoshi T (2011) Development of aerial-input and aerial-tactile-feedback system. In: Proceedings of the 2011 IEEE world haptics conference, IEEE, pp 569–573. https://doi.org/10.1109/WHC.2011.5945548, http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=5945548
Hoshi T, Takahashi M, Iwamoto T, Shinoda H (2010) Noncontact tactile display based on radiation pressure of airborne ultrasound. IEEE Trans Haptics 3(3):155–165
Hwang I, Son H, Kim JR (2017) AirPiano: enhancing music playing experience in virtual reality with mid-air haptic feedback. In: 2017 IEEE world haptics conference—WHC ’17. IEEE, pp 213–218. https://doi.org/10.1109/WHC.2017.7989903, http://ieeexplore.ieee.org/document/7989903/
Iwamoto T, Tatezono M, Shinoda H (2008) Non-contact method for producing tactile sensation using airborne ultrasound. In: Proceedings of EuroHaptics 2008. Springer, Berlin, pp 504–513. https://doi.org/10.1007/978-3-540-69057-3_64, http://www.springerlink.com/index/X41J595757401387.pdf
Korres G, Eid M (2016) Haptogram: ultrasonic point-cloud tactile stimulation. IEEE Access 4:7758–7769
Limerick H (2020) Call to interact: communicating interactivity and affordances for contactless gesture controlled public displays. In: Proceedings of the 9th ACM international symposium on pervasive displays—PerDis ’20, pp 63–70. https://doi.org/10.1145/3393712.3395338
Limerick H, Hayden R, Beattie D, Georgiou O, Müller J (2019) User engagement for mid-air haptic interactions with digital signage. In: Proceedings of the 8th ACM international symposium on pervasive displays—PerDis ’19. ACM Press, p to appear
Long B, Seah SA, Carter T, Subramanian S (2014) Rendering volumetric haptic shapes in mid-air using ultrasound. ACM Trans Graphics 33(6):Article 181. https://doi.org/10.1145/2661229.2661257, http://dl.acm.org/citation.cfm?id=2661257
Martinez J, Griffiths D, Biscione V, Georgiou O, Carter T (2018) Touchless haptic feedback for supernatural VR experiences. In: Proceedings of the 25th IEEE conference on virtual reality and 3D user interfaces, pp 629–630. https://doi.org/10.1109/VR.2018.8446522
Monnai Y, Hasegawa K, Fujiwara M, Yoshino K, Inoue S, Shinoda H (2014) HaptoMime: mid-air haptic interaction with a floating virtual screen. In: Proceedings of the 27th symposium on user interface software and technology—UIST ’14. ACM Press, pp 663–667. https://doi.org/10.1145/2642918.2647407, http://dl.acm.org/citation.cfm?id=2642918.2647407
O’Hara K, Gonzalez G, Sellen A, Penney G, Varnavas A, Mentis H, Criminisi A, Corish R, Rouncefield M, Dastur N, Carrell T (2014) Touchless interaction in surgery. Commun ACM 57(1):70–77
Rakkolainen I, Freeman E, Sand A, Raisamo R, Brewster S (2020) A survey of mid-air ultrasound haptics and its applications. IEEE Trans Haptics 14(1):2–19
Rutten I, Frier W, Van de Bogaert L, Geerts D (2019) Invisible touch: how identifiable are mid-air haptic shapes? In: Proceedings of extended abstracts on human factors in computing systems on—CHI EA ’19 p LBW0283
Sand A, Rakkolainen I, Isokoski P, Kangas J, Raisamo R, Palovuori K (2015) Head-mounted display with mid-air tactile feedback. In: Proceedings of the 21st ACM symposium on virtual reality software and technology—VRST ’15. ACM Press, pp 51–58. https://doi.org/10.1145/2821592.2821593, http://dl.acm.org/citation.cfm?doid=2821592.2821593
Shakeri G, Williamson JH, Brewster S (2018) May the force be with you: ultrasound haptic feedback for mid-air gesture interaction in cars. In: Proceedings of automotive UI 2018—AutoUI ’18. ACM Press
Sridhar S, Feit AM, Theobalt C, Oulasvirta A (2015) Investigating the dexterity of multi-finger input for mid-air text entry. In: Proceedings of the SIGCHI conference on human factors in computing systems—CHI ’15. ACM Press, pp 3643–3652. https://doi.org/10.1145/2702123.2702136, https://dl.acm.org/citation.cfm?id=2702136
Takahashi R, Hasegawa K, Shinoda H (2018) Lateral modulation of midair ultrasound focus for intensified vibrotactile stimuli. In: Proceedings of EuroHaptics 2018 in LNCS 10894—EuroHaptics ’18. Springer International Publishing, pp 276–288. https://doi.org/10.1007/978-3-319-93399-3_25, http://link.springer.com/10.1007/978-3-319-93445-7
Takahashi R, Hasegawa K, Shinoda H (2019) Tactile stimulation by repetitive lateral movement of midair ultrasound focus. IEEE Trans Haptics
Ultrahaptics (2017) Sensation Editor: The first 10 sensation templates. https://developer.ultrahaptics.com/knowledgebase/the-first-10-sensation-templates/
Ultraleap (2019) UCA tutorial 8: the forcefield sensation. https://developer.ultrahaptics.com/knowledgebase/unity-tutorial-eight/
Vo DB, Brewster S (2015) Touching the invisible: localizing ultrasonic haptic cues. In: Proceedings of world haptics conference 2015—WHC ’15, IEEE, pp 368–373. https://doi.org/10.1109/WHC.2015.7177740, http://ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=7177740
Waugh K, Robertson J (2021) Don’t touch me! a comparison of usability on touch and non-touch inputs. In: Proceedings of the IFIP conference on human-computer interaction—Interact ’21. Springer, Berlin, pp 400–404. https://doi.org/10.1007/978-3-030-85607-6_46
Wilson G, Carter T, Subramanian S, Brewster S (2014) Perception of ultrasonic haptic feedback on the hand: localisation and apparent motion. In: Proceedings of the SIGCHI conference on human factors in computing systems—CHI ’14. ACM Press, pp 1133–1142. https://doi.org/10.1145/2556288.2557033, http://dl.acm.org/citation.cfm?id=2557033
Acknowledgements
I have been fortunate enough to be a part of this community for a relatively long time. I am grateful to those who have helped me along the way, including Tom Carter, Sriram Subramanian, Stephen Brewster, Graham Wilson and many friends at Ultraleap.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this chapter
Cite this chapter
Freeman, E. (2022). Ultrasound Haptic Feedback for Touchless User Interfaces: Design Patterns. In: Georgiou, O., Frier, W., Freeman, E., Pacchierotti, C., Hoshi, T. (eds) Ultrasound Mid-Air Haptics for Touchless Interfaces. Human–Computer Interaction Series. Springer, Cham. https://doi.org/10.1007/978-3-031-04043-6_3
Download citation
DOI: https://doi.org/10.1007/978-3-031-04043-6_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-04042-9
Online ISBN: 978-3-031-04043-6
eBook Packages: Computer ScienceComputer Science (R0)