Abstract
Mid-air technology is not well studied in the context of multisensory experience. Despite increasing advances in mid-air interaction and mid-air haptics, we still lack a good understanding of how such technologies might influence human behaviour and experience. Compare this with the understanding, we currently have about physical touch, which highlights the need for more knowledge in this area. In this chapter, I describe three areas of development that consider human multisensory perception and relate these to the study and use of mid-air haptics. I focus on three main challenges of developing multisensory mid-air interactions. First, I describe how crossmodal correspondence could improve the experience of mid-air touch. Then, I outline some opportunities to introduce mid-air touch to the study of multisensory integration. Finally, I discuss how this multisensory approach can benefit applications that encourage and support a sense of agency in interaction with autonomous systems. Considering these three contributions, when developing mid-air technologies can provide a new multisensory perspective, resulting in the design of more meaningful and emotionally-loaded mid-air interactions.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Aarts H, Bijleveld E, Custers R, Dogge M, Deelder M, Schutter D, van Haren NE (2012) Positive priming and intentional binding: eye-blink rate predicts reward information effects on the sense of agency. Soc Neurosci 7(1):105–112. https://doi.org/10.1080/17470919.2011.590602
Banakou D, Slater M (2014) Body ownership causes illusory self-attribution of speaking and influences subsequent real speaking. Proc Natl Acad Sci 111(49):17678–17683. https://doi.org/10.1073/pnas.1414936111
Beck B, Di Costa S, Haggard P (2017) Having control over the external world increases the implicit sense of agency. Cognition 162:54–60. https://doi.org/10.1016/j.cognition.2017.02.002
Beiker SA (2012) Legal aspects of autonomous driving. Santa Clara l Rev 52:1145
Berberian B (2019) Man-machine teaming: a problem of agency. IFAC-PapersOnLine 51(34):118–123. https://doi.org/10.1016/j.ifacol.2019.01.049
Berberian B, Sarrazin J-C, Le Blaye P, Haggard P (2012) Automation technology and sense of control: a window on human agency. PLoS ONE 7(3):e34075. https://doi.org/10.1371/journal.pone.0034075
Borhani K, Beck B, Haggard P (2017) Choosing, doing, and controlling: implicit sense of agency over somatosensory events. Psychol Sci 28(7):882–893. https://doi.org/10.1177/0956797617697693
Brianza G, Cornelio P, Maggioni E, Obrist M (2021) Sniff before you act: exploration of scent-feature associations for designing future interactions. In: IFIP conference on human-computer interaction. Springer.https://doi.org/10.1007/978-3-030-85616-8_17
Carvalho FM, Spence C (2018) The shape of the cup influences aroma, taste, and hedonic judgements of specialty coffee. Food Qual Prefer 68:315–321. https://doi.org/10.1016/j.foodqual.2018.04.003
Castiello U, Zucco GM, Parma V, Ansuini C, Tirindelli R (2006) Cross-modal interactions between olfaction and vision when grasping. Chem Senses 31(7):665–671. https://doi.org/10.1093/chemse/bjl007
Cline ME, Herman J, Shaw ER, Morton RD (1992) Standardization of the visual analogue scale. Nurs Res. https://doi.org/10.1097/00006199-199211000-00013
Cornelio P, De Pirro S, Vi CT, Subramanian S (2017) Agency in mid-air interfaces. In: Proceedings of the 2017 CHI conference on human factors in computing systems. https://doi.org/10.1145/3025453.3025457
Cornelio P, Maggioni E, Brianza G, Subramanian S, Obrist M (2020) SmellControl: the study of sense of agency in smell. In: Proceedings of the 2020 international conference on multimodal interaction. https://doi.org/10.1145/3382507.3418810
Cornelio P, Velasco C, Obrist M (2021) Multisensory integration as per technological advances: a review. Front Neurosci 15:614. https://doi.org/10.3389/fnins.2021.652611
Crichton N (2001) Visual analogue scale (VAS). J Clin Nurs 10(5):706–706
Cwiek A, Fuchs S, Draxler C, Asu EL, Dediu D, Hiovain K, Kawahara S, Koutalidis S, Krifka M, Lippus P (2021) The bouba/kiki effect is robust across cultures and writing systems. Philos Trans Royal Soc B Biol Sci. https://doi.org/10.1098/rstb.2020.0390
De Freitas J, Censi A, Smith BW, Di Lillo L, Anthony SE, Frazzoli E (2021) From driverless dilemmas to more practical commonsense tests for automated vehicles. Proc Natl Acad Sci 118(11). https://doi.org/10.1073/pnas.2010202118
Debarba HG, Khoury J-N, Perrin S, Herbelin B, Boulic R (2018) Perception of redirected pointing precision in immersive virtual reality. https://doi.org/10.1109/VR.2018.8448285
Dematte ML, Sanabria D, Sugarman R, Spence C (2006) Cross-modal interactions between olfaction and touch. Chem Senses 31(4):291–300. https://doi.org/10.1093/chemse/bjj031
Ernst MO, Banks MS (2002) Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415(6870):429–433. https://doi.org/10.1038/415429a
Ernst MO, Bülthoff HH (2004) Merging the senses into a robust percept. Trends Cogn Sci 8(4):162–169. https://doi.org/10.1016/j.tics.2004.02.002
Etzi R, Spence C, Zampini M, Gallace A (2016) When sandpaper is ‘Kiki’ and satin is ‘Bouba’: an exploration of the associations between words, emotional states, and the tactile attributes of everyday materials. Multisensory Res 29(1–3):133–155. https://doi.org/10.1163/22134808-00002497
Evangelou G, Limerick H, Moore J (2021) I feel it in my fingers! Sense of agency with mid-air haptics. In: 2021 IEEE world haptics conference (WHC). IEEE.https://doi.org/10.1109/WHC49131.2021.9517170
Freeman E (2021) Enhancing ultrasound haptics with parametric audio effects. In: Proceedings of the 2021 international conference on multimodal interaction. https://doi.org/10.1145/3462244.3479951
Freeman E, Williamson J, Subramanian S, Brewster S (2018) Point-and-shake: selecting from levitating object displays. In: Proceedings of the 2018 CHI conference on human factors in computing systems. https://doi.org/10.1145/3173574.3173592
Frier W, Pittera D, Ablart D, Obrist M, Subramanian S (2019) Sampling strategy for ultrasonic mid-air haptics. In: Proceedings of the 2019 CHI conference on human factors in computing systems. https://doi.org/10.1145/3290605.3300351
Frier W, Seo K, Subramanian S (2016) Hilbert curves: a tool for resolution independent haptic texture. In: Proceedings of the 29th annual symposium on user interface software and technology. https://doi.org/10.1145/2984751.2984774
Frish S, Maksymenko M, Frier W, Corenthy L, Georgiou O (2019) Mid-air haptic bio-holograms in mixed reality. In: 2019 IEEE international symposium on mixed and augmented reality adjunct (ISMAR-Adjunct). IEEE.https://doi.org/10.1109/ISMAR-Adjunct.2019.00-14
Georgieff N, Jeannerod M (1998) Beyond consciousness of external reality: a “who” system for consciousness of action and self-consciousness. Conscious Cogn 7:465–477. https://doi.org/10.1006/ccog.1998.0367
Girdler A, Georgiou O (2020) Mid-air haptics in aviation—creating the sensation of touch where there is nothing but thin air. arXiv preprint arXiv:2001.01445
Haggard P (2017) Sense of agency in the human brain. Nat Rev Neurosci 18(4):196. https://doi.org/10.1038/nrn.2017.14
Haggard P, Clark S, Kalogeras J (2002) Voluntary action and conscious awareness. Nat Neurosci 5(4):382–385. https://doi.org/10.1038/nn827
Hajas D, Ablart D, Schneider O, Obrist M (2020a) I can feel it moving: science communicators talking about the potential of mid-air haptics. Frontiers Comput Sci. https://doi.org/10.3389/fcomp.2020.534974
Hajas D, Pittera D, Nasce A, Georgiou O, Obrist M (2020b) Mid-air haptic rendering of 2D geometric shapes with a dynamic tactile pointer. IEEE Trans Haptics. https://doi.org/10.1109/TOH.2020.2966445
Hanson-Vaux G, Crisinel A-S, Spence C (2013) Smelling shapes: crossmodal correspondences between odors and shapes. Chem Senses 38(2):161–166. https://doi.org/10.1093/chemse/bjs087
Harel D, Marron A, Sifakis J (2020) Autonomics: in search of a foundation for next-generation autonomous systems. Proc Natl Acad Sci 117(30):17491–17498. https://doi.org/10.1073/pnas.2003162117
Harrington K, Large DR, Burnett G, Georgiou O (2018) Exploring the use of mid-air ultrasonic feedback to enhance automotive user interfaces. In: Proceedings of the 10th international conference on automotive user interfaces and interactive vehicular applications. https://doi.org/10.1145/3239060.3239089
Hessam JF, Zancanaro M, Kavakli M, Billinghurst M (2017) Towards optimization of mid-air gestures for in-vehicle interactions. In: Proceedings of the 29th Australian conference on computer-human interaction. https://doi.org/10.1145/3152771.3152785
Ho H-N, Van Doorn GH, Kawabe T, Watanabe J, Spence C (2014) Colour-temperature correspondences: when reactions to thermal stimuli are influenced by colour. PloS One 9(3). https://doi.org/10.1371/journal.pone.0091854
Hommel B (2017) Goal-directed actions. In: The Oxford handbook of causal reasoning, pp 265–288
Iosifyan M, Korolkova O (2019) Emotions associated with different textures during touch. Conscious Cogn 71:79–85. https://doi.org/10.1016/j.concog.2019.03.012
Kasahara S, Nishida J, Lopes P (2019) Preemptive action: accelerating human reaction using electrical muscle stimulation without compromising agency. In: Proceedings of the 2019 CHI conference on human factors in computing systems. https://doi.org/10.1145/3290605.3300873
Kendall M, Stuart A (1979) Estimation: maximum likelihood. In: The advanced theory of statistics, vol 2, pp 38―81
Kervegant C, Raymond F, Graeff D, Castet J (2017) Touch hologram in mid-air. In: ACM SIGGRAPH 2017 emerging technologies, pp 1–2. https://doi.org/10.1145/3084822.3084824
Kokkinara E, Kilteni K, Blom KJ, Slater M (2016) First person perspective of seated participants over a walking virtual body leads to illusory agency over the walking. Sci Rep 6(1):1–11. https://doi.org/10.1038/srep28879
Koutsabasis P, Vogiatzidakis P (2019) Empirical research in mid-air interaction: a systematic review. Int J Human-Comput Interact 35(18):1747–1768. https://doi.org/10.1080/10447318.2019.1572352
Kühn S, Brass M, Haggard P (2013) Feeling in control: neural correlates of experience of agency. Cortex 49(7):1935–1942. https://doi.org/10.1016/j.cortex.2012.09.002
Large DR, Harrington K, Burnett G, Georgiou O (2019) Feel the noise: mid-air ultrasound haptics as a novel human-vehicle interaction paradigm. Appl Ergon 81:102909. https://doi.org/10.1016/j.apergo.2019.102909
Limerick H, Moore JW, Coyle D (2015) Empirical evidence for a diminished sense of agency in speech interfaces. In: Proceedings of the 33rd annual ACM conference on human factors in computing systems. ACM. https://doi.org/10.1145/2702123.2702379
Long B, Seah SA, Carter T, Subramanian S (2014) Rendering volumetric haptic shapes in mid-air using ultrasound. ACM Trans Graph (TOG) 33(6):1–10. https://doi.org/10.1145/2661229.2661257
Maggioni E, Cobden R, Dmitrenko D, Obrist M (2018) Smell-O-Message: integration of olfactory notifications into a messaging application to improve users’ performance. In: Proceedings of the 20th ACM international conference on multimodal interaction. https://doi.org/10.1145/3242969.3242975
Martinez Plasencia D, Hirayama R, Montano-Murillo R, Subramanian S (2020) GS-PAT: high-speed multi-point sound-fields for phased arrays of transducers. ACM Trans Graph 39(4). Article 138. https://doi.org/10.1145/3386569.3392492
Marzo A, Seah SA, Drinkwater BW, Sahoo DR, Long B, Subramanian S (2015) Holographic acoustic elements for manipulation of levitated objects. Nat Commun 6:8661. https://doi.org/10.1038/ncomms9661
Matsubayashi A, Makino Y, Shinoda H (2019) Direct finger manipulation of 3D object image with ultrasound haptic feedback. In: Proceedings of the 2019 CHI conference on human factors in computing systems. https://doi.org/10.1145/3290605.3300317
Miller MK, Mandryk RL (2016) Differentiating in-game frustration from at-game frustration using touch pressure. In: Proceedings of the 2016 ACM on interactive surfaces and spaces. ACM. https://doi.org/10.1145/2992154.2992185
Moore JW (2016) What is the sense of agency and why does it matter? Front Psychol 7:1272. https://doi.org/10.3389/fpsyg.2016.01272
Moore JW, Wegner DM, Haggard P (2009) Modulating the sense of agency with external cues. Conscious Cogn 18(4):1056–1064. https://doi.org/10.1016/j.concog.2009.05.004
Narumi T, Nishizaka S, Kajinami T, Tanikawa T, Hirose M (2011) Augmented reality flavors: gustatory display based on edible marker and cross-modal interaction. In: Proceedings of the SIGCHI conference on human factors in computing systems.https://doi.org/10.1145/1978942.1978957
Obrist M, Subramanian S, Gatti E, Long B, Carter T (2015) Emotions mediated through mid-air haptics. In: Proceedings of the 33rd annual ACM conference on human factors in computing systems. https://doi.org/10.1145/2702123.2702361
Ozkul C, Geerts D, Rutten I (2020) Combining auditory and mid-air haptic feedback for a light switch button. In: Proceedings of the 2020 international conference on multimodal interaction. https://doi.org/10.1145/3382507.3418823
Paneva V, Seinfeld S, Kraiczi M, Müller J (2020) HaptiRead: reading braille as mid-air haptic information. arXiv preprint arXiv:2005.06292. https://doi.org/10.1145/3357236.3395515
Parise CV (2016) Crossmodal correspondences: standing issues and experimental guidelines. Multisensory Res 29(1–3):7–28. https://doi.org/10.1163/22134808-00002502
Petrini K, Remark A, Smith L, Nardini M (2012) When vision is not an option: development of haptic–auditory integration. Seeing Perceiving 25:205–205. https://doi.org/10.1163/187847612X648341
Pittera D, Gatti E, Obrist M (2019) I'm sensing in the rain: spatial incongruity in visual-tactile mid-air stimulation can elicit ownership in VR users. In: Proceedings of the 2019 CHI conference on human factors in computing systems. https://doi.org/10.1145/3290605.3300362
Rakkolainen I, Freeman E, Sand A, Raisamo R, Brewster S (2020) A survey of mid-air ultrasound haptics and its applications. IEEE Trans Haptics. https://doi.org/10.1109/TOH.2020.3018754
Romanus T, Frish S, Maksymenko M, Frier W, Corenthy L, Georgiou O (2019) Mid-air haptic bio-holograms in mixed reality. In: 2019 IEEE international symposium on mixed and augmented reality adjunct (ISMAR-Adjunct). IEEE.https://doi.org/10.1109/ISMAR-Adjunct.2019.00-14
Ros F (2016) Stewart: a haptic interface designed for a self-driving car. Available from http://felixros.com/stewart2.html
Rutten I, Frier W, Van den Bogaert L, Geerts D (2019) Invisible touch: how identifiable are mid-air haptic shapes? In: Extended abstracts of the 2019 CHI conference on human factors in computing systems. https://doi.org/10.1145/3290607.3313004
Salgado Montejo A, Alvarado JA, Velasco C, Salgado CJ, Hasse K, Spence C (2015) The sweetest thing: the influence of angularity, symmetry, and the number of elements on shape-valence and shape-taste matches. Front Psychol 6:1382. https://doi.org/10.3389/fpsyg.2015.01382
Smith BC (2016) Aristotle was wrong and so are we: there are far more than five senses. K. Quinn. aeon, aeon. https://youtu.be/zWdfpwCghIw
Spence C, Gallace A (2011) Multisensory design: reaching out to touch the consumer. Psychol Mark 28(3):267–308. https://doi.org/10.1002/mar.20392
Spence C, Parise CV (2012) The cognitive neuroscience of crossmodal correspondences. i-Perception 3(7):410–412. https://doi.org/10.1068/i0540ic
Sridhar S, Feit AM, Theobalt C, Oulasvirta A (2015) Investigating the dexterity of multi-finger input for mid-air text entry. In: Proceedings of the 33rd annual ACM conference on human factors in computing systems. https://doi.org/10.1145/2702123.2702136
Stein BE, Meredith MA (1993) The merging of the senses. The MIT Press
Synofzik M, Vosgerau G, Voss M (2013) The experience of agency: an interplay between prediction and postdiction. Frontiers Psychol 4. https://doi.org/10.1422/77217
Tajadura-Jimenez A, Väljamäe A, Kuusk K (2020) Altering one’s body-perception through E-textiles and haptic metaphors. Frontiers Robot AI 7:7. https://doi.org/10.3389/frobt.2020.00007
Tan H, Zhou Y, Tao Q, Rosen J, van Dijken S (2021) Bioinspired multisensory neural network with cross-modal integration and recognition. Nat Commun 12(1):1–9. https://doi.org/10.1038/s41467-021-21404-z
Van den Bogaert L, Geerts D, Rutten I (2019) Grasping the future: identifying potential applications for mid-air haptics in the home. In: Extended abstracts of the 2019 CHI conference on human factors in computing systems. https://doi.org/10.1145/3290607.3312911
Van Doorn G, Woods A, Levitan CA, Wan X, Velasco C, Bernal-Torres C, Spence C (2017) Does the shape of a cup influence coffee taste expectations? A cross-cultural, online study. Food Qual Prefer 56:201–211. https://doi.org/10.1016/j.foodqual.2016.10.013
Velasco C, Jones R, King S, Spence C (2013) The sound of temperature: what information do pouring sounds convey concerning the temperature of a beverage. J Sens Stud 28(5):335–345. https://doi.org/10.1111/joss.12052
Velasco C, Obrist M (2020) Multisensory experiences: where the senses meet technology. Oxford University Press, Oxford. https://doi.org/10.1093/oso/9780198849629.001.0001
Vogiatzidakis P, Koutsabasis P (2018) Gesture elicitation studies for mid-air interaction: a review. Multimodal Technol Interact 2(4):65. https://doi.org/10.3390/mti2040065
Walker P, Scallon G, Francis B (2017) Cross-sensory correspondences: heaviness is dark and low-pitched. Perception 46(7):772–792. https://doi.org/10.1177/0301006616684369
Winkler P, Stiens P, Rauh N, Franke T, Krems J (2020) How latency, action modality and display modality influence the sense of agency: a virtual reality study. Virtual Reality 24(3):411–422. https://doi.org/10.1007/s10055-019-00403-y
Yoshie M, Haggard P (2013) Negative emotional outcomes attenuate sense of agency over voluntary actions. Curr Biol 23(20):2028–2032. https://doi.org/10.1016/j.cub.2013.08.034
Zhu M, He T, Lee C (2020) Technologies toward next generation human machine interfaces: from machine learning enhanced tactile sensing to neuromorphic sensory systems. Appl Phys Rev 7(3):031305. https://doi.org/10.1063/5.0016485
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this chapter
Cite this chapter
Cornelio, P. (2022). Opportunities for Multisensory Mid-Air Interactions Featuring Ultrasound Haptic Feedback. In: Georgiou, O., Frier, W., Freeman, E., Pacchierotti, C., Hoshi, T. (eds) Ultrasound Mid-Air Haptics for Touchless Interfaces. Human–Computer Interaction Series. Springer, Cham. https://doi.org/10.1007/978-3-031-04043-6_4
Download citation
DOI: https://doi.org/10.1007/978-3-031-04043-6_4
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-04042-9
Online ISBN: 978-3-031-04043-6
eBook Packages: Computer ScienceComputer Science (R0)