Abstract
Unconventional displays, such as 3D displays, projection screens formed of flowing light-scattering particles (fogscreens), and virtual reality (VR) headsets, can create illusions of images floating in mid-air. Paired with hand tracking, gestural interaction with floating user interfaces (UI) is possible on this permeable imagery, thus creating reach-through touchscreens that react and recover instantly from intersecting fingers and objects. The user can explore virtual environments and control floating UIs with hand gestures which could help, for example, in simulated training and in creating an improved feeling of immersion. However, hand-based gestural interaction with such UIs can be difficult without haptic sensations typical in daily activities. Without haptics, the level of immersion and smoothness of interaction suffers if the hands can pass through virtual objects without triggering tactile sensations. Ultrasound haptics is a method to produce a focused airborne acoustic air pressure on a user’s skin, thus creating an unobtrusive, mid-air sensation of touch. Fogscreens, VR headsets, or some other unconventional displays together with ultrasound haptics enable tactile interaction with “touchless touchscreens”. These tactile, floating UIs open new opportunities, e.g., for immersive interaction, advertisement, and entertainment. It can bring back the missing haptic feedback for these displays.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Ariga K, Fujiwara M, Makino Y, Shinoda H (2021). Workspace evaluation of long-distance midair haptic display using curved reflector. In: 2021 IEEE world haptics conference (WHC). IEEE, pp 85–90. https://doi.org/10.1109/WHC49131.2021.9517193
Benzie P, Watson J, Surman P, Rakkolainen I, Hopf K, Urey H, Sainov V, von Kopylow C (2007) A survey of 3DTV displays: techniques and technologies. IEEE Trans Circuits Syst Video Technol 17(11):1647–1658. https://doi.org/10.1109/TCSVT.2007.905377
Biswas S, Visell Y (2019) Emerging material technologies for haptics. Adv Mater Technol 4(4):1900042. https://doi.org/10.1002/admt.201900042
Coe P, Evreinov G, Sinivaara H, Hippula A, Raisamo R (2021) Haptic actuation plate for multi-layered in-vehicle control panel. Multimodal Technol Interact 5:25. https://doi.org/10.3390/mti5050025
Comeau C, Bryan JS (1961) Headsight television system provides remote surveillance. Electronics 86–90
Freeman E, Brewster S, Lantz V (2014) Tactile feedback for above-device gesture interfaces: adding touch to touchless interactions. In: Proceedings of the 16th international conference on multimodal interaction, pp 419–426. https://doi.org/10.1145/2663204.2663280
Freeman E, Anderson R, Williamson J, Wilson G, Brewster SA (2017) Textured surfaces for ultrasound haptic displays. In: ICMI 2017—Proceedings of the 19th ACM international conference on multimodal interaction, vol 2017-Jan. https://doi.org/10.1145/3136755.3143020
Gil H, Son H, Kim JR, Oakley I (2018) Whiskers: exploring the use of ultrasonic haptic cues on the face. In: Proceedings of the 2018 CHI conference on human factors in computing systems, pp 1–13. https://doi.org/10.1145/3173574.3174232
Hoshi T, Abe D, Shinoda H (2009) Adding tactile reaction to hologram. In: RO-MAN 2009—the 18th IEEE international symposium on robot and human interactive communication. IEEE, pp 7–11. https://doi.org/10.1109/ROMAN.2009.5326299
Howard T, Marchal M, Lécuyer A, Pacchierotti C (2020) PUMAH: pan-tilt ultrasound mid-air haptics for larger interaction workspace in virtual reality. IEEE Trans Haptics 13(1):38–44. https://doi.org/10.1109/TOH.2019.2963028
Inoue S, Kobayashi-Kirschvink KJ, Monnai Y, Hasegawa K, Makino Y, Shinoda H (2014) HORN. In: ACM SIGGRAPH 2014 emerging technologies on—SIGGRAPH ’14. ACM Press, New York, New York, USA, pp 1–1. https://doi.org/10.1145/2614066.2614092
Ito M, Wakuda D, Inoue S, Makino Y, Shinoda H (2016) High spatial resolution midair tactile display using 70 kHz ultrasound. In: Lecture notes in computer science (including subseries lecture notes in artificial intelligence and lecture notes in bioinformatics, vol 9774. https://doi.org/10.1007/978-3-319-42321-0_6
Iwamoto T, Tatezono M, Shinoda H (2008) Non-contact method for producing tactile sensation using airborne ultrasound. In: Haptics: perception, devices and scenarios. Springer Berlin Heidelberg, Berlin, Heidelberg, pp. 504–513. https://doi.org/10.1007/978-3-540-69057-3_64
Jepsen M (2005) Smoke, mirrors, and manufacturable displays. Computer 38(8):63–67. https://doi.org/10.1109/MC.2005.271
Jingu A, Kamigaki T, Fujiwara M, Makino Y, Shinoda H (2021) LipNotif: use of lips as a non-contact tactile notification interface based on ultrasonic tactile presentation. In: The 34th annual ACM symposium on user interface software and technology, pp 13–23. https://doi.org/10.1145/3472749.3474732
Kervegant C, Raymond F, Graeff D, Castet J (2017) Touch hologram in mid-air. In: ACM SIGGRAPH 2017 emerging technologies, pp 1–2. https://doi.org/10.1145/3084822.3084824
Kjeldsen R, Hartman J (2001) Design issues for vision-based computer interaction systems. In: ACM international conference proceeding series (Vol. 15-16-November-2001). New York, New York, USA: association for computing machinery, pp 27. https://doi.org/10.1145/971478.971511
Long B, Seah SA, Carter T, Subramanian S (2014) Rendering volumetric haptic shapes in mid-air using ultrasound. ACM Trans Graphics 33(6):1–10. https://doi.org/10.1145/2661229.2661257
Martinez J, Griffiths D, Biscione V, Georgiou O, Carter T (2018) Touchless haptic feedback for supernatural VR experiences. In: 2018 IEEE conference on virtual reality and 3D user interfaces (VR). IEEE, pp 629–630. https://doi.org/10.1109/VR.2018.8446522
Monnai Y, Hasegawa K, Fujiwara M, Yoshino K, Inoue S, Shinoda, H (2014) HaptoMime. In: Proceedings of the 27th annual ACM symposium on User interface software and technology—UIST ’14. ACM Press, New York, New York, USA, pp 663–667. https://doi.org/10.1145/2642918.2647407
Pacchierotti C, Sinclair S, Solazzi M, Frisoli A, Hayward V, Prattichizzo D (2017) Wearable haptic systems for the fingertip and the hand: taxonomy, review, and perspectives. IEEE Trans Haptics 10(4):580–600. https://doi.org/10.1109/TOH.2017.2689006
Palovuori K, Rakkolainen I, Sand A (2014) Bidirectional touch interaction for immaterial displays. In: Proceedings of the 18th international academic MindTrek conference on media business, management, content & services—AcademicMindTrek ’14. ACM Press, New York, New York, USA, pp 74–76. https://doi.org/10.1145/2676467.2676503
Rakkolainen I, Freeman E, Sand A, Raisamo R, Brewster S (2020) A survey of mid-air ultrasound haptics and its applications. IEEE Trans Haptics 14(1):2–19. https://doi.org/10.1109/TOH.2020.3018754
Sand A, Rakkolainen I, Isokoski P, Raisamo R, Palovuori K (2015a) Light-weight immaterial particle displays with mid-air tactile feedback. In: 2015a IEEE international symposium on haptic, audio and visual environments and games, HAVE 2015—Proceedings. https://doi.org/10.1109/HAVE.2015.7359448
Sand A, Rakkolainen I, Isokoski P, Kangas J, Raisamo R, Palovuori K (2015b) Head-mounted display with mid-air tactile feedback. In: Proceedings of the 21st ACM symposium on virtual reality software and technology—VRST ’15, vol 13–15-Nove, pp 51–58. https://doi.org/10.1145/2821592.2821593
Sand A, Rakkolainen I, Surakka V, Raisamo R, Brewster S (2020) Evaluating ultrasonic tactile feedback stimuli. EuroHaptics 253–261. https://doi.org/10.1007/978-3-030-58147-3_28
Schwarz J, Marais C, Leyvand T, Hudson SE, Mankoff J (2014) Combining body pose, gaze, and gesture to determine intention to interact in vision-based interfaces. In: Conference on human factors in computing systems—proceedings. Association for Computing Machinery, New York, New York, USA, pp 3443–3452. https://doi.org/10.1145/2556288.2556989
Sutherland IE (1968) A head-mounted three dimensional display. In: Proceedings of the December 9–11, 1968, fall joint computer conference, part I. Association for Computing Machinery, New York, NY, USA, pp 757–764. https://doi.org/10.1145/1476589.1476686
Suzuki S, Inoue S, Fujiwara M, Makino Y, Shinoda H (2021) AUTD3: scalable airborne ultrasound tactile display. IEEE Trans Haptics. https://doi.org/10.1109/TOH.2021.3069976
van de Camp F, Schick A, Stiefelhagen R (2013) How to click in mid-air. In: Lecture notes in computer science (including subseries lecture notes in artificial intelligence and lecture notes in bioinformatics), vol 8028 LNCS, pp 78–86. https://doi.org/10.1007/978-3-642-39351-8_9
Vi C, Ablart D, Gatti E, Velasco C, Obrist M (2017) Not just seeing, but also feeling art: mid-air haptic experiences integrated in a multisensory art exhibition. J 108. https://doi.org/10.1016/j.ijhcs.2017.06.004
Vrzakova H, Bednarik R (2013) That’s not Norma(n/l)! a detailed analysis of Midas touch in gaze-based problem-solving. In: Conference on human factors in computing systems—proceedings, vol 2013-April. Association for Computing Machinery, New York, New York, USA, pp 85–90. https://doi.org/10.1145/2468356.2468373
Walter R, Bailly G, Müller J (2013) StrikeAPose: revealing mid-air gestures on public displays. In: Conference on human factors in computing systems – proceedings, pp 841–850. https://doi.org/10.1145/2470654.2470774
Walter R, Bailly G, Valkanova N, Müller J (2014) Cuenesics: using mid-air gestures to select items on interactive public displays. In: MobileHCI 2014—Proceedings of the 16th ACM international conference on human-computer interaction with mobile devices and services. Association for Computing Machinery, Inc., New York, New York, USA, pp 299–308. https://doi.org/10.1145/2628363.2628368
Wheatstone C (1838) On some remarkable, and hitherto unobserved, phenomena of binocular vision. Philos Trans R Soc Lond 11:371–394
Yagi A, Imura M, Kuroda Y, Oshiro O (2011) 360-degree fog projection interactive display. In: SIGGRAPH Asia 2011 emerging technologies, pp 1–1. https://doi.org/10.1145/2073370.2073388
Yoo S, Parker C, Kay J, Tomitsch M (2015) To dwell or not to dwell: an evaluation of mid-air gestures for large information displays. In: OzCHI 2015: being human—conference proceedings. Association for Computing Machinery, Inc., New York, New York, USA, pp 187–191. https://doi.org/10.1145/2838739.2838819
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this chapter
Cite this chapter
Sand, A., Rakkolainen, I., Surakka, V., Raisamo, R., Brewster, S. (2022). Touchless Tactile Interaction with Unconventional Permeable Displays. In: Georgiou, O., Frier, W., Freeman, E., Pacchierotti, C., Hoshi, T. (eds) Ultrasound Mid-Air Haptics for Touchless Interfaces. Human–Computer Interaction Series. Springer, Cham. https://doi.org/10.1007/978-3-031-04043-6_8
Download citation
DOI: https://doi.org/10.1007/978-3-031-04043-6_8
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-04042-9
Online ISBN: 978-3-031-04043-6
eBook Packages: Computer ScienceComputer Science (R0)