Skip to main content
SpringerLink
Log in

Superimposing Visual Images on Mid-Air Ultrasonic Haptic Stimulation

  • Chapter
  • First Online:
Ultrasound Mid-Air Haptics for Touchless Interfaces

Part of the book series: Human–Computer Interaction Series ((HCIS))

Abstract

Mid-air haptic stimulation will be accompanied by visual information in many future applications. Vision is a fast and precise means of obtaining spatial information and is how we consume most forms of digital information. The superimposition of visual and haptic feedback at the same position would provide an intuitive interface that requires minimal learning and could lead to a more engaging and immersive user experience. In this chapter, we introduce typical strategies of visual–tactile superimposition when using an ultrasound haptics device, including: an aerial computer interface with ultrasound haptic feedback, visuo-tactile projection on the skin, holographic transmission of physical entities, and applications with head-mounted displays. We also discuss the problem of positional matching between vision and haptics which is crucial for a high-quality user experience.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
eBook
USD 16.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    MicroMirror Array Plate. A glass-plate-like device including strip mirrors perpendicular to the plate surface. The plate was produced by Asukanet Co., Ltd. as the product name AI plate. A similar device principle was also independently presented by Dr. Satoshi Maekawa, NICT.

References

  • https://www.youtube.com/watch?v=1B-7uQ5RD2A “BMW HoloActive TOUCH at CES 2017”

  • https://www.youtube.com/watch?v=Y-P1zZAcPuw “Touchable Holography” video produced by Takayuki Hoshi in Shinoda laboratory

  • https://www.youtube.com/watch?v=Bb0hNMxxewg “Visuo-Tactile Projector” video produced by Keisuke Hasegawa in Shinoda laboratory

  • https://materialized-graphics.hapislab.org/ JST CREST, materialized graphics project since 2018

  • Brice D, McRoberts T, Rafferty K (2019) A proof of concept integrated multi-systems approach for large scale tactile feedback in VR. In: De Paolis L., Bourdot P. (eds) augmented reality, virtual reality, and computer graphics, lecture notes in computer science, vol 11613

    Google Scholar 

  • Frier W, Ablart D, Chilles J, Long B, Giordano M, Obrist M, Subramanian S (2018) Using spatiotemporal modulation to draw tactile patterns in mid-air. In: 2018 proceedings of Euro haptics, Part I, pp 270–281

    Google Scholar 

  • Hasegawa K, Qiu L, Noda A, Inoue S, Shinoda H (2017) Electronically steerable ultrasound-driven long narrow air stream. Appl Phys Lett 111:064104

    Article  Google Scholar 

  • Hasegawa K, Qiu L, Shinoda H (2018) Midair ultrasound fragrance rendering. IEEE Trans vis Comput Graphics 24(4):1477–1485

    Article  Google Scholar 

  • Hasegawa K, Shinoda H (2013) Aerial display of vibrotactile sensation with high spatial-temporal resolution using large-aperture airborne ultrasound phased array. In: 2013 Proceedings of the IEEE world haptics conference, pp 31–36

    Google Scholar 

  • Hirayama R, Martinez Plasencia D, Masuda N, Subramanian S (2019) A volumetric display for visual, tactile and audio presentation using acoustic trapping. Nature 575:320–323

    Article  Google Scholar 

  • Hoshi T, Takahashi M, Iwamoto T, Shinoda H (2010) Noncontact tactile display based on radiation pressure of airborne ultrasound. IEEE Trans Haptics 3(3):155–165

    Article  Google Scholar 

  • Hoshi T, Takahashi M, Nakatsuma K, Shinoda H (2009) Touchable holography. In: 2009 proceedings of ACM SIGGRAPH emerging technologies. ACM, New York, NY, USA, Article no. 23

    Google Scholar 

  • Howard T, Marchal M, Lécuyer A, Pacchierotti C (2020) PUMAH: pan-tilt ultrasound mid-air haptics for larger interaction workspace in virtual reality. IEEE Trans Haptics 13(1):38–44

    Article  Google Scholar 

  • Howard T, Gallagher G, Lécuyer A, Pacchierotti C, Marchal M (2019) Investigating the recognition of local shapes using mid-air ultrasound haptics. In: 2019 Proceeding IEEE world haptics conference, pp 503–508

    Google Scholar 

  • Inoue S, Kobayashi-Kirschvink KJ, Monnai Y, Hasegawa K, Makino Y, Shinoda H (2014) HORN: the hapt-optic reconstruction. In: 2014 ACM SIGGRAPH emerging technologies

    Google Scholar 

  • Inoue S, Makino Y, Shinoda H (2015) Active touch perception produced by airborne ultrasonic haptic hologram. In: 2015 Proceedings of the IEEE world haptics conference (WHC), pp 362–367

    Google Scholar 

  • Ito M, Kokumai Y, Shinoda H (2019) Midair click of dual-layer haptic button. In: 2019 proceedings of the IEEE world haptics conference, pp 349–352, Tokyo, Japan

    Google Scholar 

  • Iwamoto T, Tatezono M, Shinoda H (2008) Non-contact method for producing tactile sensation using airborne ultrasound. Proc Euro Haptics 2008:504–513

    Google Scholar 

  • Kuchenbecker KJ, Fiene J, Niemeyer G (2006) Improving contact realism through event-based haptic feedback. IEEE Trans Visual Comput Graphics 12(2):219–230

    Article  Google Scholar 

  • Loomis JM (1981) Tactile pattern perception. Perception 10:5–27

    Article  Google Scholar 

  • Makino Y, Furuyama Y, Inoue S, Shinoda H (2016) HaptoClone (Haptic-Optical Clone) for mutual tele-environment by real-time 3D image transfer with midair force feedback. In: 2016 Proceedings of the chi conference on human factors in computing systems, pp 1980–1990

    Google Scholar 

  • Marchal M, Gallagher G, Lécuyer A, Pacchierotti C (2020) Can stiffness sensations be rendered in virtual reality using mid-air ultrasound haptic technologies? Proc Euro Haptics 2020:297–306

    Google Scholar 

  • Matsubayashi A, Makino Y, Shinoda H (2019a) Direct finger manipulation of 3d object image with ultrasound haptic feedback. In: 2019a proceedings of the 2019a CHI conference on human factors in computing systems, Paper no 87, pp 1–11

    Google Scholar 

  • Matsubayashi A, Oikawa H, Mizutani S, Makino Y, Shinoda H (2019b) Display of haptic shape using ultrasound pressure distribution forming cross-sectional shape.In: 2019b Proceedings of the 2019b IEEE world haptics conference, pp 419–424, Tokyo, Japan

    Google Scholar 

  • Matsubayashi A, Yamaguchi T, Makino Y, Shinoda H (2021) Rendering softness using airborne ultrasound. In: 2021 proceedings of the IEEE world haptics conference, pp 355–360

    Google Scholar 

  • Monnai Y, Hasegawa K, Fujiwara M, Yoshino K, Inoue S, Shinoda H (2014) HaptoMime: Mid-air haptic interaction with a floating virtual screen. In: 2014 proceedings of the 27th annual ACM symposium on user interface software and technology (UIST’14). ACM, New York, NY, USA, pp 663–667

    Google Scholar 

  • Morisaki T, Fujiwara M, Makino Y, Shinoda H (2021) Non-vibratory pressure sensation produced by ultrasound focus moving laterally and repetitively with fine spatial step width. IEEE trans haptics (early access)

    Google Scholar 

  • Nakajima M, Hasegawa K, Makino Y, Shinoda H (2021) Spatiotemporal pinpoint cooling sensation produced by ultrasound-driven mist vaporization on skin. 2021 IEEE trans haptics (Early access)

    Google Scholar 

  • Norasikin MA, Martinez-Plasencia D, Memoli G, Subramanian S (2019) SonicSpray: a technique to reconfigure permeable mid-air displays. In: 2019 proceedings of the 2019 ACM international conference on interactive surfaces and spaces (ISS ‘19), pp 113–122

    Google Scholar 

  • Ochiai Y, Hoshi T, Suzuki I (2017) Holographic whisper: rendering audible sound spots in three-dimensional space by focusing ultrasonic waves. In: 2017 Proceedings of the 2017 CHI conference on human factors in computing systems, pp 4314–4325

    Google Scholar 

  • Pacchierotti C, Sinclair S, Solazzi M, Frisoli A, Hayward V, Prattichizzo D (2017) Wearable haptic systems for the fingertip and the hand: taxonomy, review, and perspectives. IEEE Trans Haptics 10(4):580–600

    Article  Google Scholar 

  • Pittera D, Gatti E, Obrist M (2019) I’m sensing in the rain: spatial incongruity in visual-tactile mid-air stimulation can elicit ownership in VR users. In: 2019 Proceedings of the CHI conference on human factors in computing systems. Association for computing machinery, New York, NY, USA, Paper 132, pp 1–15

    Google Scholar 

  • Rakkolainen I, Sand A, Palovuori K (2015) Midair user interfaces employing particle screens. IEEE Comput Graph Appl 35(2):96–102

    Article  Google Scholar 

  • Rim S, Suzuki S, Toide Y, Fujiwara M, Makino Y, Shinoda H (2020) Sound-image icon with aerial haptic feedback. Proc Euro Haptics 2020:497–505

    Google Scholar 

  • Satoshi S (2019) Thermal-radiation-based haptic display using laser-emission-based radiation control. In: 2019 Proceedings of IEEE world haptics 2019, WP2P.10 (Work-in-Progress Papers)

    Google Scholar 

  • Sand A, Rakkolainen I, Isokoski P, Kangas J, Raisamo R, Palovuori K (2015) Head-mounted display with mid-air tactile feedback. In: 2015 proceedings of the 21st ACM symposium on virtual reality software and technology (VRST ‘15), pp 51–58

    Google Scholar 

  • Serizawa K, Morisaki T, Delfosse C, Fujiwara M, Makino Y, Shinoda H (2020) Super haptoclone: upper-body mutual telexistence system with haptic feedback. SIGGRAPH ’20, emerging technologies, Washington, D.C, USA (moved to a virtual conference)

    Google Scholar 

  • Shinoda H (2010) Tactile interaction with 3D images. In: 2010 the 17th international display workshops (IDW’10), INP4: 3D interactive systems, pp 1743–1746

    Google Scholar 

  • Suzuki S, Fujiwara M, Makino Y, Shinoda H (2018) Midair ultrasound haptic display with large workspace. In: 2018 AsiaHaptics 2018, Incheon, Korea

    Google Scholar 

  • Takahashi R, Hasegawa K, Shinoda H (2020) Tactile stimulation by repetitive lateral movement of midair ultrasound focus. IEEE Trans Haptics 13(2):334–342

    Article  Google Scholar 

  • Takahashi R, Hasegawa K, Shinoda H (2018) Lateral Modulation of Midair Ultrasound Focus for Intensified Vibrotactile Stimuli. In: 2018 proceedings of Euro haptics, Part II, pp 276–288

    Google Scholar 

  • Vega-Bermudez F, Johnson KO, Hsiao SS (1991) Human tactile pattern recognition: active versus passive touch, velocity effects, and patterns of confusion. J Neurophysiol 65:531–546

    Article  Google Scholar 

  • Wilson G, Carter T, Subramanian S, Brewster SA (2014) Perception of ultrasonic haptic feedback on the hand: localisation and apparent motion. In: 2014 Proceedings of the SIGCHI conference on human factors in computing systems (CHI ’14), pp 1133–1142

    Google Scholar 

  • Yoshida K, Horiuchi Y, Inoue S, Makino Y, Shinoda H (2017) HaptoCloneAR: mutual haptic-optic interactive system with 2D image superimpose. In: 2017 Proceedings of SIGGRAPH emerging technologies, Los Angeles, California, USA

    Google Scholar 

  • Yoshino K, Shinoda H (2013) Visio-acoustic screen for contactless touch interface with tactile sensation. Proc IEEE World Haptics Conf 2013:419–423

    Google Scholar 

  • Yoshino K, Hasegawa K, Shinoda H (2012) Measuring visio-tactile threshold for visio-tactile projector. Proc SICE Annu Conf 2012:1996–2000

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. The University of Tokyo, 5-1-5 Kashiwanoha, Kashiwa, Japan

    Hiroyuki Shinoda & Yasutoshi Makino

Authors
  1. Hiroyuki Shinoda
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yasutoshi Makino
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hiroyuki Shinoda .

Editor information

Editors and Affiliations

  1. Ultraleap, Bristol, UK

    Orestis Georgiou

  2. Ultraleap, Bristol, UK

    William Frier

  3. School of Computing Science, University of Glasgow, Glasgow, UK

    Euan Freeman

  4. Institut de Recherche en Informatique et Systèmes Aléatoires (IRISA), Centre national de la recherche scientifique (CNRS), Rennes, France

    Claudio Pacchierotti

  5. Pixie Dust Technologies, Inc., Tokyo, Japan

    Takayuki Hoshi

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Shinoda, H., Makino, Y. (2022). Superimposing Visual Images on Mid-Air Ultrasonic Haptic Stimulation. In: Georgiou, O., Frier, W., Freeman, E., Pacchierotti, C., Hoshi, T. (eds) Ultrasound Mid-Air Haptics for Touchless Interfaces. Human–Computer Interaction Series. Springer, Cham. https://doi.org/10.1007/978-3-031-04043-6_12

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-04043-6_12

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-04042-9

  • Online ISBN: 978-3-031-04043-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation