Skip to main content
SpringerLink
Log in

Contactless human–computer interaction system based on three-dimensional holographic display and gesture recognition

  • Research
  • Published:
Applied Physics B Aims and scope Submit manuscript

Abstract

Contactless interaction provides a hygienic, safe, and intelligent way for communication between human and computers. As computing systems continue to advance, there is a growing focus on input methods and keyboard interaction approaches. However, current research on virtual keyboard systems remains confined to two-dimensional interfaces. The existing contactless interaction implementations are generally limited by handheld tools, wearable devices, or projection screens. In this work, we propose a contactless three-dimensional (3D) virtual keyboard system, which combines gesture recognition and holographic display. A hand-tracking sensor is used to collect the user’s gestures and fingertip positions as input. A spatial light modulator is utilized to generate the corresponding holographic 3D display images. The hand-tracking sensor and display module are controlled in a synchronous manner by a personal computer. The robust and real-time system is demonstrated through user-interactive experiments with a 3D virtual keyboard. The scalability of the proposed human–computer interaction system supports a wide range of applications, including education, communication, disease prevention, information security, and virtual reality.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Code availability

https://github.com/THUHoloLab/Contactless-HCI-system.

References

  1. X. Pu, H. Guo, J. Chen, X. Wang, Y. Xi, C. Hu, Z.L. Wang, Eye motion triggered self-powered mechnosensational communication system using triboelectric nanogeneratorEye motion triggered self-powered mechnosensational communication system using triboelectric nanogenerator. Sci. Adv. 3, e1700694 (2017)

    Article  ADS  Google Scholar 

  2. W. Gao, S. Emaminejad, H.Y.Y. Nyein, S. Challa, K. Chen, A. Peck, H.M. Fahad, H. Ota, H. Shiraki, D. Kiriya, D.H. Lien, G.A. Brooks, R.W. Davis, A. Javey, Fully integrated wearable sensor arrays for multiplexed in situ perspiration analysis. Nature 529, 509–514 (2016)

    Article  ADS  Google Scholar 

  3. M. Amjadi, K.-U. Kyung, I. Park, M. Sitti, Stretchable, skin-mountable, and wearable strain sensors and their potential applications: a review. Adv. Func. Mater. 26, 1678–1698 (2016)

    Article  Google Scholar 

  4. J. Ge, X. Wang, M. Drack, O. Volkov, M. Liang, G. S. Canon Bermudez, R. Illing, C. Wang, S. Zhou, J. Fassbender, M. Kaltenbrunner, D. Makarov, A bimodal soft electronic skin for tactile and touchless interaction in real time. Nat. Commun. 10, 4405 (2019)

    Article  ADS  Google Scholar 

  5. X. Yu, Z. Xie, Y. Yu, J. Lee, A. Vazquez-Guardado, H. Luan, J. Ruban, X. Ning, A. Akhtar, D. Li, B. Ji, Y. Liu, R. Sun, J. Cao, Q. Huo, Y. Zhong, C. Lee, S. Kim, P. Gutruf, C. Zhang, Y. Xue, Q. Guo, A. Chempakasseril, P. Tian, W. Lu, J. Jeong, Y. Yu, J. Cornman, C. Tan, B. Kim, K. Lee, X. Feng, Y. Huang, J.A. Rogers, Skin-integrated wireless haptic interfaces for virtual and augmented reality. Nature 575, 473–479 (2019)

    Article  ADS  Google Scholar 

  6. L. Lu, C. Jiang, G. Hu, J. Liu, B. Yang, Flexible noncontact sensing for human-machine interaction. Adv. Mater. 33, e2100218 (2021)

    Article  Google Scholar 

  7. S. An, H. Zhu, C. Guo, B. Fu, C. Song, P. Tao, W. Shang, T. Deng, Noncontact human-machine interaction based on hand-responsive infrared structural color. Nat. Commun. 13, 1446 (2022)

    Article  ADS  Google Scholar 

  8. W. Liu, Y. Duo, J. Liu, F. Yuan, L. Li, L. Li, G. Wang, B. Chen, S. Wang, H. Yang, Y. Liu, Y. Mo, Y. Wang, B. Fang, F. Sun, X. Ding, C. Zhang, L. Wen, Touchless interactive teaching of soft robots through flexible bimodal sensory interfaces. Nat. Commun. 13, 5030 (2022)

    Article  ADS  Google Scholar 

  9. P.-O. Kristensson and S. Zhai, Relaxing stylus typing precision by geometric pattern matching. In: Proceedings of the 10th International Conference on Intelligent User Interfaces, (2005), pp. 151–158

  10. P.-O. Kristensson, Five challenges for intelligent text entry methods. AI Mag. 30, 85–85 (2009)

    Google Scholar 

  11. I.S. MacKenzie, R.W. Soukoreff, Text entry for mobile computing: models and methods, theory and practice. Hum. Comput. Interact. 17, 147–198 (2002)

    Article  Google Scholar 

  12. S. Zhai, P.-O. Kristensson, B.A. Smith, In search of effective text input interfaces for off the desktop computing. Interact. Comput. 17, 229–250 (2005)

    Article  Google Scholar 

  13. G.A.M. Vasiljevic, L.C. De Miranda, E.E.C. De Miranda, A case study of mastermind chess: comparing mouse/keyboard interaction with kinect-based gestural interface. Adv. Hum.-Comput. Interact. 2016, 1 (2016)

    Article  Google Scholar 

  14. R. Khurana, D. McIsaac, E. Lockerman, and J. Mankoff, Nonvisual interaction techniques at the keyboard surface, In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, (2018), pp. 1–12.

  15. M. McGill, S. Brewster, D.P. De Sa Medeiros, S. Bovet, M. Gutierrez, A. Kehoe, Creating and augmenting keyboards for extended reality with the keyboard augmentation toolkit. ACM Transact. Comput. Hum. Interact. 29, 1–39 (2022)

    Article  Google Scholar 

  16. S.D. Chua, K.R. Chin, S. Lim, P. Jain, Hand gesture control for human–computer interaction with deep learning. J. Electrical Eng. Technol. 17, 1961–1970 (2022)

    Article  ADS  Google Scholar 

  17. M. Schenkluhn, C. Peukert, C. Weinhardt, Typing the future: Designing multimodal AR keyboards, In: Wirtschaftsinformatik 2022 Proceedings, (2022), p. 11.

  18. M. Wang, Z. Yan, T. Wang, P. Cai, S. Gao, Y. Zeng, C. Wan, H. Wang, L. Pan, J. Yu, S. Pan, K. He, J. Lu, X. Chen, Gesture recognition using a bioinspired learning architecture that integrates visual data with somatosensory data from stretchable sensors. Nat. Electron. 3, 563–570 (2020)

    Article  Google Scholar 

  19. A. Moin, A. Zhou, A. Rahimi, A. Menon, S. Benatti, G. Alexandrov, S. Tamakloe, J. Ting, N. Yamamoto, Y. Khan, F. Burghardt, L. Benini, A.C. Arias, J.M. Rabaey, A wearable biosensing system with in-sensor adaptive machine learning for hand gesture recognition. Nat. Electron. 4, 54–63 (2020)

    Article  Google Scholar 

  20. X. Guo, X. Lu, P. Jiang, X. Bao, SrTiO3/CuNi heterostructure-based thermopile for sensitive human radiation detection and noncontact human-machine interaction. Adv. Mater. 34, 2204355 (2022)

    Article  Google Scholar 

  21. L. Lu, C. Jiang, G. Hu, J. Liu, B. Yang, Flexible noncontact sensing for human–machine interaction. Adv. Mater. 33, 2100218 (2021)

    Article  Google Scholar 

  22. L. Cao, Z. He, K. Liu, X. Sui, Progress and challenges in dynamic holographic 3D display for the metaverse. Infrared Laser Eng. 51, 20210935–20210931 (2022)

    Google Scholar 

  23. Z. He, X. Sui, G. Jin, L. Cao, Progress in virtual reality and augmented reality based on holographic display. Appl. Opt. 58, A74–A81 (2019)

    Article  ADS  Google Scholar 

  24. H. Hiura, K. Komine, J. Arai, T. Mishina, Measurement of static convergence and accommodation responses to images of integral photography and binocular stereoscopy. Opt. Express 25, 3454–3468 (2017)

    Article  ADS  Google Scholar 

  25. Z. Zhuang, L. Zhang, P. Surman, W. Song, S. Thibault, X.W. Sun, Y. Zheng, Addressable spatial light modulators for eye-tracking autostereoscopic three-dimensional display using a scanning laser. Appl. Opt. 57, 4457–4466 (2018)

    Article  ADS  Google Scholar 

  26. B. Xu, Q. Wu, Y. Bao, G. Chen, Y. Wang, S. Ren, Time-multiplexed stereoscopic display with a quantum dot-polymer scanning backlight. Appl. Opt. 58, 4526–4532 (2019)

    Article  ADS  Google Scholar 

  27. Y. Meng, Y. Lyu, L.L. Chen, Z. Yu, H. Liao, Motion parallax and lossless resolution autostereoscopic 3D display based on a binocular viewpoint tracking liquid crystal dynamic grating adaptive screen. Opt. Express 29, 35456–35473 (2021)

    Article  ADS  Google Scholar 

  28. X. Li, Q. Wu, Y. Wang, Binocular vision calibration method for a long-wavelength infrared camera and a visible spectrum camera with different resolutions. Opt. Express 29, 3855–3872 (2021)

    Article  Google Scholar 

  29. K. Kumagai, I. Yamaguchi, Y. Hayasaki, Three-dimensionally structured voxels for volumetric display. Opt. Lett. 43, 3341–3344 (2018)

    Article  ADS  Google Scholar 

  30. R. Hirayama, D. Martinez Plasencia, N. Masuda, S. Subramanian, A volumetric display for visual, tactile and audio presentation using acoustic trapping. Nature 575, 320–323 (2019)

    Article  ADS  Google Scholar 

  31. M. Chul Shin, A. Mohanty, K. Watson, G.R. Bhatt, C.T. Phare, S.A. Miller, M. Zadka, B.S. Lee, X. Ji, I. Datta, M. Lipson, Chip-scale blue light phased array. Opt. Lett. 45, 1934–1937 (2020)

    Article  ADS  Google Scholar 

  32. K. Suzuki, Y. Fukano, H. Oku, 1000-volume/s high-speed volumetric display for high-speed HMD. Opt. Express 28, 29455–29468 (2020)

    Article  ADS  Google Scholar 

  33. L. Yang, X. Sang, X. Yu, B. Liu, B. Yan, K. Wang, C. Yu, A crosstalk-suppressed dense multi-view light-field display based on real-time light-field pickup and reconstruction. Opt. Express 26, 34412–34427 (2018)

    Article  ADS  Google Scholar 

  34. Q. Ma, L. Cao, Z. He, S. Zhang, Progress of three-dimensional light-field display. Chin. Opt. Lett. 17, 111001 (2019)

    Article  Google Scholar 

  35. H.-L. Zhang, H. Deng, H. Ren, X. Yang, Y. Xing, D.-H. Li, Q.-H. Wang, Method to eliminate pseudoscopic issue in an integral imaging 3D display by using a transmissive mirror device and light filter. Opt. Lett. 45, 351–354 (2020)

    Article  ADS  Google Scholar 

  36. M. Xu, H. Hua, Systematic method for modeling and characterizing multilayer light field displays. Opt. Express 28, 1014–1036 (2020)

    Article  ADS  Google Scholar 

  37. Y. Chen, X. Sang, S. Xing, Y. Guan, H. Zhang, K. Wang, Automatic co-design of light field display system based on simulated annealing algorithm and visual simulation. Opt. Express 30, 17577–17590 (2022)

    Article  ADS  Google Scholar 

  38. S. Qi, X. Sang, B. Yan, D. Chen, P. Wang, H. Wang, X. Ye, Virtual view synthesis for 3D light-field display based on feature reprojection and fusion. Optics Commun. 519, 128383 (2022)

    Article  Google Scholar 

  39. C. Chang, K. Bang, G. Wetzstein, B. Lee, L. Gao, Toward the next-generation VR/AR optics: a review of holographic near-eye displays from a human-centric perspective. Optica 7, 1563–1578 (2020)

    Article  ADS  Google Scholar 

  40. X. Sui, Z. He, G. Jin, D. Chu, L. Cao, Band-limited double-phase method for enhancing image sharpness in complex modulated computer-generated holograms. Opt. Express 29, 2597–2612 (2021)

    Article  ADS  Google Scholar 

  41. K. Liu, Z. He, L. Cao, Pattern-adaptive error diffusion algorithm for improved phase-only hologram generation. Chin. Opt. Lett. 19, 050501 (2021)

    Article  ADS  Google Scholar 

  42. Z. Zhao, J. Duan, J. Liu, Speckle reduction in holographic display with partially spatial coherent illumination. Optics Commun. 507, 127604 (2022)

    Article  Google Scholar 

  43. T. Shimobaba, T. Ito, Random phase-free computer-generated hologram. Opt. Express 23, 9549–9554 (2015)

    Article  ADS  Google Scholar 

  44. Y. Zhao, L. Cao, H. Zhang, D. Kong, G. Jin, Accurate calculation of computer-generated holograms using angular-spectrum layer-oriented method. Opt. Express 23, 25440–25449 (2015)

    Article  ADS  Google Scholar 

  45. S. Igarashi, T. Nakamura, K. Matsushima, M. Yamaguchi, Efficient tiled calculation of over-10-gigapixel holograms using ray-wavefront conversion. Opt. Express 26, 10773–10786 (2018)

    Article  ADS  Google Scholar 

  46. D. Blinder, Direct calculation of computer-generated holograms in sparse bases. Opt. Express 27, 23124–23137 (2019)

    Article  ADS  Google Scholar 

  47. Z. Lv, Y. Xu, Y. Yang, J. Liu, Multiplane holographic augmented reality head-up display with a real–virtual dual mode and large eyebox. Appl. Opt. 61, 9962–9971 (2022)

    Article  ADS  Google Scholar 

  48. F. Lu, J. Hua, F. Zhou, Z. Xia, R. Li, L. Chen, W. Qiao, Pixelated volume holographic optical element for augmented reality 3D display. Opt. Express 30, 15929–15938 (2022)

    Article  ADS  Google Scholar 

  49. L. He, K. Liu, Z. He, L. Cao, Three-dimensional holographic communication system for the metaverse. Optics Commun. 526, 128894 (2022)

    Article  Google Scholar 

  50. T. Shimobaba, A. Shiraki, Y. Ichihashi, N. Masuda, T. Ito, Interactive color electroholography using the FPGA technology and time division switching method. IEICE Electron. Express 5, 271–277 (2008)

    Article  Google Scholar 

  51. S. Yamada, T. Kakue, T. Shimobaba, T. Ito, Interactive holographic display based on finger gestures. Sci. Rep. 8, 2010 (2018)

    Article  ADS  Google Scholar 

  52. K. Suzuki, M. Oikawa, Y. Mori, T. Kakue, T. Shimobaba, T. Ito, and N. Takada, Holographic projection system for drawing fingertip trajectory obtained from depth camera. In: Proceedings of the International Display Workshops, (2019), pp. 2–24

  53. M. Takenaka, T. Kakue, T. Shimobaba, T. Ito, Interactive holographic display for real-time drawing and erasing of 3D point-cloud images with a fingertip. IEEE Access 9, 36766–36774 (2021)

    Article  Google Scholar 

  54. V.K. Adhikarla, J. Sodnik, P. Szolgay, G. Jakus, Exploring direct 3D interaction for full horizontal parallax light field displays using leap motion controller. Sensors 15, 8642–8663 (2015)

    Article  ADS  Google Scholar 

  55. M. Yamaguchi, R. Higashida, 3D touchable holographic light-field display. Appl. Opt. 55, A178–A183 (2016)

    Article  ADS  Google Scholar 

  56. M. Yamaguchi, Full-parallax holographic light-field 3-D displays and interactive 3-D touch. Proc. IEEE 105, 947–959 (2017)

    Article  Google Scholar 

  57. X. Sang, X. Gao, X. Yu, S. Xing, Y. Li, Y. Wu, Interactive floating full-parallax digital three-dimensional light-field display based on wavefront recomposing. Opt. Express 26, 8883–8889 (2018)

    Article  ADS  Google Scholar 

  58. Y. Sando, K. Satoh, D. Barada, T. Yatagai, Real-time interactive holographic 3D display with a 360° horizontal viewing zone. Appl. Opt. 58, G1–G5 (2019)

    Article  Google Scholar 

  59. I.A. Sánchez Salazar Chavarría, T. Nakamura, M. Yamaguchi, Interactive optical 3D-touch user interface using a holographic light-field display and color information. Opt. Express 28, 36740–36755 (2020)

    Article  ADS  Google Scholar 

  60. T. Nishitsuji, T. Kakue, D. Blinder, T. Shimobaba, T. Ito, An interactive holographic projection system that uses a hand-drawn interface with a consumer CPU. Sci. Rep. 11, 147 (2021)

    Article  ADS  Google Scholar 

  61. I.A. Sánchez Salazar Chavarría, K. Shimomura, S. Takeyama, M. Yamaguchi, Interactive 3D touch and gesture capable holographic light field display with automatic registration between user and content. J. Soc. Inform. Display 30, 877–893 (2022)

    Article  Google Scholar 

  62. M. Cheok, Z. Omar, M. Jaward, A review of hand gesture and sign language recognition techniques. Int. J. Mach. Learn. Cybern. 10, 131–153 (2017)

    Article  Google Scholar 

  63. H. Cheng, L. Yang, Z. Liu, Survey on 3D hand gesture recognition. IEEE Trans. Circuits Syst. Video Technol. 26, 1659–1673 (2016)

    Article  Google Scholar 

  64. J.K. Aggarwal, L. Xia, Human activity recognition from 3D data: a review. Pattern Recogn. Lett. 48, 70–80 (2014)

    Article  ADS  Google Scholar 

  65. Intel, Intel RealSense Technology (2023). Retrieved November 12, 2023, from https://www.intelrealsense.com/

  66. Microsoft, Azure Kinect DK (2023). Retrieved November 12, 2023, from https://azure.microsoft.com/zh-cn/products/kinect-dk/

  67. Ultraleap, Leap Motion Controller (2023). Retrieved November 12, 2023, from https://www.ultraleap.com/product/leap-motion-controller/

  68. R.W. Gerchberg, A practical algorithm for the determination of plane from image and diffraction pictures. Optik 35, 237–246 (1972)

    Google Scholar 

  69. J. Wu, K. Liu, X. Sui, L. Cao, High-speed computer-generated holography using an autoencoder-based deep neural network. Opt. Lett. 46, 2908–2911 (2021)

    Article  ADS  Google Scholar 

  70. K. Liu, J. Wu, Z. He, L. Cao, 4K-DMDNet: diffraction model-driven network for 4K computer-generated holography. Opto-Electron. Adv. 6, 220135 (2023)

    Article  Google Scholar 

  71. D. Wang, N.-N. Li, Y.-L. Li, Y.-W. Zheng, Z.-Q. Nie, Z.-S. Li, F. Chu, Q.-H. Wang, Large viewing angle holographic 3D display system based on maximum diffraction modulation. Light Adv. Manuf. 4, 1–11 (2023)

    Google Scholar 

Download references

Funding

National Natural Science Foundation of China (62035003); China Postdoctoral Science Foundation (BX2021140).

Author information

Authors and Affiliations

  1. Weiyang College, Tsinghua University, Beijing, 100084, China

    Yixin Yang & Liangcai Cao

  2. Department of Precision Instrument, Tsinghua University, Beijing, 100084, China

    Yunhui Gao, Kexuan Liu, Zehao He & Liangcai Cao

Authors
  1. Yixin Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yunhui Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kexuan Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zehao He
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Liangcai Cao
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

YY and LC proposed the concept. YY and YG developed the algorithm and the experimental system. YY and KL developed the CGH algorithm. YY wrote the main manuscript. All authors reviewed the manuscript.

Corresponding author

Correspondence to Liangcai Cao.

Ethics declarations

Conflict of interest

The authors declare no conflicts of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 A video of user-interactive 3D virtual keyboard is attached (MP4 49346 KB)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yang, Y., Gao, Y., Liu, K. et al. Contactless human–computer interaction system based on three-dimensional holographic display and gesture recognition. Appl. Phys. B 129, 192 (2023). https://doi.org/10.1007/s00340-023-08128-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00340-023-08128-2

Search

Navigation