Skip to main content
SpringerLink
Log in

Gesture Recognition with Keypoint and Radar Stream Fusion for Automated Vehicles

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 Workshops (ECCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13801))

Included in the following conference series:

Abstract

We present a joint camera and radar approach to enable autonomous vehicles to understand and react to human gestures in everyday traffic. Initially, we process the radar data with a PointNet followed by a spatio-temporal multilayer perceptron (stMLP). Independently, the human body pose is extracted from the camera frame and processed with a separate stMLP network. We propose a fusion neural network for both modalities, including an auxiliary loss for each modality. In our experiments with a collected dataset, we show the advantages of gesture recognition with two modalities. Motivated by adverse weather conditions, we also demonstrate promising performance when one of the sensors lacks functionality.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://github.com/fxia22/pointnet.pytorch.

  2. 2.

    https://github.com/holzbock/st_mlp.

References

  1. Abughalieh, K.M., Alawneh, S.G.: Predicting pedestrian intention to cross the road. IEEE Access 8, 72558–72569 (2020)

    Article  Google Scholar 

  2. Ba, J.L., Kiros, J.R., Hinton, G.E.: Layer normalization. arXiv preprint arXiv:1607.06450 (2016)

  3. Baek, T., Lee, Y.G.: Traffic control hand signal recognition using convolution and recurrent neural networks. J. Comput. Des. Eng. 9(2), 296–309 (2022)

    Google Scholar 

  4. Belagiannis, V., Amann, C., Navab, N., Ilic, S.: Holistic human pose estimation with regression forests. In: Perales, F.J., Santos-Victor, J. (eds.) AMDO 2014. LNCS, vol. 8563, pp. 20–30. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-08849-5_3

    Chapter  Google Scholar 

  5. Bouazizi, A., Wiederer, J., Kressel, U., Belagiannis, V.: Self-supervised 3D human pose estimation with multiple-view geometry. In: 2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021), pp. 1–8 (2021). https://doi.org/10.1109/FG52635.2021.9667074

  6. Geng, K., Yin, G.: Using deep learning in infrared images to enable human gesture recognition for autonomous vehicles. IEEE Access 8, 88227–88240 (2020)

    Article  Google Scholar 

  7. Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)

    Article  Google Scholar 

  8. Holzbock, A., Tsaregorodtsev, A., Dawoud, Y., Dietmayer, K., Belagiannis, V.: A spatio-temporal multilayer perceptron for gesture recognition. In: 2022 IEEE Intelligent Vehicles Symposium (IV), pp. 1099–1106 (2022). https://doi.org/10.1109/IV51971.2022.9827054

  9. de Jong, R.J., de Wit, J.J., Uysal, F.: Classification of human activity using radar and video multimodal learning. IET Radar Sonar Navig. 15(8), 902–914 (2021)

    Article  Google Scholar 

  10. Kern, N., Grebner, T., Waldschmidt, C.: PointNet+ LSTM for target list-based gesture recognition with incoherent radar networks. IEEE Trans. Aerosp. Electron. Syst. (2022). https://doi.org/10.1109/TAES.2022.3179248

  11. Kern, N., Steiner, M., Lorenzin, R., Waldschmidt, C.: Robust doppler-based gesture recognition with incoherent automotive radar sensor networks. IEEE Sens. Lett. 4(11), 1–4 (2020)

    Article  Google Scholar 

  12. Kim, Y., Toomajian, B.: Hand gesture recognition using micro-doppler signatures with convolutional neural network. IEEE Access 4, 7125–7130 (2016)

    Article  Google Scholar 

  13. Lien, J., et al.: Soli: ubiquitous gesture sensing with millimeter wave radar. ACM Trans. Graph. (TOG) 35(4), 1–19 (2016)

    Article  Google Scholar 

  14. Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48

    Chapter  Google Scholar 

  15. Mishra, A., Kim, J., Cha, J., Kim, D., Kim, S.: Authorized traffic controller hand gesture recognition for situation-aware autonomous driving. Sensors 21(23), 7914 (2021)

    Article  Google Scholar 

  16. Molchanov, P., Gupta, S., Kim, K., Kautz, J.: Hand gesture recognition with 3D convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 1–7 (2015)

    Google Scholar 

  17. Molchanov, P., Gupta, S., Kim, K., Pulli, K.: Multi-sensor system for driver’s hand-gesture recognition. In: 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), vol. 1, pp. 1–8. IEEE (2015)

    Google Scholar 

  18. Ninos, A., Hasch, J., Zwick, T.: Real-time macro gesture recognition using efficient empirical feature extraction with millimeter-wave technology. IEEE Sens. J. 21(13), 15161–15170 (2021)

    Article  Google Scholar 

  19. Ohn-Bar, E., Trivedi, M.M.: Hand gesture recognition in real time for automotive interfaces: a multimodal vision-based approach and evaluations. IEEE Trans. Intell. Transp. Syst. 15(6), 2368–2377 (2014)

    Article  Google Scholar 

  20. Paszke, A., et al.: Pytorch: an imperative style, high-performance deep learning library. In: Advances in Neural Information Processing Systems 32, pp. 8024–8035. Curran Associates, Inc. (2019). http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf

  21. Pfeuffer, A., Dietmayer, K.: Robust semantic segmentation in adverse weather conditions by means of sensor data fusion. In: 2019 22th International Conference on Information Fusion (FUSION), pp. 1–8. IEEE (2019)

    Google Scholar 

  22. Pham, D.T., Pham, Q.T., Le, T.L., Vu, H.: An efficient feature fusion of graph convolutional networks and its application for real-time traffic control gestures recognition. IEEE Access 9, 121930–121943 (2021)

    Article  Google Scholar 

  23. Qi, C.R., Su, H., Mo, K., Guibas, L.J.: Pointnet: deep learning on point sets for 3D classification and segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 652–660 (2017)

    Google Scholar 

  24. Qian, K., Zhu, S., Zhang, X., Li, L.E.: Robust multimodal vehicle detection in foggy weather using complementary lidar and radar signals. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 444–453 (2021)

    Google Scholar 

  25. Quintero, R., Parra, I., Lorenzo, J., Fernández-Llorca, D., Sotelo, M.: Pedestrian intention recognition by means of a hidden Markov model and body language. In: 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC), pp. 1–7. IEEE (2017)

    Google Scholar 

  26. Rasouli, A., Tsotsos, J.K.: Autonomous vehicles that interact with pedestrians: a survey of theory and practice. IEEE Trans. Intell. Transp. Syst. 21(3), 900–918 (2019)

    Article  Google Scholar 

  27. Rohling, H.: Radar CFAR thresholding in clutter and multiple target situations. IEEE Trans. Aerosp. Electron. Syst. AES-19(4), 608–621 (1983). https://doi.org/10.1109/TAES.1983.309350

  28. Schreiber, M., Belagiannis, V., Gläser, C., Dietmayer, K.: Motion estimation in occupancy grid maps in stationary settings using recurrent neural networks. In: 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 8587–8593 (2020). https://doi.org/10.1109/ICRA40945.2020.9196702

  29. Singh, A.D., Sandha, S.S., Garcia, L., Srivastava, M.: Radhar: human activity recognition from point clouds generated through a millimeter-wave radar. In: Proceedings of the 3rd ACM Workshop on Millimeter-Wave Networks and Sensing Systems, pp. 51–56 (2019)

    Google Scholar 

  30. Skaria, S., Al-Hourani, A., Huang, D.: Radar-thermal sensor fusion methods for deep learning hand gesture recognition. In: 2021 IEEE Sensors, pp. 1–4. IEEE (2021)

    Google Scholar 

  31. Szegedy, C., et al.: Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–9 (2015)

    Google Scholar 

  32. Vandersmissen, B., Knudde, N., Jalalvand, A., Couckuyt, I., Dhaene, T., De Neve, W.: Indoor human activity recognition using high-dimensional sensors and deep neural networks. Neural Comput. Appl. 32(16), 12295–12309 (2020)

    Article  Google Scholar 

  33. Vasanelli, C., et al.: Calibration and direction-of-arrival estimation of millimeter-wave radars: a practical introduction. IEEE Antennas Propag. Mag. 62(6), 34–45 (2020). https://doi.org/10.1109/MAP.2020.2988528

    Article  Google Scholar 

  34. Wang, S., Song, J., Lien, J., Poupyrev, I., Hilliges, O.: Interacting with soli: exploring fine-grained dynamic gesture recognition in the radio-frequency spectrum. In: Proceedings of the 29th Annual Symposium on User Interface Software and Technology, pp. 851–860 (2016)

    Google Scholar 

  35. Wang, S., Jiang, K., Chen, J., Yang, M., Fu, Z., Yang, D.: Simple but effective: upper-body geometric features for traffic command gesture recognition. IEEE Trans. Hum.-Mach. Syst. 52(3), 423–434 (2021)

    Article  Google Scholar 

  36. Wharton, Z., Behera, A., Liu, Y., Bessis, N.: Coarse temporal attention network (CTA-Net) for driver’s activity recognition. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1279–1289 (2021)

    Google Scholar 

  37. Wiederer, J., Bouazizi, A., Kressel, U., Belagiannis, V.: Traffic control gesture recognition for autonomous vehicles. In: 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 10676–10683. IEEE (2020)

    Google Scholar 

  38. Winkler, V.: Range doppler detection for automotive FMCW radars. In: European Radar Conference, pp. 166–169. IEEE, Piscataway (2007). https://doi.org/10.1109/EURAD.2007.4404963

  39. Wu, Y., Kirillov, A., Massa, F., Lo, W.Y., Girshick, R.: Detectron2 (2019). https://github.com/facebookresearch/detectron2

  40. Xu, F., Xu, F., Xie, J., Pun, C.M., Lu, H., Gao, H.: Action recognition framework in traffic scene for autonomous driving system. IEEE Trans. Intell. Transp. Syst. (2021)

    Google Scholar 

  41. Zengeler, N., Kopinski, T., Handmann, U.: Hand gesture recognition in automotive human-machine interaction using depth cameras. Sensors 19(1), 59 (2018)

    Article  Google Scholar 

  42. Zhang, Z., Tian, Z., Zhou, M.: Latern: dynamic continuous hand gesture recognition using FMCW radar sensor. IEEE Sens. J. 18(8), 3278–3289 (2018)

    Article  Google Scholar 

Download references

Acknowledgment

Part of this work was supported by INTUITIVER (7547.223-3/4/), funded by State Ministry of Baden-Württemberg for Sciences, Research and Arts and the State Ministry of Transport Baden-Württemberg.

Author information

Authors and Affiliations

  1. Institute of Measurement, Control and Microtechnology, Ulm University, Albert-Einstein-Allee 41, 89081, Ulm, Germany

    Adrian Holzbock & Klaus Dietmayer

  2. Institute of Microwave Engineering, Ulm University, Albert-Einstein-Allee 41, 89081, Ulm, Germany

    Nicolai Kern & Christian Waldschmidt

  3. Department of Simulation and Graphics, Otto von Guericke University Magdeburg, Universitätsplatz 2, 39106, Magdeburg, Germany

    Vasileios Belagiannis

Authors
  1. Adrian Holzbock
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nicolai Kern
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Christian Waldschmidt
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Klaus Dietmayer
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Vasileios Belagiannis
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Adrian Holzbock .

Editor information

Editors and Affiliations

  1. IBM Research AI and MIT-IBM Watson AI Lab, Haifa, Israel

    Leonid Karlinsky

  2. Technion – Israel Institute of Technology, Haifa, Israel

    Tomer Michaeli

  3. Kyoto University, Kyoto, Japan

    Ko Nishino

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Holzbock, A., Kern, N., Waldschmidt, C., Dietmayer, K., Belagiannis, V. (2023). Gesture Recognition with Keypoint and Radar Stream Fusion for Automated Vehicles. In: Karlinsky, L., Michaeli, T., Nishino, K. (eds) Computer Vision – ECCV 2022 Workshops. ECCV 2022. Lecture Notes in Computer Science, vol 13801. Springer, Cham. https://doi.org/10.1007/978-3-031-25056-9_36

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-25056-9_36

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-25055-2

  • Online ISBN: 978-3-031-25056-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation