Skip to main content
SpringerLink
Log in

Adversarial Attacks on Skeleton-Based Sign Language Recognition

  • Conference paper
  • First Online:
Intelligent Robotics and Applications (ICIRA 2023)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 14267))

Included in the following conference series:

  • 643 Accesses

Abstract

Despite the impressive performance achieved by sign language recognition systems based on skeleton information, our research has uncovered their vulnerability to malicious attacks. In response to this challenge, we present an adversarial attack specifically designed to sign language recognition models that rely on extracted human skeleton data as features. Our attack aims to assess the robustness and sensitivity of these models, and we propose adversarial training techniques to enhance their resilience. Moreover, we conduct transfer experiments using the generated adversarial samples to demonstrate the transferability of these adversarial examples across different models. Additionally, by conducting experiments on the sensitivity of sign language recognition models, we identify the optimal experimental parameter settings for achieving the most effective attacks. This research significantly contributes to future investigations into the security of sign language recognition.

The authors would like to acknowledge the support from the National Natural Science Foundation of China (52075530).

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Yu, J., Gao, H., Zhou, D., Liu, J., Gao, Q., Ju, Z.: Deep temporal model-based identity-aware hand detection for space human-robot interaction. IEEE Transactions on Cybernetics 52(12), 13738–13751 (2022). https://doi.org/10.1109/TCYB.2021.3114031

    Article  Google Scholar 

  2. Liao, F., Liang, M., Dong, Y., Pang, T., Hu, X., Zhu, J.: Defense against adversarial attacks using high-level representation guided denoiser. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018)

    Google Scholar 

  3. Xiao, C., Li, B., yan Zhu, J., He, W., Liu, M., Song, D.: Generating adversarial examples with adversarial networks. In: IJCAI, pp. 3905–3911 (2018)

    Google Scholar 

  4. Yan, S., Xiong, Y., Lin, D.: Spatial temporal graph convolutional networks for skeleton-based action recognition. In: AAAI (2018)

    Google Scholar 

  5. Zhang, P., Lan, C., Xing, J., Zeng, W., Xue, J., Zheng, N.: View adaptive neural networks for high performance skeleton-based human action recognition. IEEE Trans. Pattern Anal. Mach. Intell. 41(8), 1963–1978 (2019)

    Article  Google Scholar 

  6. Yu, J., Gao, H., Chen, Y., Zhou, D., Liu, J., Ju, Z.: Adaptive spatiotemporal representation learning for skeleton-based human action recognition. IEEE Trans. Cognitive and Developmental Syst. 14(4), 1654–1665 (2022). https://doi.org/10.1109/TCDS.2021.3131253

    Article  Google Scholar 

  7. Cheng, K., Zhang, Y., Cao, C., Shi, L., Cheng, J., Lu, H.: Decoupling GCN with DropGraph module for skeleton-based action recognition. In: Proceedings of European Conference on Computer Vision (2020)

    Google Scholar 

  8. Xiao, Q., Qin, M., Yin, Y.: Skeleton-based chinese sign language recognition and generation for bidirectional communication between deaf and hearing people. Neural Networks, 125, 41–55 (2020)

    Google Scholar 

  9. Yan, S., Xiong, Y., Lin, D.: Spatial temporal graph convolutional networks for skeleton-based action recognition. In: Proceedings of AAAI conference on artificial intelligence 32 (2018)

    Google Scholar 

  10. Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and Harnessing Adversarial Examples. arXiv preprint arXiv:1412.6572 (20140

  11. Diao, Y., Shao, T., Yang, Y.L., et al.: BASAR: black-box attack on skeletal action recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7597–7607 (2021)

    Google Scholar 

  12. Tanaka, N., Kera, H., Kawamoto, K.: Adversarial bone length attack on action recognition. Proceedings of the AAAI Conference on Artificial Intelligence. 36(2), 2335–2343 (2022)

    Google Scholar 

  13. Wang, H., He, F., Peng, Z., et al.: Understanding the robustness of skeleton-based action recognition under adversarial attack. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14656–14665 (2021)

    Google Scholar 

  14. Koller, O., Zargaran, S., Ney, H., Bowden, R.: Deep sign: enabling robust statistical continuous sign language recognition via hybrid CNN-HMMs. Int. J. Comput. Vision 126(12), 1311–1325 (2018)

    Article  Google Scholar 

  15. Du, Y., Wang, W., Wang, L.: Hierarchical recurrent neural network for skeleton based action recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1110–1118 (2015)

    Google Scholar 

  16. Li, M., Chen, S., Chen, X., Zhang, Y., Wang, Y., Tian, Q.: Actional-structural graph convolutional networks for skeleton-based action recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3595–3603 (2019)

    Google Scholar 

  17. Shi, L., Zhang, Y., Cheng, J., Lu, H.: Twostream adaptive graph convolutional networks for skeletonbased action recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12026–12035 (2019)

    Google Scholar 

  18. Shi, L., Zhang, Y., Cheng, J., Hanqing, L.: Skeleton-based action recognition with multi-stream adaptive graph convolutional networks. IEEE Trans. Image Process. 29, 9532–9545 (2020)

    Article  MATH  Google Scholar 

  19. Song, Y.-F., Zhang, Z., Shan, C., Wang, L.: Stronger, faster and more explainable: a graph convolutional baseline for skeleton-based action recognition. In: Proceedings of ACM International Conference on Multimedia, pp. 1625–1633 (2020)

    Google Scholar 

  20. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

    Google Scholar 

  21. Liu, J., Akhtar, N., Mian, A.: Adversarial attack on skeleton-based human action recognition. IEEE Trans. Neural Networks and Learning Syst. 33(4), 1609–1622 (2020)

    Article  Google Scholar 

  22. Newman, M.E., Watts, D.J., Strogatz, S.H.: Random graph models of social networks. Proc. Natl. Acad. Sci. 99(suppl 1), 2566–2572 (2002)

    Article  MATH  Google Scholar 

  23. Cao, Z., Hidalgo, G., Simon, T., Wei, S.-E., Sheikh, Y.: Openpose: Realtime Multi-Person 2d Pose Estimation Using Part Affinity Fields. arXiv preprint arXiv:1812.08008 (2018)

  24. Zhou, Y., Han, M., Liu, L., et al.: The adversarial attacks threats on computer vision: a survey. In: 2019 IEEE 16th International Conference on Mobile Ad Hoc and Sensor Systems Workshops (MASSW). IEEE, pp. 25–30 (2019)

    Google Scholar 

  25. Zheng, T., Liu, S., Chen, C., Yuan, J., Li, B., Ren, K.: Towards Understanding the Adversarial Vulnerability of Skeleton-based Action Recognition. arXiv:2005.07151 [cs] (2020)

  26. Jiang, S., Sun, B., Wang, L., et al.: Skeleton aware multi-modal sign language recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 3413–3423 (2021)

    Google Scholar 

  27. Sincan, O.M., Keles, H.Y.: AUTSL: a large scale multi-modal turkish sign language dataset and baseline methods. IEEE Access 8, 181340–181355 (2020)

    Google Scholar 

  28. Amon, C., Fuhrmann, F., Graf, F.: Evaluation of the spatial resolution accuracy of the face tracking system for Kinect for windows V1 and V2. In: Proceedings of AAAI Conference on Artificial, pp. 16–17 (2014)

    Google Scholar 

  29. He, S., Han, M., Patel, N., et al.: Converting handwritten text to editable format via gesture recognition for education. Proceedings of the 51st ACM Technical Symposium on Computer Science Education, pp. 1369–1369 (2020)

    Google Scholar 

  30. Zhou, Y., Han, M., Liu, L., et al.: Deep learning approach for cyberattack detection. IEEE INFOCOM 2018-IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS). IEEE, pp. 262–267 (2018)

    Google Scholar 

Download references

Acknowledgement

Changting’s work was partially supported by NSFC under No. 62102363 and No. 62072404, the Zhejiang Provincial Natural Science Foundation under No. LQ21F020010, Hangzhou Innovation Team under No. TD2022011.

Author information

Authors and Affiliations

  1. School of Automation and Electrical Engineering, Shenyang Ligong University, Shenyang, 110159, China

    Yufeng Li

  2. College of Computer Science and Technology, Zhejiang University, Hangzhou, China

    Meng Han

  3. Department of Biomedical Engineering, Zhejiang University, Hangzhou, 310058, China

    Jiahui Yu

  4. College of Computer Science and Technology, Zhejiang University, Hangzhou, 310058, China

    Changting Lin

  5. School of Computing, University of Portsmouth, Portsmouth, PO13HE, UK

    Zhaojie Ju

Authors
  1. Yufeng Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Meng Han
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jiahui Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Changting Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zhaojie Ju
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Meng Han .

Editor information

Editors and Affiliations

  1. Zhejiang University, Hangzhou, China

    Huayong Yang

  2. Harbin Institute of Technology, Shenzhen, China

    Honghai Liu

  3. Zhejiang University, Hangzhou, China

    Jun Zou

  4. Huazhong University of Science and Technology, Wuhan, China

    Zhouping Yin

  5. Shenyang Institute of Automation, Shenyang, Liaoning, China

    Lianqing Liu

  6. Zhejiang University, Hangzhou, China

    Geng Yang

  7. Zhejiang University, Hangzhou, China

    Xiaoping Ouyang

  8. Harbin Institute of Technology, Shenzhen, China

    Zhiyong Wang

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Li, Y., Han, M., Yu, J., Lin, C., Ju, Z. (2023). Adversarial Attacks on Skeleton-Based Sign Language Recognition. In: Yang, H., et al. Intelligent Robotics and Applications. ICIRA 2023. Lecture Notes in Computer Science(), vol 14267. Springer, Singapore. https://doi.org/10.1007/978-981-99-6483-3_4

Download citation

  • DOI: https://doi.org/10.1007/978-981-99-6483-3_4

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-99-6482-6

  • Online ISBN: 978-981-99-6483-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation