Skip to main content
SpringerLink
Log in

3D Human Pose Estimation in Video for Human-Computer/Robot Interaction

  • Conference paper
  • First Online:
Intelligent Robotics and Applications (ICIRA 2023)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 14273))

Included in the following conference series:

Abstract

3D human pose estimation is widely used in motion capture, human-computer interaction, virtual character driving and other fields. The current 3D human pose estimation has been suffering from depth blurring and self-obscuring problems to be solved. This paper proposes a human pose estimation network in video based on a 2D lifting to 3D approach using transformer and graph convolutional network(GCN), which are widely used in natural language processing. We use transformer to obtain sequence features and use graph convolution to extract features between local joints to get more accurate 3D pose coordinates. In addition, we use the proposed 3D pose estimation network for animated character motion generation and robot motion following and design two systems of human-computer/robot interaction (HCI/HRI) applications. The proposed 3D human pose estimation network is tested on the Human3.6M dataset and outperforms the state-of-the-art models. Both HCI/HRI systems are designed to work quickly and accurately by the proposed 3D human pose estimation method.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Mandal, F.B.: Nonverbal communication in humans. J. Hum. Behav. Soc. Environ. 24(4), 417–421 (2014)

    Article  Google Scholar 

  2. Fankhauser, P., Bloesch, M., Rodriguez, D., Kaestner, R., Hutter, M., Siegwart, R.: Kinect v2 for mobile robot navigation: evaluation and modeling. In: 2015 International Conference on Advanced Robotics (ICAR), pp. 388–394. IEEE (2015)

    Google Scholar 

  3. Li, S., Chan, A.B.: 3D human pose estimation from monocular images with deep convolutional neural network. In: Cremers, D., Reid, I., Saito, H., Yang, M.-H. (eds.) ACCV 2014. LNCS, vol. 9004, pp. 332–347. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-16808-1_23

    Chapter  Google Scholar 

  4. Martinez, J., Hossain, R., Romero, J., Little, J.J.: A simple yet effective baseline for 3D human pose estimation. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2640–2649 (2017)

    Google Scholar 

  5. Pavllo, D., Feichtenhofer, C., Grangier, D., Auli, M.: 3D human pose estimation in video with temporal convolutions and semi-supervised training. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7753–7762 (2019)

    Google Scholar 

  6. Zheng, C., Zhu, S., Mendieta, M., Yang, T., Chen, C., Ding, Z.: 3D human pose estimation with spatial and temporal transformers. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11656–11665 (2021)

    Google Scholar 

  7. Wang, J., Yan, S., Xiong, Y., Lin, D.: Motion guided 3D pose estimation from videos. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12358, pp. 764–780. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58601-0_45

    Chapter  Google Scholar 

  8. Tekin, B., Márquez-Neila, P., Salzmann, M., Fua, P.: Learning to fuse 2D and 3D image cues for monocular body pose estimation. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3941–3950 (2017)

    Google Scholar 

  9. Qammaz, A., Argyros, A.A.: MocapNET: ensemble of SNN encoders for 3D human pose estimation in RGB images. In: BMVC, p. 46 (2019)

    Google Scholar 

  10. Li, W., Liu, H., Tang, H., Wang, P., Van Gool, L.: MHFormer: multi-hypothesis transformer for 3D human pose estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13147–13156 (2022)

    Google Scholar 

  11. Ge, L., et al.: 3D hand shape and pose estimation from a single RGB image. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10833–10842 (2019)

    Google Scholar 

  12. Zeng, A., Sun, X., Yang, L., Zhao, N., Liu, M., Xu, Q.: Learning skeletal graph neural networks for hard 3D pose estimation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11436–11445 (2021)

    Google Scholar 

  13. Pullen, K., Bregler, C.: Motion capture assisted animation: texturing and synthesis. In: Proceedings of the 29th Annual Conference on Computer Graphics and Interactive Techniques, pp. 501–508 (2002)

    Google Scholar 

  14. Wei, Y.: Deep-learning-based motion capture technology in film and television animation production. Secur. Commun. Netw. 2022 (2022)

    Google Scholar 

  15. Yu, X., et al.: Bayesian estimation of human impedance and motion intention for human-robot collaboration. IEEE Trans. Cybernet. 51(4), 1822–1834 (2019)

    Article  Google Scholar 

  16. Gao, Q., Liu, J., Ju, Z., Zhang, X.: Dual-hand detection for human-robot interaction by a parallel network based on hand detection and body pose estimation. IEEE Trans. Industr. Electron. 66(12), 9663–9672 (2019)

    Article  Google Scholar 

  17. Koppula, H.S., Saxena, A.: Anticipating human activities using object affordances for reactive robotic response. IEEE Trans. Pattern Anal. Mach. Intell. 38(1), 14–29 (2015)

    Article  Google Scholar 

  18. Gao, Q., Liu, J., Ju, Z.: Robust real-time hand detection and localization for space human-robot interaction based on deep learning. Neurocomputing 390, 198–206 (2020)

    Article  Google Scholar 

  19. Cheng, Y., Yi, P., Liu, R., Dong, J., Zhou, D., Zhang, Q.: Human-robot interaction method combining human pose estimation and motion intention recognition. In: 2021 IEEE 24th International Conference on Computer Supported Cooperative Work in Design (CSCWD), pp. 958–963. IEEE (2021)

    Google Scholar 

  20. Gao, Q., Liu, J., Ju, Z.: Hand gesture recognition using multimodal data fusion and multiscale parallel convolutional neural network for human-robot interaction. Expert. Syst. 38(5), e12490 (2021)

    Article  Google Scholar 

  21. Zhao, W., Wang, W., Tian, Y.: Graformer: graph-oriented transformer for 3D pose estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20438–20447 (2022)

    Google Scholar 

  22. Kipf, T.N., Welling, M.: Semi-supervised classification with graph convolutional networks. arXiv preprint: arXiv:1609.02907 (2016)

  23. Ionescu, C., Papava, D., Olaru, V., Sminchisescu, C.: Human3. 6m: large scale datasets and predictive methods for 3D human sensing in natural environments. IEEE Trans. Pattern Anal. Mach. Intell. 36(7), 1325–1339 (2013)

    Article  Google Scholar 

  24. Lutz, S., Blythman, R., Ghosal, K., Moynihan, M., Simms, C., Smolic, A.: Jointformer: single-frame lifting transformer with error prediction and refinement for 3d human pose estimation. In: 2022 26th International Conference on Pattern Recognition (ICPR), pp. 1156–1163. IEEE (2022)

    Google Scholar 

  25. Cai, Y., et al.: Exploiting spatial-temporal relationships for 3D pose estimation via graph convolutional networks. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2272–2281 (2019)

    Google Scholar 

  26. Lin, H., Chiu, Y., Wu, P.: AMPose: alternately mixed global-local attention model for 3D human pose estimation. In: ICASSP 2023–2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1–5. IEEE (2023)

    Google Scholar 

  27. Zhai, K., Nie, Q., Ouyang, B., Li, X., Yang, S.: HopFIR: hop-wise GraphFormer with intragroup joint refinement for 3D human pose estimation. arXiv preprint: arXiv:2302.14581 (2023)

  28. Redmon, J., Farhadi, A.: YOLOV3: an incremental improvement. arXiv preprint: arXiv:1804.02767 (2018)

  29. Fang, H.S., et al.: AlphaPose: whole-body regional multi-person pose estimation and tracking in real-time. IEEE Trans. Pattern Anal. Mach. Intell. (2022)

    Google Scholar 

Download references

Acknowledgement

This work was supported in part by the National Natural Science Foundation of China (62006204, 52075530), the Guangdong Basic and Applied Basic Research Foundation (2022A1515011431), and Shenzhen Science and Technology Program (RCBS20210609104516043, JSGG20210802154004014). This work is also partially supported by the AiBle project co-financed by the European Regional Development Fund.

Author information

Authors and Affiliations

  1. School of Communication and Information Engineering, Chongqing University of Posts and Telecommunications, Chongqing, 400065, China

    Rongtian Huo & Jing Qi

  2. School of Electronics and Communication Engineering, Shenzhen Campus of Sun Yat-sen University, Shenzhen, 518107, China

    Qing Gao

  3. School of Computing, University of Portsmouth, Portsmouth, PO13HE, UK

    Zhaojie Ju

Authors
  1. Rongtian Huo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qing Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jing Qi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zhaojie Ju
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Qing Gao .

Editor information

Editors and Affiliations

  1. Zhejiang University, Hangzhou, China

    Huayong Yang

  2. Harbin Institute of Technology, Shenzhen, China

    Honghai Liu

  3. Zhejiang University, Hangzhou, China

    Jun Zou

  4. Huazhong University of Science and Technology, Wuhan, China

    Zhouping Yin

  5. Shenyang Institute of Automation, Shenyang, Liaoning, China

    Lianqing Liu

  6. Zhejiang University, Hangzhou, China

    Geng Yang

  7. Zhejiang University, Hangzhou, China

    Xiaoping Ouyang

  8. Harbin Institute of Technology, Shenzhen, China

    Zhiyong Wang

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Huo, R., Gao, Q., Qi, J., Ju, Z. (2023). 3D Human Pose Estimation in Video for Human-Computer/Robot Interaction. In: Yang, H., et al. Intelligent Robotics and Applications. ICIRA 2023. Lecture Notes in Computer Science(), vol 14273. Springer, Singapore. https://doi.org/10.1007/978-981-99-6498-7_16

Download citation

  • DOI: https://doi.org/10.1007/978-981-99-6498-7_16

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-99-6497-0

  • Online ISBN: 978-981-99-6498-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation