Skip to main content
SpringerLink
Log in

Challenges and Outlook to Designing Cutting-Edge Mixed Reality Technologies of Human Pose Estimation

  • Conference paper
  • First Online:
MEDICON’23 and CMBEBIH’23 (MEDICON 2023, CMBEBIH 2023)

Part of the book series: IFMBE Proceedings ((IFMBE,volume 93))

  • 151 Accesses

Abstract

Algorithms for real-time tracking of human motion allows projects to be economical and applicable in clinics although less accurate than the gold standard VICON system. Besides, these systems may implement applications more user-friendly and without specific skills needed so clinicians and elderly could use them. The integration of this category of systems with Virtual (VR) or Mixed Reality (MR) will support diagnostic and follow up protocol in telemedicine and teleconsulting, even more for patients with specific diseases that affect movement and joint function which request specific rehabilitation protocol. Student training could benefit from these applications through the interaction with upper and lower limbs and visualization of how diseases affect anatomical movements. The goal is to obtain a comprehensive framework for camera-based algorithms of human pose estimation currently used in the medical field, especially in telerehabilitation. The suggested framework comprises 3 layers: the first layer considers pros and cons (limits) of using; the second focuses on possibility in Unity project implementation and the third layer is the integration in VR or MR. This breakthrough can be considered as a proper resource for the future development of a complete and efficient project in VR or MR choosing the suitable tool to integrate human tracking algorithms in headsets without the use of external webcams or devices.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 229.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 299.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. VisAI Lab: An overview of people/human tracking methodologies, April 27 2021, https://visailabs.com/an-overview-of-people-human-tracking-methodologies/, last accessed 2023/02/25

  2. Das, R., M., A, R.R.: A review on human motion capture. SSRN Electron. J. 151–157 (2021)

    Google Scholar 

  3. Patrona, F., Chatzitofis, A., Zarpalas, D., Daras, P.: Motion analysis: Action detection, recognition and evaluation based on motion capture data. Pattern Recognit. 76, 612–622 (2018)

    Article  Google Scholar 

  4. Merriaux, P., Dupuis, Y., Boutteau, R., Vasseur, P., Savatier, X.: A study of Vicon system positioning performance. 1–18 (2017)

    Google Scholar 

  5. Zago, M., Luzzago, M., Marangoni, T., De Cecco, M., Tarabini, M., Galli, M.: 3D Tracking of human motion using visual skeletonization and stereoscopic vision. Front. Bioeng. Biotechnol. 8, 1–11 (2020)

    Article  Google Scholar 

  6. Pirovano, M., Surer, E., Mainetti, R., Lanzi, P.L., Alberto Borghese, N.: Exergaming and rehabilitation: A methodology for the design of effective and safe therapeutic exergames. Entertain. Comput. 14, 55–65 (2016)

    Article  Google Scholar 

  7. Rosique, F., Losilla, F., Navarro, P.J.: Applying vision‐based pose estimation in a telerehabilitation application. Appl. Sci. 11 (2021)

    Google Scholar 

  8. Franzo’, M., Pascucci, S., Serrao, M., Marinozzi, F., Bini, F.: Kinect-based wearable prototype system for ataxic patients neurorehabilitation: Control group preliminary results. IEEE Med. Meas. Appl. MeMeA 2020—Conf. Proc. (2020)

    Google Scholar 

  9. Franzo’, M., Pascucci, S., Serrao, M., Marinozzi, F., Bini, F.: Kinect-based wearable prototype system for ataxic patients neurorehabilitation: Software update for exergaming and rehabilitation. 2021 IEEE Int. Symp. Med. Meas. Appl. MeMeA 2021—Conf. Proc. 0–5 (2021)

    Google Scholar 

  10. Ayvat, E., et al.: The effects of exergame on postural control in individuals with ataxia: A rater-blinded, randomized controlled, cross-over study. Cerebellum 21, 64–72 (2022)

    Article  Google Scholar 

  11. Berthold, D.P., et al.: Head-mounted display virtual reality is effective in orthopaedic training: A systematic review. Arthrosc. Sport. Med. Rehabil. 4, e1843–e1849 (2022)

    Google Scholar 

  12. Franzo’, M., Pascucci, S., Serrao, M., Marinozzi, F., Bini, F.: Exergaming in mixed reality for the rehabilitation of ataxic patients. 2022 IEEE Int. Symp. Med. Meas. Appl. MeMeA 2022—Conf. Proc. 1–5 (2022)

    Google Scholar 

  13. Franzò, M., Pica, A., Pascucci, S., Serrao, M., Marinozzi, F., Bini, F.: A proof of concept combined using mixed reality for personalized neurorehabilitation of cerebellar ataxic patients. Sensors (2023)

    Google Scholar 

  14. Franzò, M., Pica, A., Pascucci, S., Marinozzi, F., Bini, F.: Hybrid system mixed reality and marker-less motion tracking for sports rehabilitation of martial arts athletes. Appl. Sci. 13, 2587 (2023)

    Article  Google Scholar 

  15. Ansari, F.J.: A review of motion tracking algorithms. Int. J. Latest Trends Eng. Technol. 9, 100–108 (2018)

    Google Scholar 

  16. Catic, A., Gurbeta, L., Kurtovic-Kozaric, A., Mehmedbasic, S., Badnjevic, A.: Application of Neural Networks for classification of Patau, Edwards, Down, Turner and Klinefelter Syndrome based on first trimester maternal serum screening data, ultrasonographic findings and patient demographics. BMC Med. Genomics 11, 19 (2018)

    Article  Google Scholar 

  17. Sarić, R., Jokić, D., Beganović, N., Pokvić, L., Badnjević, A.: FPGA-based real-time epileptic seizure classification using Artificial Neural Network. Biomed. Signal Process. Control 62, 102106 (2020)

    Article  Google Scholar 

  18. Zhou, H., Hu, H.: Human motion tracking for rehabilitation-A survey. Biomed. Signal Process. Control 3, 1–18 (2008)

    Article  Google Scholar 

  19. Badiola-Bengoa, A., Mendez-Zorrilla, A.: A systematic review of the application of camera-based human pose estimation in the field of sport and physical exercise. Sensors 21 (2021)

    Google Scholar 

  20. Weiss, P.L., Rand, D., Katz, N., Kizony, R.: Video capture virtual reality as a flexible and effective rehabilitation tool. J. Neuroeng. Rehabil. 1, 1–12 (2004)

    Article  Google Scholar 

  21. Guo, S., Wang, S., Yang, Z., Wang, L., Zhang, H., Guo, P.: Applied sciences a review of deep learning-based visual multi-object tracking algorithms for autonomous driving deal with many objects and attempt to address the multi-object sors, how to identify multiple objects in each frame of data and a. (2022)

    Google Scholar 

  22. Hou, L., Wan, W., Hwang, J.N., Muhammad, R., Yang, M., Han, K.: Human tracking over camera networks: A review. EURASIP J. Adv. Signal Process. 2017 (2017)

    Google Scholar 

  23. Moeslund, T.B., Hilton, A., Krüger, V.: A survey of advances in vision-based human motion capture and analysis. Comput. Vis. Image Underst. 104, 90–126 (2006)

    Article  Google Scholar 

  24. You, S., Zhu, H., Li, M., Li, Y.: A Review of Visual Trackers and Analysis of its Application to Mobile Robot. (2019)

    Google Scholar 

  25. Kour, N., Sunanda, Arora, S.: Computer-vision based diagnosis of Parkinson’s disease via gait: A survey. IEEE Access 7, 156620–156645 (2019)

    Google Scholar 

  26. Lee, J., Ahn, B.: Low-cost RGB camera and mobile robot platform. Sensors (Switzerland) 20, 1–12 (2020)

    Google Scholar 

  27. Meily, S.O., Putra, I.K.G.D., Wira Buana, P.: Augmented reality using real objects tracking development. J. Inf. Syst. 17, 20–29 (2021)

    Google Scholar 

  28. Panarese, A., Colombo, R., Sterpi, I., Pisano, F., Micera, S.: Tracking motor improvement at the subtask level during robot-aided neurorehabilitation of stroke patients. Neurorehabil. Neural Repair 26, 822–833 (2012)

    Article  Google Scholar 

  29. de-la-Torre, R., Oña, E.D., Balaguer, C., Jardón, A.: Robot-aided systems for improving the assessment of upper limb spasticity: A systematic review. Sensors 20, 5251 (2020)

    Google Scholar 

  30. Cai, Z., Fernando, O.N.N., Ong, J.Y.: PoseBuddy : Pose estimation workout mobile application. Proc.—2022 Int. Conf. Cyberworlds, CW 2022. 151–154 (2022)

    Google Scholar 

  31. De Feudis, I., Buongiorno, D., Grossi, S., Losito, G., Brunetti, A., Longo, N., Di Stefano, G., Bevilacqua, V.: Evaluation of vision-based hand tool tracking methods for quality assessment and training in human-centered industry 4.0. Appl. Sci. 12, 1–19 (2022)

    Google Scholar 

  32. Liao, J., Karim, A., Jadon, S.S., Kazi, R.H., Suzuki, R.: RealityTalk: Real-Time Speech-Driven Augmented Presentation for AR Live Storytelling. Presented at the October 29 (2022)

    Google Scholar 

  33. Tsai, Y.S., Chen, S.J.: A Study on the application of walking posture for identifying persons with gait recognition. Appl. Sci. 12 (2022)

    Google Scholar 

  34. Xu, X., Gao, Y., Yan, K., Lin, X., Zou, Q.: Location-Free Human Pose Estimation. Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. 2022-June, 13127–13136 (2022)

    Google Scholar 

  35. Ye, R.Z., Subramanian, A., Diedrich, D., Lindroth, H., Pickering, B., Herasevich, V.: Effects of Image Quality on the Accuracy Human Pose Estimation and Detection of Eye Lid Opening/Closing Using Openpose and DLib. (2022)

    Google Scholar 

  36. Büker, L.C., Zuber, F., Hein, A., Fudickar, S.: Hrdepthnet: Depth image-based marker-less tracking of body joints. Sensors (Switzerland) 21, 1–17 (2021)

    Article  Google Scholar 

  37. Cao, Z., Hidalgo, G., Simon, T., Wei, S.E., Sheikh, Y.: OpenPose: Realtime multi-person 2D pose estimation using part affinity fields. IEEE Trans. Pattern Anal. Mach. Intell. 43, 172–186 (2021)

    Article  Google Scholar 

  38. Che, Y., Qi, Y.: Detection-guided 3D hand tracking for mobile AR applications. In: Proceedings—2021 IEEE International Symposium on Mixed and Augmented Reality, ISMAR 2021. pp. 386–392. Institute of Electrical and Electronics Engineers Inc. (2021)

    Google Scholar 

  39. Hernández, Ó.G., Morell, V., Ramon, J.L., Jara, C.A.: Human pose detection for robotic-assisted and rehabilitation environments. Appl. Sci. 11, (2021)

    Google Scholar 

  40. Jan, Y.F., Tseng, K.W., Kao, P.Y., Hung, Y.P.: Augmented Tai-Chi Chuan Practice Tool with Pose Evaluation. In: Proceedings—4th International Conference on Multimedia Information Processing and Retrieval, MIPR 2021. pp. 35–41. Institute of Electrical and Electronics Engineers Inc. (2021)

    Google Scholar 

  41. McGuirk, C.J.C., Baddour, N., Lemaire, E.D.: Video-based deep learning approach for 3D human movement analysis in institutional hallways: a smart hallway. Computation 9 (2021)

    Google Scholar 

  42. Yamao, K., Kubota, R.: Development of Human Pose Recognition System by Using Raspberry Pi and PoseNet Model. Proc. Isc. 2021 2021 20th Int. Symp. Commun. Inf. Technol. Quest Qual. Life Smart City 41–44 (2021)

    Google Scholar 

  43. Zhang, Y., Tian, Y., Wu, P., Chen, D.: Application of skeleton data and long short-term memory in action recognition of children with autism spectrum disorder. Sensors (Switzerland) 21, 1–17 (2021)

    Google Scholar 

  44. Choi, S., Yun, S., Ahn, B.: Implementation of automated baby monitoring: CCBeBe. Sustain. 12 (2020)

    Google Scholar 

  45. Wang, X., Wang, Y., Shi, Y., Zhang, W., Zheng, Q.: AvatarMeeting: An Augmented Reality Remote Interaction System with Personalized Avatars. In: MM 2020—Proceedings of the 28th ACM International Conference on Multimedia. pp. 4533–4535. Association for Computing Machinery, Inc. (2020)

    Google Scholar 

  46. Xiong, X., Min, W., Zheng, W.S., Liao, P., Yang, H., Wang, S.: S3D-CNN: Skeleton-based 3D consecutive-low-pooling neural network for fall detection. Appl. Intell. 50, 3521–3534 (2020)

    Article  Google Scholar 

  47. Zhang, F., Bazarevsky, V., Vakunov, A., Tkachenka, A., Sung, G., Chang, C.-L., Grundmann, M.: MediaPipe Hands: On-device Real-time Hand Tracking. (2020)

    Google Scholar 

  48. Fieraru, M., Khoreva, A., Pishchulin, L., Schiele, B.: Learning to refine human pose estimation. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. Work. 2018-June, 318–327 (2018)

    Google Scholar 

  49. Hwang, D.H., Koike, H.: ParaPara: Synthesizing pseudo-2.5D content from monocular videos for mixed reality. In: Conference on Human Factors in Computing Systems—Proceedings. Association for Computing Machinery (2018)

    Google Scholar 

  50. Chang, J.Y., Moon, G., Lee, K.M.: V2V-PoseNet: Voxel-to-Voxel Prediction Network for Accurate 3D Hand and Human Pose Estimation from a Single Depth Map. Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. 5079–5088 (2018)

    Google Scholar 

  51. Simon, T., Joo, H., Matthews, I., Sheikh, Y.: Hand keypoint detection in single images using multiview bootstrapping. Proc. - 30th IEEE Conf. Comput. Vis. Pattern Recognition, CVPR 2017. 2017-Janua, 4645–4653 (2017)

    Google Scholar 

  52. Viso.ai The Complete Guide to OpenPose in 2023, https://viso.ai/deep-learning/openpose/, last accessed 2023/02/25

  53. GitHub OpenPose's Unity Plugin for Unity users, https://github.com/CMU-Perceptual-Computing-Lab/openpose_unity_plugin, last accessed 2023/02/25

  54. Analytics Vidhya, Posture Detection using PoseNet with Real-time Deep Learning project, Raghav Agrawal Published On September 5, 2021, https://www.analyticsvidhya.com/blog/2021/09/posture-detection-using-posenet-with-real-time-deep-learning-project/, last accessed 2023/02/25

  55. TensorFlow Blog, Real-time Human Pose Estimation in the Browser with TensorFlow.js, Dan Oved, May 07, 2018, https://blog.tensorflow.org/2018/05/real-time-human-pose-estimation-in.html, last accessed 2023/02/25

  56. Toshpulatov, M., Lee, W., Lee, S., Roudsari, A.H.: Human pose, hand and mesh estimation using deep learning: A survey. J. Supercomput. 78, 7616–7654 (2022)

    Article  Google Scholar 

  57. Papandreou, G., Zhu, T., Chen, L.C., Gidaris, S., Tompson, J., Murphy, K.: Personlab: Person pose estimation and instance segmentation with a bottom-up, part-based, geometric embedding model. Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics). 11218 LNCS, 282–299 (2018)

    Google Scholar 

  58. TensorFlow Blog, Next-Generation Pose Detection with MoveNet and TensorFlow.js, Ronny Votel and Na Li, May 17, 2021, https://blog.tensorflow.org/2021/05/next-generation-pose-detection-with-movenet-and-tensorflowjs.html, last accessed 2023/02/25

  59. Medium, Yoga Pose Classification With TensorFlow’s MoveNet Model, Leah Nagy, Aug 12, 2022, https://medium.com/@leahnagy/yoga-pose-classification-with-tensorflows-movenet-model-3e5771fda292, last accessed 2023/02/25

    Google Scholar 

  60. GitHub, mediapipe, https://github.com/google/mediapipe, last accessed 2023/02/25

  61. MediaPipe Homepage, https://mediapipe.dev/, last accessed 2023/02/25

  62. New Stream, Body Motion Analysis Mobile SDK, https://www.newstream.ai/products-skeleton-detection-sdk, last accessed 2023/02/25

  63. DollarsMoCap Homepage, https://www.dollarsmocap.com/, last accessed 2023/02/25

  64. Winkler, A., Won, J., Ye, Y.: QuestSim: Human Motion Tracking from Sparse Sensors with Simulated Avatars. Proc.—SIGGRAPH Asia 2022 Conf. Pap. 1, (2022)

    Google Scholar 

  65. Romano, A., Favetta, M., Summa, S., Schirinzi, T., Bertini, E.S., Castelli, E., Vasco, G., Petrarca, M.: Upper Body Physical Rehabilitation for Children with Ataxia through IMU-Based Exergame. J. Clin. Med. 11, (2022)

    Google Scholar 

  66. Petri, K., et al.: Training using virtual reality improves response behavior in karate kumite. Sport. Eng. 22, 1–12 (2019)

    Article  Google Scholar 

  67. Petrigna, L., Musumeci, G.: The metaverse: A new challenge for the healthcare system: A scoping review. J. Funct. Morphol. Kinesiol. 7(3), 63 (2022)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Industrial Bioengineering Lab, Department of Mechanical and Aerospace Engineering, Sapienza University of Rome, Rome, Italy

    Michela Franzò, Simona Pascucci, Franco Marinozzi & Fabiano Bini

  2. Scientific Secretary of the Presidency, Italian National Institute of Health, Rome, Italy

    Simona Pascucci

Authors
  1. Michela Franzò
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Simona Pascucci
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Franco Marinozzi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Fabiano Bini
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Simona Pascucci .

Editor information

Editors and Affiliations

  1. Verlab Research Institute for Biomedical Engineering, Medical Devices, and Artificial Intelligence, Sarajevo, Bosnia and Herzegovina

    Almir Badnjević

  2. Verlab Research Institute for Biomedical Engineering, Medical Devices, and Artificial Intelligence, Sarajevo, Bosnia and Herzegovina

    Lejla Gurbeta Pokvić

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Franzò, M., Pascucci, S., Marinozzi, F., Bini, F. (2024). Challenges and Outlook to Designing Cutting-Edge Mixed Reality Technologies of Human Pose Estimation. In: Badnjević, A., Gurbeta Pokvić, L. (eds) MEDICON’23 and CMBEBIH’23. MEDICON CMBEBIH 2023 2023. IFMBE Proceedings, vol 93. Springer, Cham. https://doi.org/10.1007/978-3-031-49062-0_78

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-49062-0_78

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-49061-3

  • Online ISBN: 978-3-031-49062-0

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation