Advertisement

Multimedia Tools and Applications

, Volume 77, Issue 23, pp 31199–31220 | Cite as

Real-time dance evaluation by markerless human pose estimation

  • Yeonho Kim
  • Daijin Kim
Article
  • 129 Downloads

Abstract

This paper presents a unified framework that evaluates dance performance by markerless estimation of human poses. Dance involves complicated poses such as full-body rotation and self-occlusion, so we first develop a human pose estimation method that is invariant to these factors. The method uses ridge data and data pruning. Then we propose a metric to quantify the similarity (i.e., timing and accuracy) between two dance sequences. To validate the proposed dance evaluation method, we conducted several experiments to evaluate pose estimation and dance performance on the benchmark dataset EVAL, SMMC-10 and a large K-Pop dance database, respectively. The proposed methods achieved pose estimation accuracy of 0.9358 mAP, average pose error of 3.88 cm, and 98% concordance with experts’ evaluation of dance performance.

Keywords

Human pose estimation Dance performance evaluation 

Notes

Acknowledgments

This research was partially supported by the MSIT (Ministry of Science, ICT), Korea, under the SW Starlab support program (IITP-2017-0-00897) supervised by the IITP (Institute for Information & communications Technology Promotion).

This work was partially supported by Institute for Information & communications Technology Promotion (IITP) grant funded by the Korea government (MSIT) (IITP-2014-0-00059, Development of Predictive Visual Intelligence Technology).

Supplementary material

11042_2018_6068_MOESM1_ESM.wmv (6.8 mb)
(WMV 6.81 MB)

References

  1. 1.
    Alexander T, Szegedy C (2014) Deeppose: Human pose estimation via deep neural networks. In: The IEEE conference on computer vision and pattern recognition, pp 1653–1660Google Scholar
  2. 2.
    Baak A, Müller M, Bharaj G, Seidel H-P, Theobalt C (2011) A data-driven approach for real-time full body pose reconstruction from a depth camera. In: IEEE international conference on computer vision, pp 1092–1099Google Scholar
  3. 3.
    Cao Z, Simon T, Wei S, Sheikh Y (2017) Realtime multi-person 2D pose estimation using part affinity fields. In: The IEEE conference on computer vision and pattern recognitionGoogle Scholar
  4. 4.
    Comaniciu D, Ramesh V, Meer P (2003) Kernel-based object tracking. IEEE Trans Pattern Anal Mach Intell 25(5):564–577CrossRefGoogle Scholar
  5. 5.
    Dillencourt M, Samet H, Tamminen M (1992) A general approach to connected-component labeling for arbitrary image representations. J ACM 39(2):253–280MathSciNetCrossRefGoogle Scholar
  6. 6.
    Ganapathi V, Plagemann C, Koller D, Thrun S (2010) Real time motion capture using a single time-of-flight camera. In: IEEE conference on computer vision and pattern recognition, pp 755–762Google Scholar
  7. 7.
    Ganapathi V, Plagemann C, Koller D, Thrun S (2012) Real-time human pose tracking from range data. In: European conference on computer vision, pp 738–751CrossRefGoogle Scholar
  8. 8.
    Hough P (1959) Machine analysis of bubble chamber pictures. In: International conference on high energy accelerators and instrumentation, pp 73Google Scholar
  9. 9.
    Huang J, Altamar D (2016) Pose estimation on depth images with convolutional neural networkGoogle Scholar
  10. 10.
    Jalal A, Kamal S, Kim D (2014) A depth video sensor-based life-logging human activity recognition system for elderly care in smart indoor environments. Sensors 14(7):11735–11759CrossRefGoogle Scholar
  11. 11.
    Jalal A, Kamal S, Kim D (2015) Depth Silhouettes Context: A new robust feature for human tracking and activity recognition based on embedded HMMs. In: 12th international conference on ubiquitous robots and ambient intelligence, pp 294–299Google Scholar
  12. 12.
    Jung H, Lee S, Heo Y, Yun I (2015) Random tree walk toward instantaneous 3D human pose estimation. In: The IEEE conference on computer vision and pattern recognition, pp 2467–2474Google Scholar
  13. 13.
    Kim Y, Kim D (2015) Efficient body part tracking using ridge data and data pruning. In: IEEE-RAS 15th international conference on humanoid robots, pp 114–120Google Scholar
  14. 14.
    Lee M, Nevatia R (2009) Human pose tracking in monocular sequence using multilevel structured models. IEEE Trans Pattern Anal Mach Intell 31(1):27–38CrossRefGoogle Scholar
  15. 15.
    Ofli F, Kurillo G, Obdrzalek S, Bajcsy R, Jimison H, Pavel M (2015) Design and evaluation of an interactive exercise coaching system for older adults: lessons learned. J Biom Heath Inform 20(1):201–212CrossRefGoogle Scholar
  16. 16.
    Plagemann C, Ganapathi V, Koller D, Thrun S (2010) Real-time identification and localization of body parts from depth images. In: IEEE international conference on robotics and automation, pp 3108–3113Google Scholar
  17. 17.
    Raptis M, Kirovski D, Hoppe H (2011) Real-time classification of dance gestures from skeleton animation. In: The 2011 ACM SIGGRAPH/Eurographics symposium on computer animation 147–156Google Scholar
  18. 18.
    Reyes M, Dominguez G, Escalera S (2011) Featureweighting in dynamic timewarping for gesture recognition in depth data. In: IEEE international conference on computer vision workshops, pp 1182–1188Google Scholar
  19. 19.
    Schramm R, Jung C, Miranda E (2015) Dynamic time warping for music conducting gestures evaluation. IEEE Trans Multimedia 17(2):243–255CrossRefGoogle Scholar
  20. 20.
    Shotton J, Girshick R, Fitzgibbon A, Sharp T, Cook M, Finocchio M, Moore R, Kohli P, Criminisi A, Kipman A, Blake A (2013) Efficient human pose estimation from single depth images. IEEE Trans Pattern Anal Mach Intell 35 (12):2821–2840CrossRefGoogle Scholar
  21. 21.
    Sung J, Ponce C, Selman B, Saxena A (2011) Human Activity Detection from RGBD Images, plan, activity, and intent recognition, vol 64Google Scholar
  22. 22.
    Xia L, Chen C, Aggarwal J (2012) View invariant human action recognition using histograms of 3d joints. In: IEEE computer society conference on computer vision and pattern recognition workshops, pp 20–27Google Scholar
  23. 23.
    Yang X, Tian Y (2014) Effective 3d action recognition using eigenjoints. J Vis Commun Image Represent 25(1):2–11MathSciNetCrossRefGoogle Scholar
  24. 24.
    Ye M, Yang R (2014) Real-time simultaneous pose and shape estimation for articulated objects using a single depth camera. In: IEEE conference on computer vision and pattern recognition, pp 2345–2352Google Scholar
  25. 25.
    Yun K, Honorio J, Chattopadhyay D, Berg T, Samaras D (2012) Two-person interaction detection using body-pose features and multiple instance learning. In: IEEE computer society conference on computer vision and pattern recognition workshops, pp 28–35Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Computer Science and EngineeringPohang University of Science and TechnologyPohangSouth Korea

Personalised recommendations