Springer Nature is making SARS-CoV-2 and COVID-19 research free. View research | View latest news | Sign up for updates

Heterogeneous Multi-task Learning for Human Pose Estimation with Deep Convolutional Neural Network

Abstract

We propose a heterogeneous multi-task learning framework for human pose estimation from monocular images using a deep convolutional neural network. In particular, we simultaneously learn a human pose regressor and sliding-window body-part and joint-point detectors in a deep network architecture. We show that including the detection tasks helps to regularize the network, directing it to converge to a good solution. We report competitive and state-of-art results on several datasets. We also empirically show that the learned neurons in the middle layer of our network are tuned to localized body parts.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Notes

  1. 1.

    The full results can be viewed at http://visal.cs.cityu.edu.hk/research/hmlpe-demo/.

  2. 2.

    As pointed out in (Hara and Chellappa 2013; Pishchulin et al. 2012), the code in the Buffy toolkit does not compute PCP correctly.

  3. 3.

    Since we have different definitions of torso and head parts, we do not show the evaluation of these parts here.

References

  1. Bo, L., & Sminchisescu, C. (2010). Twin gaussian processes for structured prediction. International Journal of Computer Vision, 87(1–2), 28–52.

  2. Dalal, N., & Triggs, B. (2005) Histograms of oriented gradients for human detection. In: IEEE Conference on Computer Vision and Pattern Recognition.

  3. Dantone, M., Gall, J., Leistner, C., & van Gool L. (2013) Human pose estimation from still images using body parts dependent joint regressors. In: IEEE Conference on Computer Vision and Pattern Recognition.

  4. Eichner, M., & Ferrari, V. (2009a) Better appearance models for pictorial structures. In: British Machine Vision Conference, pp 1–11.

  5. Eichner, M., & Ferrari, V. (2009b) Upper body detector. http://groups.inf.ed.ac.uk/calvin/calvin_upperbody_detector/

  6. Eichner, M., & Ferrari, V. (2010) We are family: Joint pose estimation of multiple persons. In: European Conference.on Computer Vision.

  7. Eichner, M., & Ferrari, V. (2012). Human pose co-estimation and applications. IEEE Trans Pattern Anal Mach Intell.

  8. Eichner, M., Marin-Jimenez, M., Zisserman, A., & Ferrari, V. (2012). 2d articulated human pose estimation and retrieval in (almost) unconstrained still images. International Journal of Computer Vision, 99(2), 190–214.

  9. Evgeniou, T., Micchelli, C. A., & Pontil, M. (2005). Learning multiple tasks with kernel methods. Journal of Machine Learning Research, 6, 615–637.

  10. Farabet, C., Couprie, C., Najman, L., & LeCun, Y. (2013). Learning hierarchical features for scene labeling. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8), 1915–1929.

  11. Felzenszwalb, P. F., & Huttenlocher, D. P. (2005). Pictorial structures for object recognition. International Journal of Computer Vision, 61(1), 55–79.

  12. Gülçehrem, C., & Bengio, Y. (2013) Knowledge matters: Importance of prior information for optimization. In: International Conference on Learning Representations.

  13. Hara, K., & Chellappa, R. (2013) Computationally efficient regression on a dependency graph for human pose estimation. In: IEEE Conference on Computer Vision and Pattern Recognition.

  14. Jain, A., Tompson, J., Andriluka, M., Taylor, G. W., & Bregler, C. (2014) Learning human pose estimation features with convolutional networks. In: International Conference on Learning Representations.

  15. Johnson, S., & Everingham, M. (2011) Learning effective human pose estimation from inaccurate annotation. In: IEEE Conference on Computer Vision and Pattern Recognition.

  16. Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012) Imagenet classification with deep convolutional neural networks. In: Neural Information Processing Systems.

  17. Le, Q., Ranzato, M., Monga, R., Devin, M., Chen, K., Corrado, G., Dean, J., & Ng, A. (2012) Building high-level features using large scale unsupervised learning. In: International Conference on Machine Learning.

  18. van der Maaten, L., & Hinton, G. (2008). Visualizing Data using t-SNE. Journal of Machine Learning Research, 9, 2579–2605.

  19. Nair, V., & Hinton, G. E. (2010) Rectified linear units improve restricted boltzmann machines. In: International Conference on Machine Learning.

  20. Pishchulin, L., Jain, A., Andriluka, M., Thormaehlen, T., & Schiele, B. (2012) Articulated people detection and pose estimation: Reshaping the future. In: IEEE Conference on Computer Vision and Pattern Recognition.

  21. Pishchulin, L., Andriluka, M., Gehler, P., & Schiele, B. (2013) Poselet conditioned pictorial structures. In: IEEE Conference on Computer Vision and Pattern Recognition, pp 588–595.

  22. Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1988). Learning representations by back-propagating errors. In J. A. Anderson & E. Rosenfeld (Eds.), Neurocomputing: Foundations of research (pp. 696–699). Cambridge, MA: MIT Press.

  23. Sapp, B., & Taskar, B. (2013) Modec: Multimodal decomposable models for human pose estimation. In: IEEE Conference on Computer Vision and Pattern Recognition.

  24. Sapp, B., Toshev, A., & Taskar, B. (2010) Cascaded models for articulated pose estimation. In: European Conference on Computer Vision.

  25. Shotton, J., Fitzgibbon, A., Cook, M., Sharp, T., Finocchio, M., Moore, R., Kipman, A., & Blake, A. (2011) Real-time human pose recognition in parts from single depth images. In: IEEE Conference on Computer Vision and Pattern Recognition.

  26. Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. (2014). Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning, 15, 1929–1958.

  27. Sun, Y., Wang, X., & Tang, X. (2013) Deep convolutional network cascade for facial point detection. In: IEEE Conference on Computer Vision and Pattern Recognition.

  28. Toshev, A., & Szegedy, C. (2014) Deeppose: Human pose estimation via deep neural networks. In: IEEE Conference on Computer Vision and Pattern Recognition.

  29. Weston, J., Ratle, F., & Collobert, R. (2008) Deep learning via semi-supervised embedding. In: International Conference on Machine Learning.

  30. Yang, X., Kim, S., & Xing, E. P. (2009) Heterogeneous multitask learning with joint sparsity constraints. In: Neural Information Processing Systems.

  31. Yang, Y., & Ramanan, D. (2011) Articulated pose estimation with flexible mixtures-of-parts. In: IEEE Conference on Computer Vision and Pattern Recognition.

  32. Yang, Y., & Ramanan, D. (2013). Articulated human detection with flexible mixtures of parts. IEEE Trans Pattern Analysis and Machine Intelligence, 35(12), 2878–2890.

  33. Yu, K., Tresp, V., & Schwaighofer, A. (2005) Learning gaussian processes from multiple tasks. In: International Conference on Machine Learning, pp 1012–1019.

  34. Zeiler, M. D., & Fergus, R. (2014). Visualizing and understanding convolutional networks. In Computer Vision – ECCV 2014. Lecture Notes in Computer Science (Vol. 8689, pp. 818–833). Springer.

Download references

Acknowledgments

A.B.C. was supported by the Research Grants Council of the Hong Kong Special Administrative Region, China (CityU 123212 and CityU 110513). This work was supported by the Research Grants Council of the Hong Kong Special Administrative Region, China, under GRF 9041574 (CityU 118810), GRF 9041905 (CityU 119313).

Author information

Correspondence to Sijin Li.

Additional information

Communicated by Marc’Aurelio Ranzato, Geoffrey E. Hinton, and Yann Lecun.

Appendix

Appendix

See Figure 12.

Fig. 12
figure12figure12figure12

Visualization of patches that caused maximum activation in level-3 feature maps

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Li, S., Liu, Z. & Chan, A.B. Heterogeneous Multi-task Learning for Human Pose Estimation with Deep Convolutional Neural Network. Int J Comput Vis 113, 19–36 (2015). https://doi.org/10.1007/s11263-014-0767-8

Download citation

Keywords

  • Human Pose Estimation
  • Deep Learning