Skip to main content
SpringerLink
Log in

Model-free non-rigid head pose tracking by joint shape and pose estimation

  • Special Issue Paper
  • Published:
Machine Vision and Applications Aims and scope Submit manuscript

Abstract

Head pose estimation under non-rigid face movement is particularly useful in applications relating to eye-gaze tracking in less constrained scenarios, where the user is allowed to move naturally during tracking. Existing vision-based head pose estimation methods often require accurate initialisation and tracking of specific facial landmarks, while methods that handle non-rigid face deformations typically necessitate a preliminary training phase prior to head pose estimation. In this paper, we propose a method to estimate the head pose in real-time from the trajectories of a set of feature points spread randomly over the face region, without requiring a training phase or model-fitting of specific facial features. Conversely, our method exploits the 3-dimensional shape of the surface of interest, recovered via shape and motion factorisation, in combination with Kalman and particle filtering to determine the contribution of each feature point to the estimation of head pose based on a variance measure. Quantitative and qualitative results reveal the capability of our method in handling non-rigid face movement without deterioration of the head pose estimation accuracy.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Agudo, A., Agapito, L., Calvo, B., Montiel, J.: Good vibrations: A modal analysis approach for sequential non-rigid structure from motion. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1558–1565 (2014)

  2. Asteriadis, S., Soufleros, D., Karpouzis, K., Kollias, S.: A natural head pose and eye gaze dataset. In: Proceedings of the International Workshop on Affective-Aware Virtual Agents and Social Robots (AFFINE ’09) (2009)

  3. Poelman, C., Kanade, T.: A paraperspective factorization method for shape and motion recovery. In: European Conference on Computer Vision, pp. 97–108 (1994)

  4. Tomasi, C., Kanade, T.: Detection and tracking of point features. Tech. Rep. CMU-CS-91-132, Carnegie Mellon University (1991)

  5. Agudo, A., Montiel, J., Agapito, L., Calvo, B.: Online dense non-rigid 3D shape and camera motion recovery. In: British Machine Vision Conference, pp. 1–12 (2014)

  6. Arulampalam, M., Maskell, S., Gordon, N., Clapp, T.: A tutorial on particle filters for online nonlinear/non-Gaussian Bayesian tracking. IEEE Trans. Signal Process. 50(2), 174–188 (2002)

    Article  Google Scholar 

  7. Ba, S., Odobez, J.: A probabilistic framework for joint head tracking and pose estimation. In: Proceedings of the 7th International Conference on Pattern Recognition, vol. 4, pp. 264–267 (2004)

  8. Bartoli, A., Gay-Bellile, V., Castellani, U., Peyras, J., Olsen, S., Sayd, P.: Coarse-to-fine low-rank structure-from-motion. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8 (2008)

  9. Boston university head pose dataset. http://www.cs.bu.edu/groups/ivc/HeadTracking/ (1999)

  10. Bregler, C., Hertzmann, A., Biermann, H.: Recovering non-rigid 3D shape from image streams. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 690–696 (2000)

  11. Bronte, S., Paladini, M., Bergasa, L., Agapito, L., Arroyo, R.: Real-time sequential model-based non-rigid SFM. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 1026–1031 (2014)

  12. Chen, C., Schonfeld, D.: A particle filtering framework for joint video tracking and pose estimation. IEEE Trans. Image Process. 19(6), 1625–1634 (2010)

    Article  MathSciNet  Google Scholar 

  13. Costeira, J., Kanade, T.: Multi-body factorization method for independently moving objects. Int. J. Comput. Vis. 29, 159–179 (1997)

    Article  Google Scholar 

  14. Cristina, S., Camilleri, K.: Model-free head pose estimation based on shape factorisation and particle filtering. In: International Conference on Computer Analysis of Images and Patterns, pp. 628–639 (2015)

  15. Fanelli, G., Gall, J., Van Gool, L.: Real time head pose estimation with random regression forests. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 617–624 (2011)

  16. Funes Mora, K., Monay, F., Odobez, J.: Eyediap database: data description and gaze tracking evaluation benchmarks. In: Idiap-RR Idiap-RR-08-2014, Idiap (2014)

  17. Gurbuz, S., Oztop, E., Inoue, N.: Model free head pose estimation using stereovision. In: Pattern Recognition, pp. 33–42 (2012)

  18. Hansen, D.W., Ji, Q.: In the eye of the beholder: a survey of models for eyes and gaze. IEEE Trans. Pattern Anal. Mach. Intell. 32(3), 478–500 (2010)

    Article  Google Scholar 

  19. Harris, C., Stephens, M.: A combined corner and edge detector. In: Alvey Vision Conference, pp. 147–151 (1988)

  20. Ho, H., Chellappa, R.: Automatic head pose estimation using randomly projected dense sift descriptors. In: Proceedings of the 19th IEEE International Conference on Image Processing, pp. 153–156 (2012)

  21. Kim, J., Kim, H., Park, R.: Head pose estimation using a coplanar face model for human computer interaction. In: Proceedings of the IEEE Conference on Consumer Electronics, pp. 560–561 (2014)

  22. Kwolek, B.: Model based facial pose tracking using a particle filtering. In: Proceedings of the Geometric Modeling and Imaging—New Trends, pp. 203–208 (2006)

  23. La Cascia, M., Sclaroff, S., Athitsos, V.: Fast, reliable head tracking under varying illumination: an approach based on registration of texture-mapped 3D models. IEEE Trans. Pattern Anal. Mach. Intell. 22(4), 322–336 (2000)

    Article  Google Scholar 

  24. Lu, L., Zhang, Z., Shum, H., Liu, Z., Chen, H.: Model- and exemplar-based robust head pose tracking under occlusion and varying expression. In: IEEE Workshop on Models versus Exemplars in Computer Vision, pp. 1–8 (2001)

  25. Morita, T., Kanade, T.: A sequential factorization method for recovering shape and motion from image streams. IEEE Trans. Pattern Anal. Mach. Intell. 19, 858–867 (1997)

  26. Murphy-Chutorian, E., Trivedi, M.: Head pose estimation in computer vision: a survey. IEEE Trans. Pattern Anal. Mach. Intell. 31(4), 607–626 (2009)

    Article  Google Scholar 

  27. Paladini, M., Bartoli, A., Agapito, L.: Sequential non-rigid structure-from-motion with the 3D-implicit low-rank shape model. In: European Conference on Computer Vision, pp. 15–28 (2010)

  28. Rougier, C., Meunier, J., St-Arnaud, A., Rousseau, J.: 3D head tracking for fall detection using a single-calibrated camera. Image Vis. Comput. 31, 246–254 (2013)

    Article  Google Scholar 

  29. Sapienza, M., Camilleri, K.: Fasthpe: a recipe for quick head pose estimation. Tech. Rep. TR-SCE-2011-01, University of Malta. https://www.um.edu.mt/library/oar/handle/123456789/859 (2011)

  30. Schmid, C., Mohr, R., Bauckhage, C.: Evaluation of interest point detectors. Int. J. Comput. Vis. 37(2), 151–172 (2000)

    Article  MATH  Google Scholar 

  31. Shi, J., Tomasi, C.: Good features to track. In: Proceedings of the 1994 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 593–600 (1994)

  32. Tomasi, C., Kanade, T.: Shape and motion from image streams under orthography: a factorization method. Int. J. Comput. Vis. 9(2), 137–154 (1992)

    Article  Google Scholar 

  33. Torresani, L., Hertzmann, A., Bregler, C.: Nonrigid structure-from-motion: estimating shape and motion with hierarchical priors. IEEE Trans. Pattern Anal. Mach. Intell. 30(5), 878–892 (2008)

    Article  Google Scholar 

  34. Tran, N., Ababsa, F., Charbit, M., Feldmar, J., Petrovska-Delacretaz, D., Chollet, G.: 3D face pose and animation tracking via eigen-decomposition based bayesian approach. In: International Symposium on Advances in Visual Computing, pp. 562–571 (2013)

  35. Tuytelaars, T., Mikolajczyk, K.: Local invariant feature detectors: a survey. Found. Trends Comput. Graph. Vis. 3(3), 177–280 (2007)

    Article  Google Scholar 

  36. Viola, P., Jones, M.: Rapid object detection using a boosted cascade of simple features. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), vol. 1, pp. 511–518 (2001)

  37. Wang, G., Wu, Q.M.J.: Introduction to structure and motion factorization. In: Advances in Pattern Recognition, pp. 63–86 (2011)

  38. Wang, H., Davoine, F., Lepetit, V., Chaillou, C., Pan, C.: 3-D head tracking via invariant keypoint learning. IEEE Trans. Circuits Syst. Video Technol. 22(8), 1113–1126 (2012)

    Article  Google Scholar 

  39. Welch, G., Bishop, G.: An introduction to the kalman filter. University of North Carolina at Chapel Hill, Tech. rep. (1995)

  40. Xiao, J., Chai, J., Kanade, T.: A closed-form solution to non-rigid shape and motion recovery. In: European Conference on Computer Vision, pp. 573–587 (2004)

  41. Yan, J., Pollefeys, M.: A factorization-based approach for articulated nonrigid shape, motion and kinematic chain recovery from video. IEEE Trans. Pattern Anal. Mach. Intell. 30(5), 865–877 (2008)

    Article  Google Scholar 

  42. Zhu, X., Ramanan, D.: Face detection, pose estimation, and landmark localization in the wild. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, IEEE Biometrics Compendium, pp. 2879–2886 (2012)

Download references

Acknowledgments

This work forms part of the project Eye-Communicate funded by the Malta Council for Science and Technology through the National Research & Innovation Programme (2012) under Research Grant No. R&I-2012-057.

Author information

Authors and Affiliations

  1. Department of Systems and Control Engineering, University of Malta, Msida, MSD2080, Malta

    Stefania Cristina & Kenneth P. Camilleri

Authors
  1. Stefania Cristina
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kenneth P. Camilleri
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Stefania Cristina.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Cristina, S., Camilleri, K.P. Model-free non-rigid head pose tracking by joint shape and pose estimation. Machine Vision and Applications 27, 1229–1242 (2016). https://doi.org/10.1007/s00138-016-0791-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00138-016-0791-5

Keywords

Search

Navigation