Advertisement

Tensor Decomposition and Non-linear Manifold Modeling for 3D Head Pose Estimation

  • Dmytro Derkach
  • Adria Ruiz
  • Federico M. SuknoEmail author
Article
  • 13 Downloads

Abstract

Head pose estimation is a challenging computer vision problem with important applications in different scenarios such as human–computer interaction or face recognition. In this paper, we present a 3D head pose estimation algorithm based on non-linear manifold learning. A key feature of the proposed approach is that it allows modeling the underlying 3D manifold that results from the combination of rotation angles. To do so, we use tensor decomposition to generate separate subspaces for each variation factor and show that each of them has a clear structure that can be modeled with cosine functions from a unique shared parameter per angle. Such representation provides a deep understanding of data behavior. We show that the proposed framework can be applied to a wide variety of input features and can be used for different purposes. Firstly, we test our system on a publicly available database, which consists of 2D images and we show that the cosine functions can be used to synthesize rotated versions from an object from which we see only a 2D image at a specific angle. Further, we perform 3D head pose estimation experiments using other two types of features: automatic landmarks and histogram-based 3D descriptors. We evaluate our approach on two publicly available databases, and demonstrate that angle estimations can be performed by optimizing the combination of these cosine functions to achieve state-of-the-art performance.

Keywords

3D head pose Manifold learning Tensor decomposition 

Notes

Acknowledgements

This work is partly supported by the Spanish Ministry of Economy and Competitiveness under Project Grant TIN2017-90124-P, the Ramon y Cajal programme, and the Maria de Maeztu Units of Excellence Programme (MDM-2015-0502). Adria Ruiz work is partially funded by ANR grant ANR-16-CE23-0006.

References

  1. Ahn, B., Park, J., & Kweon, I. S. (2014). Real-time head orientation from a monocular camera using deep neural network. In Asian conference on computer vision (pp. 82–96). Springer.Google Scholar
  2. Bakry, A., & Elgammal, A. (2014). Untangling object-view manifold for multiview recognition and pose estimation. In European conference on computer vision (pp. 434–449). Springer.Google Scholar
  3. Balasubramanian, V. N., Ye, J., & Panchanathan, S. (2007). Biased manifold embedding: A framework for person-independent head pose estimation. In Computer vision and pattern recognition (CVPR) (pp. 1–7). IEEE.Google Scholar
  4. Baltrušaitis, T., Robinson, P., & Morency, L. P. (2012). 3D constrained local model for rigid and non-rigid facial tracking. In Computer vision and pattern recognition (CVPR) (pp. 2610–2617). IEEE.Google Scholar
  5. Barros, J. M. D., Mirbach, B., Garcia, F., Varanasi, K., & Stricker, D. (2018). Fusion of keypoint tracking and facial landmark detection for real-time head pose estimation. In Winter conference on applications of computer vision (WACV) (pp. 2028–2037). IEEE.Google Scholar
  6. BenAbdelkader, C. (2010). Robust head pose estimation using supervised manifold learning. In European conference on computer vision (pp. 518–531). Springer.Google Scholar
  7. Bergqvist, G., & Larsson, E. G. (2010). The higher-order singular value decomposition: Theory and an application [lecture notes]. IEEE Signal Processing Magazine, 27(3), 151–154.CrossRefGoogle Scholar
  8. Borghi, G., Fabbri, M., Vezzani, R., Calderara, S., & Cucchiara, R. (2019). Face-from-depth for head pose estimation on depth images. IEEE Transactions on Pattern Analysis and Machine Intelligence (in press).Google Scholar
  9. Borghi, G., Venturelli, M., Vezzani, R., & Cucchiara, R. (2017). Poseidon: Face-from-depth for driver pose estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4661–4670).Google Scholar
  10. Breitenstein, M. D., Kuettel, D., Weise, T., Van Gool L, & Pfister, H. (2008). Real-time face pose estimation from single range images. In Computer vision and pattern recognition (pp. 1–8). IEEE.Google Scholar
  11. Byrd, R. H., Nocedal, J., & Schnabel, R. B. (1994). Representations of quasi-newton matrices and their use in limited memory methods. Mathematical Programming, 63(1–3), 129–156.MathSciNetCrossRefzbMATHGoogle Scholar
  12. Chen, J., Wu, J., Richter, K., Konrad, J., & Ishwar, P. (2016). Estimating head pose orientation using extremely low resolution images. In Southwest symposium on image analysis and interpretation (SSIAI) (pp. 65–68). IEEEGoogle Scholar
  13. Comon, P. (2014). Tensors: A brief introduction. Signal Processing Magazine, 31(3), 44–53.CrossRefGoogle Scholar
  14. De Lathauwer, L., De Moor, B., & Vandewalle, J. (2000). A multilinear singular value decomposition. SIAM Journal on Matrix Analysis and Applications, 21(4), 1253–1278.MathSciNetCrossRefzbMATHGoogle Scholar
  15. Derkach, D., Ruiz, A., & Sukno, F. M. (2017). Head pose estimation based on 3-D facial landmarks localization and regression. In 12th IEEE international conference on automatic face and gesture recognition (FG 2017) (pp. 820–827). IEEE.Google Scholar
  16. Derkach, D., Ruiz, A., & Sukno, F. M. (2018). 3D head pose estimation using tensor decomposition and non-linear manifold modeling. In: International conference on 3D Vision (3DV) (pp. 505–513). IEEE.Google Scholar
  17. Fanelli, G., Dantone, M., Gall, J., Fossati, A., & Van Gool, L. (2013). Random forests for real time 3D face analysis. International Journal of Computer Vision, 101(3), 437–458.CrossRefGoogle Scholar
  18. Fanelli, G., Weise, T., Gall, J., & Van Gool, L. (2011). Real time head pose estimation from consumer depth cameras. In Joint pattern recognition symposium (pp. 101–110). Springer.Google Scholar
  19. Frome, A., Huber, D., Kolluri, R., Bulow, T., & Malik, J. (2004). Recognizing objects in range data using regional point descriptors. In European conference on computer vision (pp. 224–237). Springer.Google Scholar
  20. Fu, Y., & Huang, T. S. (2006). Graph embedded analysis for head pose estimation. In International conference on automatic face and gesture recognition (pp. 6–8). IEEE.Google Scholar
  21. Ghiass, R. S., Arandjelović, O., & Laurendeau, D. (2015). Highly accurate and fully automatic head pose estimation from a low quality consumer-level rgb-d sensor. In Proceedings of the 2nd workshop on computational models of social interactions: Human–Computer–Media communication (pp. 25–34). ACM.Google Scholar
  22. Gu, J., Yang, X., De Mello, S., & Kautz, J. (2017). Dynamic facial analysis: From bayesian filtering to recurrent neural network. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1548–1557).Google Scholar
  23. Johnson, A., & Hebert, M. (1999). Using spin images for efficient object recognition in cluttered 3D scenes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 21(5), 433–449.CrossRefGoogle Scholar
  24. Kolda, T. G., & Bader, B. W. (2009). Tensor decompositions and applications. SIAM Review, 51(3), 455–500.MathSciNetCrossRefzbMATHGoogle Scholar
  25. Lathuiliére, S., Juge, R., Mesejo, P., Muñoz-Salinas, R., & Horaud, R. (2017). Deep mixture of linear inverse regressions applied to head-pose estimation. In Conference on computer vision and pattern recognition (vol. 3, pp. 4817–4825).Google Scholar
  26. Lathuiliére, S., Mesejo, P., Alameda-Pineda, X., & Horaud, R. (2019). A comprehensive analysis of deep regression. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1–1 (in press).Google Scholar
  27. Lee, D., Yang, M. H., & Oh, S. (2015). Fast and accurate head pose estimation via random projection forests. In International conference on computer vision (pp. 1958–1966). IEEE.Google Scholar
  28. Lee, D., Yang, M. H., & Oh, S. (2017). Head and body orientation estimation using convolutional random projection forests. In IEEE transactions on pattern analysis and machine intelligence (pp. 1–14)Google Scholar
  29. Li, D., & Pedrycz, W. (2014). A central profile-based 3D face pose estimation. Pattern Recognition, 47(2), 525–534.CrossRefGoogle Scholar
  30. Li, S., Ngan, K. N., Paramesran, R., & Sheng, L. (2016). Real-time head pose tracking with online face template reconstruction. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(9), 1922–1928.CrossRefGoogle Scholar
  31. Liu, X., Liang, W., Wang, Y., Li, S., & Pei, M. (2016). 3D head pose estimation with convolutional neural network trained on synthetic images. In International conference on image processing (ICIP) (pp. 1289–1293). IEEE.Google Scholar
  32. Liu, X., Lu, H., & Li, W. (2010). Multi-manifold modeling for head pose estimation. In International conference on image processing (ICIP) (pp. 3277–3280). IEEE.Google Scholar
  33. Lüsi, I., Escalera, S., & Anbarjafari, G. (2016a). Human head pose estimation on SASE database using random hough regression forests. Video Analytics (pp. 137–150). Springer: Face and Facial Expression Recognition and Audience Measurement.Google Scholar
  34. Lüsi, I., Escarela, S., & Anbarjafari, G. (2016b). SASE: RGB-depth database for human head pose estimation. In European conference on computer vision (pp. 325–336). Springer.Google Scholar
  35. Lüsi, I., Jacques Junior, J. C. S., Gorbova, J., Baró X, Escalera, S., Demirel, H., Allik, J., Ozcinar, C., & Anbarjafari, G. (2017). Joint challenge on dominant and complementary emotion recognition using micro emotion features and head-pose estimation: Databases. In International conference on automatic face and gesture recognition (pp. 809–813). IEEE.Google Scholar
  36. Martin, M., Van De Camp, F., & Stiefelhagen, R. (2014). Real time head model creation and head pose estimation on consumer depth cameras. In International conference on 3D vision (3DV) (vol. 1, pp. 641–648). IEEE.Google Scholar
  37. Meyer, G. P., Gupta, S., Frosio, I., Reddy, D., & Kautz, J. (2015). Robust model-based 3D head pose estimation. In Proceedings of the IEEE international conference on computer vision (pp. 3649–3657). IEEE.Google Scholar
  38. Murphy-Chutorian, E., & Trivedi, M. M. (2009). Head pose estimation in computer vision: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(4), 607–626.CrossRefGoogle Scholar
  39. Nene, S. A., Nayar, S. K., Murase, H., et al. (1996). Columbia object image library (coil-20).Google Scholar
  40. Padeleris, P., Zabulis, X., & Argyros, A. A. (2012). Head pose estimation on depth data based on particle swarm optimization. In Computer society conference on computer vision and pattern recognition workshops (CVPRW) (pp. 42–49). IEEE.Google Scholar
  41. Papazov, C., Marks, T. K., & Jones, M. (2015). Real-time 3D head pose and facial landmark estimation from depth images using triangular surface patch features. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4722–4730).Google Scholar
  42. Patacchiola, M., & Cangelosi, A. (2017). Head pose estimation in the wild using convolutional neural networks and adaptive gradient methods. Pattern Recognition, 71, 132–143.CrossRefGoogle Scholar
  43. Peng, X., Huang, J., Hu, Q., Zhang, S., & Metaxas, D. N. (2014). Head pose estimation by instance parameterization. In International conference on pattern recognition (ICPR) (pp. 1800–1805). IEEE.Google Scholar
  44. Raytchev, B., Yoda, I., & Sakaue, K. (2004). Head pose estimation by nonlinear manifold learning. In International conference on pattern recognition (ICPR) (vol. 4, pp. 462–466). IEEE.Google Scholar
  45. Ruiz, N., Chong, E., & Rehg, J. M. (2018). Fine-grained head pose estimation without keypoints. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops (pp. 2074–2083).Google Scholar
  46. Rusu, R. B., Blodow, N., & Beetz, M. (2009). Fast point feature histograms (fpfh) for 3d registration. In International conference on robotics and automation, Citeseer (pp. 3212–3217).Google Scholar
  47. Schmidt, M. (2012). Minfunc: Unconstrained differentiable multivariate optimization in matlab. Software available at http://www.cs.ubc.ca/schmidtm/Software/minFunc.htm.
  48. Seemann, E., Nickel, K., & Stiefelhagen, R. (2004). Head pose estimation using stereo vision for human–robot interaction. In International conference on automatic face and gesture recognition (pp. 626–631). IEEE.Google Scholar
  49. Sukno, F., Waddington, J., & Whelan, P. (2012). Comparing 3D descriptors for local search of craniofacial landmarks. In International symposium on visual computing (pp. 92–103). Springer.Google Scholar
  50. Sukno, F., Waddington, J., & Whelan, P. (2013). Rotationally invariant 3D shape contexts using asymmetry patterns. International conference on computer graphics theory and applications (pp. 7–17).Google Scholar
  51. Sukno, F. M., Waddington, J. L., & Whelan, P. F. (2015). 3-D facial landmark localization with asymmetry patterns and shape regression from incomplete local features. IEEE Transactions on Cybernetics, 45(9), 1717–1730.CrossRefGoogle Scholar
  52. Sun, Y., & Yin, L. (2008). Automatic pose estimation of 3D facial models. In International conference on pattern recognition (pp. 1–4.).Google Scholar
  53. Sundararajan, K., & Woodard, D. L. (2015). Head pose estimation in the wild using approximate view manifolds. In International conference on computer vision and pattern recognition workshops (pp. 50–58). IEEE.Google Scholar
  54. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015). Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1–9).Google Scholar
  55. Takallou, H. M., & Kasaei, S. (2014). Head pose estimation and face recognition using a non-linear tensor-based model. IET Computer Vision, 8(1), 54–65.CrossRefGoogle Scholar
  56. Tan, D. J., Tombari, F., & Navab, N. (2018). Real-time accurate 3d head tracking and pose estimation with consumer rgb-d cameras. International Journal of Computer Vision, 126(2–4), 158–183.MathSciNetCrossRefGoogle Scholar
  57. Tenenbaum, J. B., & Freeman, W. T. (1997). Separating style and content. In Advances in neural information processing systems (pp. 662–668).Google Scholar
  58. Tenenbaum, J. B., & Freeman, W. T. (2000). Separating style and content with bilinear models. Neural Computation, 12(6), 1247–1283.CrossRefGoogle Scholar
  59. Tombari, F., Salti, S., & Di Stefano, L. (2010). Unique signatures of histograms for local surface description. In European conference on computer vision (pp. 356–369). Springer.Google Scholar
  60. Tulyakov, S., Vieriu, R. L., Semeniuta, S., & Sebe, N. (2014). Robust real-time extreme head pose estimation. In International conference on pattern recognition (ICPR) (pp. 2263–2268). IEEE.Google Scholar
  61. Vasilescu, M. A. O., & Terzopoulos, D. (2002). Multilinear analysis of image ensembles: Tensorfaces. In European conference on computer vision (pp. 447–460). Springer.Google Scholar
  62. Wang, B., Liang, W., Wang, Y., & Liang, Y. (2013). Head pose estimation with combined 2D SIFT and 3D HOG features. In International conference on image and graphics (ICIG) (pp. 650–655). IEEE.Google Scholar
  63. Wang, C., Guo, Y., & Song, X. (2017a). Head pose estimation via manifold learning. InTech: In Manifolds-current research areas.CrossRefGoogle Scholar
  64. Wang, C., & Song, X. (2014). Robust head pose estimation via supervised manifold learning. Neural Networks, 53, 15–25.CrossRefzbMATHGoogle Scholar
  65. Wang, K., Wu, Y., & Ji, Q. (2018). Head pose estimation on low-quality images. In International conference on automatic face and gesture recognition (FG 2018) (pp. 540–547). IEEE.Google Scholar
  66. Wang, M., Panagakis, Y., Snape, P., Zafeiriou, S., et al. (2017b). Learning the multilinear structure of visual data. In: Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4592–4600).Google Scholar
  67. Wang, Y., Liang, W., Shen, J., Jia, Y., & Yu, L. F. (2019). A deep coarse-to-fine network for head pose estimation from synthetic data. Pattern Recognition, 94, 196–206.CrossRefGoogle Scholar
  68. Xu, Y., Hao, R., Yin, W., & Su, Z. (2015). Parallel matrix factorization for low-rank tensor completion. Inverse Problems and Imaging, 9(2), 601–624.MathSciNetCrossRefzbMATHGoogle Scholar
  69. Yu, Y., Mora, K. A. F., & Odobez, J. M. (2017). Robust and accurate 3D head pose estimation through 3dmm and online head model reconstruction. In International conference on automatic face and gesture recognition (FG 2017) (pp. 711–718). IEEE.Google Scholar
  70. Zhang, H., El-Gaaly, T., Elgammal, A., & Jiang, Z. (2015). Factorization of view-object manifolds for joint object recognition and pose estimation. Computer Vision and Image Understanding, 139, 89–103.CrossRefGoogle Scholar
  71. Zhao, Q., Zhang, L., & Cichocki, A. (2015). Bayesian cp factorization of incomplete tensors with automatic rank determination. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(9), 1751–1763.CrossRefGoogle Scholar
  72. Zhu, Y., Xue, Z., & Li, C. (2014). Automatic head pose estimation with synchronized sub manifold embedding and random regression forests. International Journal of Signal Processing, Image Processing and Pattern Recognition, 7(3), 123–134.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Department of Information and Communication TechnologiesPompeu Fabra UniversityBarcelonaSpain
  2. 2.INRIA, Univ. Grenoble Alpes, Institute of Engineering, CNRSGrenobleFrance

Personalised recommendations