Skip to main content
SpringerLink
Log in

3D Facial Expression Recognition Using Multi-channel Deep Learning Framework

  • Published:
Circuits, Systems, and Signal Processing Aims and scope Submit manuscript

Abstract

Facial expression offers an important way of detecting the affective state of a human being. It plays a major role in various fields such as the estimation of students’ attention level in online education, intelligent transportation systems and interactive games. This paper proposes a facial expression recognition system in which two channels of featured images are used to represent a 3D facial scan. Features are extracted from the local binary pattern and local directional pattern using a fine-tuned pre-trained AlexNet and a shallow convolutional neural network. The feature sets are then fused together using canonical correlation analysis. The fused feature set is fed into a multi-support vector machine (mSVM) classifier to classify the expressions into seven basic categories: anger, disgust, fear, happiness, neutral, sadness and surprise. Experiments were carried out on the Bosphorus database using tenfold cross-validation with mutually exclusive training and testing samples. The results show an average accuracy of 87.69% using an mSVM classifier with a polynomial kernel and demonstrate that the system performs better by characterizing the peculiarities in facial expressions than alternative state-of-the-art approaches.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. S. Berretti, A.D. Bimbo, P. Pala, B.B. Amor, M. Daoudi, A set of selected SIFT features for 3D facial expression recognition, in International Conference on Pattern Recognition (ICPR), pp. 4125–4128 (2010)

  2. T.F. Cootes, G.J. Edwards, C.J. Taylor, Active appearance models. IEEE Trans. Pattern Anal. Mach. Intell. 23(6), 681–685 (2001)

    Article  Google Scholar 

  3. T.F. Cootes, C.J. Taylor, D.H. Cooper, J. Graham, Active shape models: their training and application. Comput. Vis. Image Underst. 61(1), 38–59 (1995)

    Article  Google Scholar 

  4. A. Jan, H. Ding, H. Meng, L. Chen, H. Li, Accurate facial parts localization and deep learning for 3D facial expression recognition, in 13th IEEE International Conference on Automatic Face & Gesture Recognition, pp. 466–472 (2018)

  5. Y. Lei, M. Bennamoun, A. El-Sallam, An efficient 3D face recognition approach based on the fusion of novel local low-level features. Pattern Recognit. 46(1), 24–37 (2013)

    Article  Google Scholar 

  6. H. Li, H. Ding, D. Huang, Y. Wang, X. Zhao, J.M. Morvan, L. Chen, An efficient multimodal 2D + 3D feature-based approach to automatic facial expression recognition. Comput. Vis. Image Underst. 140, 83–92 (2015)

    Article  Google Scholar 

  7. H. Li, J. Sun, Z. Xu, L. Chen, Multimodal 2D+3D facial expression recognition with deep fusion convolutional neural network. IEEE Trans. Multimed. 19, 2816–2831 (2017)

    Article  Google Scholar 

  8. X. Li, Q. Ruan, Y. Ming, 3D facial expression recognition based on basic geometric features, in IEEE 10th International Conference on Signal Processing (ICSP), pp. 1366–1369 (2010)

  9. X. Li, Q. Ruan, Y. Ming, A remarkable standard for estimating the performance of 3D facial expression features. Neurocomputing 82, 99–108 (2012)

    Article  Google Scholar 

  10. P. Michel, R. El Kaliouby, Real time facial expression recognition in video using support vector machines, in 5th International Conference on Multimodal interfaces (ICMI) (2003)

  11. I. Mpiperis, S. Malassiotis, M.G. Strintzis, Bilinear models for 3-D face and facial expression recognition. IEEE Trans. Inf. Forensics Secur. 3(3), 498–511 (2008)

    Article  Google Scholar 

  12. D. O’Shaughnessy, Recognition and processing of speech signals using neural networks. Circuits Syst. Signal Process. 1, 2 (2019). https://doi.org/10.1007/s00034-019-01081-6

    Article  Google Scholar 

  13. A. Savran, N. Alyüz, H. Dibeklioğlu, O. Çeliktutan, B. Gökberk, B. Sankur, L. Akarun, Bosphorus database for 3D face analysis, in European workshop on Biometrics and Identity Management, pp. 47–56 (2008)

    Google Scholar 

  14. H. Soyel, H. Demirel, Facial expression recognition using 3D facial feature distances, in Proceedings of the International Conference on Image Analysis and Recognition, pp. 831–838 (2007)

  15. Q.S. Sun, S.G. Zeng, Y. Liu, P.A. Heng, D.S. Xia, A new method of feature fusion and its application in image recognition. Pattern Recognit. 38, 2437–2448 (2005)

    Article  Google Scholar 

  16. Y. Sun, X. Chen, M. Rosato, L. Yin, Tracking vertex flow and model adaptation for three-dimensional spatiotemporal face analysis. IEEE Trans. Syst. Man Cybernet. Part A Syst. Hum. 40(3), 461–474 (2010)

    Article  Google Scholar 

  17. M. Taner Eskil, K.S. Benli, Facial expression recognition based on anatomy. Comput. Vis. Image Underst. 119, 1–14 (2014)

    Article  Google Scholar 

  18. H. Tang, T. Huang, 3D facial expression recognition based on properties of line segments connecting facial feature points, in 8th IEEE International Conference on Automatic Face & Gesture Recognition, pp. 1–6 (2008)

  19. Y.V. Venkatesh, A.A. Kassim, O.V. Ramana Murthy, A novel approach to classification of facial expressions from 3D-mesh datasets using modified PCA. Pattern Recognit. Lett. 30(12), 1128–1137 (2009)

    Article  Google Scholar 

  20. J. Wang, L. Yin, X. Wei, Y. Sun, 3D facial expression recognition based on primitive surface feature distribution. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. 2, 1399–1406 (2006)

    Google Scholar 

  21. Y. Wang, M. Gupta, S. Zhang, S. Wang, X. Gu, D. Samaras, P. Huang, High resolution tracking of non-rigid motion of densely sampled 3D data using harmonic maps. Int. J. Comput. Vis. 76(3), 283–300 (2008)

    Article  Google Scholar 

  22. X. Yang, D. Huang, Y. Wang, L. Chen, Automatic 3D facial expression recognition using geometric scattering representation, in 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition, pp. 1–6 (2015)

  23. T. Yun, L. Guan, Human emotion recognition using real 3D visual features from Gabor library, in IEEE International Workshop on Multimedia Signal Processing (MMSP), pp. 505–510 (2010)

  24. K. Yurtkan, H. Demirel, Feature selection for improved 3D facial expression recognition. Pattern Recognit. Lett. 38, 26–33 (2014)

    Article  Google Scholar 

  25. X. Zhao, E. Dellandréa, J. Zou, L. Chen, A unified probabilistic framework for automatic 3D facial expression analysis based on a Bayesian belief inference and statistical feature models. Image Vis. Comput. 31(3), 231–245 (2013)

    Article  Google Scholar 

  26. Z. Zeng, M. Pantic, G. Roisman, T. Huang, A survey of affect recognition methods: audio, visual, and spontaneous expressions. IEEE Trans. Pattern Anal. Mach. Intell. 31(1), 39–58 (2007)

    Article  Google Scholar 

  27. L. Zhang, D. Tjondronegoro, Facial expression recognition using facial movement features. IEEE Trans. Affect. Comput. 2(4), 219–229 (2011). https://doi.org/10.1109/T-AFFC.2011

    Article  Google Scholar 

  28. X. Zhou, H. Seibert, C. Busch, W. Funk, A 3D face recognition algorithm using histogram-based features, in Eurographics Workshop on 3D Object Retrieval (2008)

Download references

Author information

Authors and Affiliations

  1. Kamaraj College of Engineering and Technology, Virudhunagar, India

    R. Ramya

  2. Mepco Schlenk Engineering College, Sivakasi, India

    K. Mala & S. Selva Nidhyananthan

Authors
  1. R. Ramya
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. K. Mala
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. S. Selva Nidhyananthan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to R. Ramya.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ramya, R., Mala, K. & Selva Nidhyananthan, S. 3D Facial Expression Recognition Using Multi-channel Deep Learning Framework. Circuits Syst Signal Process 39, 789–804 (2020). https://doi.org/10.1007/s00034-019-01144-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00034-019-01144-8

Keywords

Search

Navigation