Skip to main content
SpringerLink
Log in

Ensemble of Deep Neural Networks with Probability-Based Fusion for Facial Expression Recognition

  • Published:
Cognitive Computation Aims and scope Submit manuscript

Abstract

Convolutional neural network (CNN) is a very effective method to recognize facial emotions. However, the preprocessing and selection of parameters of these methods heavily depend on the human experience and require a large amount of trial-and-errors. This paper presents an ensemble of convolutional neural networks method with probability-based fusion for facial expression recognition, where the architecture of CNN was adapted by using the convolutional rectified linear layer as the first layer and multiple hidden maxout layers. It was constructed by randomly varying parameters and architecture around the optimal values for CNN, where each CNN as the base classifier was trained to output a probability for each class. These probabilities were then fused through the probability-based fusion method. The conducted experiments on benchmark data sets validated our method, which had better accuracy than the compared methods. The proposed method was novel and efficient for facial expression recognition.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  1. Ayes A, Blewitt W. Models for computational emotions from psychological theories using type-II fuzzy logic. Cogn Comput. 2015;7:309–32.

    Article  Google Scholar 

  2. Naji M, Firoozabadi M, Azadfallah P. Classification of music induced emotions based on information fusion of forehead biosignals and electrocardiogram. Cogn Comput. 2014;6:41–52.

    Article  Google Scholar 

  3. Littlewort G, Whitehill J, Wu T, Fasel I, Frank M, Movellan J, Bartlett M. 2011. The computer expression recognition toolbox (CERT). In: IEEE Int’l Conf. on automatic face and gesture recognition; p. 1–2.

  4. Mehrabian A. Communication without words. Psychol Today 1968;2(4):53–56.

    Google Scholar 

  5. Sandbach G, Zafeiriou S, Pantic M, Yin L. Static and dynamic 3D facial expression recognition: a comprehensive survey. Image Vis Comput. 2012;30(10):683–97.

    Article  Google Scholar 

  6. Sun Y, Wen G, et al. Weighted spectral features based on local Hu moments for speech emotion recognition. Biomed Signal Process Control 2015;18:80–90.

    Article  Google Scholar 

  7. Eleftheriadis S, Rudovic O, Pantic M. Discriminative shared Gaussian processes for multiview and view-invariant facial expression recognition. IEEE Trans Image Process 2015;24(1):189–204.

    Article  PubMed  Google Scholar 

  8. Zhang W, Zhang Y, Ma L, Guan J, Gong S. Multimodal learning for facial expression recognition. Pattern Recog 2015;48:3191–202.

    Article  Google Scholar 

  9. Pires P, Mendes L, Mendes J, Rodrigues R, Pereira A. Integrated e-healthcare system for elderly support. Cogn Comput. 2016;8:368–84.

    Article  Google Scholar 

  10. Fong B, Westerink J. Affective computing in consumer electronics. IEEE Trans Affect Comput. 2012;3(2): 129–31.

    Article  Google Scholar 

  11. Agarwal B, Poria S, Mittal N, Gelbukh A, Hussain A. Concept level sentiment analysis with dependency-based semantic parsing: a novel approach. Cogn Comput. 2015;7:87–99.

    Article  Google Scholar 

  12. Vinciarelli A, Esposito A, Andre E, Bonin F, Chetouani M, Cohn JF, Cristani M, Fuhrmann F, Gilmartin E, Hammal Z, Heylen D, Kaiser R, Koutsombogera M, Potamianos A, Renals S, Riccardi G, Salah AA. Open challenges in modelling, analysis and synthesis of human behaviour in human-human and human-machine interactions. Cogn Comput. 2015;7:397–413.

    Article  Google Scholar 

  13. Sadeghi H, Raie A A. Suitable models for face geometry normalization infacial expression recognition. J Electron Imag. 2015;24(1):013005.

    Article  Google Scholar 

  14. Tian Y, Kanade T, Cohn J F. Facial expression analysis. In: Handbook of face recognition. Springer; p. 247–275. 2005.

  15. Geetha A, Ramalingam V, Palanivel S, Palaniappan B. Facial expression recognition—a real time approach. Expert Syst Appl. 2009;36(1):303–8.

    Article  Google Scholar 

  16. Happy S L, Routray A. Automatic facial expression recognition using features of salient facial patches. IEEE Trans Affect Comput. 2015;6(1):1–12.

    Article  Google Scholar 

  17. Lv Y, Feng Z, Xu C. Facial expression recognition via deep learning. In: SMARTCOMP; p. 303–308. 2015.

  18. Gu W, et al. Facial expression recognition using radial encoding of local Gabor features and classifier synthesis. Pattern Recogn. 2012;45(1):80–91.

    Article  Google Scholar 

  19. Fauer S, Schwenker F. Neural network ensembles in reinforcement learning. Neural Process Lett. 2015;41: 55–69.

    Article  Google Scholar 

  20. Editorial. Editorial introduction to the neural networks special issue on deep learning of representations. Neural Netw 2015;64:1–3.

    Article  Google Scholar 

  21. Krizhevsky A, Sutskever I, Hinton G. Imagenet classification with deep convolutional neural networks. Adv Neural Inf Process Syst. 2012;25:1106–14.

    Google Scholar 

  22. Jung H, Lee S, Park S, Kim B. Development of deep learning-based facial expression recognition system. Workshop on frontiers of computer vision. 2015.

  23. Eigen D, Rolfe J, Fergus R, LeCun Y. Understanding deep architectures using a recursive convolutional network. International conference on learning representations. 2014.

  24. LeCun Y, Bottou L, Bengio Y, Haffner P. Gradient-based learning applied to document recognition. Proc IEEE 1998;86(11):2278–2324.

    Article  Google Scholar 

  25. Sun Y, Wang X, Tang X. 2014. Deep learning facial representation from predicting 10,000 classes. In: CVPR; p. 1891–1898.

  26. Taigman, et al. 2014. Deepface: closing the gap to human level performance in face verification. In: CVPR; p. 1701–1708.

  27. Hinton G, Srivastava N. Improving neural networks by preventing co-adaptation of feature detectors. Comput Sci. 2012;3(4):212–23.

    Google Scholar 

  28. Schmidhuber J. Deep learning in neural networks: an overview. Neural Netw Official J Int Neural Netw Soc 2014;61:85–117.

    Article  Google Scholar 

  29. Meguid M A E, Levine M. Fully automated recognition of spontaneous facial expressions in videos using random forest classifiers. IEEE Trans Affect Comput 2014;5(5):141–54.

    Article  Google Scholar 

  30. Ying C, Shiqing Z, Xiaoming Z. Facial expression recognition via non-negative least-squares sparse coding. Information (Switzerland) 2014;5(2):305–18.

    Google Scholar 

  31. Chathura R, De Silva, Ranganath S, De Silva L C. 2008. Cloud basis function neural network: a modified RBF network architecture for holistic facial expression recognition, Vol. 41.

  32. Sandbach G, Zafeiriou S, Pantic M, Yin L. Static and dynamic 3D facial expression recognition: a comprehensive survey. Image Vis Comput. 2012;30:683–97.

    Article  Google Scholar 

  33. Wan S, Aggarwal JK. Spontaneous facial expression recognition: a robust metric learning approach. Pattern Recogn 2014;47:1859–68.

    Article  Google Scholar 

  34. Mousavia R, Eftekhari M. A new ensemble learning methodology based on hybridization of classifier ensemble selection approaches. Appl Soft Comput. 2015;37:652–66.

    Article  Google Scholar 

  35. Ijjina E P, Mohan C K. 2015. Hybrid deep neural network model for human action recognition. Appl Soft Comput.

  36. He S, Wang S, Wuwei, Fu L, Ji Q. Facial expression recognition using deep Boltzmann machine from thermal infrared images. In: Humaine association conference on affective computing and intelligent interaction; p. 239–244. 2013.

  37. Susskind J M, Hinton G E, Movellan J R, Anderson A K. Generating facial expressions with deep belief nets. In: Kordic V, editor. Affective computing, emotion modelling, synthesis and recognition; 2008. p. 421–440.

    Google Scholar 

  38. Ranzato M, Susskind J, Mnih V, Hinton G. 2011. On deep generative models with applications to recognition. In: CVPR; p. 2857–2864.

  39. Rifai S, Bengio Y, Courville A, Vincent P, Mirza M. Disentangling factors of variation for facial expression recognition. In: ECCV; p. 808–822. 2012.

  40. Ranzato M, Mnih V, Susskind J, Hinton G. Modeling natural images using gated mrfs. IEEE TPAMI 2013;35(9):2206–16.

    Article  Google Scholar 

  41. Cheng Y, Jiang B, Jia K. A deep structure for facial expression recognition under partial occlusion. In: Tenth international conference on intelligent information hiding and multimedia signal processing; p. 211–214. 2014.

  42. Liu M, Li S, Shann S, Chen X. AU-inspired deep networks for facial expression feature learning. Neurocomputing 2015;159:126–136.

    Article  Google Scholar 

  43. Li W, Li M, Su Z. A deep-learning approach to facial expression recognition with candid images. In: 14th IAPR international conference on machine vision applications. 2015.

  44. Liu M, Wang R, Huang Z, Shan S, Chen X. Partial least squares regression on grassmannian manifold for emotion recognition. In: 15th ACM on International conference on multimodal interaction; p. 525–530. 2013.

  45. Tang Y. Deep learning using linear support vector machines. In: Workshop on challenges in representation learning in ICML. 2013.

  46. Tariq U, Lin K-H, Li Z, Zhou X, Wang Z. Recognizing emotions from an ensemble of features. IEEE Trans Syst Man Cybern Part B: Cybern 2012;42(4):1017–26.

    Article  CAS  Google Scholar 

  47. Xibin J, Yanhua Z, Ali PD, Binte H. Multi-classifier fusion based facial expression recognition approach. KSII Trans Int Inf Syst 2014;8(1):196–212.

    Google Scholar 

  48. Zhou X, Xie L, Zhang P, Zhang Y. An ensemble of deep neural networks for object tracking. In: IEEE International conference on image processing; p. 843–847. 2014.

  49. Qiu X, Zhang L, Ren Y, Suganthan P N. Ensemble deep learning for regression and time series forecasting. In: 2014 IEEE Symposium on computational intelligence in ensemble learning; p. 1–6. 2014.

  50. Liu M, Wang R, Li S, Shan S, Huang Z, Chen X. Combining multiple kernel methods on riemannian manifold for emotion recognition in the wild. In: 16th International conference on multimodal interaction; p. 494–501. 2014.

  51. Liu P, et al. 2014. Facial expression recognition via a boosted deep belief network. In: CVPR; p. 1805–1812.

  52. Kahou S E, Bouthillier X, Lamblin P, et al. EmoNets: multimodal deep learning approaches for emotion recognition in video. J Multimodal User Interf 2016;10:99–111.

    Article  Google Scholar 

  53. Frazäo X, Alexandre L A. Weighted convolutional neural network ensemble. In: CIARP; p. 674–681. 2014.

  54. Ciresan DC, Meier U, Schmidhuber J. Multi-column deep neural networks for image classification. In: IEEE Conference on computer vision and pattern recognition (CVPR); p. 3642–3649. 2012.

  55. Lyksborg M, Puonti O, Agn M, Larsen R. An ensemble of 2D convolutional neural networks for tumor segmentation. Lect Notes Comput Sci 2015;9127(1):201–211.

    Article  Google Scholar 

  56. Wang H, Cruz-Roa A, Basavanhally A, Gilmore H, Shih N, Feldman M, Tomaszewski J, Gonzalez F, Madabhushi A. Cascaded ensemble of convolutional neural networks and handcrafted features for mitosis detection. Proc SPIE 2014;9041(2):90410B-90410B-10.

    Google Scholar 

  57. Tajbakhsh N, Gurudu S R, Liang J. Automatic polyp detection in colonoscopy videos using an ensemble of convolutional neural networks. In: 12th IEEE International symposium on biomedical imaging; p. 16–19. 2015.

  58. Goodfellow I J, Warde-Farley D, Mirza M, Courville A, Bengio Y. Maxout networks. In: 30th International conference on machine learning; p. 1319–1327. 2013.

  59. Dahl G, Sainath T, Hinton G. 2013. Improving deep neural networks for LVCSR using rectified linear units and dropout. In: ICASSP.

  60. Lyons MJ, Budynek J, Akamatsu S. Automatic classification of single facial images. IEEE Trans Pattern Anal Mach Intell. 1999;21(12):1357–1362.

    Article  Google Scholar 

  61. Lucey P, et al. The extended Cohn-Kanade dataset (CK+): a complete expression dataset for action unit and emotion-specified expression. In: Workshop on CVPR for human communicative behavior analysis; p. 94–101. 2010.

  62. Goodfellow I J, Erhan D, Carrier P L, et al. Challenges in representation learning: a report on three machine learning contests. Neural Inf Process 2013;23(1):117–124.

    Google Scholar 

  63. Zhang C-X, Zhang J-S, Ji N-N, Guo G. Learning ensemble classifiers via restricted Boltzmann machines. Pattern Recogn Lett. 2014;36:161–170.

    Article  Google Scholar 

  64. Zhang L, Tjondronegoro D, Chandran V. Facial expression recognition experiments with data from television broadcasts and the World Wide Web. Image Vis Comput. 2014;32(2):107–119.

    Article  CAS  Google Scholar 

  65. Goodfellow I, Warde-Farley D, Lamblin P, Dumoulin V, Mirza M, Pascanu R, Bergstra J, Bastien F, Bengio Y. 2013. Pylearn2: a machine learning research library. arXiv preprint arXiv:1308.4214.

  66. Valstar MF, Mehu M, Jiang B, Pantic M, Scherer K. Meta-analysis of the first facial expression recognition challenge. IEEE Trans Syst Man Cybern 2012;42(4):966–791.

    Article  CAS  Google Scholar 

  67. Mayer C, Eggers M, Radig B. Cross-database evaluation for facial expression. Pattern Recogn Image Anal. 2014;24(1):124–32.

    Article  Google Scholar 

  68. Zhu R, Zhang T, Zhao Q, Wu Z. A transfer learning approach to cross-database facial expression recognition. In: International conference on biometrics; p. 293–298. 2015.

  69. Zhou J, Xu T, Gan J. Feature extraction based on local directional pattern with svm decision-level fusion for facial expression recognition. Int J Biosci Biotechnol. 2013;5(2):101–110.

    Google Scholar 

  70. Shan C, Gong Sh, McOwan P W. Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis Comput. 2009;27(6):803–816.

    Article  Google Scholar 

  71. Kim Y, Lee H, Provost EM. Deep learning for robust feature generation in audiovisual emotion recognition. In: ICASSP; p. 3687–3691. 2013.

  72. Lysiak R, Kurzynski M, Woloszynski T. Optimal selection of ensemble classifiers using measures of competence and diversity of base classifiers. Neurocomputing 2014;126:29–35.

    Article  Google Scholar 

  73. Dhall A, Ramana Murthy O V, Goecke R, Joshi J, Gedeon T. Video and image based emotion recognition challenges in the wild: EmotiW 2015. In: ACM International conference on multimodal interaction (ICMI). 2015.

  74. Kim B-K, Roh J, Dong S-Y, Lee S-Y. Hierarchical committee of deep convolutional neural networks for robust facial expression recognition. J Multimodal User Interf 2016;1–17.

  75. Mollahosseini A, Chan D, Mahoor M H. Going deeper in facial expression recognition using deep neural networks. In: 2016 IEEE Winter conference on applications of computer vision (WACV); p. 1–10. 2016.

Download references

Author information

Authors and Affiliations

  1. School of Computer Science and Engineering, South China University of Technology, Guangzhou, China

    Guihua Wen, Zhi Hou, Huihui Li, Danyang Li, Lijun Jiang & Eryang Xun

Authors
  1. Guihua Wen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhi Hou
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Huihui Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Danyang Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Lijun Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Eryang Xun
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Guihua Wen.

Ethics declarations

Conflict of interests

The authors declare that they have no conflict of interest.

Informed Consent

Informed consent was obtained from all individual participants included in the study.

Funding

This study was supported by the China National Science Foundation (60973083, 61273363), Science and Technology Planning Project of Guangdong Province (2014A010103009, 2015A020217002), and Guangzhou Science and Technology Planning Project(201504291154480).

Ethical approval

This paper does not contain any studies with human or animals participants.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wen, G., Hou, Z., Li, H. et al. Ensemble of Deep Neural Networks with Probability-Based Fusion for Facial Expression Recognition. Cogn Comput 9, 597–610 (2017). https://doi.org/10.1007/s12559-017-9472-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12559-017-9472-6

Keywords

Search

Navigation