A Review of Local Feature Algorithms and Deep Learning Approaches in Facial Expression Recognition with Tensorflow and Keras

  • Kennedy ChengetaEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11524)


In facial expression identification classification and lower processing times are key in choosing the algorithms to use in the facial detection, preprocessing, feature extraction or classification step. Facial expression recognition is based on deep learning, feature and holistic algorithms. Feature based algorithms like local binary patterns, local directional patterns (LDP) extract features from various facial components like nose, mouth or ears into a histogram. Deep learning involves using convolutional neural networks for image analysis with several hidden layers as opposed to artificial neural or shallow networks. The most popular models are AlexNet, VGG-Face and GoogleNet. The study evaluates computational accuracy and efficiency of deep learning algorithms and compares them to local feature based algorithms. The FER2013, Yale Faces, AT&T Database of Faces, JAFFE and CK+ datasets were used for analysis. Popular frameworks deep learning frameworks called Keras and Tensorflow backends are used to classify data and give better accuracy than a variant of local binary patterns. The processing time is shorter for feature based algorithms than the deep learning algorithms. To improve time on the deep learning approaches the study used pre-trained models to achieve greater accuracy with low execution times as well. A combination of preprocessed multi block binary patterns, PCA, multilayer perceptron, support vector machines and extra trees classifier gave competitive results to the superior established convolutional network for small datasets within a percentage range. Preprocessing used canny edge detection and histogram equalization.


Keras Tensorflow Multi-block local binary patterns 


  1. 1.
    Pramerdorfer, C., Kampel, M.: Facial expression recognition using convolutional neural networks: state of the art (2016). arXiv preprint arXiv:1612.02903
  2. 2.
    Tang, Y.: Deep learning using linear support vector machines. arXiv preprint arXiv:1306.0239 (2013)
  3. 3.
    Levi, G., Hassner, T.: Emotion recognition in the wild via convolutional neural networks and mapped binary patterns. In: ACM International Conference on Multimodal Interaction, pp. 503–510. ACM, November 2015Google Scholar
  4. 4.
    Mollahosseini, A., Chan, D., Mahoor, M.H.: Going deeper in facial expression recognition using deep neural networks. In: IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1–10. IEEE, March 2016Google Scholar
  5. 5.
    Zhao, X., Shi, X., Zhang, S.: Facial expression recognition via deep learning. IETE Tech. Rev. 32(5), 347–355 (2015)CrossRefGoogle Scholar
  6. 6.
    Hemalatha, G., Sumathi, C.P.: A study of techniques for facial detection and expression classification. Int. J. Comput. Sci. Eng. Surv. 5(2), 27 (2014)Google Scholar
  7. 7.
    Xu, M., Cheng, W., Zhao, Q., Ma, L., Xu, F.: Facial expression recognition based on transfer learning from deep convolutional networks. In: 11th International Conference on Natural Computation (ICNC), pp. 702–708. IEEE, August 2015Google Scholar
  8. 8.
    Kahou, S.E., et al.: Emonets: multimodal deep learning approaches for emotion recognition in video. J. Multimodal User Interfaces 10(2), 99–111 (2016)CrossRefGoogle Scholar
  9. 9.
    Tripathi, S., Acharya, S., Sharma, R.D., Mittal, S., Bhattacharya, S.: Using deep and convolutional neural networks for accurate emotion classification on DEAP dataset. In: Twenty-Ninth IAAI Conference, February 2017Google Scholar
  10. 10.
    Cai, Z., Gu, Z., Yu, Z.L., Liu, H., Zhang, K.: A real-time visual object tracking system based on Kalman filter and MB-LBP feature matching. Multimedia Tools Appl. 75(4), 2393–2409 (2016)CrossRefGoogle Scholar
  11. 11.
    Girish, G.N., CL, S.N., Das, P.K.: Face recognition using MB-LBP and PCA: a comparative study. In 2014 International Conference on Computer Communication and Informatics, pp. 1–6. IEEE, January 2014Google Scholar
  12. 12.
    Dhavalikar, A.S., Kulkarni, R.K.: Face detection and facial expression recognition system. In: 2014 International Conference on Electronics and Communication Systems (ICECS), pp. 1–7. IEEE, February 2014Google Scholar
  13. 13.
    Zhou, S., Yin, J.: Face detection using multi-block local gradient patterns and support vector machine. J. Comput. Inf. Syst. 10(4), 1767–1776 (2014)Google Scholar
  14. 14.
    Zhang, L., Chu, R., Xiang, S., Liao, S., Li, S.Z.: Face detection based on multi-block LBP representation. In: Lee, S.-W., Li, S.Z. (eds.) ICB 2007. LNCS, vol. 4642, pp. 11–18. Springer, Heidelberg (2007). Scholar
  15. 15.
    Iftikhar, S., Younas, R., Nasir, N., Zafar, K.: Detection and classification of facial expressions using artificial neural network. J. Inf. Technol. Electr. Eng. 3, 18–22 (2014)Google Scholar
  16. 16.
    Chaudhari, M.V., Student, M.E., Bhusaval, S.S.G.B.C.O.E.T., Patil, Y.S., Patil, D.D.: Facial expression recognition using ANN & Gabor filter 1(6) (2017)Google Scholar
  17. 17.
    Pietikinen, M., Hadid, A., Zhao, G., Ahonen, T.: Computer Vision Using Local Binary Patterns, vol. 40. Springer, Heidelberg (2011). Scholar
  18. 18.
    Poornima, P., Radhapriya, S.: Survey of automatic facial recognition based on classification schemes 4(10) (2017). ISSN: 2454-6933Google Scholar
  19. 19.
    Ruiz, L.Z., Alomia, R.P.V., Dantis, A.D.Q., San Diego, M.J.S., Tindugan, C.F., Serrano, K.K.D.: Human emotion detection through facial expressions for commercial analysis. In: Conference on Humanoid, Nanotechnology, (HNICEM) (2017)Google Scholar
  20. 20.
    Han, J., Kamber, M.: Data Mining: Concepts and Techniques. Morgan Kaufmann, San Francisco (2001)Google Scholar
  21. 21.
    Gulli, A., Pal, S.: Deep Learning with Keras. Packt Publishing Ltd., Birmingham (2017)Google Scholar
  22. 22.
    Breuer, R., Kimmel, R.: A deep learning perspective on the origin of facial expressions (2017)Google Scholar
  23. 23.
    Xia, X.L., Xu, C., Nan, B.: Facial expression recognition based on tensorflow platform. In: ITM Web of Conferences, vol. 12, p. 01005. EDP Sciences (2017)Google Scholar
  24. 24.
    The Japanese Female Facial Expression (JAFFE) Database.
  25. 25.
    Kanade, T., Tian, Y., Cohn, J.F.: Comprehensive database for facial expression analysis. In: IEEE International Conference Automatic Face GestureRecognition (2000)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Computer Science DepartmentUniversity of KwaZulu NatalDurbanSouth Africa

Personalised recommendations