Cluster Computing

, Volume 21, Issue 1, pp 549–567 | Cite as

Facial appearance and texture feature-based robust facial expression recognition framework for sentiment knowledge discovery

  • Muhammad Sajjad
  • Adnan Shah
  • Zahoor Jan
  • Syed Inayat Shah
  • Sung Wook Baik
  • Irfan MehmoodEmail author


Facial sentiment analysis has been an enthusiastic research area for the last two decades. A fair amount of work has been done by researchers in this field due to its utility in numerous applications such as facial expression driven knowledge discovery. However, developing an accurate and efficient facial expression recognition system is still a challenging problem. Although many efficient recognition systems have been introduced in the past, the recognition rate is not satisfactory in general due to inherent limitations including light, pose variations, noise, and occlusion. In this paper, a hybrid approach of facial expression based sentiment analysis has been presented combining local and global features. Feature extraction is performed fusing the histogram of oriented gradients (HOG) descriptor with the uniform local ternary pattern (U-LTP) descriptor. These features are extracted from the entire face image rather than from individual components of faces like eyes, nose, and mouth. The most suitable set of HOG parameters are selected after analyzing them experimentally along with the ULTP descriptor, boosting performance of the proposed technique over face images containing noise and occlusions. Face sentiments are analyzed classifying them into seven universal emotional expressions: Happy, Angry, Fear, Disgust, Sad, Surprise, and Neutral. Extracted features via HOG and ULTP are fused into a single feature vector and this feature vector is fed into a Multi-class Support Vector Machine classifier for emotion classification. Three types of experiments are conducted over three public facial image databases including JAFFE, MMI, and CK+ to evaluate the recognition rate of the proposed technique during experimental evaluation; recognition accuracy in percent, i.e., 95.71, 98.20, and 99.68 are achieved for JAFFE, MMI, and CK+, respectively.


Facial expression recognition Sentiment based knowledge discovery Histogram of oriented gradient Uniform local ternary pattern Support vector machine 



This research was supported by the MISP (Ministry of Science, ICT & Future Planning), Korea, under National program for Excellence in SW (2015-0-00938) supervised by the IITP (Institute for Information & communications Technology Promotion).

Compliance with ethical standards

Conflicts of interest

The authors declare no conflict of interest.


  1. 1.
    Mehrabian, A.: Nonverbal Communication. Transaction Publishers, Los Angeles (1972)Google Scholar
  2. 2.
    Ekman, P., Friesen, W.V., Hager, J.C.: A technique for the measurement of facial action. In: Facial action coding system (FACS), p. 22. Palo Alto, Consulting (1978)Google Scholar
  3. 3.
    Zhang, Y., Ji, Q.: Active and dynamic information fusion for facial expression understanding from image sequences. IEEE Trans. Pattern Anal. Mach. Intell. 27(5), 699–714 (2005)CrossRefGoogle Scholar
  4. 4.
    Shan, C., Gong, S., McOwan, P.W.: Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis. Comput. 27(6), 803–816 (2009)CrossRefGoogle Scholar
  5. 5.
    Song, K.T., Chien, S.C.: Facial expression recognition based on mixture of basic expressions and intensities. In: 2012 IEEE International Conference on Systems, Man, and Cybernetics (SMC), IEEE (2012)Google Scholar
  6. 6.
    Dhall, A., et al.: Emotion recognition using PHOG and LPQ features. In: Automatic Face & Gesture Recognition and Workshops (FG 2011), 2011 IEEE International Conference on, IEEE (2011)Google Scholar
  7. 7.
    Apte, S.: Facial Emotion IdentificationGoogle Scholar
  8. 8.
    Viola, P., Jones, M.J.: Robust real-time face detection. Int. J. Comput. Vis. 57(2), 137–154 (2004)CrossRefGoogle Scholar
  9. 9.
    Viola, P., Jones, M.: Rapid object detection using a boosted cascade of simple features. In: Computer Vision and Pattern Recognition, 2001. CVPR 2001. Proceedings of the 2001 IEEE Computer Society Conference on, IEEE (2001)Google Scholar
  10. 10.
    Rivera, A.R., Castillo, J.R., Chae, O.O.: Local directional number pattern for face analysis: face and expression recognition. IEEE Trans. Image Process. 22(5), 1740–1752 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  11. 11.
    Jeyashree, T., et al.: An efficient algorithm for face and expression recognition. Int. J. Sci. Technol. 2(3), 172 (2014)Google Scholar
  12. 12.
    Turk, M.A., Pentland, A.P.: Face recognition using eigenfaces. In: Computer Vision and Pattern Recognition, 1991. Proceedings CVPR’91., IEEE Computer Society Conference on, IEEE (1991)Google Scholar
  13. 13.
    Hsu, C.W., Chang, C.C., Lin, C.J.: A practical guide to support vector classification (2003)Google Scholar
  14. 14.
    Kozma, L.: k Nearest Neighbors algorithm (kNN). Helsinki University of Technology (2008)Google Scholar
  15. 15.
    Lee, Y.H., Han, W., Kim, Y.: Emotional recognition from facial expression analysis using bezier curve fitting. In: 2013 16th International Conference on Network-Based Information Systems, IEEE (2013)Google Scholar
  16. 16.
    Al-Shabi, M., Cheah, W.P., Connie, T.: Facial expression recognition using a hybrid CNN-SIFT aggregator. arXiv preprint arXiv:1608.02833 (2016)
  17. 17.
    Wang, X.H., Liu, A., Zhang, S.Q.: New facial expression recognition based on FSVM and KNN. Optik-Int. J. Light Electron Opt. 126(21), 3132–3134 (2015)CrossRefGoogle Scholar
  18. 18.
    Tong, Y., Chen, R., Cheng, Y.: Facial expression recognition algorithm using LGC based on horizontal and diagonal prior principle. Optik-Int. J. Light Electron Opt. 125(16), 4186–4189 (2014)CrossRefGoogle Scholar
  19. 19.
    Luo, Y., Wu, C.M., Zhang, Y.: Facial expression recognition based on fusion feature of PCA and LBP with SVM. Optik-Int. J. Light Electron Opt. 124(17), 2767–2770 (2013)CrossRefGoogle Scholar
  20. 20.
    Happy, S., Routray, A.: Robust facial expression classification using shape and appearance features. In: Advances in Pattern Recognition (ICAPR), 2015 Eighth International Conference on, IEEE (2015)Google Scholar
  21. 21.
    Carcagnì, P., et al.: Facial expression recognition and histograms of oriented gradients: a comprehensive study. SpringerPlus 4(1), 1 (2015)CrossRefGoogle Scholar
  22. 22.
    Kumar, S., Bhuyan, M., Chakraborty, B.K.: Extraction of informative regions of a face for facial expression recognition. IET Comput. Vis. 10(6), 567–576 (2016)CrossRefGoogle Scholar
  23. 23.
    Guo, Y., et al.: EI3D: expression-invariant 3D face recognition based on feature and shape matching. Pattern Recognit. Lett. 83, 403–412 (2016)CrossRefGoogle Scholar
  24. 24.
    Sajjad, M., Ejaz, N., Baik, S.W.: Multi-kernel based adaptive interpolation for image super-resolution. Multimed. Tools Appl. 72(3), 2063–2085 (2014)CrossRefGoogle Scholar
  25. 25.
    Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), IEEE (2005)Google Scholar
  26. 26.
    Geng, C., Jiang, X.: SIFT features for face recognition. In: Computer Science and Information Technology, 2009. ICCSIT 2009. 2nd IEEE International Conference on, IEEE (2009)Google Scholar
  27. 27.
    Dreuw, P., et al. SURF-Face: face recognition under viewpoint consistency constraints. In: BMVC (2009)Google Scholar
  28. 28.
    Tan, X., Triggs, B.: Enhanced local texture feature sets for face recognition under difficult lighting conditions. IEEE Trans. Image Process. 19(6), 1635–1650 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  29. 29.
    Ren, J., Jiang, X., Yuan, J.: Relaxed local ternary pattern for face recognition. In: ICIP (2013)Google Scholar
  30. 30.
    Ojala, T., Pietikainen, M., Maenpaa, T.: Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans. Pattern Anal. Mach. Intell. 24(7), 971–987 (2002)CrossRefzbMATHGoogle Scholar
  31. 31.
    Lucey, P., et al.: The extended cohn-kanade dataset (ck+): a complete dataset for action unit and emotion-specified expression. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops, IEEE (2010)Google Scholar
  32. 32.
    Cohen, I., et al.: Learning Bayesian network classifiers for facial expression recognition both labeled and unlabeled data. In: Computer Vision and Pattern Recognition, 2003. Proceedings. 2003 IEEE Computer Society Conference on, IEEE (2003)Google Scholar
  33. 33.
    Cohen, I., et al.: Facial expression recognition from video sequences: temporal and static modeling. Comput. Vis. Image Underst. 91(1), 160–187 (2003)CrossRefGoogle Scholar
  34. 34.
    Bartlett, M.S., et al.: Recognizing facial expression: machine learning and application to spontaneous behavior. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), IEEE (2005)Google Scholar
  35. 35.
    Valstar, M., Pantic, M.: Fully automatic facial action unit detection and temporal analysis. In: 2006 Conference on Computer Vision and Pattern Recognition Workshop (CVPRW’06), IEEE (2006)Google Scholar
  36. 36.
    Valstar, M.F., Patras, I., Pantic, M.: Facial action unit detection using probabilistic actively learned support vector machines on tracked facial point data. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05)-Workshops, IEEE (2005)Google Scholar
  37. 37.
    Pantic, M., et al.: Web-based database for facial expression analysis. In: 2005 IEEE international conference on multimedia and Expo, IEEE (2005)Google Scholar
  38. 38.
    Lyons, M., et al.: Coding facial expressions with gabor wavelets. In: Automatic Face and Gesture Recognition, 1998. Proceedings. Third IEEE International Conference on, IEEE (1998)Google Scholar
  39. 39.
    Chao, W.L., Ding, J.J., Liu, J.Z.: Facial expression recognition based on improved local binary pattern and class-regularized locality preserving projection. Signal Process. 117, 1–10 (2015)CrossRefGoogle Scholar
  40. 40.
    Gu, W., et al.: Facial expression recognition using radial encoding of local Gabor features and classifier synthesis. Pattern Recognit. 45(1), 80–91 (2012)CrossRefGoogle Scholar
  41. 41.
    Donia, M.M., Youssif, A.A., Hashad, A.: Spontaneous facial expression recognition based on histogram of oriented gradients descriptor. Comput. Inf. Sci. 7(3), 31 (2014)Google Scholar
  42. 42.
    Zhang, L., Tjondronegoro, D.: Facial expression recognition using facial movement features. IEEE Trans. Affect. Comput. 2(4), 219–229 (2011)CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2017

Authors and Affiliations

  • Muhammad Sajjad
    • 1
  • Adnan Shah
    • 1
  • Zahoor Jan
    • 1
  • Syed Inayat Shah
    • 2
  • Sung Wook Baik
    • 3
  • Irfan Mehmood
    • 4
    Email author
  1. 1.Digital Image Processing Laboratory, Department of Computer ScienceIslamia College PeshawarPeshawarPakistan
  2. 2.Department of MathematicsIslamia College PeshawarPeshawarPakistan
  3. 3.Digital Contents Research Institute, Department of SoftwareSejong UniversitySeoulSouth Korea
  4. 4.Department of Computer Science and EngineeringSejong UniversitySeoulSouth Korea

Personalised recommendations