Skip to main content
SpringerLink
Log in

Interval graph of facial regions with common intersection salient points for identifying and classifying facial expression

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Measuring the facial expression is an important research and used in many real-time applications. Various methods are proposed in the academia and industry for a decade and still continue to have research potential. This paper proposes a novel scheme by using Interval graph of facial regions. It is assumed that common intersecting salient points of facial regions can be used for estimating the emotions. The facial region is decomposed in four sub regions and the Interval graph is extracted for each region. The common salient points and degree of deformation and direction of deformation are measured for vertical, horizontal and diagonal directions. These values are considered as feature vectors. The well-known datasets such as JAFEE and CK++ are used for evaluating the performance of various classification algorithms and estimating their average classification accuracy. The average classification of the proposed approach is 95.9% and 94.7% for CK++ dataset and JAFEE dataset respectively. The performance of the proposed approach is better when compared to other state of art approaches.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Bashir Y, Aslam A, Kamran M, Qureshi M I, Jahangir A, Rafiq M, Bibi N, Muhammad N (2017) On forgotten topological indices of some Dendrimers structure. Molecules 24;22(6):867. https://doi.org/10.3390/molecules22060867

  2. Batur AU, Hayes MH (2005) Adaptive active appearance models. IEEE Trans Image Process 14(11):1707–1721

    Article  Google Scholar 

  3. Castrillón M, Déniz O, Hernández D, Lorenzo J (2011) A comparison of face and facial feature detectors based on the viola–jones general object detection framework. Mach Vis Appl 22(3):481–494

    Google Scholar 

  4. Castrillón-Santana M, Hernández-Sosa D, Lorenzo-Navarro J (2012) Combining face and facial feature detectors for face detection performance improvement. In: Alvarez L., Mejail M., Gomez L., Jacobo J. (eds) Progress in pattern recognition, image analysis, computer vision, and applications. Lecture Notes in Computer Science, Springer, Berlin, Heidelberg, 7441:82–89

  5. Chen X, Huang T (2003) Facial expression recognition: a clustering-based approach. Pattern Recogn Lett 24:1295–1302

    Article  Google Scholar 

  6. Chen HT, Liu TL, Fuh CS (2004) Probabilistic tracking with adaptive feature selection. International Conference on Pattern Recognition, Washington DC 2:736–739

    Google Scholar 

  7. Chien SI, Choi I (2000) Face and Facial Landmarks Location Based on Log-Polar Mapping. In: Lee SW, Bülthoff HH, Poggio T. (eds) Biologically Motivated Computer Vision. BMCV 2000. Lecture notes in computer science, vol 1811,Springer, Berlin, Heidelberg, pp 379–386

  8. Cohen I, Sebe N, Cozman F, Cirelo M, Huang T (2003) Learning Bayesian network classifiers for facial expression recognition using both labelled and unlabeled data. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Madison, WI, USA, pp I–I.

  9. Corcoran P, Ionita MC, Bacivarov I (2007) Next generation face tracking technology using AAM techniques. International Symposium on Signals, Systems and Circuits 1:1–4

    Google Scholar 

  10. Cowie R, Douglas-Cowie E, Tsapatsoulis N, Votsis G, Kollias S, Fellenz W, Taylor JG (2001) Emotion recognition in human-computer interaction. IEEE Signal Process Mag 18(1):32–80

    Article  Google Scholar 

  11. Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. In: computer vision and pattern recognition (CVPR 2005), vol. 1, pp 886–893

  12. Darwin C (1872) The expression of the emotions in man and animals. Murray J, London

    Book  Google Scholar 

  13. Ding Y, Zhao Q, Li B, Yuan X (2017) Facial Expression Recognition From Image Sequence Based on LBP and Taylor Expansion. Special Section On Sequential Data Modeling And Its Emerging Applications, IEEE Access, vol 5

  14. Donahue J, Jia Y, Vinyals O et al (2014) Decaf: a deep convolutional activation feature for generic visual recognition. In: 31st international conference on machine learning. PMLR 32(1):647–655

    Google Scholar 

  15. Ekman P, Friesen WV et al (1978) Facial action coding system: a technique for the measurement of facial movement. Consulting Psychologists Press, Palo Alto

    Google Scholar 

  16. Fasel B, Luettin J (2000) Recognition of asymmetric facial action unit activities and intensities. In: 15th International Conference on Pattern Recognition, Barcelona, Spain, vol. 1, pp 1100–1103

  17. Fasel B, Luettin J (2003) Automatic facial expression analysis: a survey. Pattern Recogn 36(1):259–275

    Article  Google Scholar 

  18. Felzenszwalb PF, Girshick RB, McAllester D, Ramanan D (2010) Object detection with discriminatively trained part-based models. IEEE Transactions on Pattern Analytical and Machine Intelligence 32(9):1627–1645

    Article  Google Scholar 

  19. Freedman D (2004) Active contours for tracking distributions. IEEE Trans Image Process 13(4):518–526

    Article  Google Scholar 

  20. Geetha A, Ramalingam V, Palanivel S (2009) Facial expression recognition- a real time approach. Expert Systems with Applications, Elsevier 36(1):303–308

    Article  Google Scholar 

  21. Happy SL, Routray A (2015) Automatic facial expression recognition using features of salient facial patches. IEEE Trans Affect Comput 6(1):1–12. https://doi.org/10.1109/TAFFC.2014.2386334

    Article  Google Scholar 

  22. HaqMU SA, MahmoodZ SAA (2019) Boosting the face recognition performance of ensemble based LDA for pose. Non-uniform Illuminations, and Low-Resolution ImagesKSII Transactions on Internet and Information Systems 13(6):3144–3166

    Google Scholar 

  23. He C, Mao H, Jin L (2011) Realistic smile expression recognition using biologically inspired features. In: Wang D, Reynolds M (eds) Advances in Artificial Intelligence (AI 2011), Lecture notes in computer science, Springer, Berlin, Heidelberg, vol 7106

  24. Hussain A, Khan MS, Nazir M, Iqbal MA (2012) Survey of various feature extraction and classification techniques for facial expression recognition. In:11th WSEAS international conference on Electronics, Hardware, Wireless and Optical Communications (WSEAS) international conference on Signal Processing, Robotics and Automation, and proceedings of the 4th WSEAS international conference on Nanotechnology, pp 138–142

  25. Jabid T, Kabir MH, Chae O (2010) Robust facial expression recognition based on local directional pattern. ETRI J 32(5):784–794

    Article  Google Scholar 

  26. Kahou SE, Froumenty P, Pal C (2015) Facial expression analysis based on high dimensional binary features. In: Agapito L., Bronstein M., Rother C. (eds) computer vision - ECCV 2014 workshops. ECCV 2014. Lecture Notes in Computer Science, Springer, Cham, vol 8926, pp 135–147

  27. KhanH SM, BibiN SJH, HaiderS A, ZainabS UM, BashirY MN (2019) Localization of radiance transformation for image Dehazing in wavelet domain. Neurocomputing 381:141–151. https://doi.org/10.1016/j.neucom.2019.10.005

    Article  Google Scholar 

  28. Kim YK, Lim JG, Kim MH (2015) Comparison of lip image feature extraction methods for improvement of isolated word recognition rate. Adv. Sci. Lett. 107:57–61. https://doi.org/10.14257/astl.2015.107.14

  29. Ko KE, Sim KB (2010) Emotion recognition in facial image sequences using a combination of AAM with FACS and DBN. In: the 3rd international conference on intelligent robotics and applications (ICIRA’10), springer-Verlag, Berlin, Heidelberg, vol. 1, pp 702–712

  30. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Pereira F, Burges, C, Bottou L, Weinberger K (eds). Adv Neural Inf Proces Syst 25(2): 1097–1105. https://doi.org/10.1145/3065386

  31. Lee YH, Kim CG, Kim Y, Whangbo TK (2013) Facial landmarks detection using improved active shape model on android platform. Multimed Tools Appl 74(20):8821–8830. https://doi.org/10.1007/s11042-013-1565-y

    Article  Google Scholar 

  32. Lewis TW, Powers DMW (2001) Lip feature extraction using red exclusion. In: selected papers from the pan-Sydney area workshop on visual information processing (VIP ‘00). Sydney, Australia, vol 2:61–67

    Google Scholar 

  33. Liang S, Wu J et al. (2013) Improved detection of landmarks on 3d human face data. In: 35th international conference of the IEEE engineering in medicine and biology society (EMBC), pp 6482–6485

  34. Lienhart R, Maydt J (2002) An extended set of haar-like features for rapid object detection. In: international conference on image processing, vol. 1, pp I–900

  35. Littlewort G, Whitehill J et al. (2011) The computer expression recognition toolbox (cert). In: IEEE international conference on automatic face gesture recognition and workshops (FG 2011), pp 298–305

  36. Lowe DG (1999) Object recognition from local scale-invariant features. In: 7th IEEE international conference on computer vision, vol. 2, pp 1150–1157

  37. Lucey P, Cohn JF et al. (2010) The extended Cohn-Kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression. In: IEEE computer society conference on computer vision and pattern recognition workshops (CVPRW), pp 94–101

  38. Lyons M, Akamatsu S, Kamachi M, Gyoba J (1998) Coding facial expressions with gabor wavelets. In: 3rd IEEE international conference on automatic face gesture recognition, Nara, Japan, pp 200–205

  39. Lyons MJ, Budynek J, Akamatsu S (1999) Automatic classification of single facial images. IEEE transaction on pattern analysis and machine Intelligence 21(12):1357–1362. https://doi.org/10.1109/34.817413.

  40. Ma L, Khorasani K (2004) Facial expression recognition using constructive feed forward neural networks. IEEE Transactions on Systems, Man, and cybernetics 34(3):1588–1595

    Article  Google Scholar 

  41. Mahmood Z, Muhammad N,Bibi N,Ali T (2017) A review on state-of-the-art face recognition approaches. Fractals 25(2) 1750025 (19 pages). https://doi.org/10.1142/S0218348X17500256

  42. Mahmood Z, Bibi N, Usman M. Khan U (2019) Mobile cloud based-framework for sports applications. Multidimensional Systems and Signal Processing;1–29. https://doi.org/10.1007/s11045-019-00639-6

  43. Matthews I, Cootes T, Bangham J, Cox S, Harvey R (2002) Extraction of visual features for lip reading. IEEE Trans Pattern Anal Mach Intell 24(2):198–213

    Article  Google Scholar 

  44. Nguyen HT, Smeulders AWM (2004) Fast occluded object tracking by a robust appearance filter. IEEE Trans Pattern Anal Mach Intell 26(8):1099–1104

    Article  Google Scholar 

  45. Ouellet S (2014) Real-time emotion recognition for gaming using deep convolutional network features. http://arxiv.org/abs/1408.3750.

  46. Pali V, Goswami S, Bhaiya L (2014) An extensive survey on feature extraction techniques for facial image processing. In: 6th international conference on computational intelligence and communication networks, pp 142–148

  47. Pardàs M, Bonafonte A, Landabaso J (2002) Emotion recognition based on MPEG4 facial animation parameters. In: IEEE International Conference on Acoustics, Speech, and Signal Processing, Orlando, FL, pp IV-3624-IV-3627

  48. Pentland A, Moghaddam B, Starner T (1994) View-based and modular eigen spaces for face recognition. In: IEEE conference on computer vision and pattern recognition. Seattle, NA, pp 84–91

    Google Scholar 

  49. Qi C, Li M, et al. (2018) Facial Expressions Recognition Based on Cognition and Mapped Binary Patterns. Special Section On Trends, Perspectives And Prospects Of Machine Learning Applied To Biomedical Systems In Internet Of Medical Things, IEEE Access, vol 6

  50. Shan C, Gong S, McOwan PW (2009) Facial expression recognition based on local binary patterns: a comprehensive study. Image vision Computing 27(6):803–816. https://doi.org/10.1016/j.imavis.2008.08.005

    Article  Google Scholar 

  51. Shen X, Wu WY (2015). An algorithm of lips secondary positioning and feature extraction based on YCBCR color space. In: International Conference on Advances in Mechanical Engineering and Industrial Informatics Atlantis Press. https://doi.org/10.2991/ameii-15.2015.271

  52. Su C, Deng J, Yang Y, Wang G (2010) Expression recognition methods based on feature fusion. In: Yao Y, Sun R, Poggio T, Liu J, Zhong N, Huang J. (eds) Brain informatics (BI 2010). Lecture Notes in Computer Science, Springer, Berlin, Heidelberg, vol 6334, pp 346–356

  53. Susskind JM, Littlewort G, Bartlett MS, Movellan J, Anderson AK (2007) Human and computer recognition of facial expressions of emotion. Neuropsychologia, Elsevier 45(1):152–162

    Article  Google Scholar 

  54. Swaminathan A, Vadivel A, Arock M (2020) FERCE: facial expression recognition for combined emotions using FERCE algorithm. IETE J Res:1–16. https://doi.org/10.1080/03772063.2020.1756471

  55. Tomasi C, Manduchi R (1998) Bilateral filtering for gray and color images. In: IEEE 6th international conference on computer vision. Bombay, India, pp 839–846

    Google Scholar 

  56. Viola P, Jones M (2004) Robust real-time face detection. Int J Comput Vis 57(2):137–154

    Article  Google Scholar 

  57. Wang L, Li RF, Wang K, Chen J (2015) Feature representation for facial expression recognition based on FACS and LBP. Int J Autom Comput 11(5):459–468

    Article  Google Scholar 

  58. Yan S, Shan S, Chen X, Gao W (2008) Locally assembled binary (LAB) feature with feature-centric cascade for fast and accurate face detection. In: IEEE conference on computer vision and pattern recognition (CVPR 2008), pp 1–7

  59. Yang MT, Cheng YJ, Shih YC (2011) Facial expression recognition for learning status analysis. Springer, Berlin Heidelberg, pp 131–138

    Google Scholar 

  60. Yang W, Sun C, Zheng W, Ricanek K (2017) Gender classification using 3D statistical models. Multimed Tools Appl 76(3):4491–4503. https://doi.org/10.1007/s11042-016-3446-7

    Article  Google Scholar 

  61. ZhouF KS, FowlkesC C, ChenT LB (2020) Fine-grained facial expression analysis using dimensional emotion model. Neurocomputing 392(7):38–49

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of CSE, Dayananda Sagar University, Bangalore, India

    V. Suma Avani & S. G. Shaila

  2. Department of CSE, SRM University, Amaravati, Andra Pradesh, India

    A. Vadivel

Authors
  1. V. Suma Avani
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. S. G. Shaila
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. A. Vadivel
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to A. Vadivel.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Avani, V.S., Shaila, S.G. & Vadivel, A. Interval graph of facial regions with common intersection salient points for identifying and classifying facial expression. Multimed Tools Appl 80, 3367–3390 (2021). https://doi.org/10.1007/s11042-020-09806-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-09806-5

Keywords

Search

Navigation