Skip to main content
SpringerLink
Log in

Landmark triangulation-induced Altitude Signature for change detection of human emotion from face image sequence

  • S.I. : Multifaceted Intelligent Computing Systems (MICS)
  • Published:
Innovations in Systems and Software Engineering Aims and scope Submit manuscript

Abstract

Video-based facial expression recognition is a potential alternative for detecting the transitional processes of human emotion. In this paper, we present the landmark triangulation-induced altitude signatures to depict gradual changes in human emotion from face video frames. In our proposed approach, we consider the geometry-based triangulation technique which results from the triangles formed by important landmark points on face images. In this work, we measure three altitudes from all triangles and generate three different variations of altitude signature viz. AS1, AS2 and AS3 for the classification of six basic emotions viz. anger (AN), disgust (DI), fear (FE), happiness (HA), sadness (SA) and surprise (SU) in several ways. The performance capability of our proposed recognition system is tested on three benchmark face video datasets: Extended Cohn–Kanade (CK+), M&M Initiative (MMI), and Multimedia Understanding Group (MUG) through the application of Multilayer Perceptron (MLP) classifier and also validated by tenfold cross-validation method. Experimental results, 98.77%, 93.06%, and 98.75% on CK+, MMI, and MUG, respectively, are found very impressive. System efficacy is vindicated by showing the superiority of the proposed technique over the other existing techniques.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Zabih R, Woodfill J (1994) Non-parametric local transforms for computing visual correspondence. In: European conference on computer vision, pp 151–158. Springer

  2. Mehrabian A (2007) Nonverbal communication. New Brunswick

  3. Tao J, Tan T (2005) Affective computing: a review. In: International conference on affective computing and intelligent interaction. Springer, pp 981–995

  4. Beat F, Juergen L (2003) Automatic facial expression analysis: a survey. Pattern Recognit 36(1):259–275

    Article  Google Scholar 

  5. Ekman P, Friesen Wallace V (1971) Constants across cultures in the face and emotion. J Personal Soc Psychol 17(2):124

    Article  Google Scholar 

  6. Caifeng S, Shaogang G, McOwan Peter W (2009) Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis Comput 27(6):803–816

    Article  Google Scholar 

  7. Pantic M, Rothkrantz LJM (2004) Facial action recognition for facial expression analysis from static face images. IEEE Trans Syst Man Cybern Part B (Cybernetics) 34(3):1449–1461

  8. Naveen Kumar HN, Jagadeesha S, Jain Amith K (2016) Human facial expression recognition from static images using shape and appearance feature. In: 2016 2nd international conference on applied and theoretical computing and communication technology (iCATccT), pp 598–603. IEEE

  9. Akkoca BS, Gökmen M (2014) Facial expression recognition from static images. In: 2014 22nd signal processing and communications applications conference (SIU), pp 1291–1294. IEEE

  10. Ghimire D, Jeong S, Lee J, Park SH (2017) Facial expression recognition based on local region specific features and support vector machines. Multimed Tools Appl 76(6):7803–7821

    Article  Google Scholar 

  11. Irene K, Ioannis P (2006) Facial expression recognition in image sequences using geometric deformation features and support vector machines. IEEE Trans Image Process 16(1):172–187

    MathSciNet  Google Scholar 

  12. Lajevardi SM, Lech M (2008) Facial expression recognition from image sequences using optimized feature selection. In: 2008 23rd international conference image and vision computing New Zealand, pp 1–6. IEEE

  13. Haris RA (2018) Face expression recognition using artificial neural network (ANN) model back propagation. In: Proceedings of the joint workshop KO2PI and the 1st international conference on advance & scientific innovation, pp 126–134. ICST (Institute for Computer Sciences, Social-Informatics and ...)

  14. Yang S, Bhanu B (2012) Understanding discrete facial expressions in video using an emotion avatar image. IEEE Trans Syst Man Cybern Part B (Cybernetics) 42(4):980–992

  15. Zavaschi Thiago HH, Britto Jr Alceu S, Oliveira Luiz ES, Koerich Alessandro L (2013) Fusion of feature sets and classifiers for facial expression recognition. Expert Syst Appl 40(2):646–655

    Article  Google Scholar 

  16. Wan C, Tian Y, Liu S (2012) Facial expression recognition in video sequences. In: Proceedings of the 10th world congress on intelligent control and automation, pp 4766–4770. IEEE

  17. Liu X, Cheng T (2003) Video-based face recognition using adaptive hidden Markov models. In: 2003 IEEE computer society conference on computer vision and pattern recognition, 2003. Proceedings, vol 1, p I. IEEE

  18. Pantic M, Patras I (2006) Dynamics of facial expression: recognition of facial actions and their temporal segments from face profile image sequences. IEEE Trans Syst Man Cybern Part B (Cybernetics) 36(2):433–449

  19. Khairuni AKS, Hisham JM, Jussi P, Rajendran P (2016) Spatiotemporal feature extraction for facial expression recognition. IET Image Process 10(7):534–541

    Article  Google Scholar 

  20. Yaddaden Y, Adda M, Bouzouane A, Gaboury S, Bouchard B (2017) Facial expression recognition from video using geometric features

  21. Anwar S, Ayoub A-H, Robert N, Moftah E (2014) Frame-based facial expression recognition using geometrical features. Adv Human-Comput Interact 2014:4

    Google Scholar 

  22. Nuri Ö, Mehmet BG (2021) New geometric based features for facial expression recognition. Technical report, EasyChair

  23. Murugappan M, Mutawa A (2021) Facial geometric feature extraction based emotional expression classification using machine learning algorithms. Plos One 16(2):e0247131

    Article  Google Scholar 

  24. Singh A, Khan MA, Baghel N (2020) Face emotion identification by fusing neural network and texture features: facial expression. In: 2020 international conference on contemporary computing and applications (IC3A), pp 187–190. IEEE

  25. Chang L, Kaoru H, Junjie M, Zhiyang J, Yaping D (2021) Facial expression recognition using hybrid features of pixel and geometry. IEEE Access 9:18876–18889

    Article  Google Scholar 

  26. Deepak G, Joonwhoan L, Ze-Nian L, Sunghwan J (2017) Recognition of facial expressions based on salient geometric features and support vector machines. Multimed Tools Appl 76(6):7921–7946

    Article  Google Scholar 

  27. Asit B, Paramartha D (2019) Influence of shape and texture features on facial expression recognition. IET Image Process 13(8):1349–1363

    Article  Google Scholar 

  28. Asit B, Paramartha D (2019) Facial expression recognition using distance and texture signature relevant features. Appl Soft Comput 77:88–105

    Article  Google Scholar 

  29. Lei Z, Zengcai W, Zhang G (2017) Facial expression recognition from video sequences based on spatial-temporal motion local binary pattern and gabor multiorientation fusion histogram. Mathematical Problems in Engineering

  30. Deepak G, Joonwhoan L (2014) Extreme learning machine ensemble using bagging for facial expression recognition. J Inf Process Syst 10(3):443–458

    Article  Google Scholar 

  31. Happy SL, Routray A (2015) Robust facial expression classification using shape and appearance features. In: 2015 eighth international conference on advances in pattern recognition (ICAPR), pp 1–5. IEEE

  32. Cootes Timothy F, Taylor Christopher J, Cooper David H, Jim G (1995) Active shape models-their training and application. Comput Vis Image Underst 61(1):38–59

    Article  Google Scholar 

  33. Tzimiropoulos G, Pantic M (2013) Optimization problems for fast AAM fitting in-the-wild. In: Proceedings of the IEEE international conference on computer vision, pp 593–600

  34. Barman A, Dutta P (2021) Facial expression recognition using distance and shape signature features. Pattern Recognit Lett 145:254–261

  35. Ying-Ming W, Wang H-W, Lu Y-L, Yen S, Hsiao Y-T (2012) Facial feature extraction and applications: a review. In: Asian conference on intelligent information and database systems. Springer, pp 228–238

  36. Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I (2010) The extended cohn-kanade dataset (ck+): a complete dataset for action unit and emotion-specified expression. In: 2010 IEEE computer society conference on computer vision and pattern recognition-workshops, pp 94–101. IEEE

  37. Aifanti N, Papachristou C, Delopoulos A (2010) The mug facial expression database. In: 11th international workshop on image analysis for multimedia interactive services WIAMIS 10, pp 1–4. IEEE

  38. Valstar M, Pantic M (2010) Induced disgust, happiness and surprise: an addition to the mmi facial expression database. In: Proceedings of 3rd international workshop on EMOTION (satellite of LREC): Corpora for research on emotion and affect, p 65. Paris, France

  39. Amin M, Hassan A, Farzad T (2015) Video-based facial expression recognition by removing the style variations. IET Image Process 9(7):596–603

    Article  Google Scholar 

  40. Yongqiang L, Shangfei W, Yongping Z, Qiang J (2013) Simultaneous facial feature tracking and facial expression recognition. IEEE Trans Image Process 22(7):2559–2573

    Article  Google Scholar 

  41. Yogachandran R, Phan Raphael C-W, Chambers Jonathon A, Parish David J (2012) Facial expression recognition in the encrypted domain based on local fisher discriminant analysis. IEEE Trans Affect Comput 4(1):83–92

    Google Scholar 

  42. Shiqing Z, Xianzhang P, Yueli C, Xiaoming Z, Limei L (2019) Learning affective video features for facial expression recognition via hybrid deep learning. IEEE Access 7:32297–32304

    Article  Google Scholar 

  43. Anima M, Laxmidhar B, Subramanian Venkatesh K (2014) Emotion recognition from geometric facial features using self-organizing map. Pattern Recognit 47(3):1282–1293

    Article  Google Scholar 

Download references

Acknowledgements

The authors express their gratitude to Dr. A. Delopoulos and Prof. Maja Pantic for providing the MUG and MMI database free for carrying out this work. The authors gratefully acknowledge the support of the Department of Science and Technology, Ministry of Science and Technology, Government of India, for providing the DST-INSPIRE Fellowship (INSPIRE Reg. no. IF160285, Ref. No.: DST/INSPIRE Fellowship/[IF160285]) to carry out the research. The authors are also thankful to the Department of Computer and System Sciences, Visva-Bharati, Santiniketan for giving the research infrastructure.

Author information

Authors and Affiliations

  1. Visva-Bharati University, Santiniketan, WestBengal, India

    Md Nasir, Paramartha Dutta & Avishek Nandi

Authors
  1. Md Nasir
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Paramartha Dutta
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Avishek Nandi
    View author publications

    You can also search for this author in PubMed Google Scholar

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Nasir, M., Dutta, P. & Nandi, A. Landmark triangulation-induced Altitude Signature for change detection of human emotion from face image sequence. Innovations Syst Softw Eng (2021). https://doi.org/10.1007/s11334-021-00415-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11334-021-00415-5

Keywords

Search

Navigation