Skip to main content
SpringerLink
Log in

Development of a Toolkit for Online Analysis of Facial Emotion

  • Conference paper
  • First Online:
XXVI Brazilian Congress on Biomedical Engineering

Part of the book series: IFMBE Proceedings ((IFMBE,volume 70/1))

Abstract

Facial emotion recognition is an important topic in the fields of computer vision and artificial intelligence, since has the potential to be a powerful tool to develop a wide variety of academic and commercial applications, such as human-computer interaction systems. Facial expression communication is especially effective because visual expressions are one of the main information channels in interpersonal communication. This work presents the development of a toolkit for Matlab, which allows online analysis of facial emotion. To facilitate the design of this system, four modules are implemented, which allow: data acquisition, feature extraction, expression classifier and graphic report of analysis. The results show that the six basic emotion classes were recognized by the computational system, with accuracy of 63.0% and 68.8% for LDA and KNN classifiers, respectively. These results are close to the success rates of other systems found in the literature that have an average of 63.2% accuracy. The use of the platform and the methods implemented in this work can benefit automatic emotion recognition applications, which require online processing and evaluation of human emotion objectively and non-intrusively.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Maat, L., Pantic, M.: Gaze-x: adaptive, affective, multimodal interface for single-user office scenarios. In: Artifical Intelligence for Human Computing. Springer (2007)

    Google Scholar 

  2. Devault, D., Artstein, R., Benn, G., Dey, T., Fast, E., Gainer, A., Morency, L.: Virtual human interviewer for healthcare decisión support. AAMAS (2014)

    Google Scholar 

  3. Ishiguro, H., Ono, T., Imai, M., Maeda, T., Kanda, T., Nakatsu, R.: Robovie: an interactive humanoid robot. Ind. Robot. Int. J. 28(6), 498–504 (2001)

    Article  Google Scholar 

  4. Kapoor, A., Burleson, W., Picard, R.W.: Automatic prediction of frustration. IJHCS 65(8), 724–736 (2007)

    Google Scholar 

  5. Blom, P.M., Bakkes, S., Tan, C.T., Whiteson, S., Roijers, D., Valenti, R., Gevers, T.: Towards personalised gaming via facial expression recognition. AIIDE (2014)

    Google Scholar 

  6. Irani, R., Nasrollahi, K., Simon, M.O., Corneanu, C.A., Escalera, S., Bahnsen, C., Lundtoft, D.H., Moeslund, T.B., Pedersen, T., Klitgaard, M.: Spatiotemporal analysis of rgb-dt facial images for multimodal pain level recognition. CVPR Workshops (2015)

    Google Scholar 

  7. Ryan, A., Cohn, J., Lucey, S., Saragih, J., Lucey, P., Torre, F., Ross, A.: Automated facial expression recognition system. ICCST (2009)

    Google Scholar 

  8. Vural, E., Cetin, M., Ercil, A., Littlewort, G., Bartlett, M., Movellan, J.: Drowsy driver detection through facial movement analysis. In: Human–Computer Interaction (2007)

    Google Scholar 

  9. Scherer, S., Stratou, G., Mahmoud, M., Boberg, J., Gratch, J., Rizzo, A., Morency, L.: Automatic behavior descriptors for psychological disorder analysis. In: 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG). IEEE (2013)

    Google Scholar 

  10. Koesltra, S., Mühl, C., Soleyman, M., Lee, J. S., Yazdani, A., Ebrahimi, T., Pun, T., Nijholt, A., Patras, I.: DEAP: a database for emotion analysis using physiological signals. Trans. Affect. Comput. 3(1) (2012)

    Google Scholar 

  11. Kaulard, K., Cunningham, D.W., Bülthoff, H.H., Wallraven, C.: The MPI facial expression database: a validated database of emotional and conversational facial expressions. PLoS ONE 7, e32321 (2012)

    Article  Google Scholar 

  12. Ekman, P., Friesen, W.: The Facial Action Coding System: A Technique For The Measurement of Facial Movement. Consulting Psychologists Press, Inc., San Francisco, CA (1978)

    Google Scholar 

  13. Essa, I., Pentland, A.: Coding, analysis, interpretation, and recognition of facial expressions. IEEE Trans. Pattern Anal. Mach. Intell. 19(7), 757–763 (1997)

    Article  Google Scholar 

  14. Bartlett, M., Hager, J., Ekman, P., Sejnowski, T.: Measuring facial expressions by computer image analysis. Psychophysiology 36, 253–264 (1999)

    Article  Google Scholar 

  15. Tadas, B., Peter, R., Louis-Philippe, M.: OpenFace: an open source facial behavior analysis toolkit. In: IEEE Winter Conference on Applications of Computer Vision (2016)

    Google Scholar 

  16. Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I.: The extended Cohn-Kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. San Francisco, CA, USA, pp. 94–101 (2010)

    Google Scholar 

  17. Tadas, B., Marwa, M., Peter, R.: Cross-dataset learning and person-specific normalisation for automatic action unit detection in facial expression recognition and analysis challenge. In: IEEE International Conference on Automatic Face and Gesture Recognition (2015)

    Google Scholar 

  18. Jiawei, H., Micheline, K., Jian, P.: Data Mining: Concepts and Techniques. Elsevier, USA, Third edition (2012)

    MATH  Google Scholar 

  19. Rivera, H., Goulart, C., Favarato, A., Valadão, C., Caldeira, E., Bastos, T.: Development of an automatic expression recognition system based on facial action coding system. In: Simpósio Brasileiro de Automação Inteligente (SBAI2017) (2017)

    Google Scholar 

  20. Huang, M.H., Wang, Z.W., Ying, Z.L.: A new method for facial expression recognition based on sparse representation plus LBP. In: Proceedings of the International Congress on Image and Signal Processing, pp. 1750–1754. Yantai, China. Sensors 2018, 18, 401 20 of 20 (2010)

    Google Scholar 

  21. Zhen, W., Zilu, Y.: Facial expression recognition based on local phase quantization and sparse representation. In: Proceedings of the IEEE International Conference on Natural Computation, pp. 222–225. Chongqing, China (2012)

    Google Scholar 

  22. Zhang, S., Zhao, X., Lei, B.: Robust facial expression recognition via compressive sensing. Sensors 3747–3761 (2012)

    Article  Google Scholar 

  23. Zhao, G., Pietikainen, M.: Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans. Pattern Anal. Mach. Intell. 29, 915–928 (2007)

    Article  Google Scholar 

  24. Jiang, B., Valstar, M.F., Pantic, M.: Action unit detection using sparse appearance descriptors in space-time video volumes. In: Proceedings of the IEEE International Conference and Workshops on Automatic Face & Gesture Recognition, pp. 314–321. Santa Barbara, CA, USA (2011)

    Google Scholar 

  25. Lee, S.H., Baddar, W.J., Ro, Y.M.: Collaborative expression representation using peak expression and intra class variation face images for practical subject-independent emotion recognition in videos. Pattern Recognit. 54, 52–67 (2016)

    Article  Google Scholar 

  26. Rivera, H., Bastos, T.: Development of a Multisensorial System for Emotion Recognition. Universidade Federal do Espírito Santo, Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (2017)

    Google Scholar 

  27. Rivera, H., Goulart, C., Caldeira, E., Bastos, T.: Using eye-tracking for study about valence and emotional facial expressions. In: Congresso Brasileiro de Engenharia Biomédica, 2016, Foz de Iguaçu. Anais do Congresso Brasileiro de Engenharia Biomédica (2016)

    Google Scholar 

  28. Rivera, H., Bissoli, A., Goulart, C., Caldeira, E., Bastos, T.: Development of matlab toolbox for eye tracking systems. In: Congresso Brasileiro de Automática, 2016, Vitória. Anais do Congresso Brasileiro de Automática (2016)

    Google Scholar 

  29. Walecki, R., Rudovic, O.: Deep structured learning for facial expression intensity esti-mation. Image Vis. Comput. 259, 143–154 (2017)

    Google Scholar 

  30. Kahou, S.E., Michalski, V., Konda, K.: Recurrent neural networks for emotion recognition in video. In: Proceedings of the ACM on International Conference on Multi-modal Interaction, pp. 467–474. Seattle, WD, USA (2015)

    Google Scholar 

Download references

Acknowledgements

The authors would like to thank Federal University of Espirito Santo (UFES), Ryerson University, CNPq, CAPES and ELAP Program for financial support and scholarships.

Author information

Authors and Affiliations

  1. Postgraduate Program in Electrical Engineering, Federal University of Espirito Santo, Vitoria, 29075-910, Brazil

    Hamilton Rivera, Carlos Valadão, Eliete Caldeira & Teodiano Freire Bastos-Filho

  2. Department of Electrical and Computer Engineering, Ryerson University, Toronto, Canada

    Hamilton Rivera & Sridhar Krishnan

Authors
  1. Hamilton Rivera
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Carlos Valadão
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Eliete Caldeira
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Sridhar Krishnan
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Teodiano Freire Bastos-Filho
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hamilton Rivera .

Editor information

Editors and Affiliations

  1. INMETRO-Instituto Nacional de Metrologia, Qualidade e Tecnologia, Duque de Caxias, Rio de Janeiro, Brazil

    Rodrigo Costa-Felix

  2. Federal University of Rio de Janeiro, Rio de Janeiro, Brazil

    João Carlos Machado

  3. INMETRO-Instituto Nacional de Metrologia, Qualidade e Tecnologia, Duque de Caxias, Rio de Janeiro, Brazil

    André Victor Alvarenga

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Rivera, H., Valadão, C., Caldeira, E., Krishnan, S., Bastos-Filho, T.F. (2019). Development of a Toolkit for Online Analysis of Facial Emotion. In: Costa-Felix, R., Machado, J., Alvarenga, A. (eds) XXVI Brazilian Congress on Biomedical Engineering. IFMBE Proceedings, vol 70/1. Springer, Singapore. https://doi.org/10.1007/978-981-13-2119-1_95

Download citation

  • DOI: https://doi.org/10.1007/978-981-13-2119-1_95

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-13-2118-4

  • Online ISBN: 978-981-13-2119-1

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation