Skip to main content
SpringerLink
Log in

Performance Comparison of Facial Emotion Recognition: A Transfer Learning-Based Driver Assistance Framework for In-Vehicle Applications

  • Published:
Circuits, Systems, and Signal Processing Aims and scope Submit manuscript

Abstract

This research proposes facial emotion recognition (FER) using a transfer learning-based approach for a safe drive. To deal with the increasing road accidents, it is important to keep an eye on the facial expressions of the driver. It aims to develop a FER system that will monitor the facial expressions of the driver to identify their emotions and provide immediate assistance for safety. The developed system will be useful for in-vehicle embedded system applications, primarily using custom architectural designs through transfer learning techniques with reduced parameters. Pre-trained convolutional neural network (CNN) models such as AlexNet, SqueezeNet, and VGG19 are used to develop transfer learning frameworks. The performance of the proposed transfer learning-based model has been evaluated on publicly available benchmark databases such as FER2013, JAFFE, KDEF, CK+, SFEW, and KMU-FED. The performance is compared with previously reported techniques to observe the efficacy of the proposed model. The proposed transfer learning-based model has shown good performance on the FER benchmark database. The pre-trained VGG19 model showed better performance in most benchmark databases than the pre-trained AlexNet and SqueezeNet models. However, the pre-trained VGG19 model only achieved the best performance accuracy of 99.7% with the KMU-FED database and achieved comparable performance to the rest of the benchmark database. Furthermore, compared with state-of-the-art technologies, the pre-trained AlexNet and pre-trained SqueezeNet models show improved performance of 99.4% only for KMU-FED datasets and comparable accuracy to other databases.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Data Availability Statement

The authors confirm that the data supporting the findings of the proposed technique in the article title “Performance Comparison of Facial Emotion Recognition: A Transfer Learning-Based Driver Assistance Framework for In-Vehicle Applications” are openly available in the “CK+, FER2013, JAFFE, KDEF, SFEW and KMU-FED” benchmark databases. The said databases can be accessed on web page https://www.kaggle.com/shawon10/ckplus,

https://www.kaggle.com/c/challenges-in-representation-learning-facial-expression-recognition-challenge,

https://zenodo.org/record/3451524#.YtVt2HZBxD8,

https://www.kdef.se/,

https://cs.anu.edu.au/few/,

and https://cvpr.kmu.ac.kr/KMU-FED.htm.

These databases were last accessed by the authors for the implementation used on 17th Nov 2022.

Notes

  1. CNN:Convolutional Neural Networks, LA-Net:lightweight attention Deep-CNN, SCAE:Stacked Convolutional Auto-Encoder, LMRF:lightweight multilayer random forest, LEMHI:local enhanced motion history image, LSTM:Long Short-Term Memory, WRF:Weighted random forest, GGDA:Grassmann Graph embedding Discriminant Analysis, RELM:Regularized Extreme Learning Machine, SVM:Support Vector Machine, PCANet:Principal Component Analysis Network, LDANet:Linear Discriminant Analysis Network, LPQ:Local Phase Quantization, PHOG:Pyramid of Histogram of Oriented Gradients, Action Unit (AU), LBP:Local Binary Pattern, REL-GAN:Reenactment-based Expression-Representation Learning Generative Adversarial Network.

References

  1. Facial Expression Recognition 2013 Dataset (FER2013), (Online). Available: https://www.kaggle.com/c/challenges-in-representation-learning-facial-expression-recognition-challenge

  2. KMU-FED. Accessed: Feb. 22, 2022. (Online). Available: https://cvpr.kmu.ac.kr/KMU-FED.htm

  3. A. Aggarwal, S. Garg, R. Madaan, R. Kumar, Comparison of different machine learning and deep learning emotion detection models, in Intelligent Computing and Communication Systems. (Springer, 2021), pp. 401–408

  4. M. Akhand, S. Roy, N. Siddique, M.A.S. Kamal, T. Shimamura, Facial emotion recognition using transfer learning in the deep CNN. Electronics 10(9), 1036 (2021)

    Article  Google Scholar 

  5. K. Ali, C.E. Hughes, Face reenactment based facial expression recognition. in International Symposium on Visual Computing, (Springer, 2020), pp. 501–513

  6. H. Alshamsi, V. Kepuska, H. Meng, Real time automated facial expression recognition app development on smart phones. in Proceedings of the IEEE 8th Annual Information Technology, Electronics and Mobile Communication Conference (IEMCON), (2017), pp. 384–392

  7. Y.K. Bhatti, A. Jamil, N. Nida, M.H. Yousaf, S. Viriri, S.A. Velastin, Facial expression recognition of instructor using deep features and extreme learning machine. Computational Intelligence and Neuroscience (2021). https://doi.org/10.1155/2021/5570870

    Article  Google Scholar 

  8. C.M. Bishop, N.M. Nasrabadi, Pattern Recognition and Machine Learning, vol. 4 (Springer, Berlin, 2006)

    Google Scholar 

  9. C. Bisogni, A. Castiglione, S. Hossain, F. Narducci, S. Umer, Impact of deep learning approaches on facial expression recognition in healthcare industries. IEEE Trans. Ind. Inform. 18(8), 5619–5627 (2022)

    Article  Google Scholar 

  10. R.D. Blomberg, R.C. Peck, H. Moskowitz, M. Burns, D. Fiorentino, The long beach/fort lauderdale relative risk study. J. Saf. Res. 40(4), 285–292 (2009)

    Article  Google Scholar 

  11. J.D. Bodapati, U. Srilakshmi, N. Veeranjaneyulu, Fernet: a deep CNN architecture for facial expression recognition in the wild. J. Inst. Eng. (India): Ser. B 103(2), 439–448 (2022)

    Google Scholar 

  12. R.A. Borgalli, S. Surve, Deep learning framework for facial emotion recognition using CNN architectures. in 2022 International Conference on Electronics and Renewable Systems (ICEARS), IEEE (2022). pp. 1777–1784

  13. S. Chand, A. Singh, R. Bhatia, I. Kaur, K. Seeja, Real-time facial emotion recognition using deep learning. in Intelligent Computing and Communication Systems (Springer, Berlin 2021)m pp. 219–226

  14. C. Darwin, The Expression of the Emotions in Man and Animals by Charles Darwin (John Murray, London, 1872)

    Book  Google Scholar 

  15. A. Das, H. Gjerde, S.S. Gopalan, P.T. Normann, Alcohol, drugs, and road traffic crashes in India: a systematic review. Traffic Injury Prev. 13(6), 544–553 (2012)

    Article  Google Scholar 

  16. A. Dhall, R. Goecke, S. Lucey, T. Gedeon, Australian National University, Canberra, Australia, Technical Report TR-CS-11. Acted facial expressions in the wild database. 2, 1 (2011)

  17. A. Dhall, R. Goecke, S. Lucey, T. Gedeon, Static facial expression analysis in tough conditions: Data, evaluation protocol and benchmark. in IEEE International Conference on Computer Vision Workshops (ICCV Workshops) (2011), pp. 2106–2112

  18. H. Ding, S.K. Zhou, R. Chellappa, Facenet2expnet: regularizing a deep face recognition net for expression recognition. In Proceedings of the IEEE 12th International Conference on Automatic Face Gesture Recogn. (FG 2017) (2017), pp. 118–126

  19. P. Ekman, Biological and cultural contributions to body and facial movement, in The anthropology of the body (Academic Press Inc., 1977), pp. 39–84

  20. P. Ekman, An argument for basic emotions. Cogn. Emot. 6(3–4), 169–200 (1992)

    Article  Google Scholar 

  21. X. Fan, M. Jiang, A.R. Shahid, H. Yan, Hierarchical scale convolutional neural network for facial expression recognition. Cogn. Neurodyn. 16(4), 847–858 (2022)

    Article  Google Scholar 

  22. A. Gholamy, V. Kreinovich, O. Kosheleva, Why 70/30 or 80/20 relation between training and testing sets: a pedagogical explanation. Int. J. Intell. Syst. Technol. Appl. Stat. 11(2), 105–111 (2018)

    Google Scholar 

  23. P. Giannopoulos, I. Perikos, I. Hatzilygeroudis, Deep learning approaches for facial emotion recognition: a case study on fer-2013. in Advances in Hybridization of Intelligent Methods (Springer, 2018), pp. 1–16

  24. H. Gjerde, P.T. Normann, A.S. Christophersen, S.O. Samuelsen, J. Mørland, Alcohol, psychoactive drugs and fatal road traffic accidents in Norway: a case–control study. Accid. Anal. Prev. 43(3), 1197–1203 (2011)

    Article  Google Scholar 

  25. X. Glorot, A. Bordes, Y. Bengio, Deep sparse rectifier neural networks. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pp. 315–323. JMLR Workshop and Conference Proceedings (2011)

  26. T.S. Gunawan, A. Ashraf, B.S. Riza, E.V. Haryanto, R. Rosnelly, M. Kartiwi, Z. Janin, Development of video-based emotion recognition using deep learning with google colab. TELKOMNIKA 18(5), 2463–2471 (2020)

    Article  Google Scholar 

  27. F.N. Iandola, S. Han, M.W. Moskewicz, K. Ashraf, W.J. Dally, K. Keutzer, SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and \(<\) 0.5 MB model size. arXiv preprint arXiv:1602.07360 (2016)

  28. C.M.A. Ilyas, M. Rehm, K. Nasrollahi, Y. Madadi, T.B. Moeslund, V. Seydi, Deep transfer learning in human–robot interaction for cognitive and physical rehabilitation purposes. Pattern Anal. Appl. 25(3), 653–677 (2022)

    Article  Google Scholar 

  29. M. Jeong, B.C. Ko, Driver’s facial expression recognition in real-time for safe driving. Sensors 18(12), 4270 (2018)

    Article  Google Scholar 

  30. M. Jeong, J. Nam, B.C. Ko, Lightweight multilayer random forests for monitoring driver emotional status. IEEE Access 8, 60344–60354 (2020)

    Article  Google Scholar 

  31. V.H. Kamble, M.P. Dale, Machine learning approach for longitudinal face recognition of children, in Machine Learning for Biometrics (Elsevier, Amsterdam 2022), pp. 1–27

  32. M.N. Kartheek, M.V. Prasad, R. Bhukya, Windmill graph based feature descriptors for facial expression recognition. Optik 260, 169053 (2022)

    Article  Google Scholar 

  33. D. Keltner, P. Ekman, Emotion: an overview, in Encyclopedia of Psychology, vol. 3 (Oxford University Press, 2000), pp. 162–167

  34. U. Khalil, T. Javid, A. Nasir, Automatic road accident detection techniques: a brief survey, in Proceedings of the International Symposium Wireless Systems and Networks (ISWSN) (2017), pp. 1–6

  35. A. Khattak, M.Z. Asghar, M. Ali, U. Batool, An efficient deep learning technique for facial emotion recognition. Multimedia Tools Appl. 18, 1649–1683 (2021)

    Google Scholar 

  36. A., Krishnadas, S. Nithin, A comparative study of machine learning and deep learning algorithms for recognizing facial emotions, in Proceedings of the IEEE 2nd International International Conference on Electronics and Sustainable Communication Systems (ICESC), (IEEE, 2021), pp. 1506–1512

  37. N. Kumari, R. Bhatia, Efficient facial emotion recognition model using deep convolutional neural network and modified joint trilateral filter. Soft Comput. 26(16), 7817–7830 (2022)

    Article  Google Scholar 

  38. J.H. Lacey, et al., national roadside survey of alcohol and drug use by drivers: drug results. Technical report, United States. National Highway Traffic Safety Administration (2007–2009)

  39. A. Latif, T.S. Gunawan, M. Kartiwi, F. Arifin, H. Mansor, Development of image-based emotion recognition using convolutional neural networks. in Proceedings of the IEEE 7th International Conference on Smart Instrumentation, Measurement and Applications (ICSIMA) (2021), pp. 47–52

  40. A. Leone, A. Caroppo, A. Manni, P. Siciliano, Vision-based road rage detection framework in automotive safety applications. Sensors 21(9), 2942 (2021)

    Article  Google Scholar 

  41. G. Levi, T. Hassner, Emotion recognition in the wild via convolutional neural networks and mapped binary patterns, in Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, pp. 503–510 (2015)

  42. S. Li, W. Deng, Deep facial expression recognition: a survey. IEEE Trans. Affect. Comput. 13, 1195–1215 (2020)

    Article  MathSciNet  Google Scholar 

  43. S. Li, W. Deng, J. Du, Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2852–2861 (2017)

  44. C.F. Liew, T. Yairi, Facial expression recognition and analysis: a comparison study of feature descriptors. IPSJ Trans. Comput. Vis. Appl. 7, 104–120 (2015)

    Article  Google Scholar 

  45. J. Liu, H. Wang, Y. Feng, An end-to-end deep model with discriminative facial features for facial expression recognition. IEEE Access 9, 12158–12166 (2021)

    Article  Google Scholar 

  46. M. Liu, S. Li, S. Shan, X. Chen, Au-aware deep networks for facial expression recognition. in 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG) (IEEE 2013), pp. 1–6

  47. Z. Lu, X. Jiang, A. Kot, Enhance deep learning performance in face recognition, in Proeedings of the IEEE 2nd International Conference on Image, Vision and Computing (ICIVC) (2017), pp. 244–248

  48. P. Lucey, J.F. Cohn, T. Kanade, J. Saragih, Z. Ambadar, I. Matthews, The extended cohn-kanade dataset (CK+): A complete dataset for action unit and emotion-specified expression, in Proceedings of the IEEE Computer Society Conferences Computer Vision and Pattern Recognition Workshops (2010), pp. 94–101

  49. D. Lundqvist, A. Flykt, A. Öhman, The Karolinska directed emotional faces (KDEF). CD ROM from Department of Clinical Neuroscience, Psychology section, Karolinska Institutet 91(630), 2 (1998)

  50. M.J. Lyons, J. Budynek, S. Akamatsu, Automatic classification of single facial images. IEEE Trans. Pattern Anal. Mach. Intell. 21(12), 1357–1362 (1999)

    Article  Google Scholar 

  51. H. Ma, T. Celik, H.C. Li, Lightweight attention convolutional neural network through network slimming for robust facial expression recognition. Signal Imag. Video Process. 15, 1507–1515 (2021)

    Article  Google Scholar 

  52. S. Minaee, M. Minaei, A. Abdolrashidi, Deep-emotion: facial expression recognition using attentional convolutional network. Sensors 21(9), 3046 (2021)

    Article  Google Scholar 

  53. H.W. Ng, V.D. Nguyen, V. Vonikakis, S. Winkler, Deep learning for emotion recognition on small datasets using transfer learning, in Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, pp. 443–449 (2015)

  54. Pham, L., Vu, T.H., Tran, T.A.: Facial expression recognition using residual masking network, in Proceedings of the IEEE 25th Indian Council of Philosophical Research (ICPR), pp. 4513–4519 (2021)

  55. Pretrained AlexNet convolutional neuralnetwork-MATLABalexnet-MathWorks India: (Accessed September 21, 2021)

  56. J.G. Ramaekers, G. Berghaus, M. van Laar, O.H. Drummer, Dose related risk of motor vehicle crashes after cannabis use. Drug Alcohol Depend. 73(2), 109–119 (2004)

    Article  Google Scholar 

  57. A. Ruiz-Garcia, M. Elshaw, A. Altahhan, V. Palade, Stacked deep convolutional auto-encoders for emotion recognition from facial expressions. in Proceedings of the IEEE International Joint Conference on Neural Networks (IJCNN) (2017), pp. 1586–1593

  58. G.K. Sahoo, S.K. Das, P. Singh, A deep learning-based distracted driving detection solution implemented on embedded system. Multimedia Tools Appl. 1–24 (2022). https://doi.org/10.1007/s11042-022-13450-6

  59. G.K. Sahoo, S.K. Das, P. Singh, Deep learning-based facial emotion recognition for driver healthcare, in Proceedings of the IEEE 28th National Conference on Communications (NCC) (2022), pp. 154–159

  60. A. Sajjanhar, Z. Wu, Q. Wen, Deep learning models for facial expression recognition, in Proceedings of the IEEE Digital Image Computing: Techniques and Applications (DICTA) (2018), pp. 1–6

  61. K. Saleh, M. Hossny, S. Nahavandi, Driving behavior classification based on sensor data fusion using lstm recurrent neural networks, in 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC) (IEEE, 2017), pp. 1–6

  62. S.A.R. Sekaran, C.P. Lee, K.M. Lim, Facial emotion recognition using transfer learning of alexnet, In Proceedings of the IEEE 9th International Conference on Information and Communication Technology (ICoICT) (2021). pp. 170–174

  63. J. Shao, Y. Qian, Three convolutional neural network models for facial expression recognition in the wild. Neurocomputing 355, 82–92 (2019)

    Article  Google Scholar 

  64. Z. Sun, H. Zhang, S. Ma, Z. Hu, Combining filtered dictionary representation based deep subspace filter learning with a discriminative classification criterion for facial expression recognition. Artif. Intell. Rev. 55(8), 6547–6566 (2022)

    Article  Google Scholar 

  65. M. Usman, S. Latif, J. Qadir, Using deep autoencoders for facial expression recognition. in Proceedings of the IEEE 13th International Conference on Emerging Technologies (ICET) (2017), pp. 1–6

  66. M. Valstar, M. Pantic, et al., Induced disgust, happiness and surprise: an addition to the mmi facial expression database, in Proceedingins of the 3rd International Workshop on EMOTION (satellite of LREC): Corpora for Research on Emotion and Affect (Paris, France. 2010), p. 65

  67. B. Verma, A. Choudhary, A framework for driver emotion recognition using deep learning and Grassmann manifolds, in Proceedings of the IEEE 21st International Conference on Intelligent Transportation Systems (ITSC) (2018), pp. 1421–1426

  68. J.M. Walsh, J.J. Gier, A.S. Christopherson, A.G. Verstraete, Drugs and driving. Traffic Inj. Prev. 5(3), 241–253 (2004)

    Article  Google Scholar 

  69. B. Yan, Z. Xiao, P. Yuan, K. Cai, Q. Chen, Facial expression recognition with convolutional neural networks via a data augmentation strategy (Association for the Advancement of Artificial Intelligence, 2021)

  70. J. Zhi, T. Song, K. Yu, F. Yuan, H. Wang, G. Hu, H. Yang, Multi-attention module for dynamic facial emotion recognition. Information 13(5), 207 (2022)

    Article  Google Scholar 

  71. D. Zhu, Y. Fu, X. Zhao, X. Wang, H. Yi, Facial emotion recognition using a novel fusion of convolutional neural network and local binary pattern in crime investigation. Comput. Intell. Neurosci. (2022). https://doi.org/10.1155/2022/2249417

    Article  Google Scholar 

Download references

Author information

Author notes

  1. Santos Kumar Das and Poonam Singh have contributed equally to this work.

Authors and Affiliations

  1. Department of Electronics and Communication Engineering, National Institute of Technology Rourkela, Rourkela, Odisha, 769008, India

    Goutam Kumar Sahoo, Santos Kumar Das & Poonam Singh

Authors
  1. Goutam Kumar Sahoo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Santos Kumar Das
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Poonam Singh
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Goutam Kumar Sahoo.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sahoo, G.K., Das, S.K. & Singh, P. Performance Comparison of Facial Emotion Recognition: A Transfer Learning-Based Driver Assistance Framework for In-Vehicle Applications. Circuits Syst Signal Process 42, 4292–4319 (2023). https://doi.org/10.1007/s00034-023-02320-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00034-023-02320-7

Keywords

Search

Navigation