Skip to main content
SpringerLink
Log in

Effective Facial Expression Recognition Through Multimodal Imaging for Traumatic Brain Injured Patient’s Rehabilitation

  • Conference paper
  • First Online:
Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2018)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 997))

Abstract

This article presents the issues related to applying computer vision techniques to identify facial expressions and recognize the mood of Traumatic Brain Injured (TBI) patients in real life scenarios. Many TBI patients face serious problems in communication and activities of daily living. These are due to restricted movement of muscles or paralysis with lesser facial expression along with non-cooperative behaviour, and inappropriate reasoning and reactions. All these aforementioned attributes contribute towards the complexity of the system for the automatic understanding of their emotional expressions. Existing systems for facial expression recognition are highly accurate when tested on healthy people in controlled conditions. However, their performance is not yet verified on the TBI patients in the real environment. In order to test this, we devised a special arrangement to collect data from these patients. Unlike the controlled environment, it was very challenging because these patients have large pose variations, poor attention and concentration with impulsive behaviours. In order to acquire high-quality facial images from videos for facial expression analysis, effective techniques of data preprocessing are applied. The extracted images are then fed to a deep learning architecture based on Convolution Neural Network (CNN) and Long Short-Term Memory (LSTM) network to exploit the spatiotemporal information with 3D face frontalization. RGB and thermal imaging modalities are used and the experimental results show that better quality of facial images and larger database enhance the system performance in facial expressions and mood recognition of TBI patients under natural challenging conditions. The proposed approach hopefully facilitates the physiotherapists, trainers and caregivers to deploy fast rehabilitation activities by knowing the positive mood of the patients.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 64.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 84.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Stuss, D.T., Levine, B.: Adult clinical neuropsychology: lessons from studies of the frontal lobes. Annu. Rev. Psychol. 53, 401–433 (2002). PMID: 11752491

    Article  Google Scholar 

  2. Levin, H.S., et al.: Relationship of depth of brain lesions to consciousness and outcome after closed head injury. J. Neurosurg. 69, 861–866 (1988)

    Article  Google Scholar 

  3. Khan, F., Baguley, I.J., Cameron, I.D.: 4: Rehabilitation after traumatic brain injury. Med. J. Aust. 178(6), 290–295 (2003). Wiley Online Library

    Google Scholar 

  4. Koskinen, S., Hokkinen, E.M., Sarajuuri, J., Alaranta, H.: Applicability of the ICF checklist to traumatically brain-injured patients in post-acute rehabilitation settings. J. Rehabil. Med. 39(6), 467–472 (2007). Medical Journals Limited

    Article  Google Scholar 

  5. Taylor, C.A., Bell, J.M., Breiding, M.J., Xu, L.: Traumatic brain injury–related emergency department visits, hospitalizations, and deaths–United States, 2007 and 2013. MMWR Surveill. Summ. 66(9), 1 (2017). Centers for Disease Control and Prevention

    Article  Google Scholar 

  6. Ilyas, C.M.A., Haque, M.A., Rehm, M., Nasrollahi, K., Moeslund, T.B.: Facial expression recognition for traumatic brain injured patients. In: Proceedings of the 13th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 4: VISAPP, INSTICC, SciTePress, pp. 522–530 (2018)

    Google Scholar 

  7. Dang, B., Chen, W., He, W., Chen, G.: Rehabilitation treatment and progress of traumatic brain injury dysfunction. Neural Plast. 2017, 6 (2017)

    Article  Google Scholar 

  8. Bird, J., Parente, R.: Recognition of nonverbal communication of emotion after traumatic brain injury. NeuroRehabilitation 34, 39–43 (2014)

    Google Scholar 

  9. Bender, A., et al.: Longterm-rehabilitation in patients with acquired brain injury. Dtsch. Ärztebl. Int. 113, 634–641 (2016)

    Google Scholar 

  10. Hyett, M.P., Parker, G.B., Dhall, A.: The utility of facial analysis algorithms in detecting Melancholia. In: Kawulok, M., Celebi, M.E., Smolka, B. (eds.) Advances in Face Detection and Facial Image Analysis, pp. 359–375. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-25958-1_13

    Chapter  Google Scholar 

  11. Chen, Y.: Face perception in schizophrenia spectrum disorders: interface between cognitive and social cognitive functioning. In: Ritsner, M. (ed.) Handbook of Schizophrenia Spectrum Disorders, vol. II, pp. 111–120. Springer, Dordrecht (2011). https://doi.org/10.1007/978-94-007-0831-0_5

    Chapter  Google Scholar 

  12. Klonovs, J., et al.: Monitoring technology. Distributed Computing and Monitoring Technologies for Older Patients. SCS, pp. 49–84. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-27024-1_4

    Chapter  Google Scholar 

  13. Haque, M.A., Irani, R., Nasrollahi, K., Moeslund, T.B.: Facial video-based detection of physical fatigue for maximal muscle activity. IET Comput. Vision 10, 323–329 (2016)

    Article  Google Scholar 

  14. Ilyas, C.M.A., Rehm, M., Nasrollahi, K., Moeslund, T.B.: Rehabilitation of traumatic brain injured patients: patient mood analysis from multimodal video. In: IEEE Signal Processing Society. IEEE Xplore (2018)

    Google Scholar 

  15. Rodriguez, P., et al.: Deep pain: exploiting long short-term memory networks for facial expression classification. IEEE Trans. Cybern. 99, 1–11 (2017)

    Article  Google Scholar 

  16. Bellantonio, M., et al.: Spatio-temporal pain recognition in CNN-based super-resolved facial images. In: Nasrollahi, K., et al. (eds.) FFER/VAAM -2016. LNCS, vol. 10165, pp. 151–162. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-56687-0_13

    Chapter  Google Scholar 

  17. Lauterbach, M.D., Notarangelo, P.L., Nichols, S.J., Lane, K.S., Koliatsos, V.E.: Diagnostic and treatment challenges in traumatic brain injury patients with severe neuropsychiatric symptoms: insights into psychiatric practice. Neuropsychiatr. Dis. Treat. 11, 1601–1607 (2015)

    Article  Google Scholar 

  18. Lander, K., Metcalfe, S.: The influence of positive and negative facial expressions on face familiarity. Memory 15, 63–69 (2007). PMID: 17479925

    Article  Google Scholar 

  19. Tian, Y.I., Kanade, T., Cohn, J.F.: Recognizing action units for facial expression analysis. IEEE Trans. Pattern Anal. Mach. Intell. 23, 97–115 (2001)

    Article  Google Scholar 

  20. Pantic, M., Patras, I.: Dynamics of facial expression: recognition of facial actions and their temporal segments from face profile image sequences. IEEE Trans. Syst. Man Cybern. Part B (Cybern.) 36, 433–449 (2006)

    Article  Google Scholar 

  21. Jiang, B., Valstar, M., Martinez, B., Pantic, M.: A dynamic appearance descriptor approach to facial actions temporal modeling. IEEE Trans. Cybern. 44, 161–174 (2014)

    Article  Google Scholar 

  22. Ghimire, D., Lee, J.: Geometric feature-based facial expression recognition in image sequences using multi-class adaboost and support vector machines. Sensors 13, 7714–7734 (2013)

    Article  Google Scholar 

  23. Haque, M.A., Nasrollahi, K., Moeslund, T.B.: Constructing facial expression log from video sequences using face quality assessment. In: 2014 International Conference on Computer Vision Theory and Applications (VISAPP), vol. 2, pp. 517–525 (2014)

    Google Scholar 

  24. Poursaberi, A., Noubari, H.A., Gavrilova, M., Yanushkevich, S.N.: Gauss-laguerre wavelet textural feature fusion with geometrical information for facial expression identification. EURASIP J. Image Video Process. 2012, 17 (2012)

    Article  Google Scholar 

  25. Saeed, A., Al-Hamadi, A., Niese, R., Elzobi, M.: Frame-based facial expression recognition using geometrical features. Adv. Hum.-Comput. Interact. 2014, 1–13 (2014)

    Article  Google Scholar 

  26. Lyons, M.J., Budynek, J., Akamatsu, S.: Automatic classification of single facial images. IEEE Trans. Pattern Anal. Mach. Intell. 21, 1357–1362 (1999)

    Article  Google Scholar 

  27. li Tian, Y.: Evaluation of face resolution for expression analysis. In: 2004 Conference on Computer Vision and Pattern Recognition Workshop, pp. 82–82 (2004)

    Google Scholar 

  28. Li, H., Hua, G.: Hierarchical-pep model for real-world face recognition. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4055–4064 (2015)

    Google Scholar 

  29. Huang, Z., Wang, R., Shan, S., Chen, X.: Face recognition on large-scale video in the wild with hybrid euclidean-and-riemannian metric learning. Pattern Recogn. 48, 3113–3124 (2015). Discriminative Feature Learning from Big Data for Visual Recognition

    Article  Google Scholar 

  30. Yang, J., Ren, P., Chen, D., Wen, F., Li, H., Hua, G.: Neural aggregation network for video face recognition. CoRR abs/1603.05474 (2016)

    Google Scholar 

  31. Tang, Y.: Deep learning using support vector machines. CoRR abs/1306.0239 (2013)

    Google Scholar 

  32. Kahou, S.E., et al.: Combining modality specific deep neural networks for emotion recognition in video. In: Proceedings of the 15th ACM on International Conference on Multimodal Interaction, ICMI 2013, pp. 543–550. ACM, New York, NY, USA (2013)

    Google Scholar 

  33. Liu, P., Han, S., Meng, Z., Tong, Y.: Facial expression recognition via a boosted deep belief network. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1805–1812 (2014)

    Google Scholar 

  34. Yu, Z., Zhang, C.: Image based static facial expression recognition with multiple deep network learning. In: Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, ICMI 2015, pp. 435–442. ACM, New York, NY, USA (2015)

    Google Scholar 

  35. Liu, M., Li, S., Shan, S., Wang, R., Chen, X.: Deeply learning deformable facial action parts model for dynamic expression analysis. In: Cremers, D., Reid, I., Saito, H., Yang, M.-H. (eds.) ACCV 2014. LNCS, vol. 9006, pp. 143–157. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-16817-3_10

    Chapter  Google Scholar 

  36. Kim, B.K., Roh, J., Dong, S.Y., Lee, S.Y.: Hierarchical committee of deep convolutional neural networks for robust facial expression recognition. J. Multimodal User Interfaces 10, 173–189 (2016)

    Article  Google Scholar 

  37. Ofodile, I., et al.: Automatic recognition of deceptive facial expressions of emotion. CoRR abs/1707.04061 (2017)

    Google Scholar 

  38. Liu, M., Wang, R., Li, S., Shan, S., Huang, Z., Chen, X.: Combining multiple kernel methods on riemannian manifold for emotion recognition in the wild. In: Proceedings of the 16th International Conference on Multimodal Interaction, ICMI 2014, pp. 494–501. ACM, New York, NY, USA (2014)

    Google Scholar 

  39. Fan, Y., Lu, X., Li, D., Liu, Y.: Video-based emotion recognition using CNN-RNN and C3D hybrid networks. In: Proceedings of the 18th ACM International Conference on Multimodal Interaction, ICMI 2016, pp. 445–450. ACM, New York, NY, USA (2016)

    Google Scholar 

  40. Ranganathan, H., Chakraborty, S., Panchanathan, S.: Multimodal emotion recognition using deep learning architectures. In: 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1–9 (2016)

    Google Scholar 

  41. Noroozi, F., Marjanovic, M., Njegus, A., Escalera, S., Anbarjafari, G.: Audio-visual emotion recognition in video clips. IEEE Trans. Affect. Comput. 10, 1–1 (2017)

    Article  Google Scholar 

  42. Uddin, M.Z., Hassan, M.M., Almogren, A., Alamri, A., Alrubaian, M., Fortino, G.: Facial expression recognition utilizing local direction-based robust features and deep belief network. IEEE Access 5, 4525–4536 (2017)

    Article  Google Scholar 

  43. Zhao, X., Zhang, S.: Facial expression recognition based on local binary patterns and kernel discriminant isomap. Sensors 11, 9573–9588 (2011)

    Article  Google Scholar 

  44. Albiol, A., Monzo, D., Martin, A., Sastre, J., Albiol, A.: Face recognition using HOG-EBGM. Pattern Recogn. Lett. 29, 1537–1543 (2008)

    Article  Google Scholar 

  45. Berretti, S., Ben Amor, B., Daoudi, M., del Bimbo, A.: 3D facial expression recognition using sift descriptors of automatically detected keypoints. Vis. Comput. 27, 1021 (2011)

    Article  Google Scholar 

  46. Shan, C., Gong, S., McOwan, P.W.: Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis. Comput. 27, 803–816 (2009)

    Article  Google Scholar 

  47. Uddin, M.Z., Hassan, M.M.: A depth video-based facial expression recognition system using radon transform, generalized discriminant analysis, and hidden markov model. Multimed. Tools Appl. 74, 3675–3690 (2015)

    Article  Google Scholar 

  48. de Vries, G.-J., Pauws, S., Biehl, M.: Facial expression recognition using learning vector quantization. In: Azzopardi, G., Petkov, N. (eds.) CAIP 2015. LNCS, vol. 9257, pp. 760–771. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-23117-4_65

    Chapter  Google Scholar 

  49. Ravichander, A., Vijay, S., Ramaseshan, V., Natarajan, S.: Automated human facial expression recognition using extreme learning machines. In: Cao, J., Mao, K., Wu, J., Lendasse, A. (eds.) Proceedings of ELM-2015 Volume 2. PALO, vol. 7, pp. 209–222. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-28373-9_18

    Chapter  Google Scholar 

  50. Farfade, S.S., Saberian, M.J., Li, L.J.: Multi-view face detection using deep convolutional neural networks. In: Proceedings of the 5th ACM on International Conference on Multimedia Retrieval, ICMR 2015, pp. 643–650. ACM (2015)

    Google Scholar 

  51. Triantafyllidou, D., Tefas, A.: Face detection based on deep convolutional neural networks exploiting incremental facial part learning. In: 2016 23rd International Conference on Pattern Recognition (ICPR), pp. 3560–3565 (2016)

    Google Scholar 

  52. Yoshihara, H., Seo, M., Ngo, T.H., Matsushiro, N., Chen, Y.W.: Automatic feature point detection using deep convolutional networks for quantitative evaluation of facial paralysis. In: 2016 9th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI), pp. 811–814 (2016)

    Google Scholar 

  53. Kharghanian, R., Peiravi, A., Moradi, F.: Pain detection from facial images using unsupervised feature learning approach. In: 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 419–422 (2016)

    Google Scholar 

  54. Haque, M.A., Nasrollahi, K., Moeslund, T.B.: Quality-aware estimation of facial landmarks in video sequences. In: 2015 IEEE Winter Conference on Applications of Computer Vision, pp. 678–685 (2015)

    Google Scholar 

  55. Barman, A., Chatterjee, A., Bhide, R.: Cognitive impairment and rehabilitation strategies after traumatic brain injury. Indian J. Psychol. Med. 38, 172–181 (2016)

    Article  Google Scholar 

  56. McKenna, K., Cooke, D.M., Fleming, J., Jefferson, A., Ogden, S.: The incidence of visual perceptual impairment in patients with severe traumatic brain injury. Brain Inj. 20, 507–518 (2006)

    Article  Google Scholar 

  57. Tsaousides, T., Gordon, W.A.: Cognitive rehabilitation following traumatic brain injury: assessment to treatment. Mt. Sinai J. Med. J. Transl. Personalized Med. 76, 173–181 (2009)

    Article  Google Scholar 

  58. Hartley, R., Zisserman, A.: Multiple View Geometry in Computer Vision, 2nd edn. Cambridge University Press, New York (2003)

    MATH  Google Scholar 

  59. Xiong, X., la Torre, F.D.: Supervised descent method and its applications to face alignment. In: 2013 IEEE Conference on Computer Vision and Pattern Recognition, pp. 532–539 (2013)

    Google Scholar 

  60. Irani, R., Nasrollahi, K., Dhall, A., Moeslund, T.B., Gedeon, T.: Thermal super-pixels for bimodal stress recognition. In: 2016 Sixth International Conference on Image Processing Theory, Tools and Applications (IPTA), pp. 1–6 (2016)

    Google Scholar 

  61. Hassner, T., Harel, S., Paz, E., Enbar, R.: Effective face frontalization in unconstrained images. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4295–4304 (2015)

    Google Scholar 

  62. Ji, S., Xu, W., Yang, M., Yu, K.: 3D convolutional neural networks for human action recognition. IEEE Trans. Pattern Anal. Mach. Intell. 35, 221–231 (2013)

    Article  Google Scholar 

  63. Parkhi, O.M., Vedaldi, A., Zisserman, A.: Deep face recognition. In: British Machine Vision Conference (2015)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Visual Analysis of People Laboratory and Interaction laboratory Aalborg University (AAU), Aalborg, Denmark

    Chaudhary Muhammad Aqdus Ilyas, Mohammad A. Haque, Kamal Nasrollahi & Thomas B. Moeslund

  2. Robotics Aalborg U, Aalborg University, Aalborg, Denmark

    Matthias Rehm

Authors
  1. Chaudhary Muhammad Aqdus Ilyas
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mohammad A. Haque
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Matthias Rehm
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Kamal Nasrollahi
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Thomas B. Moeslund
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Chaudhary Muhammad Aqdus Ilyas , Mohammad A. Haque , Matthias Rehm , Kamal Nasrollahi or Thomas B. Moeslund .

Editor information

Editors and Affiliations

  1. University of Strasbourg, Strasbourg, France

    Dominique Bechmann

  2. University of Genoa, Genoa, Italy

    Manuela Chessa

  3. University of Lisbon, Lisbon, Portugal

    Ana Paula Cláudio

  4. Research Innovation Center, Apple Inc., San Jose, CA, USA

    Francisco Imai

  5. Linnaeus University, Växjö, Kronobergs Län, Sweden

    Andreas Kerren

  6. LISA - ISTIA, University of Angers, Angers, France

    Paul Richard

  7. University of Groningen, Groningen, The Netherlands

    Alexandru Telea

  8. Jean Monnet University, Saint-Etienne, France

    Alain Tremeau

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Ilyas, C.M.A., Haque, M.A., Rehm, M., Nasrollahi, K., Moeslund, T.B. (2019). Effective Facial Expression Recognition Through Multimodal Imaging for Traumatic Brain Injured Patient’s Rehabilitation. In: Bechmann, D., et al. Computer Vision, Imaging and Computer Graphics Theory and Applications. VISIGRAPP 2018. Communications in Computer and Information Science, vol 997. Springer, Cham. https://doi.org/10.1007/978-3-030-26756-8_18

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-26756-8_18

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-26755-1

  • Online ISBN: 978-3-030-26756-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation