Using Emotion Recognition in Intelligent Interface Design for Elderly Care

  • Salik Khanal
  • Arsénio Reis
  • João Barroso
  • Vitor Filipe
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 746)


In the later stages of the aging process, an elderly person might need the help of a family member or a caregiver. Technology can be used to help to take care of elderly persons. Autonomous systems, using special interfaces, can collect information from elderly people, which might be useful to predict and recognize health related problems or physical security problems in real time. The emerging technology of image processing, in particular, the emotion recognition, can be a good option to use in elderly care support systems. In this article, we implemented a Microsoft Azure – Emotion SDK to recognize emotion of elderly that able to detect faces and recognize emotions in real time and to be used for elderly care support. The analysis is done with an online video stream, which analyzes facial expression, so that in case of a critical emotion, e.g., if an elderly is very sad or crying, it will inform a caregiver or related entity. From the experiment, we concluded that emotion recognition is a reliable technology to be implemented in real time elderly care.


Human-computer interaction Elderly care Image processing Emotion recognition Microsoft cognitive services 



This work was supported by Project “NanoSTIMA: Macro-to-Nano Human Sensing: Towards Integrated Multimodal Health Monitoring and Analytics/NORTE-01-0145-FEDER-000016” financed by the North Portugal Regional Operational Programme (NORTE 2020), under the PORTUGAL 2020 Partnership Agreement, and through the European Regional Development Fund (ERDF).


  1. 1.
    Hertog, S.: World Population Ageing 2017. United Nations Department of Economic and Social Affairs, Population Division, New York, USA (2017)Google Scholar
  2. 2.
    Carvalho, A.C.: Sensos 2011. Resultados Provisorious, Lisbon (2011)Google Scholar
  3. 3.
    Reis, A., Paredes, H., Barroso, I., Monteiro, M., Rodrigues, V., Khanal, S.R., Barroso, J.: Autonomous systems to support social activity of elderly people - a prospective approach to a system design. In: International Conference on Technology and Innovation on Sports, Health and Wellbeing, TISHW 2016, 1–3 December, 2016. UTAD, Vila Real (2016).
  4. 4.
    He, D., Li, Z., Gao, X., Li, M., Yin, Y., Lu, K.: The research of elderly care system based on video image processing system. IEEE, Santa Clara, CA, USA, pp. 254–258 (2016) Google Scholar
  5. 5.
    Reis, A., Paulino, D., Paredes, H., Barroso, J.: Using intelligent personal assistants to strengthen the elderlies’ social bonds. In: Universal Access in Human–Computer Interaction. Human and Technological Environments, January 2017, pp. 593–602 (2017). ISBN 978-3-319-58699-1CrossRefGoogle Scholar
  6. 6.
    Reis A., Barroso, I., Monteiro, M., Khanal, S.R., Rodrigues, V., Filipe, V., Paredes, H., Barroso, J.: Designing autonomous systems interactions with elderly people. In: Universal Access in Human–Computer Interaction. Human and Technological Environments, January 2017, pp. 603–611 (2017). ISBN 978-3-319-58699-1CrossRefGoogle Scholar
  7. 7.
    Gnanavel, R., Anjana, P., Nappinnai, K.S., Sahari, N.P.: Smart home system using a wireless sensor network for elderly care. IEEE, Chennai, India (2016)Google Scholar
  8. 8.
    Paulino, D., Reis, A., Barroso, J., Paredes, H.: Mobile devices to monitor physical activity and health data. In: 12th Iberian Conference on Information Systems and Technologies (CISTI), June 2017.
  9. 9.
    Faulkner, J., Eston, R.J.: Perceived exertion research in the 21st century: developments, reflections and questions for the future. J. Exerc. Sci. Fitness 6(1), 1–12 (2008)Google Scholar
  10. 10.
    Huanga, D.H., Chioua, W.K., Chenb, B.H.: Judgment of perceived exertion by static and dynamic facial expression. In: Triennial Congress of the IEA, Melbourne, pp. 1–7 (2015)Google Scholar
  11. 11.
    Leat, M., Mei, S.J.: Quantitative assessment of perceived visibility enhancement with image processing for single face images: a preliminary study. Invest. Ophthalmol. Vis. Sci. 50, 4502–4508 (2008)Google Scholar
  12. 12.
    Chaudhuri, S., Thompson, H., Demiris, G.: Fall detection devices and their use with older adults: a systematic review. J. Geriatr. Phys. Ther. 34(4), 178–196 (2014)CrossRefGoogle Scholar
  13. 13.
    Yu, X.: Approaches and principles of fall detection for elderly and patient. IEEE, Singapore (2008)Google Scholar
  14. 14.
    Docampo, G.N.: Heart rate estimation using facial video information, Pontevedra (2012)Google Scholar
  15. 15.
    Liukkonen, T.N., Tuomas, M., Hanna, A., Toni, H., Reetta, R., Paula, P.: Motion tracking exergames for elderly users. IADIS Int. J. Comput. Sci. Inf. Syst. 10(2), 52–64 (2015) Google Scholar
  16. 16.
    Abreu, J., Rebelo, S., Paredes, H., Barroso, J., Martins, P., Reis, A., Filipe, V.: Assessment of microsoft kinect in the monitoring and rehabilitation of stroke patients. In: Rocha, Á., Correia, A.M., Adeli, H., Reis, L.P., Costanzo, S. (eds.) Recent Advances in Information Systems and Technologies, vol. 2, pp. 167–174. Springer International Publishing, Cham (2017). ISBN 978-3-319-56537-8Google Scholar
  17. 17.
    Ekman, P., Friensen, W.V., Ancoli, S.: Facial signs of emotion experiences. J. Pers. Soc. Psychol. 39, 1125–1134 (1980)CrossRefGoogle Scholar
  18. 18.
    Tivatansakul, S., Ohkura, M., Puangpontip, S., Achalakul, T.: Emotional healthcare system: emotion detection by facial expressions using Japanese database. IEEE, Colchester, UK, pp. 41–47 (2014)Google Scholar
  19. 19.
    Santos, C., Santos, V., Tavares, A., Varajão, J.: Project management success in health–the need of additional research in public health projects. Procedia Technol. 16, 1080–1085 (2014)CrossRefGoogle Scholar
  20. 20.
    Liu, M., Li, S., Shan, S., Chen, X.: AU-inspired deep networks for facial expression feature learning. Neurocomputing 159, 126–136 (2015)CrossRefGoogle Scholar
  21. 21.
    Ng, H.W., Nguyen, V.D., Vonikakis, V., Winkler, S.: Deep learning for emotion recognition on small datasets using transfer learning. In: International Conference on Multimodal Interaction, Seattle, Washington, pp. 443–449 (2015)Google Scholar
  22. 22.
    Nugroho, L.E., Kurnianingsih, Lazuardi, L., Widyawan, Ferdiana, R., Selo: Contempo: a home care model to enhance the wellbeing of elderly people. In: IEEE-EMBS International Conference 2014 on Biomedical and Health Informatics (BHI) (2014)Google Scholar
  23. 23.
    Shaukat, A., Ahsan, M., Hassan, A., Riaz, F.: Daily sound recognition for elderly people using ensemble methods. IEEE, Xiamen, China, pp. 418–424 (2014)Google Scholar
  24. 24.
    Reis, A., Lains, J., Paredes, H., Filipe, V., Abrantes, C., Ferreira, F., Mendes, R., Amorim, P., Barroso, J.: Developing a system for post-stroke rehabilitation: an exergames approach. In: Antona, M., Stephanidis, C. (eds.) Universal Access in Human-Computer Interaction. Users and Context Diversity, July 2016, pp. 403–413. Springer International Publishing (2016). ISBN 978-3-319-40237-6CrossRefGoogle Scholar
  25. 25.
    Ebner, N.C., Riediger, M., Linderberger, U.: FACES—a database of facial expressions in young, middle-aged, and older women and men: development and validation. Behav. Res. Methods 42(1), 351–362 (2010)CrossRefGoogle Scholar
  26. 26.
  27. 27.
    Viola, P., Jones, M.J.: Robust real-time object detection. Int. J. Comput. Vis., 1–25, July 2001 Google Scholar
  28. 28.
    Felisberto, F., Laza, R., Fdez-Riverola, F., Pereira, A.: A distributed multiagent system architecture for body area networks applied to healthcare monitoring. In: BioMed Research International (2015)CrossRefGoogle Scholar
  29. 29.
    Marcelino, I., Pereira, A.: Elder care modular solution. In: Second International Conference on Advances in Human-Oriented and Personalized Mechanisms, Technologies, and Services, CENTRIC 2009, pp. 1–6. IEEE, September 2009Google Scholar
  30. 30.
    Serrão, M., Shahrabadi, S., Moreno, M., José, J., Rodrigues, J.I., Rodrigues, J.M.F., du Buf, J.M.H.: Computer vision and GIS for the navigation of blind persons in buildings. Int. J. Univ. Access Inf. Soc. 1–14 (2015).

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Salik Khanal
    • 1
    • 2
  • Arsénio Reis
    • 1
    • 2
  • João Barroso
    • 1
    • 2
  • Vitor Filipe
    • 1
    • 2
  1. 1.INESC TECPortoPortugal
  2. 2.University of Trás-os-Montes e Alto DouroVila RealPortugal

Personalised recommendations