First-person Video Analysis for Evaluating Skill Level in the Humanitude Tender-Care Technique

  • Atsushi NakazawaEmail author
  • Yu Mitsuzumi
  • Yuki Watanabe
  • Ryo Kurazume
  • Sakiko Yoshikawa
  • Miwako Honda
Open Access


In this paper, we describe a wearable first-person video (FPV) analysis system for evaluating the skill levels of caregivers. This is a part of our project that aims to quantize and analyze the tender-care technique known as Humanitude by using wearable sensing and AI technology devices. Using our system, caregivers can evaluate and elevate their care levels by themselves. From the FPVs of care sessions taken by wearable cameras worn by caregivers, we obtained the 3D facial distance, pose and eye-contact states between caregivers and receivers by using facial landmark detection and deep neural network (DNN)-based eye contact detection. We applied statistical analysis to these features and developed algorithms that provide scores for tender-care skill. In experiments, we first evaluated the performance of our DNN-based eye contact detection by using eye contact datasets prepared from YouTube videos and FPVs that assume conversational scenes. We then performed skill evaluations by using Humanitude training scenes involving three novice caregivers, two Humanitude experts and seven middle-level students. The results showed that our eye contact detection outperformed existing methods and that our skill evaluations can estimate the care skill levels.


Dementia Care Deep neural network (DNN) Skill evaluation Wearable system Computer vision First person video 



All the experiments were conducted in compliance with the protocol which was reviewed and approved by the ethical committee of Unit for Advanced Studies of the Human Mind, Kyoto University (Permit Number: 30-P-4). This work was supported by JST CREST Grant Number JPMJCR17A5 and JSPS KAKENHI 17H01779, Japan.


  1. 1.
    Acton, G.J., Kang, J.: Interventions to reduce the burden of caregiving for an adult with dementia: a meta-analysis. Res. Nurs. Health 24(5), 349–360 (2001)CrossRefGoogle Scholar
  2. 2.
    Adelman, R.D., Tmanova, L.L., Delgado, D., Dion, S., Lachs, M.S.: Caregiver burden: a clinical review. Jama 311(10), 1052–1060 (2014)CrossRefGoogle Scholar
  3. 3.
    Alzheimer’s Society: Factsheet: Communicating., [Online; accessed 18-Nov-2016] (2016)
  4. 4.
    Baltrusaitis, T., Robinson, P., Morency, L.P.: (2016) OpenFace: An Open source facial behavior analysis toolkit. 2016 IEEE Winter Conference on Applications of Computer Vision WACV. (2016)
  5. 5.
    Bertasius, G., Park, H.S., Stella, X.Y., Shi, J.: Am I a Baller? Basketball Performance Assessment from First-Person Videos. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2196–2204. IEEE (2017)Google Scholar
  6. 6.
    Binetti, N., Harrison, C., Coutrot, A., Johnston, A., Mareschal, I.: Pupil dilation as an index of preferred mutual gaze duration. Royal Society Open Science 3(7)., (2016)
  7. 7.
    Biquand, S., Zittel, B.: Care giving and nursing, work conditions and humanitude®;. Work 41(Supplement 1), 1828–1831 (2012)Google Scholar
  8. 8.
    Boseley, S.: Dementia research funding to more than double to £66m by 2015. The Guardian (2012)Google Scholar
  9. 9.
    Campbell, L.C., Keefe, F.J., Scipio, C., McKee, D.C., Edwards, C.L., Herman, S.H., Johnson, L.E., Colvin, O.M., McBride, C.M., Donatucci, C.: Facilitating research participation and improving quality of life for african american prostate cancer survivors and their intimate partners: a pilot study of telephone-based coping skills training. Cancer: Interdiscip. Int. J. Amer. Cancer Soc. 109(S2), 414–424 (2007)CrossRefGoogle Scholar
  10. 10.
    Casado, B., Sacco, P.: Correlates of caregiver burden among family caregivers of older korean americans. J. Gerontol. Ser. B: Psychol. Sci. Soc. Sci. 67(3), 331–336 (2011)CrossRefGoogle Scholar
  11. 11.
    Chong, E., Chanda, K., Ye, Z., Southerland, A., Ruiz, N., Jones, R.M., Rozga, A., Rehg, J.M.: Detecting gaze towards eyes in natural social interactions and its use in child assessment. Proc. ACM Interact. Mob. Wearab. Ubiquit. Technol. 1(3), 43 (2017)CrossRefGoogle Scholar
  12. 12.
    Clyburn, L.D., Stones, M.J., Hadjistavropoulos, T., Tuokko, H., et al.: Predicting caregiver burden and depression in alzheimer’s disease. J. Gerontol. Ser. B 55(1), S2–S13 (2000)CrossRefGoogle Scholar
  13. 13.
    Coon, D.W., Thompson, L., Steffen, A., Sorocco, K., Gallagher-Thompson, D.: Anger and depression management: psychoeducational skill training interventions for women caregivers of a relative with dementia. Gerontologist 43(5), 678–689 (2003)CrossRefGoogle Scholar
  14. 14.
    Dozat, T.: Incorporating nesterov momentum into adam., [Online; accessed 25-Aug-2018] (2016)
  15. 15.
    Dunkin, J.J., Anderson-Hanley, C.: Dementia caregiver burden: a review of the literature and guidelines for assessment and intervention. Neurology 51(1 Suppl 1), S53–S60 (1998)CrossRefGoogle Scholar
  16. 16.
    Etters, L., Goodall, D., Harrison, B.E.: Caregiver burden among dementia patient caregivers: a review of the literature. J. Am. Acad. Nurse Pract. 20(8), 423–428 (2008)CrossRefGoogle Scholar
  17. 17.
    Fathi, A., Hodgins, J.K., Rehg, J.M.: Social Interactions: a First-Person Perspective. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1226–1233. IEEE (2012)Google Scholar
  18. 18.
    Gineste, Y., Pellissier, J.: Humanitude: comprendre la vieillesse, prendre soin des hommes vieux. A. Colin (2007)Google Scholar
  19. 19.
    Given, B., Sherwood, P.R., Given, C.W.: What knowledge and skills do caregivers need? J. Soc. Work. Educ. 44(sup3), 115–123 (2008)CrossRefGoogle Scholar
  20. 20.
    Honda, M., Ito, M., Ishikawa, S., Takebayashi, Y., Tierney, L.: Reduction of behavioral psychological symptoms of dementia by multimodal comprehensive care for vulnerable geriatric patients in an acute care hospital: a case series. Case reports in medicine 2016 (2016)Google Scholar
  21. 21.
    Ishikawa, S., Ito, M., Honda, M., Takebayashi, Y.: The skill representation of a multimodal communication care method for people with dementia. JJAP Conf. Proc. 011616, 4 (2016)Google Scholar
  22. 22.
    Ito, M., Honda, M.: An examination of the influence of humanitude caregiving on the behavior of older adults with dementia in japan. In: Proceedings of the 8th International Association of Gerontology and Geriatrics European Region Congress (2015)Google Scholar
  23. 23.
    Kim, H., Chang, M., Rose, K., Kim, S.: Predictors of caregiver burden in caregivers of individuals with dementia. J. Adv. Nurs. 68(4), 846–855 (2012)CrossRefGoogle Scholar
  24. 24.
    Krafka, K., Khosla, A., Kellnhofer, P., Kannan, H., Bhandarkar, S., Matusik, W., Torralba, A.: Eye Tracking for Everyone. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)Google Scholar
  25. 25.
    Larson, E.B., Yaffe, K., Langa, K.M.: New insights into the dementia epidemic. Engl. J. Med. 369(24), 2275–2277 (2013)CrossRefGoogle Scholar
  26. 26.
    Law, H., Ghani, K., Deng, J.: Surgeon technical skill assessment using computer vision based analysis. In: Machine Learning for Healthcare Conference, pp. 88–99 (2017)Google Scholar
  27. 27.
    Lu, F., Sugano, Y., Okabe, T., Sato, Y.: Gaze estimation from eye appearance: a head pose-free method via eye image synthesis. IEEE Trans. Image Process. 24(11), 3680–3693 (2015)MathSciNetCrossRefzbMATHGoogle Scholar
  28. 28.
    Ma, M., Fan, H., Kitani, K.M.: Going deeper into first-person activity recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1894–1903 (2016)Google Scholar
  29. 29.
    Ministry of Health, Labour and Welfare, Japan: Supply and demand estimation for nursing care personnel for 2025., [Online; accessed 11-Aug-2018] (2015)
  30. 30.
    Mitsuzumi, Y., Nakazawa, A., Nishida, T.: Deep Eye Contact Detector: Robust Eye Contact Bid Detection Using Convolutional Neural Network. In: Proceedings of the British Machine Vision Conference (BMVC) (2017)Google Scholar
  31. 31.
    Northouse, L.L., Katapodi, M.C., Song, L., Zhang, L., Mood, D.W.: Interventions with family caregivers of cancer patients: meta-analysis of randomized trials. CA: Cancer J. Clin. 60(5), 317–339 (2010)Google Scholar
  32. 32.
    Ostwald, S.K., Hepburn, K.W., Caron, W., Burns, T., Mantell, R.: Reducing caregiver burden: a randomized psychoeducational intervention for caregivers of persons with dementia. Gerontologist 39(3), 299–309 (1999)CrossRefGoogle Scholar
  33. 33.
    Papastavrou, E., Kalokerinou, A., Papacostas, S.S., Tsangari, H., Sourtzi, P.: Caring for a relative with dementia: family caregiver burden. J. Adv. Nurs. 58(5), 446–457 (2007)CrossRefGoogle Scholar
  34. 34.
    Petric, F., Miklić, D., kovačić, Z.: Probabilistic eye contact detection for the robot-assisted asd diagnostic protocol. In: Lončarić S., Cupec, R. (eds.) Proceedings of the Croatian Compter Vision Workshop, Year 4, Center of Excellence for Computer Vision, pp 3–8. University of Zagreb, Osijek (2016)Google Scholar
  35. 35.
    Povithead: Pivothead KUDU., [Online; accessed 29-Aug-2016] (2016)
  36. 36.
    Pupil Labs: Pupil labs camera system., [Online; accessed 25-Aug-2018] (2018)
  37. 37.
    Singh, S., Arora, C., Jawahar, C.: First person action recognition using deep learned descriptors. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 2620–2628 (2016)Google Scholar
  38. 38.
    Smith BA, Yin Q, Feiner SK, Nayar SK: Gaze locking: passive eye contact detection for human-object interaction. In: Proceedings of the 26th annual ACM symposium on User interface software and technology, pp. 271–280. ACM (2013)Google Scholar
  39. 39.
    Soo Park, H., Shi, J., et al.: Force from motion: decoding physical sensation in a first person video. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3834–3842 (2016)Google Scholar
  40. 40.
    Su, S., Hong, J.P., Shi, J., Park, H.S.: Predicting Behaviors of Basketball Players from First Person Videos. In: CVPR, vol. 2, pp. 3 (2017)Google Scholar
  41. 41.
    Win, K.K., Chong, M.S., Ali, N., Chan, M., Lim, W.S.: Burden among family caregivers of dementia in the oldest-old: an exploratory study. Front. Med. 4, 205 (2017)CrossRefGoogle Scholar
  42. 42.
    World Health Organization, et al.: Dementia: Fact sheet N 362 (2012)Google Scholar
  43. 43.
    Ye, Z., Li, Y., Fathi, A., Han, Y., Rozga, A., Abowd, G.D., Rehg, J.M.: Detecting eye contact using wearable eye-tracking glasses. In: Proceedings of the 2012 ACM Conference on Ubiquitous Computing, pp. 699–704. ACM (2012a)Google Scholar
  44. 44.
    Ye, Z., Li, Y., Fathi, A., Han, Y., Rozga, A., Abowd, G.D., Rehg, J.M.: Detecting eye contact using wearable eye-tracking glasses. In: Proceedings of the 2012 ACM Conference on Ubiquitous Computing, UbiComp’12, pp 699–704. ACM, New York, (2012b)
  45. 45.
    Ye, Z., Li, Y., Liu, Y., Bridges, C., Rozga, A., Rehg, J.M.: Detecting Bids for Eye Contact Using a Wearable Camera. In: 2015 11Th IEEE International Conference and Workshops On Automatic Face and Gesture Recognition (FG), vol. 1, pp. 1–8. IEEE (2015)Google Scholar
  46. 46.
    Zhang, X., Sugano, Y., Fritz, M., Bulling, A.: Appearance-based gaze estimation in the wild. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4511–4520 (2015)Google Scholar
  47. 47.
    Zhang, X., Sugano, Y., Fritz, M., Bulling, A.: It’s written all over your face: Full-face appearance-based gaze estimation. arXiv:1611.08860, (2016)
  48. 48.
    Zhang, X., Sugano, Y., Bulling, A.: Everyday eye contact detection using unsupervised gaze target discovery. In: Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology, pp. 193–203. ACM (2017)Google Scholar

Copyright information

© The Author(s) 2019

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (, which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Authors and Affiliations

  1. 1.Graduate School of InformaticsKyoto UniversityKyotoJapan
  2. 2.NTT Communication Science LaboratoriesNippon Telegraph and Telephone Corporation (NTT)KanagawaJapan
  3. 3.Faculty of Information Science and Electrical EngineeringKyushu UniversityFukuokaJapan
  4. 4.Kokoro Research CenterKyoto UniversityKyotoJapan
  5. 5.Geriatric Research DivisionNational Hospital Organization Tokyo Medical CenterTokyoJapan

Personalised recommendations