Skip to main content
Download PDF

First-person Video Analysis for Evaluating Skill Level in the Humanitude Tender-Care Technique

Journal of Intelligent & Robotic Systems volume 98, pages 103–118 (2020)Cite this article

Abstract

In this paper, we describe a wearable first-person video (FPV) analysis system for evaluating the skill levels of caregivers. This is a part of our project that aims to quantize and analyze the tender-care technique known as Humanitude by using wearable sensing and AI technology devices. Using our system, caregivers can evaluate and elevate their care levels by themselves. From the FPVs of care sessions taken by wearable cameras worn by caregivers, we obtained the 3D facial distance, pose and eye-contact states between caregivers and receivers by using facial landmark detection and deep neural network (DNN)-based eye contact detection. We applied statistical analysis to these features and developed algorithms that provide scores for tender-care skill. In experiments, we first evaluated the performance of our DNN-based eye contact detection by using eye contact datasets prepared from YouTube videos and FPVs that assume conversational scenes. We then performed skill evaluations by using Humanitude training scenes involving three novice caregivers, two Humanitude experts and seven middle-level students. The results showed that our eye contact detection outperformed existing methods and that our skill evaluations can estimate the care skill levels.

References

  1. Acton, G.J., Kang, J.: Interventions to reduce the burden of caregiving for an adult with dementia: a meta-analysis. Res. Nurs. Health 24(5), 349–360 (2001)

    Article  Google Scholar 

  2. Adelman, R.D., Tmanova, L.L., Delgado, D., Dion, S., Lachs, M.S.: Caregiver burden: a clinical review. Jama 311(10), 1052–1060 (2014)

    Article  Google Scholar 

  3. Alzheimer’s Society: Factsheet: Communicating. https://www.alzheimers.org.uk/site/scripts/documents_info.php?documentID=130, [Online; accessed 18-Nov-2016] (2016)

  4. Baltrusaitis, T., Robinson, P., Morency, L.P.: (2016) OpenFace: An Open source facial behavior analysis toolkit. 2016 IEEE Winter Conference on Applications of Computer Vision WACV. https://doi.org/10.1109/WACV.2016.7477553 (2016)

  5. Bertasius, G., Park, H.S., Stella, X.Y., Shi, J.: Am I a Baller? Basketball Performance Assessment from First-Person Videos. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2196–2204. IEEE (2017)

  6. Binetti, N., Harrison, C., Coutrot, A., Johnston, A., Mareschal, I.: Pupil dilation as an index of preferred mutual gaze duration. Royal Society Open Science 3(7). https://doi.org/10.1098/rsos.160086, http://rsos.royalsocietypublishing.org/content/3/7/160086.full.pdf (2016)

  7. Biquand, S., Zittel, B.: Care giving and nursing, work conditions and humanitude®;. Work 41(Supplement 1), 1828–1831 (2012)

    Article  Google Scholar 

  8. Boseley, S.: Dementia research funding to more than double to £66m by 2015. The Guardian (2012)

  9. Campbell, L.C., Keefe, F.J., Scipio, C., McKee, D.C., Edwards, C.L., Herman, S.H., Johnson, L.E., Colvin, O.M., McBride, C.M., Donatucci, C.: Facilitating research participation and improving quality of life for african american prostate cancer survivors and their intimate partners: a pilot study of telephone-based coping skills training. Cancer: Interdiscip. Int. J. Amer. Cancer Soc. 109(S2), 414–424 (2007)

    Article  Google Scholar 

  10. Casado, B., Sacco, P.: Correlates of caregiver burden among family caregivers of older korean americans. J. Gerontol. Ser. B: Psychol. Sci. Soc. Sci. 67(3), 331–336 (2011)

    Article  Google Scholar 

  11. Chong, E., Chanda, K., Ye, Z., Southerland, A., Ruiz, N., Jones, R.M., Rozga, A., Rehg, J.M.: Detecting gaze towards eyes in natural social interactions and its use in child assessment. Proc. ACM Interact. Mob. Wearab. Ubiquit. Technol. 1(3), 43 (2017)

    Article  Google Scholar 

  12. Clyburn, L.D., Stones, M.J., Hadjistavropoulos, T., Tuokko, H., et al.: Predicting caregiver burden and depression in alzheimer’s disease. J. Gerontol. Ser. B 55(1), S2–S13 (2000)

    Article  Google Scholar 

  13. Coon, D.W., Thompson, L., Steffen, A., Sorocco, K., Gallagher-Thompson, D.: Anger and depression management: psychoeducational skill training interventions for women caregivers of a relative with dementia. Gerontologist 43(5), 678–689 (2003)

    Article  Google Scholar 

  14. Dozat, T.: Incorporating nesterov momentum into adam. http://cs229.stanford.edu/proj2015/054_report.pdf, [Online; accessed 25-Aug-2018] (2016)

  15. Dunkin, J.J., Anderson-Hanley, C.: Dementia caregiver burden: a review of the literature and guidelines for assessment and intervention. Neurology 51(1 Suppl 1), S53–S60 (1998)

    Article  Google Scholar 

  16. Etters, L., Goodall, D., Harrison, B.E.: Caregiver burden among dementia patient caregivers: a review of the literature. J. Am. Acad. Nurse Pract. 20(8), 423–428 (2008)

    Article  Google Scholar 

  17. Fathi, A., Hodgins, J.K., Rehg, J.M.: Social Interactions: a First-Person Perspective. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1226–1233. IEEE (2012)

  18. Gineste, Y., Pellissier, J.: Humanitude: comprendre la vieillesse, prendre soin des hommes vieux. A. Colin (2007)

  19. Given, B., Sherwood, P.R., Given, C.W.: What knowledge and skills do caregivers need? J. Soc. Work. Educ. 44(sup3), 115–123 (2008)

    Article  Google Scholar 

  20. Honda, M., Ito, M., Ishikawa, S., Takebayashi, Y., Tierney, L.: Reduction of behavioral psychological symptoms of dementia by multimodal comprehensive care for vulnerable geriatric patients in an acute care hospital: a case series. Case reports in medicine 2016 (2016)

  21. Ishikawa, S., Ito, M., Honda, M., Takebayashi, Y.: The skill representation of a multimodal communication care method for people with dementia. JJAP Conf. Proc. 011616, 4 (2016)

    Google Scholar 

  22. Ito, M., Honda, M.: An examination of the influence of humanitude caregiving on the behavior of older adults with dementia in japan. In: Proceedings of the 8th International Association of Gerontology and Geriatrics European Region Congress (2015)

  23. Kim, H., Chang, M., Rose, K., Kim, S.: Predictors of caregiver burden in caregivers of individuals with dementia. J. Adv. Nurs. 68(4), 846–855 (2012)

    Article  Google Scholar 

  24. Krafka, K., Khosla, A., Kellnhofer, P., Kannan, H., Bhandarkar, S., Matusik, W., Torralba, A.: Eye Tracking for Everyone. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)

  25. Larson, E.B., Yaffe, K., Langa, K.M.: New insights into the dementia epidemic. Engl. J. Med. 369(24), 2275–2277 (2013)

    Article  Google Scholar 

  26. Law, H., Ghani, K., Deng, J.: Surgeon technical skill assessment using computer vision based analysis. In: Machine Learning for Healthcare Conference, pp. 88–99 (2017)

  27. Lu, F., Sugano, Y., Okabe, T., Sato, Y.: Gaze estimation from eye appearance: a head pose-free method via eye image synthesis. IEEE Trans. Image Process. 24(11), 3680–3693 (2015)

    MathSciNet  Article  Google Scholar 

  28. Ma, M., Fan, H., Kitani, K.M.: Going deeper into first-person activity recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1894–1903 (2016)

  29. Ministry of Health, Labour and Welfare, Japan: Supply and demand estimation for nursing care personnel for 2025. https://www.mhlw.go.jp/stf/houdou/0000088998.html, [Online; accessed 11-Aug-2018] (2015)

  30. Mitsuzumi, Y., Nakazawa, A., Nishida, T.: Deep Eye Contact Detector: Robust Eye Contact Bid Detection Using Convolutional Neural Network. In: Proceedings of the British Machine Vision Conference (BMVC) (2017)

  31. Northouse, L.L., Katapodi, M.C., Song, L., Zhang, L., Mood, D.W.: Interventions with family caregivers of cancer patients: meta-analysis of randomized trials. CA: Cancer J. Clin. 60(5), 317–339 (2010)

    Google Scholar 

  32. Ostwald, S.K., Hepburn, K.W., Caron, W., Burns, T., Mantell, R.: Reducing caregiver burden: a randomized psychoeducational intervention for caregivers of persons with dementia. Gerontologist 39(3), 299–309 (1999)

    Article  Google Scholar 

  33. Papastavrou, E., Kalokerinou, A., Papacostas, S.S., Tsangari, H., Sourtzi, P.: Caring for a relative with dementia: family caregiver burden. J. Adv. Nurs. 58(5), 446–457 (2007)

    Article  Google Scholar 

  34. Petric, F., Miklić, D., kovačić, Z.: Probabilistic eye contact detection for the robot-assisted asd diagnostic protocol. In: Lončarić S., Cupec, R. (eds.) Proceedings of the Croatian Compter Vision Workshop, Year 4, Center of Excellence for Computer Vision, pp 3–8. University of Zagreb, Osijek (2016)

  35. Povithead: Pivothead KUDU. http://www.pivothead.com/, [Online; accessed 29-Aug-2016] (2016)

  36. Pupil Labs: Pupil labs camera system. https://pupil-labs.com/pupil/, [Online; accessed 25-Aug-2018] (2018)

  37. Singh, S., Arora, C., Jawahar, C.: First person action recognition using deep learned descriptors. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 2620–2628 (2016)

  38. Smith BA, Yin Q, Feiner SK, Nayar SK: Gaze locking: passive eye contact detection for human-object interaction. In: Proceedings of the 26th annual ACM symposium on User interface software and technology, pp. 271–280. ACM (2013)

  39. Soo Park, H., Shi, J., et al.: Force from motion: decoding physical sensation in a first person video. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3834–3842 (2016)

  40. Su, S., Hong, J.P., Shi, J., Park, H.S.: Predicting Behaviors of Basketball Players from First Person Videos. In: CVPR, vol. 2, pp. 3 (2017)

  41. Win, K.K., Chong, M.S., Ali, N., Chan, M., Lim, W.S.: Burden among family caregivers of dementia in the oldest-old: an exploratory study. Front. Med. 4, 205 (2017)

    Article  Google Scholar 

  42. World Health Organization, et al.: Dementia: Fact sheet N 362 (2012)

  43. Ye, Z., Li, Y., Fathi, A., Han, Y., Rozga, A., Abowd, G.D., Rehg, J.M.: Detecting eye contact using wearable eye-tracking glasses. In: Proceedings of the 2012 ACM Conference on Ubiquitous Computing, pp. 699–704. ACM (2012a)

  44. Ye, Z., Li, Y., Fathi, A., Han, Y., Rozga, A., Abowd, G.D., Rehg, J.M.: Detecting eye contact using wearable eye-tracking glasses. In: Proceedings of the 2012 ACM Conference on Ubiquitous Computing, UbiComp’12, pp 699–704. ACM, New York, https://doi.org/10.1145/2370216.2370368 (2012b)

  45. Ye, Z., Li, Y., Liu, Y., Bridges, C., Rozga, A., Rehg, J.M.: Detecting Bids for Eye Contact Using a Wearable Camera. In: 2015 11Th IEEE International Conference and Workshops On Automatic Face and Gesture Recognition (FG), vol. 1, pp. 1–8. IEEE (2015)

  46. Zhang, X., Sugano, Y., Fritz, M., Bulling, A.: Appearance-based gaze estimation in the wild. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4511–4520 (2015)

  47. Zhang, X., Sugano, Y., Fritz, M., Bulling, A.: It’s written all over your face: Full-face appearance-based gaze estimation. arXiv:1611.08860, https://perceptual.mpi-inf.mpg.de/wp-content/blogs.dir/12/files/2016/11/zhang16_arxiv.pdf (2016)

  48. Zhang, X., Sugano, Y., Bulling, A.: Everyday eye contact detection using unsupervised gaze target discovery. In: Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology, pp. 193–203. ACM (2017)

Download references

Acknowledgements

All the experiments were conducted in compliance with the protocol which was reviewed and approved by the ethical committee of Unit for Advanced Studies of the Human Mind, Kyoto University (Permit Number: 30-P-4). This work was supported by JST CREST Grant Number JPMJCR17A5 and JSPS KAKENHI 17H01779, Japan.

Author information

Affiliations

  1. Graduate School of Informatics, Kyoto University, Kyoto, Japan

    Atsushi Nakazawa & Yuki Watanabe

  2. NTT Communication Science Laboratories, Nippon Telegraph and Telephone Corporation (NTT), Kanagawa, Japan

    Yu Mitsuzumi

  3. Faculty of Information Science and Electrical Engineering, Kyushu University, Fukuoka, Japan

    Ryo Kurazume

  4. Kokoro Research Center, Kyoto University, Kyoto, Japan

    Sakiko Yoshikawa

  5. Geriatric Research Division, National Hospital Organization Tokyo Medical Center, Tokyo, Japan

    Miwako Honda

Authors
  1. Atsushi Nakazawa
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yu Mitsuzumi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yuki Watanabe
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ryo Kurazume
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Sakiko Yoshikawa
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Miwako Honda
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Atsushi Nakazawa.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Nakazawa, A., Mitsuzumi, Y., Watanabe, Y. et al. First-person Video Analysis for Evaluating Skill Level in the Humanitude Tender-Care Technique. J Intell Robot Syst 98, 103–118 (2020). https://doi.org/10.1007/s10846-019-01052-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10846-019-01052-8

Keywords

  • Dementia
  • Care
  • Deep neural network (DNN)
  • Skill evaluation
  • Wearable system
  • Computer vision
  • First person video