Towards Episodic Memory Support for Dementia Patients by Recognizing Objects, Faces and Text in Eye Gaze

  • Takumi ToyamaEmail author
  • Daniel Sonntag
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9324)


Humans forget events or certain types of information sometimes even when the information is important. There are patients who suffer from diseases such as dementia that result in memory loss. Using an eye tracker and image analysis modules, we develop a system that recognizes everyday objects, faces and text that the user looks at. This system can be effectively used to support an individual user’s memory by logging certain types of everyday information that the user perceives and tries to organize in his or her memory. We present a technical implementation of an episodic memory support.


Face Recognition Query Image Recognition Result Dementia Patient Visual Content 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Ahonen, T., Hadid, A., Pietikainen, M.: Face Description with Local Binary Patterns: Application to Face Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 28, 2037–2041 (2006)CrossRefzbMATHGoogle Scholar
  2. 2.
    Bulling, A., Roggen, D., Tröster, G.: Wearable EOG goggles: Seamless sensing and context-awareness in everyday environments. Journal of Ambient Intelligence and Smart Environments 1, 157–171 (2009)Google Scholar
  3. 3.
    Czaja, S.J., Sharit, J.: The aging of the population: Opportunities and challenges for human factors engineering. The Bridge: Linking Engineering and Society 39(1), 34–40 (2009)Google Scholar
  4. 4.
    Haigh, K.Z., Yanco, H.: Automation as caregiver: a survey of issues and technologies. In: AAAI-02 Workshop on Automation as Caregiver: The Role of Intelligent Technology in Elder Care, pp. 39–53 (2002)Google Scholar
  5. 5.
    Ishiguro, Y., Mujibiya, A., Miyaki, T., Rekimoto, J.: Aided eyes: eye activity sensing for daily life. In: The 1st Augmented Human International Conference (AH 2010), pp. 1–7 (2010)Google Scholar
  6. 6.
    Iwamura, M., Kunze, K., Kato, Y., Utsumi, Y., Kise, K.: Haven’t we met before? A realistic memory assistance system to remind you of the person in front of you. In: Proceedings of the 5th Augmented Human International Conference (AH 2014) (2014)Google Scholar
  7. 7.
    Kawamura, T., Kono, Y., Kidode, M.: Wearable interfaces for a video diary: towards memory retrieval, exchange, and transportation. In: Proceedings of the Sixth International Symposium on Wearable Computers, (ISWC 2002), pp. 31–38 (2002)Google Scholar
  8. 8.
    Label, L.: Dementia Facts and Statistics (2009).
  9. 9.
    Lee, M.L., Dey, A.K.: Lifelogging memory appliance for people with episodic memory impairment. In: Proceedings of the 10th International Conference on Ubiquitous Computing - UbiComp 2008, vol. 344, p. 44 (2008)Google Scholar
  10. 10.
    Lowe, D.: Object recognition from local scale-invariant features. In: Proceedings of the Seventh IEEE International Conference on Computer Vision, vol. 2 (1999)Google Scholar
  11. 11.
    Marling, C., Whitehouse, P.: Case-based reasoning in the care of alzheimers disease patients. In: Case-Based Reasoning Research and Development, vol. 208, pp. 702–715 (2001)Google Scholar
  12. 12.
    Orlosky, J., Toyama, T., Sonntag, D., Kiyokawa, K.: Using eye-gaze and visualization to augment memory. In: Streitz, N., Markopoulos, P. (eds.) DAPI 2014. LNCS, vol. 8530, pp. 282–291. Springer, Heidelberg (2014) CrossRefGoogle Scholar
  13. 13.
    Pollack, M.E.: Intelligent Technology for an Aging Population: The Use of AI to Assist Elders with Cognitive Impairment. Ai Magazine 26, 9–24 (2005)Google Scholar
  14. 14.
    Qvarfordt, P., Zhai, S.: Conversing with the user based on eye-gaze patterns. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems - CHI 2005, p. 221 (2005)Google Scholar
  15. 15.
    Simons, J.S., Graham, K.S., Galton, C.J., Patterson, K., Hodges, J.R.: Semantic knowledge and episodic memory for faces in semantic dementia. Neuropsychology 15(1), 101–114 (2001)CrossRefGoogle Scholar
  16. 16.
    Sonntag, D.: Collaborative multimodality. KI-Künstliche Intelligenz 26(2), 161–168 (2012)CrossRefGoogle Scholar
  17. 17.
    Sonntag, D., Toyama, T.: On-body IE: a head-mounted multimodal augmented reality system for learning and recalling faces. In: Proceedings of the 9th International Conference on Intelligent Environments (IE 2013), pp. 151–156. IEEE, Atgens (2013)Google Scholar
  18. 18.
    Toyama, T., Kieninger, T., Shafait, F., Dengel, A.: Gaze guided object recognition using a head-mounted eye tracker. In: Proceedings of the Symposium on Eye Tracking Research and Applications - ETRA 2012, pp. 91–98 (2012)Google Scholar
  19. 19.
    Viola, P., Jones, M.: Robust Real-Time Face Detection. International Journal of Computer Vision 57, 137–154 (2004)CrossRefGoogle Scholar
  20. 20.
    Zillich, M., Krüger, N.: Special issue on bio-inspired vision systems. KI 29(1), 5–7 (2015)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  1. 1.DFKI GmbHSaarbrückenGermany

Personalised recommendations