Assisting individuals with Alzheimer’s disease using mobile augmented reality with voice interaction: an acceptance experiment with individuals in the early stages

  • Keynes Masayoshi KannoEmail author
  • Edgard Afonso Lamounier Jr.
  • Alexandre Cardoso
  • Ederaldo José Lopes
  • Saadallah Azor Fakhouri Filho
Original Article



This article presents an innovative system that provides assistance to individuals with Alzheimer’s disease (AD) through the use of a mobile augmented reality (MAR) interface. By using MAR with voice commands and virtual buttons adapted to enable a better human-computer interaction (HCI), this study proposes a system that assists in pharmacological and non-pharmacological treatment. The authors have conducted an experiment to evaluate the acceptance of this system in individuals with AD in the early stages.


Six individuals with AD participated in the experiment. Three sessions were held to assess acceptance of the system in assistance with pharmacological treatment. Three other sessions were conducted to evaluate non-pharmacological activity. We measured the time spent and the number of interventions required for the participant to complete the task. We performed an observational analysis of the experiment and applied a questionnaire at the end of the evaluation so that the caregiver could answer questions together with the research participant.


All participants were able to execute the proposed activities. During the sessions, most participants showed improvements in their performance. The observational analysis and questionnaire results show a good acceptance of this system in terms of the volunteers.


This work proposed an alternative for assisting individuals with AD during their pharmacological and non-pharmacological treatment, innovating by means of interacting with medications and photo albums. Through the use of voice commands and a custom MAR interface, a good acceptance of this system by individuals with AD in early stage has been identified.


Mobile Augmented Reality Speech recognition Alzheimer disease Assistive technology 



We would like to thank all the hospital professionals, geriatricians, neurologists, and psychologists who supported this study with their knowledge and insight. We wish to express gratitude to the volunteers and caretakers.

Compliance with ethical standards

In addition, this work was approved by the Ethics Committee of the Federal University of Uberlândia (#71365417.6.0000.5152). Informed consent was obtained from all individual participants included in the study.

Conflict of interest

The authors declare that they have no conflict of interest.


  1. Aisen PS, Petersen RC, Donohue MC, Gamst A, Raman R, Thomas RG, et al. Clinical core of the Alzheimer’s disease neuroimaging initiative: progress and plans. Alzheimers. Dement. United States. 2010;6(3):239–46.CrossRefGoogle Scholar
  2. Al-khafaji NJ, Al-shaher MA, Al-khafaji MJ, Asmail MAA. Use BuildAR to Help the Alzheimer ’ s disease Patients. Int. Conf. E-Technologies Bus. Web. SDIWC digital library; 2013. p. 280–4.Google Scholar
  3. Alzheimer’s Association [Internet]. Alzheimer’s Dement. 2018 [cited 2018 Sep 10]. Available from:
  4. Chang YJ, Kang YS, Huang PC. An augmented reality (AR)-based vocational task prompting system for people with cognitive impairments. Res. Dev. Disabil. 2013 Oct [cited 2017 Aug 23];34(10):3049–56. Available from: Scholar
  5. Colucci L, Bosco M, Fasanaro AM, Gaeta GL, Ricci G, Amenta F. Alzheimer’s disease costs: what we know and what we should take into account. J. Alzheimers. Dis. Netherlands. 2014;42(4):1311–24.CrossRefGoogle Scholar
  6. Coppola J, Kowtko M, Yamagata C, Joyce S. Applying mobile application development to help dementia and Alzheimer patients. Csis.Pace.Edu. 2013;1–7. Available from: Accessed 15 Mar 2019.
  7. Décio OC. XML - Guia de Consulta Rápida. 1st ed. São Paulo: Novatec; 2000.Google Scholar
  8. Dubois B, Feldman HH, Jacova C, Cummings JL, Dekosky ST, Barberger-Gateau P, et al. Revising the definition of Alzheimer’s disease: a new lexicon. Lancet. Neurol. England. 2010;9:1118–27.CrossRefGoogle Scholar
  9. Feleke SA, Mulatu MA, Yesmaw YS. Medication administration error: magnitude and associated factors among nurses in Ethiopia. BMC Nurs. 2015;14:53 Available from: Scholar
  10. Hamidia M, Zenati N, Belghit H, Guetiteni K, Achour N. Voice Interaction Using Gaussian Mixture Models for Augmented Reality Applications. 4th Int. Conf. Electr. Eng. [Internet]. Boumerdes; 2015. p. 1–4. Available from:
  11. Hervás R, Bravo J, Fontecha J. An assistive navigation system based on augmented reality and context awareness for people with mild cognitive impairments. IEEE J. Biomed. Heal. Informatics. 2014;18(1):368–74.CrossRefGoogle Scholar
  12. Hocking J. Unity in Action: Multiplatform Game Development in C# with Unity 5 [Internet]. Shelter Island, NY: Manning Publications Company; 2015. Available from:,ISBN:9781617292323&hl=pt-BR&source=gbs_api 
  13. Hsiao KF, Rashvand HF. Data modeling mobile augmented reality: integrated mind and body rehabilitation. Multimed. Tools Appl. 2015;74(10):3543–60.CrossRefGoogle Scholar
  14. Hwangbo H, Yoon SH, Jin BS, Han YS, Ji YG. A Study of pointing performance of elderly users on smartphones. Int. J. Human–Computer Interact. 2013;29(9):604–18. Taylor & Francis. Available from. Scholar
  15. James BD, Leurgans SE, Hebert LE, Scherr PA, Yaffe K, Bennett DA. Contribution of Alzheimer disease to mortality in the United States. Neurology United States. 2014;82(12):1045–50.CrossRefGoogle Scholar
  16. Kalai Chelvam Y, Zamin N. M3DITRACK3R: A design of an automated patient tracking and medicine dispensing mobile robot for senior citizens. I4CT 2014 - 1st Int. Conf. Comput. Commun. Control Technol. Proc. 2014;(I4ct):36–41.Google Scholar
  17. Kenzo, Kang Y, Huang P. Research in developmental disabilities an augmented reality ( AR ) -based vocational task prompting system for people with cognitive impairments. Res Dev Disabil. 2013;34(10):3049–56. Elsevier Ltd; Available from. Scholar
  18. King JB, Jones KG, Goldberg E, Rollins M, MacNamee K, Moffit C, et al. Increased functional connectivity after listening to favored music in adults with Alzheimer dementia. J. Prev. Alzheimer’s Dis. 2019;6(1):56–62.Google Scholar
  19. Koutkias V, Kilintzis V, Beredimas N, Maglaveras N. Leveraging medication safety through mobile computing: decision support and guidance services for adverse drug event prevention. Proc. 2014 4th Int. Conf. Wirel. Mob. Commun. Healthc. - "Transforming Healthc. Through Innov. Mob. Wirel. Technol. MOBIHEALTH 2014. 2015;19–22.Google Scholar
  20. Lecheta RR. Google Android [Internet]. 5th ed. São Paulo: Novatec; 2015. Available from:
  21. Likert R. A Technique for the measurement of attitudes. Arch. Psichol. 1932;1:1–55.Google Scholar
  22. Linowes J, Babilinski K. Augmented Reality for Developers: Build practical augmented reality applications with Unity, ARCore, ARKit, and Vuforia. Birmingham, UK: Packt Publishing; 2017. Google Scholar
  23. Mondol AS, Emi IA, Stankovic JA. MedRem : An Interactive Medication Reminder and Tracking System on Wrist Devices. 2016 IEEE Wirel. Heal. [Internet]. Bethesda, MD; 2016. p. 1–8. Available from:
  24. Morel A, Bormans K, Rombouts K. Memory palaces to improve quality of life in dementia. 2015 Conf. Rais. Aware. Soc. Environ. Role Eng. (Re)Training Eng. Particip. Des. 2015;(6):80–4. Available from: Accessed 15 Mar 2019.
  25. Noh S, Han J, Jo J, Choi A. Virtual companion based mobile user interface:: An Intelligent and Simplified Mobile User Interface for the Elderly Users. 2017 Int. Symp. Ubiquitous Virtual Real. [Internet]. Nara; 2017. p. 8–9. Available from:
  26. Ohsaga A, Kondoh K. Bedside medication safety management system using a PDA and RFID tags. 2013 7th Int. Symp. Med. Inf. Commun. Technol. 2013;85–9. Available from: Accessed 15 Mar 2019.
  27. Pavarini S, de Melo L, Manara Silva V, Orlandi F, Zazzetta M, Lúcia Alves Filizola C, et al. Caring for elders with Alzheimer’s disease: experiences of family caregivers. Rev. Eletrônica Enferm. [Internet]. 2008;10:580–90. Available from: Scholar
  28. Plassman BL, Langa KM, Fisher GG, Heeringa SG, Weir DR, Ofstedal MB, et al. Prevalence of dementia in the United States: the aging, demographics, and memory study. Neuroepidemiology. 2007;29(1–2):125–32.CrossRefGoogle Scholar
  29. Prince M, Comas-Herrera A, Knapp M, Guerchet M, Karagiannidou M. World Alzheimer Report 2016 Improving healthcare for people living with dementia. Coverage, Quality and costs now and in the future. 2016; 1–140. Available from: Accessed 15 Mar 2019.
  30. Quintana E, Favela J. Augmented reality annotations to assist persons with Alzheimers and their caregivers. Pers. Ubiquitous Comput. 2013;17(6):1105–16.CrossRefGoogle Scholar
  31. Samson S. Efficacy of non-pharmacological interventions in dementia: methodological requirements. Alzheimer’s Dement. J. Alzheimer’s Assoc. Elsevier; 2017 1;13(7):P1209. Available from: CrossRefGoogle Scholar
  32. Schoenmakers B, Buntinx F, Delepeleire J. Factors determining the impact of care-giving on caregivers of elderly patients with dementia. A systematic literature review. Maturitas. Ireland. Ireland. 2010;66(2):191–200.Google Scholar
  33. Silva BM, Lopes IM, Marques MB, Rodrigues JJPC, Proença ML. A mobile health application for outpatients medication management. IEEE Int. Conf. Commun. 2013:4389–93.Google Scholar
  34. Smith AL, Chaparro BS. Smartphone text input method performance, usability, and preference with younger and older adults. Hum Factors. 2015;57(6):1015–28. Available from:. Scholar
  35. Toyama T, Sonntag D. Towards episodic memory support for dementia patients by recognizing objects, faces and text in eye gaze. In: Hölldobler S, Peñaloza R, Rudolph S, editors. KI 2015 Adv. Artif. Intell. 38th Annu. Ger. Conf. AI, Dresden, Ger. Sept. 21-25, 2015, Proc. Cham: Springer International Publishing; 2015. p. 316–23.CrossRefGoogle Scholar
  36. Trojanowski JQ, Arnold SE, Karlawish JH, Naylor M, Brunden KR, Lee VMY. A model for improving the treatment and care of Alzheimer’s disease patients through interdisciplinary research. Alzheimer’s Dementia1. John Q. Trojanowski, Steven E. Arnold, Jason H. Karlawish, Mary Naylor, Kurt R. Brunden, Virginia M Y Lee. 2012. A Model Improv. Treat. care Alzheimer’s Dis. patients through Interdiscip. Res. Elsevier Ltd; 2012;8(6):564–73. Available from: CrossRefGoogle Scholar
  37. Tsai PH, Chuang YT, Chou TS, Liu JWS, Shih CS. iNuC: an intelligent mobile nursing cart. Proc. 2009 2nd Int. Conf. Biomed. Eng. Informatics, BMEI 2009. 2009.Google Scholar
  38. Tsolaki M, Kounti F, Agogiatou C. Effectiveness of pharmacological and non pharmacological therapy in patients with MCI and mild Alzheimer’s disease. Alzheimer’s Dement. 2009;5(4):P98.CrossRefGoogle Scholar
  39. VanDyk J, Tomlinson T. Pro Drupal 7 Development. Apress, 2010; 2010.Google Scholar
  40. Vu Nguyen K. Special issue: Alzheimer’s disease. AIMS Neurosci. 2018;5(1):74–80 Available from: Scholar
  41. Wood S, Mccrindle RJ. Augmented reality discovery and information system for people with memory loss. Int. Conf. Disabil. Real. Assoc. Technol. Laval 2012 ICDVRAT. Laval, Fr; 2012. [cited 2019 Mar 15] p. 10–2. Available from:, S. and Mccrindle, R.J., Augmented reality discovery and information system for people with memory loss, 9th Intl Conf. Disability, Virtual Reality  Associated Technologies, Laval, France, 2012. 

Copyright information

© Sociedade Brasileira de Engenharia Biomedica 2019

Authors and Affiliations

  1. 1.Group of Virtual and Augmented Reality, Faculty of Electrical EngineerFederal University of UberlândiaUberlândiaBrazil
  2. 2.Laboratory of Experimental Psychology, Institute of PsychologyFederal University of UberlândiaUberlândiaBrazil
  3. 3.Faculty of MedicineFederal University of UberlândiaUberlândiaBrazil

Personalised recommendations