Elderly Speech-Gaze Interaction

State of the Art and Challenges for Interaction Design
  • Cengiz Acartürk
  • João Freitas
  • Mehmetcal Fal
  • Miguel Sales Dias
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9175)


Elderly people face problems when using current forms of Human-Computer Interaction (HCI). Developing novel and natural methods of interaction would facilitate resolving some of those issues. We propose that HCI can be improved by combining communication modalities, in particular, speech and gaze, in various ways. This study presents elderly speech-gaze interaction as a novel method in HCI, a review of literature for its potential of use, and discusses possible domains of application for further empirical investigations.


Multimodal Gaze Eye tracking Speech Elderly Interaction 



This work was partially funded by Marie Curie Actions IRIS (ref. 610986, FP7-PEOPLE-2013-IAPP) and METU Scientific Research Project scheme BAP–08-11-2012-121 Investigation of Cognitive Processes in Multimodal Communication.


  1. 1.
    Dias, M.S., Pires, C.G., Pinto, F.M., Teixeira, V.D., Freitas, J.: Multimodal user interfaces to improve social integration of elderly and mobility impaired. Stud. Heal. Technol. Informatics. 177, 14–25 (2012)Google Scholar
  2. 2.
    Phang, C.W., Sutanto, J., Kankanhalli, A., Li, Y., Tan, B.C.Y., Teo, H.-H.: Senior citizens’ acceptance of information systems: a study in the context of e-government services. IEEE Trans. Eng. Manag. 53, 555–569 (2006)CrossRefGoogle Scholar
  3. 3.
    Cisek, E., Triche, K.: Depression and social support among older adult computer users. In: 113th Annual Convention of the American Psychological Association (2005)Google Scholar
  4. 4.
    Oliveira, C., Albuquerque, L., Hämäläinen, A., Pinto, F.M., Dias, M.S., Júdice, A., Freitas, J., Pires, C., Teixeira, V., Calado, A., Braga, D., Teixeira, A.: Tecnologias de Fala para Pessoas Idosas. Laboratório Vivo de Usabilidade (Living Usability Lab), pp. 167–181. ARC Publishing (2013)Google Scholar
  5. 5.
    Stephanidis, C., Akoumianakis, D., Sfyrakis, M., Paramythis, A.: Universal accessibility in HCI: process-oriented design guidelines and tool requirements. In: Proceedings of the 4th ERCIM Workshop on User Interfaces for all, Stockholm, pp. 19–21 (1998)Google Scholar
  6. 6.
    Fox, S.: Are “wired seniors” sitting ducks? Pew Internet & American Life Project (2006)Google Scholar
  7. 7.
    Teixeira, V., Pires, C., Pinto, F., Freitas, J., Dias, M.S., Rodrigues, E.M.: Towards elderly social integration using a multimodal human-computer interface. In: Proceedings of International Living Usability Lab Workshop on AAL Latest Solutions, Trends and Applications (AAL 2012), pp. 3–13 (2012)Google Scholar
  8. 8.
    Jessen, M.: Speaker classification in forensic phonetics and acoustics. In: Müller, C. (ed.) Speaker Classification 2007. LNCS (LNAI), vol. 4343. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  9. 9.
    Linville, S.E.: Vocal Aging. Singular Thomson Learning, CA (2001)Google Scholar
  10. 10.
    Wilpon, J.G., Jacobsen, C.N.: A study of speech recognition for children and the elderly. In: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 1996), pp. 349–352. IEEE (1996)Google Scholar
  11. 11.
    Helfrich, H.: Age Markers in Speech. Cambridge University Press, Cambridge (1979)Google Scholar
  12. 12.
    Pellegrini, T., Hämäläinen, A., de Mareüil, P.B., Tjalve, M., Trancoso, I., Candeias, S., Dias, M.S., Braga, D.: A corpus-based study of elderly and young speakers of european portuguese: acoustic correlates and their impact on speech recognition performance. In: Proceedings of (INTERSPEECH 2013), pp. 852–856 (2013)Google Scholar
  13. 13.
    Stover, S.E., Haynes, W.O.: Topic manipulation and cohesive adequacy in conversations of normal adults between the ages of 30 and 90. Clin. Linguist. Phon. 3, 137–149 (1989)CrossRefGoogle Scholar
  14. 14.
    Albuquerque, L., Oliveira, O., Teixeira, T., Sá-Couto, P., Freitas, J., Dias, M.S.: Impact of age in the production of european portuguese vowels. In: 15th Annual Conference of the International Speech Communication Association (INTERSPEECH 2014), Singapore (2014)Google Scholar
  15. 15.
    Xue, S.A., Hao, G.J.: Changes in the human vocal tract due to aging and the acoustic correlates of speech productiona pilot study. J. Speech Lang. Hear. Res. 46, 689–701 (2003)CrossRefGoogle Scholar
  16. 16.
    Pellegrini, T., Trancoso, I., Hämäläinen, A., Calado, A., Dias, M.S., Braga, D.: Impact of age in asr for the elderly: preliminary experiments in European Portuguese. In: Torre Toledano, D., Ortega Giménez, A., Teixeira, A., González Rodr\’ıguez, J., Hernández Gómez, L., San Segundo Hernández, R., Ramos Castro, D. (eds.) IberSPEECH 2012. CCIS, vol. 328, pp. 139–147. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  17. 17.
    Schultz, T.: Speaker characteristics. In: Müller, C. (ed.) Speaker Classification 2007. LNCS (LNAI), vol. 4343, pp. 47–74. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  18. 18.
    Anderson, S., Liberman, N., Bernstein, E., Foster, S., Cate, E., Levin, B., Hudson, R.: Recognition of elderly speech and voice-driven document retrieval. In: IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 1999), pp. 145–148. IEEE (1999)Google Scholar
  19. 19.
    Baba, A., Yoshizawa, S., Yamada, M., Lee, A., Shikano, K.: Elderly acoustic model for large vocabulary continuous speech recognition. IEICE Trans. Inf. Syst. J85-D-2(3), 390–397 (2002)Google Scholar
  20. 20.
    Vipperla, R., Wolters, M., Georgila, K., Renals, S.: Speech input from older users in smart environments: challenges and perspectives. In: Stephanidis, C. (ed.) UAHCI 2009, Part II. LNCS, vol. 5615, pp. 117–126. Springer, Heidelberg (2009)Google Scholar
  21. 21.
    Hämäläinen, A., Avelar, J., Rodrigues, S. Dias, M., Kolesinski, A., Fegyó, T., Németh, G., Csobánka, P., Lan, K., Hewson, D.: The EASR corpora of European Portuguese, French, Hungarian and polish elderly speech. In: LREC, pp. 1458–1464 (2014)Google Scholar
  22. 22.
    Hämäläinen, A., Pinto, F., Dias, M., Júdice, A., Freitas, J., Pires, C., Teixeira, V., Calado, A., Braga, D.: The first European Portuguese elderly speech corpus. In: Proceedings of IberSPEECH, Madrid, Spain (2012)Google Scholar
  23. 23.
    Olincy, A., Ross, R.G., Youngd, D.A., Freedman, R.: Age diminishes performance on an antisaccade eye movement task. Neurobiol. Aging 18, 483–489 (1997)CrossRefGoogle Scholar
  24. 24.
    Munoz, D.P., Broughton, J.R., Goldring, J.E., Armstrong, I.T.: Age-related performance of human subjects on saccadic eye movement tasks. Exp. Brain Res. 121, 391–400 (1998)CrossRefGoogle Scholar
  25. 25.
    Munoz, D.P., Everling, S.: Look away: the anti-saccade task and the voluntary control of eye movement. Nat. Rev. Neurosci. 5, 218–228 (2004)CrossRefGoogle Scholar
  26. 26.
    Bowling, A., Draper, A.: Using saccadic eye movements to assess cognitive decline with ageing. In: Horsley, M., Eliot, M., Knight, B.A., Reilly, R. (eds.) Current Trends in Eye Tracking Research, pp. 237–244. Springer International Publishing, Switzerland (2014)CrossRefGoogle Scholar
  27. 27.
    Trick, L.M., Enns, J.T.: Lifespan changes in attention: the visual search task. Cogn. Dev. 13, 369–386 (1998)CrossRefGoogle Scholar
  28. 28.
    Scialfa, C.T., Thomas, D.M., Joffe, K.M.: Age differences in the useful field of view: an eye movement analysis. Optom. Vis. Sci. Official Publ. Am. Acad. Optom. 71, 736–742 (1994)CrossRefGoogle Scholar
  29. 29.
    Beurskens, R., Bock, O.: Age-related decline of peripheral visual processing: the role of eye movements. Exp. Brain Res. 217, 117–124 (2012)CrossRefGoogle Scholar
  30. 30.
    Kliegl, R., Nuthmann, A., Engbert, R.: Tracking the mind during reading: the influence of past, present, and future words on fixation durations. J. Exp. Psychol. Gen. 135, 12–35 (2006)CrossRefGoogle Scholar
  31. 31.
    Rayner, K.: Eye movements in reading and information processing: 20 years of research. Psychol. Bull. 124, 372–422 (1998)CrossRefGoogle Scholar
  32. 32.
    Rayner, K., Pollatsek, A., Ashby, J., Clifton, C.: The Psychology of Reading, 2nd edn. Psychology Press, Abingdon (2012)Google Scholar
  33. 33.
    Holmqvist, K., Nyström, M., Andersson, R., Dewhurst, R., Halszka, J., van de Weijer, J.: Eye Tracking: A Comprehensive Guide to Methods and Measures. Oxford University Press, Oxford (2011)Google Scholar
  34. 34.
    Rayner, K., Castelhano, M.S., Yang, J.: Eye movements and the perceptual span in older and younger readers. Psychol. Aging 24, 755–760 (2009)CrossRefGoogle Scholar
  35. 35.
    Rayner, K., Yang, J., Schuett, S., Slattery, T.J.: The effect of foveal and parafoveal masks on the eye movements of older and younger readers. Psychol. Aging 29, 205–212 (2014)CrossRefGoogle Scholar
  36. 36.
    Maltz, M., Shinar, D.: Eye movements of younger and older drivers. Hum. Factors 4, 15–25 (1999)CrossRefGoogle Scholar
  37. 37.
    Ho, G., Scialfa, C.T., Caird, J.K., Graw, T.: Visual search for traffic signs: the effects of clutter, luminance, and aging. Hum. Factors 432, 194–207 (2001)CrossRefGoogle Scholar
  38. 38.
    Schieber, F., Gilland, J.: Age differences in the useful field of view during real-world driving. In: Proceedings of the Human Factors and Ergonomic Society Annual Meeting, vol. 49, pp. 182–185. Sage Publications (2005)Google Scholar
  39. 39.
    Cantin, M.L.M.T.V., Teasdale, M.S.N.: Aging yields a smaller number of fixations and a reduced gaze amplitude when driving in a simulator. Adv. Transp. Stud. Int. J. Special Issue, 21–30 (2006)Google Scholar
  40. 40.
    Bitsios, P., Prettyman, R., Szabadi, E.: Changes in autonomic function with age: a study of pupillary kinetics in healthy young and old people. Age Ageing 25, 432–438 (1996)CrossRefGoogle Scholar
  41. 41.
    Kuchinsky, S.E., Ahlstrom, J.B., Vaden, K.I., Cute, S.L., Humes, L.E., Dubno, J.R., Eckert, M.A.: Pupil size varies with word listening and response selection difficulty in older adults with hearing loss. Psychophysiology 50, 23–34 (2013)CrossRefGoogle Scholar
  42. 42.
    Mather, M., Carstensen, L.L.: Aging and motivated cognition: the positivity effect in attention and memory. Trends Cogn. Sci. 9, 496–502 (2005)CrossRefGoogle Scholar
  43. 43.
    McGurk, H., MacDonald, J.: Hearing lips and seeing voices. Nature 264, 746–748 (1976)CrossRefGoogle Scholar
  44. 44.
    Oviatt, S.: Ten myths of multimodal interaction. Commun. ACM 42, 74–81 (1999)CrossRefGoogle Scholar
  45. 45.
    Quek, F., McNeill, D., Bryll, R., Duncan, S., Ma, X.-F., Kirbas, C., McCullough, K.E., Ansari, R.: Multimodal human discourse: gesture and speech. ACM Trans. Comput. Interact. 9, 171–193 (2002)CrossRefGoogle Scholar
  46. 46.
    Hakkani-Tür, D., Slaney, M., Celikyilmaz, A., Heck, L.: Eye gaze for spoken language understanding in multi-modal conversational interactions. In: Proceedings of the 16th International Conference on Multimodal Interaction, pp. 263–266. ACM (2014)Google Scholar
  47. 47.
    Cooke, N.J., Russell, M.: Gaze-contingent automatic speech recognition. Signal Process. IET 2, 369–380 (2008)CrossRefGoogle Scholar
  48. 48.
    Cooke, N., Shen, A., Russell, M.: Exploiting a “gaze-Lombard effect” to improve ASR performance in acoustically noisy settings. In: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2014), pp. 1754–1758. IEEE (2014)Google Scholar
  49. 49.
    Prasov, Z., Chai, J.Y.: What’s in a gaze? the role of eye-gaze in reference resolution in multimodal conversational interfaces. In: Proceedings of the 13th International Conference on Intelligent User Interfaces, pp. 20–29. ACM (2008)Google Scholar
  50. 50.
    Heck, L.P., Hakkani-Tür, D., Chinthakunta, M., Tür, G., Iyer, R., Parthasarathy, P., Stifelman, L., Shriberg, E., Fidler, A.: Multi-modal conversational search and browse. In: SLAM Workshop, pp. 96–101 (2013)Google Scholar
  51. 51.
    Slaney, M., Rajan, R., Stolcke, A., Parthasarathy, P.: Gaze-enhanced speech recognition. In: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2014), pp. 3236–3240. IEEE (2014)Google Scholar
  52. 52.
    Vertanen, K.: Efficient correction interfaces for speech recognition (2009)Google Scholar
  53. 53.
    Vertanen, K., MacKay, D.J.C.: Speech dasher: fast writing using speech and gaze. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 595–598. ACM (2010)Google Scholar
  54. 54.
    Vertanen, K., MacKay, D.J.C.: Speech dasher: a demonstration of text input using speech and approximate pointing. In: Proceedings of the 16th International ACM SIGACCESS Conference on Computers & Accessibility, pp. 353–354. ACM (2014)Google Scholar
  55. 55.
    Slaney, M., Stolcke, A., Hakkani-Tür, D.: The relation of eye gaze and face pose: Potential impact on speech recognition. In: Proceedings of the 16th International Conference on Multimodal Interaction, pp. 144–147. ACM (2014)Google Scholar
  56. 56.
    Malins, J.G., Joanisse, M.F.: The roles of tonal and segmental information in Mandarin spoken word recognition: an eyetracking study. J. Mem. Lang. 62, 407–420 (2010)CrossRefGoogle Scholar
  57. 57.
    Mitterer, H., Reinisch, E.: No delays in application of perceptual learning in speech recognition: evidence from eye tracking. J. Mem. Lang. 69, 527–545 (2013)CrossRefGoogle Scholar
  58. 58.
    Kennington, C., Kousidis, S., Schlangen, D.: Interpreting situated dialogue utterances: an update model that uses speech, gaze, and gesture information. In: Proceedings of the SIGDIAL 2013 (2013)Google Scholar
  59. 59.
    Misu, T., Raux, A., Lane, I., Devassy, J., Gupta, R.: Situated multi-modal dialog system in vehicles. In: Proceedings of the 6th Workshop on Eye Gaze in Intelligent Human Machine Interaction: Gaze in Multimodal Interaction, pp. 25–28. ACM (2013)Google Scholar
  60. 60.
    Dalmaijer, E.S.: Is the low-cost EyeTribe eye tracker any good for research? PeerJ PrePrints (Preprint, 2015).,, Accessed 25 Jan 2015
  61. 61.
    Dowiasch, S., Marx, S., Einhauser, W., Bremmer, F.: Effects of aging on eye movements in the real world. Frontiers Hum. Neurosci. 9, 46 (2015)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Cengiz Acartürk
    • 1
  • João Freitas
    • 2
  • Mehmetcal Fal
    • 1
  • Miguel Sales Dias
    • 2
    • 3
  1. 1.Informatics InstituteMiddle East Technical UniversityAnkaraTurkey
  2. 2.Microsoft Language Development CenterLisbonPortugal
  3. 3.Instituto Universitário de Lisboa (ISCTE-IUL), ISTAR-IULLisboaPortugal

Personalised recommendations