Applications of the Multimodal Interaction Architecture in Ambient Assisted Living

  • António TeixeiraEmail author
  • Nuno Almeida
  • Carlos Pereira
  • Miguel Oliveira e Silva
  • Diogo Vieira
  • Samuel Silva


Developing applications for ambient assisted living (AAL) scenarios requires dealing with diverse user groups, heterogeneous environments, and a large plethora of devices. These requirements pose several challenges on how to design and develop user interaction with the proposed applications and services. In this context, the versatility provided by multimodal interaction (MMI) is paramount and the adopted architecture should be instrumental in harnessing its full potential. This chapter offers an insight on how AAL challenges can be tackled by multimodal-based solutions. It presents the authors’ views and research outcomes in multimodal application development for AAL grounded on an architecture for MMI aligned with the W3C recommendations.


Ambient assisted living (AAL) Multimodal interaction (MMI) Ubiquitous AAL for All Speech for interaction 



The research was partially funded by IEETA Research Unit funding (Incentivo/EEI/UI0127/2014), Marie Curie IAPP project IRIS (ref. 610986, FP7-PEOPLE-2013-IAPP), project Smart Phones for Seniors (S4S), a QREN project (QREN 21541), co-funded by COMPETE and FEDER, project PaeLIFE (AAL-08-1-2001-0001), and project AAL4ALL (AAL/0015/2009).

The authors thank all W3C MMI recommendations contributors for their insightful and inspiring approaches to MMI.


  1. 1.
    Steg, H., Strese, H., Loroff, C., Hull, J., & Schmidt, S. (2006). Ambient assisted living–European overview report, March 2006. EU Specific Support Action.Google Scholar
  2. 2.
    Kung, A., & Jean-Bart, B. (2010). Making AAL platforms a reality. Ambient Intelligence, 6439, Lecture Notes in Computer Science, 187–196.Google Scholar
  3. 3.
    Mikulecký, P., Lišková, T., Čech, P., & Bureš, V. (2009). Ambient intelligence perspectives: Selected papers from the First International Ambient Intelligence Forum 2008. Amsterdam: IOS Press.Google Scholar
  4. 4.
    Hämäläinen, A., Teixeira, A., Almeida, N., Meinedo, H., Fegyó, T., & Dias, M. S. (2015). Multilingual speech recognition for the elderly: The AALFred personal life assistant. Proceedings of the 6th International Conference on Software Development and Technologies for Enhancing Accessibility and Fighting Infoexclusion (DSAI 2015) June 10–12, 2015 Fraunhofer FIT, Sankt Augustin, GermanyGoogle Scholar
  5. 5.
    Teixeira, A., Ferreira, F., Almeida, N., Silva, S., Rosa, A. F., Pereira, J. C., et al. (2016). Design and development of Medication Assistant: older adults centred design to go beyond simple medication reminders. Universal Access in the Information Society.Google Scholar
  6. 6.
    Jara, A. J., Zamora, M. A., & Skarmeta, A. F. G. (2011). An internet of things–based personal device for diabetes therapy management in ambient assisted living (AAL). Personal and Ubiquitous Computing, 15(4), 431–440.CrossRefGoogle Scholar
  7. 7.
    Signoretti, A., Martins, A. I., Almeida, N., Vieira, D., Rosa, A. F., Costa, C. M. M., & Teixeira, A. (2015). Trip 4 all: A gamified app to provide a new way to elderly people to travel. Procedia Computer Science, 67, 301–311.CrossRefGoogle Scholar
  8. 8.
    Li, R., Lu, B., & McDonald-Maier, K. D. (2015). Cognitive assisted living ambient system: A survey. Digital Communications and Networks, 1(4), 229–252.CrossRefGoogle Scholar
  9. 9.
    Dasios, A., Gavalas, D., Pantziou, G., & Konstantopoulos, C. (2015). Hands-on experiences in deploying cost-effective ambient-assisted living systems. Sensors, 15(6), 14487–14512.CrossRefGoogle Scholar
  10. 10.
    Blackman, S., Matlo, C., Bobrovitskiy, C., Waldoch, A., Fang, M. L., Jackson, P., Mihailidis, A., Nygård, L., Astell, A., & Sixsmith, A. (2016). Ambient assisted living technologies for aging well: A scoping review. Journal of Intelligent Systems, 25(1), 55–69.CrossRefGoogle Scholar
  11. 11.
    Gubbi, J., Buyya, R., Marusic, S., & Palaniswami, M. (2013). Internet of Things (IoT): A vision, architectural elements, and future directions. Future Generation Computer Systems, 29(7), 1645–1660.CrossRefGoogle Scholar
  12. 12.
    Feki, M. A., Kawsar, F., Boussard, M., & Trappeniers, L. (2013). The internet of things: The next technological revolution. Computer (Long Beach California), 46(2), 24–25.Google Scholar
  13. 13.
    Gomes, B., Muniz, L., da Silva e Silva, F. J., Ríos, L. E. T., & Endler, M. (2015). A comprehensive cloud-based IoT software infrastructure for ambient assisted living. In 2015 International Conference on Cloud Technologies and Applications (CloudTech), pp. 1–8.Google Scholar
  14. 14.
    Moschetti, A., Fiorini, L., Aquilano, M., Cavallo, F., & Dario, P. (2014). Preliminary findings of the AALIANCE2 ambient assisted living roadmap. In S. Longhi, P. Siciliano, M. Germani, & A. Monteriù (Eds.), Ambient assisted living: Italian forum 2013 (pp. 335–342). Cham: Springer.CrossRefGoogle Scholar
  15. 15.
    Teixeira, A. (2014). A critical analysis of speech-based interaction in healthcare robots: Making a case for the increased use of speech in medical and assistive robots. In A. Neustein (Ed.), Speech and automata in health care (pp. 1–29). Boston: De Gruyter.Google Scholar
  16. 16.
    Teixeira, A., Braga, D., Coelho, L., & Fonseca, A. (2009). Speech as the basic interface for assistive technology. In International Conference on Software Development for Enhancing Accessibility and Fighting Info-Exclusion. Lisbon: UTAD.Google Scholar
  17. 17.
    Wu, D., Cai, Y., & Guizani, M. (2015). Asynchronous flow scheduling for green ambient assisted living communications. IEEE Communications Magazine, 53(1), 64–70.CrossRefGoogle Scholar
  18. 18.
    Memon, M., Wagner, S. R., Pedersen, C. F., Beevi, F. H. A., & Hansen, F. O. (2014). Ambient assisted living healthcare frameworks, platforms, standards, and quality attributes. Sensors, 14(3), 4312–4341.CrossRefGoogle Scholar
  19. 19.
    Gövercin, M., Meyer, S., Schellenbach, M., Steinhagen-Thiessen, E., Weiss, B., & Haesner, M. (2016). SmartSenior@home: Acceptance of an integrated ambient assisted living system. Results of a clinical field trial in 35 households. Informatics for Health and Social Care, 41(4), 430–447.CrossRefGoogle Scholar
  20. 20.
    Teixeira, A., Almeida, N., Pereira, C., Silva, M. O., & Pereira, J. C. (2013). Serviços de Suporte à Interação Multimodal. In A. Teixeira, A. Queirós, & N. Rocha (Eds.), Laboratório Vivo de Usabilidade (pp. 151–165). Portugal: ARC Publishing.Google Scholar
  21. 21.
    Teixeira, A., Pereira, C., e Silva, M. O., Pacheco, O., Neves, A., & Casimiro, J. (2011). AdaptO—Adaptive multimodal output. In Proceedings of PECCS. Vila Moura, Algarve: INSTICC.Google Scholar
  22. 22.
    Teixeira, A., Pereira, C., e Silva, M. O., Alvarelhão, J., Silva, A., Cerqueira, M., et al. (2013). New telerehabilitation services for the elderly. In M. M. Cruz-Cunha (Ed.), Handbook of research on ICTs for healthcare and social services: Developments and applications (pp. 109–132). IGI Global, Hershey, Pennsylvania.Google Scholar
  23. 23.
    Johnston, M. (2009). Building multimodal applications with EMMA. In Proceedings of the 2009 International Conference on Multimodal Interfaces—ICMI-MLMI’09, Cambridge, MA, USA (p. 47).Google Scholar
  24. 24.
    Baggia, P., Burnett, D. C., Carter, J., Dahl, D. A., McCobb, G., & Raggett, D. EMMA: Extensible MultiModal Annotation markup language. W3C Recommendation. Accessed 10 Feb 2009.
  25. 25.
    Dahl, D. A. (2013). The W3C multimodal architecture and interfaces standard. Journal on Multimodal User Interfaces, 7(3), 171–182. doi: 10.1007/s12193-013-0120-5.CrossRefGoogle Scholar
  26. 26.
    Teixeira, A., Francisco, P., Almeida, N., Pereira, C., & Silva, S. (2014). Services to support use and development of speech input for multilingual multimodal applications for mobile scenarios. In The Ninth International Conference on Internet and Web Applications and Services (ICIW 2014), Track WSSA—Web Services-based Systems and Applications. IARIA, Paris.Google Scholar
  27. 27.
    Almeida, N., Teixeira, A., Rosa, A. F., Braga, D., Freitas, J., Dias, M. S., et al. (2015). Giving voices to multimodal applications. In M. Kurosu (Ed.), Human–computer interaction: Interaction technologies is a LNCS volume of the Proceedings of the 17th International Conference HCI International Los Angeles, CA, USA, August 2–7, 2015 (pp. 273–283). Cham: Springer.Google Scholar
  28. 28.
    World Health Organization (2002). Active ageing: A policy framework. A contribution of the World Health Organization to the Second United Nations World Assembly on Ageing, Madrid, Spain, April 2002, WHO/NMH/NPH/02.8.
  29. 29.
    Kairy, D., & Lehoux, P. (2009). A systematic review of clinical outcomes, clinical process, healthcare utilization and costs associated with telerehabilitation. Disability and Rehabilitation, 31(6), 427–447.CrossRefGoogle Scholar
  30. 30.
    Burdea, G. (2002). Keynote address: Virtual rehabilitation-benefits and challenges. In 1st International Workshop on Virtual Reality Rehabilitation (Mental Health, Neurological, Physical, Vocational) VRMHR, Lausanne, Switzerland (pp. 1–11). Vila Moura, Algarve, INSTICC Portugal.Google Scholar
  31. 31.
    Siegel, C., Hochgatterer, A., & Dorner, T. E. (2014). Contributions of ambient assisted living for health and quality of life in the elderly and care services—a qualitative analysis from the experts’ perspective of care service professionals. BMC Geriatrics, 14(1), 112.CrossRefGoogle Scholar
  32. 32.
    Teixeira, A., Pereira, C., Oliveira e Silva, M., Almeida, N., Pinto, J., Teixeira, C., et al. (2012). Health@home scenario: Creating a new support system for home telerehabilitation. In Proceedings of the 2nd International Living Usability Lab Workshop on AAL Latest Solutions, Trends and Applications, AAL 2012, in Conjunction with BIOSTEC 2012 (pp. 37–47). INSTICC: Vila Moura.Google Scholar
  33. 33.
    Pereira, C., Almeida, N., Martins, A. I., Silva, S., Rosa, A. F., Silva, M. O., et al. (2015). Evaluation of complex distributed multimodal applications evaluating a telerehabilitation system when it really matters. In J. Zhou & G. Salvendy (Eds.), Human aspects of IT for the aged population: Design for everyday life (part II) (Proceedings of the 1st International Conference on Human Aspects of IT for the Aged Population, part of HCI International 2015). Los Angeles, CA: Springer. published as LNCS volume 9194.Google Scholar
  34. 34.
    Vieira, D. (2015). Enhanced multimodal interaction framework and applications. Master thesis, Universidade de Aveiro, Aveiro.Google Scholar
  35. 35.
    Vieira, D., Freitas, J. D., Acartürk, C., Teixeira, A., Sousa, L., Silva, S., et al. (2015). Read that article: Exploring synergies between gaze and speech interaction. In Proceedings of the 17th International ACM SIGACCESS Conference on Computers and Accessibility (pp. 341–342). New York, NY.Google Scholar

Copyright information

© Springer International Publishing Switzerland 2017

Authors and Affiliations

  • António Teixeira
    • 1
    Email author
  • Nuno Almeida
    • 1
  • Carlos Pereira
    • 1
  • Miguel Oliveira e Silva
    • 1
  • Diogo Vieira
    • 1
  • Samuel Silva
    • 1
  1. 1.DETI—Department of Electronics, Telecommunications and InformaticsIEETA—Institute of Electronics and Informatics of Aveiro, University of AveiroAveiroPortugal

Personalised recommendations