Multi-Device Applications Using the Multimodal Architecture

  • Nuno Almeida
  • Samuel SilvaEmail author
  • António Teixeira
  • Diogo Vieira


Nowadays, users have access to a multitude of devices at their homes, workplaces or that they can carry around. Each of these devices, given its features (e.g., interaction modalities, screen size), might be more suitable for particular users, tasks, and contexts. While having one application installed in several devices might be common, they mostly work isolated, not exploring the possibilities of several devices working together to provide a more versatile and richer interaction scenario. Adopting a multimodal interaction (MMI) architecture based on the W3C recommendations, beyond the advantages to the design and development of MMI, provides, we argue, an elegant approach to tackle multi-device interaction scenarios. In this regard, this chapter conveys our views and research outcomes addressing this subject, presenting concrete application examples.


Multi-device applications Multimodal interaction Ambient assisted living Interactive visualization 



The work presented in this chapter has been partially funded by IEETA Research Unit funding (Incentivo/EEI/UI0127/2014), Marie Curie IAPP project IRIS (ref. 610986, FP7-PEOPLE-2013-IAPP), project PaeLife (AAL-08-1-2001-0001), and QREN projects Smart Phones for Seniors (S4S), AAL4ALL and EMIF—European Medical Information Framework (EU FP7), co-funded by COMPETE and FEDER.

The authors thank all W3C MMI recommendations contributors for their insightful and inspiring approaches to MMI.


  1. 1.
    Ghiani, G., Polet, J., Antila, V., & Mäntyjärvi, J. (2015). Evaluating context-aware user interface migration in multi-device environments. Journal of Ambient Intelligence and Humanized Computing, 6(2), 259–277.CrossRefGoogle Scholar
  2. 2.
    Dahl, D. A. (2013). The W3C multimodal architecture and interfaces standard. Journal on Multimodal User Interfaces, 7(3), 171–182.CrossRefGoogle Scholar
  3. 3.
    Freitas, J., Candeias, S., Dias, M. S., Lleida, E., Ortega, A., Teixeira, A., et al. (2014). The IRIS project: A liaison between industry and academia towards natural multimodal communication. In Proceedings of Iberspeech, Las Palmas de Gran Canaria, Spain, pp. 338–347.Google Scholar
  4. 4.
    Rekimoto, J. (1998). A multiple device approach for supporting whiteboard-based interactions. In Proceedings of the Conference on Human Factors in Computing Systems (CHI’98), Los Angeles, CA, pp. 344–351.Google Scholar
  5. 5.
    Diehl, J., & Borchers, J. O. (2014). Supporting multi-device iteraction in the wild by exposing application state. (PhD thesis, No. RWTH-CONV-144030). Aachen:Fachgruppe Informatik.Google Scholar
  6. 6.
    Teixeira, A., Hämäläinen, A., Avelar, J., Almeida, N., Németh, G., Fegyó, T., et al. (2013). Speech-centric multimodal interaction for easy-to-access online services: A personal life assistant for the elderly. In Proceedings DSAI 2013, Procedia Computer Science, Vigo, Spain, pp. 389–397.Google Scholar
  7. 7.
    Hämäläinen, A., Teixeira, A., Almeida, N., Meinedo, H., Fegyó, T., & Dias, M. S. (2015). Multilingual speech recognition for the elderly: the AALFred personal life assistant. Procedia Computer Science, 67, 283–292.CrossRefGoogle Scholar
  8. 8.
    Teixeira, A. J. S., Pereira, C., Oliveira e Silva, M., Alvarelhão, J., Silva, A., Cerqueira, M., et al. (2013). New telerehabilitation services for the elderly. In I. M. Miranda & M. M. Cruz-Cunha (Eds.), Handbook of research on ICTs for healthcare and social services: Developments and applications. Hershey, PA: IGI Global.Google Scholar
  9. 9.
    Ferreira, F., Almeida, N., Rosa, A. F., Oliveira, A., Casimiro, J., Silva, S., et al. (2013). Elderly centered design for Interaction—the case of the S4S Medication Assistant. In 5th International Conference on Software Development and Technologies for Enhancing Accessibility and Fighting Info-exclusion, DSAI, Vigo, Spain.Google Scholar
  10. 10.
    Leal, A., Teixeira, A., & Silva, S. (2016). On the creation of a persona to support the development of technologies for children with autism spectrum disorder. In Proc. HCI International LNCS 9739, Toronto, Canada, 213–223. doi:  10.1007/978-3-319-40238-3_21
  11. 11.
    Almeida, N., Silva, S., Santos, B. S., & Teixeira, A. (2016). Interactive, multi-device visualization supported by a multimodal interaction framework: Proof of concept. In Proc. HCI International. LNCS 9754, Toronto, Canada, 279–289. doi:  10.1007/978-3-319-39943-0_27
  12. 12.
    Almeida, N., Silva, S., & Teixeira, A. (2014). Design and development of speech interaction: a methodology. In Proc. HCI International, LNCS 8511, Crete, Greece, 370–381.Google Scholar
  13. 13.
    Teixeira, A., Francisco, P., Almeida, N., Pereira, C., & Silva, S. (2014). Services to support use and development of speech input for multilingual multimodal applications for mobile scenarios. In The Ninth International Conference on Internet and Web Applications and Services (ICIW 2014), Track WSSA—Web Services-based Systems and Applications, Paris, France.Google Scholar
  14. 14.
    Vieira, D., Freitas, J. D., Acartürk, C., Teixeira, A., Sousa, L., Silva, S., Candeias, S., and Sales Dias, M. (2015). "Read that article": Exploring synergies between gaze and speech interaction. In Proc. 17th International ACM SIGACCESS Conference on Computers & Accessibility (ASSETS '15). ACM, New York, NY, USA, 341–342. doi:  10.1145/2700648.2811369
  15. 15.
    Wiechno, P., Dahl, D., Ashimura, K., & Tumuluri, R. (2012). Registration & discovery of multimodal modality components in multimodal systems: Use cases and requirements. [Online]. Accessed 1 Jan 2016.
  16. 16.
    Almeida, N., Silva, S., & Teixeira, A. J. S. (2014). Multimodal multi-device application supported by an SCXML state chart machine. In Workshop on Engineering Interactive Systems with SCXML, The sixth ACM SIGCHI Symposium on Computing Systems, Toronto, Canada.Google Scholar
  17. 17.
    Almeida N., & Teixeira A. (2013). Enhanced interaction for the elderly supported by the W3C Multimodal Architecture. In Proc. 5a Conf. Nacional sobre Interacção, Vila Real, Portugal.Google Scholar
  18. 18.
    Teixeira, A., Almeida, N., Pereira, C., e Silva, M. O., & Pereira, J. C. (2013). Serviços de Suporte à Interação Multimodal. In A. Teixeira, A. Queirós, & N. Rocha (Eds.), Laboratório Vivo de Usabilidade (pp. 151–165). ARC Publishing.Google Scholar
  19. 19.
    Teixeira, A., Almeida, N., Pereira, C., e Silva, M. O., Vieira, D., & Silva, S. (2016). Applications in ambient assisted living. In D. Dahl (Ed.), Multimodal Interaction with W3C Standards. Springer.Google Scholar
  20. 20.
    Barnett, J., Akolkar, R., Auburn, R. J., Bodell, M., Burnett, D. C., Carter, J., et al. (2015), State chart XML (SCXML): State machine notation for control abstraction. W3C Recommendation. Accessed 29 Jul 2016.
  21. 21.
    Baggia, P., Burnett, D. C., Carter, J., Dahl, D. A., McCobb, G., & Raggett, D. (2009). EMMA: Extensible multimodal annotation markup language. Accessed 1 Jan 2016.
  22. 22.
    Lee, B., Isenberg, P., Riche, N. H., & Carpendale, S. (2012). Beyond mouse and keyboard: Expanding design considerations for information visualization interactions. IEEE Transactions on Visualization and Computer Graphics, 18(12), 2689–2698.CrossRefGoogle Scholar
  23. 23.
    Ward, M. O., Grinstein, G., & Keim, D. (2010). Interactive data visualization: Foundations, techniques, and applications. Natick, MA: CRC Press.zbMATHGoogle Scholar
  24. 24.
    Roberts, J. C., Ritsos, P. D., Badam, S. K., Brodbeck, D., Kennedy, J., & Elmqvist, N. (2014). Visualization beyond the desktop—the next big thing. IEEE Computer Graphics and Applications, 34(6), 26–34.CrossRefGoogle Scholar
  25. 25.
    Jaimes, A., & Sebe, N. (2007). Multimodal human-computer interaction: A survey. Computer Vision and Image Understanding, 108(1–2), 116–134.CrossRefGoogle Scholar
  26. 26.
    Schmidt, B. (2014). Facilitating data exploration in casual mobile settings with multi-device interaction. Universitat Stuttgart, Holzgartenstr. 16, 70174 Stuttgart.Google Scholar
  27. 27.
    Chung, H., North, C., Self, J. Z., Chu, S., & Quek, F. (2014). VisPorter: Facilitating information sharing for collaborative sensemaking on multiple displays. Personal and Ubiquitous Computing, 18(5), 1169–1186.CrossRefGoogle Scholar
  28. 28.
    Isenberg, P., Elmqvist, N., Scholtz, J., Cernea, D., Ma, K.-L., & Hagen, H. (2011). Collaborative visualization: Definition, challenges, and research agenda. Information Visualization, 10(4), 310–326.CrossRefGoogle Scholar
  29. 29.
    Pereira, C., Almeida, N., Martins, A. I., Silva, S., Rosa, A. F., Oliveira e Silva, M., & Teixeira, A. (2015). Evaluation of complex distributed multimodal applications: evaluating a telerehabilitation system when it really matters. In Proc. HCI International, LNCS 9194, Los Angeles, CA, USA, 146–157, doi: 10.1007/978-3-319-20913-5_14
  30. 30.
    Bostock, M., Ogievetsky, V., & Heer, J. (2011). D3: Data-driven documents. IEEE Transactions on Visualization and Computer Graphics, 17(12), 2301–2309.CrossRefGoogle Scholar
  31. 31.
    Barnett, J., Dahl, D., Tumuluri, R., Kharidi, N., & Ashimura, K. (2016). Discovery and registration of multimodal modality components: State handling. [Online]. Accessed 15 Mar 2016.

Copyright information

© Springer International Publishing Switzerland 2017

Authors and Affiliations

  • Nuno Almeida
    • 1
  • Samuel Silva
    • 1
    Email author
  • António Teixeira
    • 1
  • Diogo Vieira
    • 1
  1. 1.DETI—Department of Electronics, Telecommunications and InformaticsIEETA—Institute of Electronics and Informatics of Aveiro, University of AveiroAveiroPortugal

Personalised recommendations