Interactive, Multi-device Visualization Supported by a Multimodal Interaction Framework: Proof of Concept

  • Nuno Almeida
  • Samuel SilvaEmail author
  • Beatriz Sousa Santos
  • António Teixeira
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9754)


Nowadays, users can interact with a system using a wide variety of modalities, such as touch and speech. Nevertheless, multimodal interaction has yet to be explored for interactive visualization scenarios. Furthermore, users have access to a wide variety of devices (e.g., smartphones, tablets) that could be harnessed to provide a more versatile visualization experience, whether by providing complementary views or by enabling multiple users to jointly explore the visualization using their devices. In our effort to gather multimodal interaction and multi-device support for visualization, this paper describes our first approach to an interactive multi-device system, based on the multimodal interaction architecture proposed by the W3C, enabling interactive visualization using different devices and representations. It allows users to run the application in different types of devices, e.g., tablets or smartphones, and the visualizations can be adapted to multiple screen sizes, by selecting different representations, with different levels of detail, depending on the device characteristics. Groups of users can rely on their personal devices to synchronously visualize and interact with the same data, maintaining the ability to use a custom representation according to their personal needs. A preliminary evaluation was performed, mostly to collect users’ first impressions and guide future developments. Although the results show a moderate user satisfaction, somehow expected at this early stage of development, user feedback allowed the identification of important routes for future improvement, particularly regarding a more versatile navigation along the data and the definition of composite visualizations (e.g., by gathering multiple representations on the same screen).


Multi-device applications Multimodal interaction Interactive visualization 



The work presented in this chapter has been partially funded by IEETA Research Unit funding (Incentivo/EEI/UI0127/2014) and Marie Curie IAPP project IRIS (ref. 610986, FP7-PEOPLE-2013-IAPP).


  1. 1.
    Lee, B., Isenberg, P., Riche, N.H., Carpendale, S.: Beyond mouse and keyboard: expanding design considerations for information visualization interactions. IEEE Trans. Vis. Comput. Graph. 18, 2689–2698 (2012)CrossRefGoogle Scholar
  2. 2.
    Ward, M.O., Grinstein, G., Keim, D.: Interactive Data Visualization: Foundations, Techniques, and Applications. CRC Press, Natick (2010)zbMATHGoogle Scholar
  3. 3.
    Roberts, J.C., Ritsos, P.D., Badam, S.K., Brodbeck, D., Kennedy, J., Elmqvist, N.: Visualization beyond the desktop–the next big thing. IEEE Comput. Graph. Appl. 34, 26–34 (2014)CrossRefGoogle Scholar
  4. 4.
    Jaimes, A., Sebe, N.: Multimodal human-computer interaction: a survey. Comput. Vis. Image Underst. 108, 116–134 (2007)CrossRefGoogle Scholar
  5. 5.
    Lee, J.-H., Poliakoff, E., Spence, C.: The effect of multimodal feedback presented via a touch screen on the performance of older adults. In: Altinsoy, M., Jekosch, U., Brewster, S. (eds.) HAID 2009. LNCS, vol. 5763, pp. 128–135. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  6. 6.
    Schmidt, B.: Facilitating data exploration in casual mobile settings with multi-device interaction (2014).
  7. 7.
    Chung, H., North, C., Self, J.Z., Chu, S., Quek, F.: VisPorter: facilitating information sharing for collaborative sensemaking on multiple displays. Pers. Ubiquitous Comput. 18, 1169–1186 (2014)CrossRefGoogle Scholar
  8. 8.
    Isenberg, P., Elmqvist, N., Scholtz, J., Cernea, D., Ma, K.-L., Hagen, H.: Collaborative visualization: definition, challenges, and research agenda. Inf. Vis. 10, 310–326 (2011)CrossRefGoogle Scholar
  9. 9.
    Almeida, N., Silva, S., Teixeira, A.: Design and development of speech interaction: a methodology. In: Kurosu, M. (ed.) HCI 2014, Part II. LNCS, vol. 8511, pp. 370–381. Springer, Heidelberg (2014)Google Scholar
  10. 10.
    Almeida, N., Silva, S., Teixeira, A.J.S.: Multimodal multi-device application supported by an SCXML state chart machine. In: Workshop on Engineering Interactive Systems with SCXML, the Sixth ACM SIGCHI Symposium on Computing Systems (2014)Google Scholar
  11. 11.
    Almeida, N., Teixeira, A.: Enhanced interaction for the elderly supported by the W3C multimodal architecture. In: Proceedings of 5a Conferência Nacional sobre Interacção (2013)Google Scholar
  12. 12.
    Teixeira, A.J.S., Almeida, N., Pereira, C., e Silva, M.O.: W3C MMI architecture as a basis for enhanced interaction for ambient assisted living. In: Get Smart: Smart Homes, Cars, Devices and the Web, W3C Workshop on Rich Multimodal Application Development. Metropolitan Area, New York (2013)Google Scholar
  13. 13.
    Badam, S., Elmqvist, N.: PolyChrome: a cross-device framework for collaborative web visualization. In: Proceedings of Ninth ACM International Conference on Interactive Tabletops and Surfaces (2014)Google Scholar
  14. 14.
    Heikkinen, T., Goncalves, J., Kostakos, V., Elhart, I., Ojala, T.: Tandem browsing toolkit: distributed multi - display interfaces with web technologies, pp. 142–147 (2014)Google Scholar
  15. 15.
    Hamilton, P., Wigdor, D.J.: Conductor: enabling and understanding cross-device interaction. In: Proceedings of the 32nd Annual ACM Conference on Human Factors in Computing Systems - CHI 2014, pp. 2773–2782. ACM Press, New York (2014)Google Scholar
  16. 16.
    Woźniak, P., Lischke, L., Schmidt, B., Zhao, S., Fjeld, M.: Thaddeus: a dual device interaction space for exploring information visualisation. In: Proceedings of the 8th Nordic Conference on Human-Computer Interaction, pp. 41–50 (2014)Google Scholar
  17. 17.
    Houben, S., Marquardt, N.: WATCHCONNECT: A toolkit for prototyping smartwatch-centric cross-device applications. In: Proceedings of 33rd Annual ACM Conference on Human Factors in Computing Systems (2015)Google Scholar
  18. 18.
    Kernchen, R., Meissner, S., Moessner, K., Cesar, P., Vaishnavi, I., Boussard, M., Hesselman, C.: Intelligent multimedia presentation in ubiquitous multidevice scenarios. IEEE Multimed. 17, 52–63 (2010)CrossRefGoogle Scholar
  19. 19.
    Weibel, N., Oda, R.: Hiperface: a multichannel architecture to explore multimodal interactions with ultra-scale wall displays. In: ICSE 2011: Proceedings of the 33rd International Conference on Software Engineering (2011)Google Scholar
  20. 20.
    Berti, S., Paternò, F.: Migratory multimodal interfaces in multidevice environments. In: Proceedings of 7th International Conference Multimodal interfaces. ACM (2005)Google Scholar
  21. 21.
    Blumendorf, M., Roscher, D., Albayrak, S.: Dynamic user interface distribution for flexible multimodal interaction. In: International Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multimodal Interaction on - ICMI-MLMI 2010. p. 1. ACM Press, New York (2010)Google Scholar
  22. 22.
    Paterno, F.: Multimodality and multi-device interfaces. In: W3C Workshop on Multimodal Interaction, Sophia Antipolis (2004)Google Scholar
  23. 23.
    Shen, C., Esenther, A., Forlines, C., Ryall, K.: Three modes of multisurface interaction and visualization. In: Information Visualization and Interaction Techniques for Collaboration Across Multiple Displays Workshop associated with CHI (2006)Google Scholar
  24. 24.
    Seyed, A.: Examining user experience in multi-display environments (2013)Google Scholar
  25. 25.
    Dahl, D.A.: The W3C multimodal architecture and interfaces standard. J. Multimod. User Interfaces 7(3), 171–182 (2013)CrossRefGoogle Scholar
  26. 26.
    Bodell, M., Dahl, D., Kliche, I., Larson, J., Porter, B., Raggett, D., Raman, T., Rodriguez, B.H., Selvaraj, M., Tumuluri, R., Wahbe, A., Wiechno, P., Yudkowsky, M.: Multimodal architecture and interfaces: W3C recommendation (2012)Google Scholar
  27. 27.
    Baggia, P., Burnett, D.C., Carter, J., Dahl, D.A., McCobb, G., Raggett, D.: EMMA: Extensible multimodal annotation markup language (2009)Google Scholar
  28. 28.
    Pereira, C., Almeida, N., Martins, A.I., Silva, S., Rosa, A.F., Oliveira e Silva, M., Teixeira, A.: Evaluation of complex distributed multimodal applications: evaluating a telerehabilitation system when it really matters. In: Zhou, J., Salvendy, G. (eds.) ITAP 2015. LNCS, vol. 9194, pp. 146–157. Springer, Heidelberg (2015)CrossRefGoogle Scholar
  29. 29.
    Pinelle, D., Gutwin, C., Greenberg, S.: Task analysis for groupware usability evaluation. ACM Trans. Comput. Interact. 10, 281–311 (2003)CrossRefGoogle Scholar
  30. 30.
    Lewis, J.R., Sauro, J.: The factor structure of the system usability scale. In: Kurosu, M. (ed.) HCD 2009. LNCS, vol. 5619, pp. 94–103. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  31. 31.
    Vieira, D.: Enhanced multimodal interaction framework and applications. Master thesis, Aveiro, Universidade de Aveiro, (2015)Google Scholar
  32. 32.
    Vieira, D., Freitas, J.D., Acartürk, C., Teixeira, A., Sousa, L., Silva, S., Candeias, S., Dias, M.S.: Read That Article: Exploring synergies between gaze and speech interaction, pp. 341–342 (2015)Google Scholar
  33. 33.
    Almeida, N., Teixeira, A., Rosa, A.F., Braga, D., Freitas, J., Dias, M.S., Silva, S., Avelar, J., Chesi, C., Saldanha, N.: Giving voices to multimodal applications. In: Kurosu, M. (ed.) Human-Computer Interaction. LNCS, vol. 9170, pp. 273–283. Springer, Heidelberg (2015)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • Nuno Almeida
    • 1
    • 2
  • Samuel Silva
    • 1
    • 2
    Email author
  • Beatriz Sousa Santos
    • 1
    • 2
  • António Teixeira
    • 1
    • 2
  1. 1.DETI – Department of Electronics, Telecommunications and InformaticsUniversity of AveiroAveiroPortugal
  2. 2.IEETA – Institute of Electronics and Informatics Engineering of AveiroUniversity of AveiroAveiroPortugal

Personalised recommendations