Management of Multimodal User Interaction in Companion-Systems

  • Felix Schüssel
  • Frank Honold
  • Nikola Bubalo
  • Michael Weber
  • Anke Huckauf
Part of the Cognitive Technologies book series (COGTECH)


While interacting, human beings continuously adapt their way of communication to their surroundings and their communication partner. Although present context-aware ubiquitous systems gather a lot of information to maximize their functionality, they predominantly offer rather static ways to communicate. In order to fulfill the user’s communication needs and demands, ubiquitous sensors’ varied information could be used to dynamically adapt the user interface. Considering such an adaptive user interface management as a major and relevant component for a Companion-Technology, we also have to cope with emotional and dispositional user input as a source of implicit user requests and demands. In this chapter we demonstrate how multimodal fusion based on evidential reasoning and probabilistic fission with adaptive reasoning can act together to form a highly adaptive and model-driven interactive system component for multimodal interaction. The presented interaction management (IM) can handle uncertain or ambiguous data throughout the complete interaction cycle with a user. In addition, we present the IM’s architecture and its model-driven concept. Finally, we discuss its role within the framework of the other constituents of a Companion-Technology.



This work was done within the Transregional Collaborative Research Centre SFB/TRR 62 “Companion-Technology for Cognitive Technical Systems” funded by the German Research Foundation (DFG).


  1. 1.
    Ameri Ekhtiarabadi, A., Akan, B., Çürüklu, B., Asplund, L.: A general framework for incremental processing of multimodal inputs. In: Proceedings of the 13th International Conference on Multimodal Interface, ICMI ’11, pp. 225–228. ACM, New York (2011). doi:10.1145/2070481.2070521Google Scholar
  2. 2.
    Atrey, P., Hossain, M.A., El Saddik, A., Kankanhalli, M.: Multimodal fusion for multimedia analysis: a survey. Multimedia Systems 16, 345–379 (2010)CrossRefGoogle Scholar
  3. 3.
    Bertrand, G., Nothdurft, F., Honold, F., Schüssel, F.: CALIGRAPHI - Creation of Adaptive Dialogues Using a Graphical Interface. In: 2011 IEEE 35th Annual Computer Software and Applications Conference (COMPSAC), pp. 393–400. IEEE, New York (2011). doi:10.1109/COMPSAC.2011.58Google Scholar
  4. 4.
    Biundo, S., Wendemuth, A.: Companion-technology for cognitive technical systems. Künstl. Intell. (2016). doi:10.1007/s13218-015-0414-8Google Scholar
  5. 5.
    Blumendorf, M., Roscher, D., Albayrak, S.: Dynamic user interface distribution for flexible multimodal interaction. In: ICMI and the Workshop on Machine Learning for Multimodal Interaction. ACM, New York (2010). doi:10.1145/1891903.1891930CrossRefGoogle Scholar
  6. 6.
    Brandes, U., Eiglsperger, M., Herman, I., Himsolt, M., Marshall, M.: GraphML progress report: structural layer proposal. In: Mutzel, P., Jünger, M., Leipert, S. (eds.) Graph Drawing. Lecture Notes in Computer Science, vol. 2265, pp. 501–512. Springer, Berlin, Heidelberg (2002). doi:10.1007/3-540-45848-4_59Google Scholar
  7. 7.
    Calvary, G., Coutaz, J., Thevenin, D., Bouillon, L., Florins, M., Limbourg, Q., Souchon, N., Vanderdonckt, J., Marucci, L., Paternò, F., Santoro, C.: The Cameleon reference framework. Technical Report 1.1. CAMELEON Reference Framework Working Group (2002)Google Scholar
  8. 8.
    Cohen, P.R., Johnston, M., McGee, D., Oviatt, S., Pittman, J., Smith, I., Chen, L., Clow, J.: QuickSet: multimodal interaction for distributed applications. In: Proceedings of the Fifth ACM International Conference on Multimedia, MULTIMEDIA ’97, pp. 31–40. ACM, New York (1997). doi:10.1145/266180.266328Google Scholar
  9. 9.
    Costa, D., Duarte, C.: Adapting multimodal fission to user’s abilities. In: Proceedings of the 6th International Conference on Universal Access in Human-Computer Interaction: Design for All and EInclusion - Volume Part I, UAHCI’11, pp. 347–356. Springer, Berlin (2011)Google Scholar
  10. 10.
    Cutugno, F., Leano, V.A., Rinaldi, R., Mignini, G.: Multimodal framework for mobile interaction. In: Proceedings of the International Working Conference on Advanced Visual Interfaces, AVI ’12, pp. 197–203. ACM, New York (2012). doi:10.1145/2254556.2254592Google Scholar
  11. 11.
    Dey, A.K., Abowd, G.D.: Towards a better understanding of context and context-awareness. In: HUC ’99: Proceedings of the 1st International Symposium on Handheld and Ubiquitous Computing, pp. 304–307. Springer, Berlin (1999)Google Scholar
  12. 12.
    Duarte, C., Carriço, L.: A conceptual framework for developing adaptive multimodal applications. In: IUI ’06: Proceedings of the 11th International Conference on Intelligent User Interfaces, pp. 132–139. ACM, New York (2006). doi:10.1145/1111449.1111481Google Scholar
  13. 13.
    Dumas, B., Lalanne, D., Ingold, R.: Prototyping multimodal interfaces with the SMUIML modeling language. In: CHI 2008 Workshop on User Interface Description Languages for Next Generation User Interfaces, CHI 2008, Frienze, pp. 63–66 (2008)Google Scholar
  14. 14.
    Dumas, B., Lalanne, D., Guinard, D., Koenig, R., Ingold, R.: Strengths and weaknesses of software architectures for the rapid creation of tangible and multimodal interfaces. In: TEI ’08: Proceedings of the 2nd International Conference on Tangible and Embedded Interaction, pp. 47–54. ACM, New York (2008). doi:10.1145/1347390.1347403Google Scholar
  15. 15.
    Dumas, B., Lalanne, D., Oviatt, S.: Multimodal interfaces: a survey of principles, models and frameworks. In: Lalanne, D., Kohlas, J. (eds.) Human Machine Interaction – Research Results of the MMI Program. Lecture Notes in Computer Science, vol. 5440/2009, chap. 1, pp. 3–26. Springer, Berlin, Heidelberg (2009). doi:10.1007/978-3-642-00437-7_1Google Scholar
  16. 16.
    Dumas, B., Signer, B., Lalanne, D.: Fusion in multimodal interactive systems: an HMM-based algorithm for user-induced adaptation. In: Proceedings of the 4th ACM SIGCHI Symposium on Engineering Interactive Computing Systems, EICS ’12, pp. 15–24. ACM, New York (2012). doi:10.1145/2305484.2305490Google Scholar
  17. 17.
    Foster, M.E.: State of the art review: multimodal fission. Public Deliverable 6.1, University of Edinburgh (2002). COMIC ProjectGoogle Scholar
  18. 18.
    Geier, T., Reuter, S., Dietmayer, K., Biundo, S.: Goal-based person tracking using a first-order probabilistic model. In: Proceedings of the Ninth UAI Bayesian Modeling Applications Workshop (UAI-AW 2012) (2012)Google Scholar
  19. 19.
    Hina, M.D., Tadj, C., Ramdane-Cherif, A., Levy, N.: A Multi-Agent based Multimodal System Adaptive to the User’s Interaction Context. Multi-Agent Systems – Modeling, Interactions, Simulations and Case Studies, chap. 2, pp. 29–56. InTech (2011)Google Scholar
  20. 20.
    Holzapfel, H., Nickel, K., Stiefelhagen, R.: Implementation and evaluation of a constraint-based multimodal fusion system for speech and 3D pointing gestures. In: Proceedings of the 6th International Conference on Multimodal Interfaces, ICMI ’04, pp. 175–182. ACM, New York (2004). doi:10.1145/1027933.1027964Google Scholar
  21. 21.
    Honold, F., Schüssel, F., Weber, M.: Adaptive probabilistic fission for multimodal systems. In: Proceedings of the 24th Australian Computer-Human Interaction Conference, OzCHI ’12, pp. 222–231. ACM, New York (2012). doi:10.1145/2414536.2414575Google Scholar
  22. 22.
    Honold, F., Schüssel, F., Weber, M., Nothdurft, F., Bertrand, G., Minker, W.: Context models for adaptive dialogs and multimodal interaction. In: 2013 9th International Conference on Intelligent Environments (IE), pp. 57–64. IEEE, New York (2013). doi:10.1109/IE.2013.54Google Scholar
  23. 23.
    Honold, F., Schüssel, F., Munding, M., Weber, M.: Tangible context modelling for rapid adaptive system testing. In: 2013 9th International Conference on Intelligent Environments (IE), pp. 278–281. IEEE, Athens (2013). doi:10.1109/IE.2013.9Google Scholar
  24. 24.
    Honold, F., Bercher, P., Richter, F., Nothdurft, F., Geier, T., Barth, R., Hörnle, T., Schüssel, F., Reuter, S., Rau, M., Bertrand, G., Seegebarth, B., Kurzok, P., Schattenberg, B., Minker, W., Weber, M., Biundo, S.: Companion-technology: towards user- and situation-adaptive functionality of technical systems. In: 2014 10th International Conference on Intelligent Environments (IE), pp. 378–381. IEEE, New York (2014). doi:10.1109/IE.2014.60Google Scholar
  25. 25.
    Honold, F., Schüssel, F., Weber, M.: The automated interplay of multimodal fission and fusion in adaptive HCI. In: IE’14: Proceedings of the 10th International Conference on Intelligent Environments, pp. 170–177. IEEE, Shanghai (2014). doi:10.1109/IE.2014.32Google Scholar
  26. 26.
    Johnston, M., Bangalore, S.: Finite-state multimodal integration and understanding. Nat. Lang. Eng. 11, 159–187 (2005). doi:10.1017/S1351324904003572CrossRefGoogle Scholar
  27. 27.
    Koons, D.B., Sparrell, C.J., Thorisson, K.R.: Integrating simultaneous input from speech, gaze, and hand gestures. In: Maybury, M.T. (ed.) Intelligent Multimedia Interfaces, chap. 11, pp. 257–276. American Association for AI, Menlo Park, CA (1993)Google Scholar
  28. 28.
    Krahnstoever, N., Kettebekov, S., Yeasin, M., Sharma, R.: A real-time framework for natural multimodal interaction with large screen displays. In: Proceedings of the 4th IEEE International Conference on Multimodal Interfaces, ICMI ’02, pp. 349–354. IEEE Computer Society, Washington, DC (2002). doi:10.1109/ICMI.2002.1167020Google Scholar
  29. 29.
    Lalanne, D., Nigay, L., Palanque, P., Robinson, P., Vanderdonckt, J., Ladry, J.F.: Fusion engines for multimodal input: a survey. In: Proceedings of the 2009 International Conference on Multimodal Interfaces, ICMI-MLMI ’09, pp. 153–160. ACM, New York (2009). doi:10.1145/1647314.1647343Google Scholar
  30. 30.
    LaViola, J.J. Jr., Buchanan, S., Pittman, C.: Multimodal Input for Perceptual User Interfaces, chap. 9, pp. 285–312. Wiley, New York (2014). doi:10.1002/9781118706237.ch9Google Scholar
  31. 31.
    Mansoux, B., Nigay, L., Troccaz, J.: Output multimodal interaction: the case of augmented surgery. In: Bryan-Kinns, N., Blanford, A., Curzon, P., Nigay, L. (eds.) People and Computers XX – Engage. BCS Conference Series, vol. 5, pp. 177–192. Springer/ACM, London/New York (2006). doi:10.1007/978-1-84628-664-3_14Google Scholar
  32. 32.
    Martin, J.C.: Tycoon: theoretical framework and software tools for multimodal interfaces. In: Lee, J. (ed.) Intelligence and Multimodality in Multimedia Interfaces. AAAI Press, Palo Alto, CA (1998)Google Scholar
  33. 33.
    Neal, J.G., Thielman, C.Y., Dobes, Z., Haller, S.M., Shapiro, S.C.: Natural language with integrated deictic and graphic gestures. In: Proceedings of the Workshop on Speech and Natural Language, HLT ’89, pp. 410–423. Association for Computational Linguistics, Stroudsburg, PA (1989). doi:10.3115/1075434.1075499Google Scholar
  34. 34.
    Nigay, L., Coutaz, J.: A generic platform for addressing the multimodal challenge. In: CHI ’95: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 98–105. ACM, New York (1995). doi:10.1145/223904.223917Google Scholar
  35. 35.
    Norman, D.A.: The Design of Everyday Things. iBasic Books, New York (2002)Google Scholar
  36. 36.
    Nothdurft, F., Honold, F., Zablotskaya, K., Diab, A., Minker, W.: Application of verbal intelligence in dialog systems for multimodal interaction. In: 2014 10th International Conference on Intelligent Environments (IE), pp. 361–364. IEEE, Shanghai (2014). doi:10.1109/IE.2014.59. (Short paper)Google Scholar
  37. 37.
    Oviatt, S.: Multimodal Interfaces, 2nd edn., chap. 21, pp. 413–432. CRC, Boca Raton (2007). doi:10.1201/9781410615862.ch21Google Scholar
  38. 38.
    Pfleger, N.: Context based multimodal fusion. In: ICMI ’04: Proceedings of the 6th International Conference on Multimodal Interfaces, pp. 265–272. ACM, New York (2004). doi:10.1145/1027933.1027977Google Scholar
  39. 39.
    Portillo, P.M., García, G.P., Carredano, G.A.: Multimodal fusion: a new hybrid strategy for dialogue systems. In: Proceedings of the 8th International Conference on Multimodal Interfaces, ICMI ’06, pp. 357–363. ACM, New York (2006). doi:10.1145/1180995.1181061Google Scholar
  40. 40.
    Reddy, B.S., Basir, O.A.: Concept-based evidential reasoning for multimodal fusion in human-computer interaction. Appl. Soft Comput. 10(2), 567–577 (2010). doi:10.1016/j.asoc.2009.08.026CrossRefGoogle Scholar
  41. 41.
    Roscher, D., Blumendorf, M., Albayrak, S.: A meta user interface to control multimodal interaction in smart environments. In: Proceedings of the 14th International Conference on Intelligent User Interfaces, IUI ’09, pp. 481–482. ACM, New York (2009). doi:10.1145/1502650.1502725Google Scholar
  42. 42.
    Rousseau, C., Bellik, Y., Vernier, F., Bazalgette, D.: A framework for the intelligent multimodal presentation of information. Signal Process. 86(12), 3696–3713 (2006). doi:10.1016/j.sigpro.2006.02.041CrossRefMATHGoogle Scholar
  43. 43.
    Schüssel, F., Honold, F., Weber, M.: Influencing factors on multimodal interaction during selection tasks. J. Multimodal User Interfaces 7(4), 299–310 (2013). doi:10.1007/s12193-012-0117-5CrossRefGoogle Scholar
  44. 44.
    Schüssel, F., Honold, F., Weber, M.: Using the transferable belief model for multimodal input fusion in companion systems. In: Schwenker, F., Scherer, S., Morency, L.P. (eds.) Multimodal Pattern Recognition of Social Signals in Human-Computer-Interaction. Lecture Notes in Computer Science, vol. 7742, pp. 100–115. Springer, Berlin, Heidelberg (2013). doi:10.1007/978-3-642-37081-6_12CrossRefGoogle Scholar
  45. 45.
    Smets, P.: The combination of evidence in the transferable belief model. IEEE Trans. Pattern Anal. Mach. Intell. 12(5), 447–458 (1990). doi:10.1109/34.55104CrossRefGoogle Scholar
  46. 46.
    Smets, P.: Data fusion in the transferable belief model. In: Proceedings of the Third International Conference on Information Fusion. FUSION 2000, vol. 1, pp. PS21–PS33. IEEE, New York (2000). doi:10.1109/IFIC.2000.862713Google Scholar
  47. 47.
    Vanderdonckt, J., Limbourg, Q., Michotte, B., Bouillon, L., Trevisan, D., Florins, M.: USIXML: a user interface description language for specifying multimodal user interfaces. In: Proceedings of W3C Workshop on Multimodal Interaction WMI’2004, pp. 1–7 (2004)Google Scholar
  48. 48.
    Wahlster, W.: User and discourse models for multimodal communication. In: Sullivan, J.W., Tyler, S.W. (eds.) Intelligent User Interfaces, pp. 45–67. ACM, New York (1991). doi:10.1145/107215.128691Google Scholar
  49. 49.
    Wahlster, W. (ed.): SmartKom: Foundations of Multimodal Dialogue Systems. Springer, Berlin (2006). doi:10.1007/3-540-36678-4Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Felix Schüssel
    • 1
  • Frank Honold
    • 1
  • Nikola Bubalo
    • 2
  • Michael Weber
    • 1
  • Anke Huckauf
    • 2
  1. 1.Institute of Media InformaticsUlm UniversityUlmGermany
  2. 2.General PsychologyUlm UniversityUlmGermany

Personalised recommendations