Journal on Multimodal User Interfaces

, Volume 7, Issue 4, pp 299–310 | Cite as

Influencing factors on multimodal interaction during selection tasks

Original Paper


When developing multimodal interactive systems it is not clear which importance should be given to which modality. In order to study influencing factors on multimodal interaction, we conducted a Wizard of Oz study on a basic recurrent task: 53 subjects performed diverse selections of objects on a screen. The way and modality of interaction was not specified nor predefined by the system, and the users were free in how and what to select. Natural input modalities like speech, gestures, touch, and arbitrary multimodal combinations of these were recorded as dependent variables. As independent variables, subjects’ gender, personality traits, and affinity towards technical devices were surveyed, as well as the system’s varying presentation styles of the selection. Our statistical analyses reveal gender as a momentous influencing factor and point out the role of individuality for the way of interaction, while the influence of the system output seems to be quite limited. This knowledge about the prevalent task of selection will be useful for designing effective and efficient multimodal interactive systems across a wide range of applications and domains.


Natural human computer interaction  Multimodal interaction User study Gender differences Selection task Input Output 


  1. 1.
    Bellik Y, Rebaï I, Machrouh E, Barzaj Y, Jacquet C, Pruvost G, Sansonnet JP (2009) Multimodal interaction within ambient environments: an exploratory study. In: Proceedings of the 12th IFIP TC 13 international conference on human-computer interaction: part II, INTERACT ’09, Springer, Berlin, pp 89–92. doi: 10.1007/978-3-642-03658-3-13
  2. 2.
    Chelazzi L (1999) Serial attention mechanisms in visual search: a critical look at the evidence. Psychol Res 62:195–219Google Scholar
  3. 3.
    Costa PT, McCrae RR (1992) Professional manual: revised NEO personality inventory (NEO-PI-R) and NEO five-factor inventory (NEO-FFI). Psychological Assessment ResourcesGoogle Scholar
  4. 4.
    De Angeli A, Gerbino W, Cassano G, Petrelli D (1998) Visual display, pointing, and natural language: the power of multimodal interaction. In: Proceedings of the working conference on advanced visual interfaces, AVI ’98, ACM, New York, NY, USA, pp 164–173. doi: 10.1145/948496.948519
  5. 5.
    Frank AU (1998) Formal models for cognition—taxonomy of spatial location description and frames of reference. In: Freksa C, Habel C, Wender KF (eds) Spatial cognition. Springer, Berlin, pp 293–312Google Scholar
  6. 6.
    Gerhard U, Borkenau P, Ostendorf F (1993) NEO-Fünf-Faktoren inventar (NEO-FFI) nach Costa und McCrae. Zeitschrift für Klinische Psychologie und Psychotherapie 28(2):145–146. doi: 10.1026/0084-5345.28.2.145
  7. 7.
    Huang X, Oviatt SL, Lunsford R (2006) Combining user modeling and machine learning to predict users’ multimodal integration patterns. In: Renals S, Bengio S, Fiscus JG (eds) Machine learning for multimodal interaction, 3rd international workshop, MLMI 2006, Bethesda, MD, USA, 1–4 May 2006, revised selected papers, lecture notes in computer science, vol 4299/2006, Springer, Berlin, pp 50–62. doi:10.1007/11965152-5
  8. 8.
    Jöst M, Häußler J, Merdes M, Malaka R (2005) Multimodal interaction for pedestrians: an evaluation study. In: Proceedings of the 10th international conference on intelligent user interfaces, IUI ’05, ACM, New York, NY, USA, pp 59–66. doi:10.1145/1040830.1040852
  9. 9.
    Karrer K, Glaser C, Clemens C, Bruder C (2009) Technikaffinität erfassen-der Fragebogen TA-EG. Der Mensch im Mittelpunkt technischer Systeme 8. Berliner Werkstatt Mensch-Maschine-Systeme 7. bis 9. Oktober 2009 (ZMMS Spektrum) 22(29):196–201Google Scholar
  10. 10.
    Kieffer S, Carbonell N (2007) How really effective are multimodal hints in enhancing visual target spotting? some evidence from a usability study. CoRR abs/0708.3575.
  11. 11.
    Kieffer S, Carbonell N (2007) Oral messages improve visual search. CoRR abs/0709.0428Google Scholar
  12. 12.
    Kipp M (2001) Anvil—a generic annotation tool for multimodal dialogue. In: Proceedings of EUROSPEECH 2001, Aalborg, Denmark, pp 1367–1370Google Scholar
  13. 13.
    Körner A, Drapeau M, Albani C, Geyer M, Schmutzer G, Brähler E (2008) Deutsche Normierung des NEO-Fünf-Faktoren-Inventars (NEO-FFI). Zeitschrift für Medizinische Psychologie 17(2–3):133–144 Google Scholar
  14. 14.
    Mairesse F, Walker MA, Mehl MR, Moore RK (2007) Using linguistic cues for the automatic recognition of personality in conversation and text. J Artif Intell Res 30:457–501Google Scholar
  15. 15.
    Mignot C, Valot C, Carbonell N (1993) An experimental study of future “natural” multimodal human-computer interaction. In: INTERACT ’93 and CHI ’93 conference companion on Human factors in computing systems, CHI ’93, ACM, New York, NY, USA, pp 67–68. doi: 10.1145/259964.260075
  16. 16.
    Oviatt S (1999) Ten myths of multimodal interaction. Commun ACM 42(11):74–81CrossRefGoogle Scholar
  17. 17.
    Oviatt S, Coulston R, Lunsford R (2004) When do we interact multimodally?: cognitive load and multimodal communication patterns. In: ICMI ’04, proceedings of the 6th international conference on multimodal interfaces, ACM, New York, NY, USA, pp 129–136. doi: 10.1145/1027933.1027957
  18. 18.
    Ratzka A (2008) Explorative studies on multimodal interaction in a pda- and desktop-based scenario. In: Proceedings of the 10th international conference on multimodal interfaces, ICMI ’08, ACM, New York, NY, USA, pp 121–128.
  19. 19.
    Reis T, de SM, Carrio L (2008) Multimodal interaction: real context studies on mobile digital artefacts. In: Pirhonen A, Brewster S (eds) Haptic and audio interaction design, lecture notes in computer science, vol 5270, Springer, Berlin, pp 60–69. doi: 10.1007/978-3-540-87883-4-7
  20. 20.
    Ren X, Zhang G, Dai G (2000) An experimental study of input modes for multimodal human-computer interaction. In: Tan T, Shi Y, Gao W (eds) Advances in multimodal interfaces ICMI 2000, lecture notes in computer science, vol 1948, Springer, Berlin, pp 49–56. doi:10.1007/3-540-40063-X-7
  21. 21.
    Wasinger R, Krüger A (2006) Modality preferences in mobile and instrumented environments. In: Proceedings of the 11th international conference on Intelligent user interfaces, IUI ’06, ACM, New York, NY, USA, pp 336–338. doi: 10.1145/1111449.1111529

Copyright information

© OpenInterface Association 2012

Authors and Affiliations

  1. 1.Institute of Media InformaticsUlm UniversityUlmGermany

Personalised recommendations