Original Paper

Journal on Multimodal User Interfaces

, Volume 7, Issue 4, pp 299-310

First online:

Influencing factors on multimodal interaction during selection tasks

  • Felix SchüsselAffiliated withInstitute of Media Informatics, Ulm University Email author 
  • , Frank HonoldAffiliated withInstitute of Media Informatics, Ulm University
  • , Michael WeberAffiliated withInstitute of Media Informatics, Ulm University

Rent the article at a discount

Rent now

* Final gross prices may vary according to local VAT.

Get Access


When developing multimodal interactive systems it is not clear which importance should be given to which modality. In order to study influencing factors on multimodal interaction, we conducted a Wizard of Oz study on a basic recurrent task: 53 subjects performed diverse selections of objects on a screen. The way and modality of interaction was not specified nor predefined by the system, and the users were free in how and what to select. Natural input modalities like speech, gestures, touch, and arbitrary multimodal combinations of these were recorded as dependent variables. As independent variables, subjects’ gender, personality traits, and affinity towards technical devices were surveyed, as well as the system’s varying presentation styles of the selection. Our statistical analyses reveal gender as a momentous influencing factor and point out the role of individuality for the way of interaction, while the influence of the system output seems to be quite limited. This knowledge about the prevalent task of selection will be useful for designing effective and efficient multimodal interactive systems across a wide range of applications and domains.


Natural human computer interaction Multimodal interaction User study Gender differences Selection task Input Output