Towards Cognitive-Aware Multimodal Presentation: The Modality Effects in High-Load HCI
In this study, we argue that multimodal presentations should be created in a cognitive-aware manner, especially in a high-load HCI situation where the user task challenges the full capacity of the human cognition. An experiment was conducted to investigate the cognitive effects of modalities, using a high-load task. The performance measurements and subjective reports consistently confirm a significant modality impact on cognitive workload, stress and performance. A relation between modality usage and physiological states was not found, due to the insufficient sensitivity and individual differences of the physiological measurements. The findings of this experiment can be well explained by several modality-related cognitive theories. We further integrate these theories into a suitability prediction model, which can systematically predict how suitable a certain modality usage is for this presentation task. The model demonstrates a possible approach towards cognitive-aware modality planning and can be modified for other applications.
KeywordsCognitive-aware multimodal presentation modality planning cognitive load stress performance high-load HCI
Unable to display preview. Download preview PDF.
- 1.Andre, E.: The Generation of Multimedia Presentations. In: Dale, R., Somers, H.L., Moisl, H. (eds.) Handbook of Natural Language Processing. Marcel Dekker, Inc., USA (2000)Google Scholar
- 2.Bachvarova, Y., van Dijk, B., Nijholt, A.: Towards a Unified Knowledge-Based Approach to Modality Choice. In: Multimodal Output Generation (MOG), pp. 7–15 (2007) Google Scholar
- 3.Baddeley, A.D.: Essentials of Human Memory. Psychology Press, USA (1999)Google Scholar
- 6.Cao, Y., Theune, M., Nijholt, A.: Modality Effects on Cognitive Load and Performance in High-Load Information Presentation. In: Intelligent User Interface (IUI), pp. 335–344 (2009)Google Scholar
- 10.Kramer, A.F.: Physiological Metrics of Mental Workload: A Review of Recent Progress. In: Damos, D.L. (ed.) Multiple-Task Performance. CRC Press, USA (1991)Google Scholar
- 13.Paivio, A.: Mental Representations: A Dual Coding Approach. Oxford University Press, USA (1986)Google Scholar
- 15.Scerbo, M.W., Freeman, F.G., Mikulka, P.J., Parasuraman, R., Di Nocero, F.: The Efficacy of Psychophysiological Measures for Implementing Adaptive Technology. TP-2001-211018, NASA Langley Research Center, Hampton (2001)Google Scholar
- 16.Stern, R.M., Ray, W.J., Quigley, K.S.: Psychophysiological Recording, 2nd edn. Oxford University Press, UK (2001)Google Scholar