Towards Cognitive-Aware Multimodal Presentation: The Modality Effects in High-Load HCI

  • Yujia Cao
  • Mariët Theune
  • Anton Nijholt
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5639)


In this study, we argue that multimodal presentations should be created in a cognitive-aware manner, especially in a high-load HCI situation where the user task challenges the full capacity of the human cognition. An experiment was conducted to investigate the cognitive effects of modalities, using a high-load task. The performance measurements and subjective reports consistently confirm a significant modality impact on cognitive workload, stress and performance. A relation between modality usage and physiological states was not found, due to the insufficient sensitivity and individual differences of the physiological measurements. The findings of this experiment can be well explained by several modality-related cognitive theories. We further integrate these theories into a suitability prediction model, which can systematically predict how suitable a certain modality usage is for this presentation task. The model demonstrates a possible approach towards cognitive-aware modality planning and can be modified for other applications.


Cognitive-aware multimodal presentation modality planning cognitive load stress performance high-load HCI 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Andre, E.: The Generation of Multimedia Presentations. In: Dale, R., Somers, H.L., Moisl, H. (eds.) Handbook of Natural Language Processing. Marcel Dekker, Inc., USA (2000)Google Scholar
  2. 2.
    Bachvarova, Y., van Dijk, B., Nijholt, A.: Towards a Unified Knowledge-Based Approach to Modality Choice. In: Multimodal Output Generation (MOG), pp. 7–15 (2007) Google Scholar
  3. 3.
    Baddeley, A.D.: Essentials of Human Memory. Psychology Press, USA (1999)Google Scholar
  4. 4.
    Baddeley, A.D., Hitch, G.J.: Working Memory. The Psychology of Learning and Motivation: Advances in Research and Theory 8, 47–89 (1974)CrossRefGoogle Scholar
  5. 5.
    Boucsein, W., Haarmann, A., Schaefer, F.: Combining Skin Conductance and Heart Rate Variability for Adaptive Automation During Simulated IFR Flight. In: Harris, D. (ed.) HCII 2007 and EPCE 2007. LNCS, vol. 4562, pp. 639–647. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  6. 6.
    Cao, Y., Theune, M., Nijholt, A.: Modality Effects on Cognitive Load and Performance in High-Load Information Presentation. In: Intelligent User Interface (IUI), pp. 335–344 (2009)Google Scholar
  7. 7.
    Carr, T.H., McCauley, C., Sperber, R.D., Parmelee, C.M.: Words, Pictures, and Priming: On Semantic Activation, Conscious Identification, and the Automaticity of Information Processing. J. Exp. Psychol. Hum. Percept. Perform. 8, 757–777 (1982)CrossRefGoogle Scholar
  8. 8.
    Clark, J.M., Paivio, A.: Dual Coding Theory and Education. Educational Psychology Review 3, 149–210 (1991)CrossRefGoogle Scholar
  9. 9.
    Hart, S.G., Staveland, L.E.: Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. Human Mental Workload 1, 139–183 (1988)CrossRefGoogle Scholar
  10. 10.
    Kramer, A.F.: Physiological Metrics of Mental Workload: A Review of Recent Progress. In: Damos, D.L. (ed.) Multiple-Task Performance. CRC Press, USA (1991)Google Scholar
  11. 11.
    Malik, M.: Heart Rate Variability - Standards of Measurement, Physiological Interpretation, and Clinical Use. Circulation 93, 1043–1065 (1996)CrossRefGoogle Scholar
  12. 12.
    Mayer, R.E., Moreno, R.: Nine Ways to Reduce Cognitive Load in Multimedia Learning. Educational Psychologist 38, 45–52 (2003)CrossRefGoogle Scholar
  13. 13.
    Paivio, A.: Mental Representations: A Dual Coding Approach. Oxford University Press, USA (1986)Google Scholar
  14. 14.
    Potter, M.C., Faulconer, B.A.: Time to Understand Pictures and Words. Nature 253, 437–438 (1975)CrossRefGoogle Scholar
  15. 15.
    Scerbo, M.W., Freeman, F.G., Mikulka, P.J., Parasuraman, R., Di Nocero, F.: The Efficacy of Psychophysiological Measures for Implementing Adaptive Technology. TP-2001-211018, NASA Langley Research Center, Hampton (2001)Google Scholar
  16. 16.
    Stern, R.M., Ray, W.J., Quigley, K.S.: Psychophysiological Recording, 2nd edn. Oxford University Press, UK (2001)Google Scholar
  17. 17.
    Verwey, W.B., Veltman, H.A.: Detecting Short Periods of Elevated Workload: A Comparison of Nine Workload Assessment Techniques. Applied Experimental Psychology 2, 270–285 (1996)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  • Yujia Cao
    • 1
  • Mariët Theune
    • 1
  • Anton Nijholt
    • 1
  1. 1.Human Media Interaction GroupUniversity of TwenteEnschedeThe Netherlands

Personalised recommendations