A Flexible Dual Task Paradigm for Evaluating an Embodied Conversational Agent: Modality Effects and Reaction Time as an Index of Cognitive Load

  • Catherine J. Stevens
  • Guillaume Gibert
  • Yvonne Leung
  • Zhengzhi Zhang
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6895)


A new experimental method based on the dual task paradigm is used to evaluate speech intelligibility of an embodied conversational agent (ECA). The experiment consists of the manipulation of auditory-visual (AV) versus auditory-only (A) presentation of speech. In the dual task, participants perform two tasks concurrently. The secondary task is sensitive to cognitive processing demands of the primary task. In the primary task participants either shadowed words or named the superordinate categories to which words belonged, as the word items were spoken by the ECA under A or AV conditions. Reaction time (RT) on the secondary task–swatting a fly on the ECA face–was affected by the difficulty of the concurrent task. The secondary RT was affected by modality of presentation of the primary task. Using a relatively primitive ECA, RT on the secondary task was significantly slower when shadowing occurred in AV versus A conditions. The benefits of this evaluation system, that returns quantitative behavioural data and self-report ratings, are discussed.


Evaluation Embodied Conversational Agent Dual Task Divided Attention Reaction Time Shadowing 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Ibister, K., Doyle, P.: Design and evaluation of embodied conversational agents: a proposed taxonomy. In: Proc 1st Intl Joint Conf Autonomous Agents and Multi-Agent System, AAMAS 2002, Bologna, Italy (2002)Google Scholar
  2. 2.
    Catrambone, R., Stasko, J., Xiao, J.: ECA as user interface paradigm: experimental findings within a framework for research. In: Ruttkay, Z., Pelachaud, C. (eds.) From Brows to Trust: Evaluating Embodied Conversational Agents, pp. 239–267. Springer, The Netherlands (2005)Google Scholar
  3. 3.
    Buisine, S., Abrillian, S., Martin, J.-C.: Evaluation of multimodal behaviour of embodied agents. In: Ruttkay, Z., Pelachaud, C. (eds.) From Brows to Trust: Evaluating Embodied Conversational Agents, pp. 217–238. Springer, The Netherlands (2005)Google Scholar
  4. 4.
    Koller, A., Striegnitz, K., Byron, D., Cassell, J., Dale, R., Dalzel-Job, S., Oberlander, J., Moore, J.: Validating the web-based evaluation of NLG systems. In: Proc ACL-IJCNLP, Singapore (2009)Google Scholar
  5. 5.
    Ruttkay, Z., Pelachaud, C. (eds.): From brows to trust: evaluating embodied conversational agents. Kluwer Academic Publishers, Dordrecht (2005)Google Scholar
  6. 6.
    Sharp, H., Rogers, Y., Preece, J.: Interaction design: Beyond human-computer interaction, 2nd edn. John Wiley & Sons, Ltd., Chichester (2007)Google Scholar
  7. 7.
    Stevens, C., Lees, N., Vonwiller, J., Burnham, D.: On-line experimental methods to evaluate text-to-speech (TTS) synthesis: Effects of voice gender and signal quality on intelligibility, naturalness and preference. Computer Speech & Language 19, 129–146 (2005)CrossRefGoogle Scholar
  8. 8.
    Kopp, S., Gesellensetter, L., Krämer, N.C., Wachsmuth, I.: A conversational agent as museum guide – design and evaluation of a real-world application. In: Panayiotopoulos, T., Gratch, J., Aylett, R.S., Ballin, D., Olivier, P., Rist, T. (eds.) IVA 2005. LNCS (LNAI), vol. 3661, pp. 329–343. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  9. 9.
    Bailly, G., Raidt, S., Elisei, F.: Gaze, conversational agents and face-to-face communication. Speech Comm. 52, 598–612 (2010)CrossRefGoogle Scholar
  10. 10.
    Badin, P., Tarabalka, Y., Elisei, F., Bailly, G.: Can you ‘read’ tongue movements? Evaluation of the contribution of tongue display to speech understanding. Speech Comm. 52, 493–503 (2010)CrossRefGoogle Scholar
  11. 11.
    Karatekin, C., Couperus, J.W., Marcus, D.J.: Attention allocation in the dual-task paradigm as measured through behavioral and psychophysiological responses. Psychophysiol. 41, 175–185 (2004)CrossRefGoogle Scholar
  12. 12.
    Pashler, H., Johnston, J.C.: Attentional limitations in dual-task performance. In: Pashler, H. (ed.) Attention, pp. 155–189. Psychology Press, East Sussex (1998)Google Scholar
  13. 13.
    Johnston, W.A., Heinz, S.P.: Flexibility and capacity demands of attention. J. Experimental Psychol.: General 107, 420–435 (1978)CrossRefGoogle Scholar
  14. 14.
    Wickens, C.D.: Multiple resources and performance prediction. Theoretical Issues in Ergonomic Science 3, 159–177 (2002)CrossRefGoogle Scholar
  15. 15.
    Fisk, A.D., Derrick, W.L., Schneider, W.: A methodological assessment and evaluation of dual-task paradigms. Current Psychological Research & Reviews 5, 315–327 (1986)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Catherine J. Stevens
    • 1
  • Guillaume Gibert
    • 1
  • Yvonne Leung
    • 1
  • Zhengzhi Zhang
    • 1
  1. 1.MARCS Auditory LaboratoriesUniversity of Western SydneyPenrithAustralia

Personalised recommendations