Generating context-sensitive ECA responses to user barge-in interruptions

  • Nigel Crook
  • Debora Field
  • Cameron Smith
  • Sue Harding
  • Stephen Pulman
  • Marc Cavazza
  • Daniel Charlton
  • Roger Moore
  • Johan Boye
Original Paper


We present an Embodied Conversational Agent (ECA) that incorporates a context-sensitive mechanism for handling user barge-in. The affective ECA engages the user in social conversation, and is fully implemented. We will use actual examples of system behaviour to illustrate. The ECA is designed to recognise and be empathetic to the emotional state of the user. It is able to detect, react quickly to, and then follow up with considered responses to different kinds of user interruptions. The design of the rules which enable the ECA to respond intelligently to different types of interruptions was informed by manually analysed real data from human–human dialogue. The rules represent recoveries from interruptions as two-part structures: an address followed by a resumption. The system is robust enough to manage long, multi-utterance turns by both user and system, which creates good opportunities for the user to interrupt while the ECA is speaking.


ECA Spoken dialogue system User barge-in Interruption Recovery 



This work was partially funded by the COMPANIONS project ( sponsored by the European Commission (EC) as part of the Information Society Technologies (IST) programme under EC grant number IST-FP6-034434. We thank the University of Augsburg (Prof. Elisabeth André) for supplying a version of the EmoVoice [25] system. Other contributors to the prototype described in this paper are Ramon Granell, Simon Dobnik, Karo Moilanen and Manjari Chandran-Ramesh (University of Oxford), Raúl Santos de la Cámara (Telefonica ID, Madrid), Markku Turunen (University of Tampere) and Enrico Zovato (Loquendo, Torino)


  1. 1.
    Cavazza M, Santos de la Cámara R, Turunen M (The COMPANIONS Consortium) (2010) How was your day? A companion ECA. In: Proceedings of the 9th international conference on autonomous agents and multiagent systems (AAMAS2010), Toronto, Canada, May 10–14, 2010, pp 1629–1630 Google Scholar
  2. 2.
    Young S (2010) Still talking to machines (cognitively speaking), 2010. In: Proc Interspeech, Chiba, Japan, 26–30 September, 2010 Google Scholar
  3. 3.
    Lemon O, Georgila K, Henderson J, Stuttle M (2006) An ISU dialogue system exhibiting reinforcement learning of dialogue policies: generic slot-filling in the TALK in-car system. In: Proceedings of the eleventh conference of the European chapter of the association for computational linguistics: posters & demonstrations, EACL ’06, Morristown, NJ, USA. Association for Computational Linguistics, Stroudsburg, pp 119–122 CrossRefGoogle Scholar
  4. 4.
    Allen J, Chambers N, Ferguson G, Galescu L, Jung H, Swift M, Taysom W (2007) Plow: a collaborative task learning agent. In: Proceedings of the 22nd national conference on artificial intelligence, vol 2. AAAI Press, Menlo Park, pp 1514–1519 Google Scholar
  5. 5.
    West C, Zimmerman D (1983) Small insults: A study of interruptions in cross-sex conversations between unacquainted persons. In: Thorne B, Kramarae C, Henley N (eds) Language, gender and society. Newbury House, Cambridge, pp 102–117 Google Scholar
  6. 6.
    Lakoff RT (1995) Cries and whispers: the shattering of the silence. In: Hall K, Bucholtz M (eds) Gender articulated: language and the socially constructed self. Routledge, New York, pp 25–50 Google Scholar
  7. 7.
    Sacks H, Schegloff EA, Jefferson G (1974) A simplest systematics for the organization of turn-taking for conversation. Language 50(4):696–735 CrossRefGoogle Scholar
  8. 8.
    Coates J (1993) Women, men, and language: a sociolinguistic account of gender differences in language, 2nd edn. Longman, London/New York Google Scholar
  9. 9.
    Bevacqua E, Pammi S, Hyniewska SJ, Schröder M, Pelachaud C (2010) Multimodal backchannels for embodied conversational agents. In: Allbeck JM, Badler NI, Bickmore TW, Pelachaud C, Safonova A (eds) IVA. Lecture notes in computer science, vol 6356. Springer, Berlin, pp 194–200 Google Scholar
  10. 10.
    Morency L-P, de Kok I, Gratch J (2008) Predicting listener backchannels: a probabilistic multimodal approach. In: Prendinger H, Lester JC, Ishizuka M (eds) IVA. Lecture notes in computer science, vol 5208. Springer, Berlin, pp 176–190 Google Scholar
  11. 11.
    Zimmerman D, West C (1975) Sex roles, interruptions and silences in conversation. In: Thorne B, Henly N (eds) Language and sex: difference and dominance. Newbury House, Cambridge, pp 10–129 Google Scholar
  12. 12.
    Murray SO (1985) Toward a model of members’ methods for recognizing interruptions. Lang Soc 14(1):31–40 CrossRefGoogle Scholar
  13. 13.
    Raux A, Eskenazi M (2007) A multi-layer architecture for semi-synchronous event-driven dialogue management. In: ASRU, Kyoto, Japan, pp 514–519 Google Scholar
  14. 14.
    Barnett J, Singh M (1996) Designing a portable spoken dialogue system. In: Maier E, Mast M, LuperFoy S (eds) ECAI workshop on dialogue processing in spoken language systems. Lecture notes in computer science, vol 1236. Springer, Berlin, pp 156–170 CrossRefGoogle Scholar
  15. 15.
    Rose RC, Kim HK (2003) A hybrid barge-in procedure for more reliable turn-taking in human-machine dialog systems. In: Proceedings of the automatic speech recognition and understanding workshop Google Scholar
  16. 16.
    Balentine B, Morgan DP (1999) How to build speech recognition applications—a style guide for telephony dialogs. Enterprise Integration Group, San Ramon Google Scholar
  17. 17.
    Setlur AR, Sukkar RA (1998) Recognition-based word counting for reliable barge-in and early endpoint detection in continuous speech recognition. In: Proceeding of the international conference on spoken language processing, pp 2135–2138 Google Scholar
  18. 18.
    Matsuyama K, Komatani K, Ogata T, Okuno HG (2009) Enabling a user to specify an item at any time during system enumeration—item identification for barge-in-able conversational dialogue systems. In: Proceedings of the 10th annual conference of the international speech communication association (INTERSPEECH 2009), Brighton UK, 6–10 September 2009, pp 252–255 Google Scholar
  19. 19.
    Komatani K, Rudnicky AI (2009) Predicting barge-in utterance errors by using implicitly-supervised ASR accuracy and barge-in rate per user. In: Proceedings of the ACL-IJCNLP conference short papers, Suntec, Singapore, August 2009. Association for Computational Linguistics, Stroudsburg, pp 89–92 CrossRefGoogle Scholar
  20. 20.
    Brooks RA (1985) A robust layered control system for a mobile robot. Technical report, Massachusetts Institute of Technology, Cambridge, MA, USA Google Scholar
  21. 21.
    Brooks RA (1995) Intelligence without representation. In: Computation & intelligence: collected readings. American Association for Artificial Intelligence, Menlo Park, pp 343–362 Google Scholar
  22. 22.
    Moore RK (2007) Presence: A human-inspired architecture for speech-based human-machine interaction. IEEE Trans Comput 56(9):1176–1188 MathSciNetCrossRefGoogle Scholar
  23. 23.
    Reidsma D, de Kok T, Neiberg D, Pammi S, van Straalen B, Truong K, van Welbergen H (2011) Continuous interaction with a virtual human. J Multimodal User Interfaces 4:97–118 CrossRefGoogle Scholar
  24. 24.
    Santos de la Cámara R, Turunen M, Hakulinen J, Field D (2010) How was your day? an architecture for multimodal ECA systems, 2010. In: Proc 11th annual meeting of the special interest group on discourse and dialogue (SIGDIAL), 24–25 September, 2010. University of Tokyo, Tokyo, pp 47–50 Google Scholar
  25. 25.
    Vogt T, André E, Bee N (2008) Emovoice—a framework for online recognition of emotions from voice. In: Proceedings of the 4th IEEE tutorial and research workshop on perception and interactive technologies for speech-based systems: perception in multimodal dialogue systems, PIT ’08. Springer, Berlin, pp 188–199 Google Scholar
  26. 26.
    Moilanen K, Pulman S (2007) Sentiment composition. In: Proceedings of the recent advances in natural language processing international conference (RANLP-2007), Borovets, Bulgaria, 27–29 September 2007, pp 378–382 Google Scholar
  27. 27.
    Bremond C (1973) Logique du Récit. Editions du Seuil, Paris Google Scholar
  28. 28.
    Cavazza M, Smith C, Charlton D, Crook N, Boye J, Pulman S, Moilanen K, Pizzi D, Santos de la Cámara R, Turunen M (2010) Persuasive dialogue based on a narrative theory: an ECA implementation. In: Proceedings of the fifth international conference on persuasive technology (Persuasive 2010), Copenhagen, Denmark, 7–10 June 2010 Google Scholar
  29. 29.
    Smith C, Crook N, Boye J, Charlton D, Dobnik S, Pizzi D, Cavazza M, Pulman S, Santos de la Cámara R, Turunen M (2010) Interaction strategies for an affective conversational agent. In: Proc of the 10th int. conf. on intelligent virtual agents (IVA 2010), Philadelphia, PA, September 2010 Google Scholar
  30. 30.
    Hernández A, López B, Pardo D, Santos R, Hernández L, Relaño Gil J, Rodríguez M (2008) Modular definition of multimodal ECA communication acts to improve dialogue robustness and depth of intention. In: Proc 1st functional markup language workshop, 7th international joint conference on autonomous agents and multiagent systems (AAMAS 2008), Estoril, Portugal, 12–16 May 2008 Google Scholar
  31. 31.
    López B, Hernández A, Pardo D, Santos R, Rodríguez M (2008) ECA gesture strategies for robust SLDS. In: Proc artificial intelligence and simulation behaviour convention (AISB 2008) symposium on multimodal output generation, Aberdeen, UK, 1–4 April, 2008 Google Scholar
  32. 32.
    Danieli M, Zovato E (2010) The affective dimension of speech acts and voice expressiveness. In: Pettorino M, Giannini A, Chiari I, Dovetto Fr (eds) Spoken communication. Cambridge Scholars Publishing, Newcastle upon Tyne, pp 191–204 Google Scholar
  33. 33.
    Stoness S, Tetreault J, Allen J (2004) Incremental parsing with reference interaction. In: ACL workshop on incremental parsing, pp 18–25 Google Scholar
  34. 34.
    Aist G, Allen J, Campana E, Gallo C, Stoness S, Swift M, Tanenhaus M (2007) Incremental understanding in human-computer dialogue and experimental evidence for advantages over nonincremental methods. In: Proceedings of the 11th workshop on the semantics and pragmatics of dialogue, Trento, Italy, 30 May–1 June 2007, pp 149–154 Google Scholar
  35. 35.
    Brick T, Scheutz M (2007) Incremental natural language processing for HRI. In: Proceedings of the ACM/IEEE international conference on Human-robot interaction, Arlington, Virginia, USA, pp 263–270 CrossRefGoogle Scholar
  36. 36.
    Skantze G, Schlangen D (2009) Incremental dialogue processing in a micro-domain. In: Proceedings of the 12th conference of the European chapter of the ACL (EACL 2009), Athens, Greece, April 2009, pp 745–753, Google Scholar
  37. 37.
    Schlangen D, Skantze G (2009) A general, abstract model of incremental dialogue processing. In: Proc of the 12th conference of the European chapter of the ACL (EACL 2009), Athens, Greece, April 2009, pp 710–718 Google Scholar
  38. 38.
    Starkey D (1972) Some signals and rules for taking speaking turns in conversations. J Pers Soc Psychol 23:283–292 CrossRefGoogle Scholar
  39. 39.
    Wiemann JM, Knapp ML (1975) Turn-taking in conversations. J Commun 25:75–92 Google Scholar
  40. 40.
    Schegloff EA (2000) Overlapping talk and the organization of turn-taking for conversation. Lang Soc 29(1):1–63 CrossRefGoogle Scholar
  41. 41.
    Kennedy CW, Camden CT (1983) A new look at interruptions. West J Commun 47:45–58 Google Scholar
  42. 42.
    Roger D (1989) 4: Experimental studies of dyadic turn-taking behaviour. In: Roger D, Bull P (eds) Conversation: an interdisciplinary perspective. Multilingual Matters, Clevedon Google Scholar
  43. 43.
    Hutchby I (1992) Confrontation talk: Aspects of interruption in argument sequences on talk radio. Interdiscip J Study Discourse 12:343–372 CrossRefGoogle Scholar
  44. 44.
    Walker M, Whittaker S (1990) Mixed initiative in dialogue: An investigation into discourse segmentation. In: Proc. 28th annual meeting of the ACL, pp 70–79 Google Scholar
  45. 45.
    Heins R, Franzke M, Durian M, Bayya A (1997) Turn-taking as a design principle for barge-in in spoken language systems. Int J Speech Technol 2:155–164. doi: 10.1007/BF02208827 CrossRefGoogle Scholar
  46. 46.
    Oth RKEN, Kieling A, Kuhn T, Mast M, Niemann H, Ott K, Batliner A (1994) Prosody takes over: towards a prosodically guided dialog system. Speech Commun 15(15):155–167 Google Scholar
  47. 47.
    Austin JL (1962) How to do things with words, 2nd edn. Oxford University Press, New York Google Scholar
  48. 48.
    Bunt HC (2000) Dynamic interpretation and dialogue theory. In: Taylor MM, Neel F, Bouwhuis DG (eds) The structure of multimodal dialogue, vol 2. North-Holland, Amsterdam, pp 139–166 Google Scholar
  49. 49.
    Traum DR (2000) 20 questions on dialogue act taxonomies. J Semant 17:7–30 CrossRefGoogle Scholar
  50. 50.
    Jurafsky D, Shriberg E, Biasca D (1997) Switchboard SWBD-DAMSL shallow-discourse-function annotation coders manual, draft 13. University of Colorado, Boulder. Institute of Cognitive Science Technical Report 97-02 Google Scholar
  51. 51.
    Thomason RH (1990) Accommodation, meaning, and implicature: interdisciplinary foundations for pragmatics. In: Intentions and communication, pp 325–363 Google Scholar
  52. 52.
    Lewis D (1979) Scorekeeping in a language game. J Philos Log 8:339–359. Reprinted in Lewis, D (1983) Philosophical papers, vol. I. Oxford University Press, New York/Oxford, pp 233–249 CrossRefGoogle Scholar
  53. 53.
    Stalnaker R (1972) Pragmatics. In: Davidson D, Harman G (eds) Semantics of natural language. Synthese library, vol 40. Reidel, Dordrecht, pp 380–397 CrossRefGoogle Scholar

Copyright information

© OpenInterface Association 2012

Authors and Affiliations

  • Nigel Crook
    • 1
  • Debora Field
    • 2
  • Cameron Smith
    • 3
  • Sue Harding
    • 2
  • Stephen Pulman
    • 1
  • Marc Cavazza
    • 3
  • Daniel Charlton
    • 3
  • Roger Moore
    • 2
  • Johan Boye
    • 4
  1. 1.Department of Computing and Communication TechnologiesOxford Brookes UniversityOxfordUK
  2. 2.Department of Computer ScienceUniversity of SheffieldSheffieldUK
  3. 3.School of ComputingTeesside UniversityMiddlesbroughUK
  4. 4.School of Computer ScienceKTH Royal Institute of TechnologyStockholmSweden

Personalised recommendations