Systems Engineering and Conversational Agents

  • James O’Shea
  • Zuhair Bandar
  • Keeley Crockett
Part of the Intelligent Systems Reference Library book series (ISRL, volume 10)


This chapter describes Conversational Agents (CAs) in the context of Systems Engineering. A CA is a computer program which interacts with a user through natural language dialogue and provides some form of service. CA technology has two points of interest to systems engineers: the use of systems engineering techniques in CA research and the application of CAs in project development. CAs offer the opportunity to automate more complex applications than are feasible with conventional web interfaces. Currently such applications require a human expert in the domain to mediate between the user and the application. The CA effectively replaces the human expert. This chapter reviews the current capabilities of various CA technologies, outlines a development methodology for systems engineering practitioners interested in developing real world applications and suggests a number of directions for systems engineers who wish to participate in CA research.


Conversational agent systems engineering dialogue evaluation methodology semantic similarity short text 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Turing, A.M.: Computing Machinery and Intelligence. Mind, New Series 59(236), 433–460 (1950)CrossRefMathSciNetGoogle Scholar
  2. 2.
    Gunderson, K.: The Imitation Game. Mind, New Series 73(290), 234–245 (1964)CrossRefGoogle Scholar
  3. 3.
    Searle, J.R.: Minds, brains and programs. Behavioural and Brain Sciences 3, 417–424 (1980)CrossRefGoogle Scholar
  4. 4.
    Block, N.: Psychologism and behaviourism. The Philosophical Review LXXXX(1), 5–43 (1981)CrossRefGoogle Scholar
  5. 5.
    Searle, J.R.: Mind, Language and Society. Weidenfield & Nicholson (1999)Google Scholar
  6. 6.
    Bickmore, T., Giorgino, T.: Health dialog systems for patients and consumers. J. Biomed. Inform. 39(5), 556–571 (2006)CrossRefGoogle Scholar
  7. 7.
    Cassell, J., et al.: Embodied CAs (2000)Google Scholar
  8. 8.
    Gorin, A.L., Riccardi, G., Wright, J.H.: How may I help you? Speech Communication 23, 113–127 (1997)CrossRefGoogle Scholar
  9. 9.
    Graesser, A.C., et al.: AutoTutor: An Intelligent Tutoring System With Mixed Initiative Dialogue. IEEE Transactions on Education 48(4), 612–618 (2005)CrossRefGoogle Scholar
  10. 10.
    Owda, M., Bandar, Z., Crockett, K.: Conversation-Based Natural Language Interface to Relational Databases. In: IEEE/WIC/ACM International Conferences on Web Intelligence and Intelligent Agent Technology - Workshops (2007)Google Scholar
  11. 11.
    Glass, J., et al.: A Framework for Developing Conversational User Interfaces. In: Fourth International Conference on Computer-Aided Design of User Interfaces, Funchal, Isle of Madeira, Portugal, 347–358 (2004)Google Scholar
  12. 12.
    McGeary, Z., et al.: Online Self-service: The Slow Road to Search Effectiveness. In: Customer Relationship Management (2005)Google Scholar
  13. 13.
    Hunt, M.J.: Figures of Merit for Assessing Connected Word Recognisers. Speech Communication 9, 239–336 (1990)CrossRefGoogle Scholar
  14. 14.
    Hosom, J.-P.: Automatic Speech Recognition at CSLU (2003), (cited October 20, 2010)
  15. 15.
    Hunt, A.: comp.speech FAQ Section 6 (1997), (cited October 20, 2010)
  16. 16.
    Raut, C.K.: Discriminative Adaptive Training and Bayesian Inference for Speech Recognition. In: Emmanuel College, University of Cambridge (2009)Google Scholar
  17. 17.
    Hillard, D.L.: Automatic Sentence Structure Annotation for Spoken Language Processing. In: Electrical Engineering, University of Washington (2008)Google Scholar
  18. 18.
    Zdenek, S.: Passing Loebner’s Turing test: A case of Conflicting Discourse Functions. Minds and Machines 11, 53–76 (2001)CrossRefMATHGoogle Scholar
  19. 19.
    Jackson, P., Moulinier, I.: Natural Language Processing for Online Applications, 2nd edn. Natural Language Processing. John Benjamins Publishing Company, Amsterdam (2007)Google Scholar
  20. 20.
    Zdravkova, K.: Conceptual Framework for an Intelligent chatterbot. In: 22nd International Conference Information Technology Interfaces ITI 2000, 189–194 (2000)Google Scholar
  21. 21.
    Minker, W., Bennacef, S., Gauvain, J.-L.: A stochastic case frame approach for natural language understanding. In: Fourth International Conference on Spoken Language, ICSLP 1996, Philadelphia, PA, 1013–1016 (1996)Google Scholar
  22. 22.
    Farquhar, A., Fikes, R., Rice, J.: The Ontolingua Server: a Tool for Collaborative Ontology Construction. Journal of Human-Computer Studies 46, 707–728 (1997)CrossRefGoogle Scholar
  23. 23.
    Sagae, K., et al.: Towards Natural Language Understanding of Partial Speech Recognition Results in Dialogue Systems. In: NAACL HLT 2009, 53–56. Association for Computational Linguistics, Boulder (2009)Google Scholar
  24. 24.
    Young, S., et al.: The Hidden Information State model: A practical framework for POMDP-based spoken dialogue management. Computer Speech and Language, Special Issue on Evaluation 24(2), 150–174 (2010)Google Scholar
  25. 25.
    Lefevre, F., et al.: k-Nearest Neighbor Monte-Carlo Control Algorithm for POMDP-Based Dialogue Systems. In: The SIGDIAL 2009 Conference: The 10th Annual Meeting of the Special Interest Group on Discourse and Dialogue, London, UK, 272–275 (2009)Google Scholar
  26. 26.
    Bohus, D., Horvitz, E.: Learning to Predict Engagement with a Spoken Dialog System in Open-World Settings. In: The SIGDIAL 2009 Conference: The 10th Annual Meeting of the Special Interest Group on Discourse and Dialogue, London, UK (2009)Google Scholar
  27. 27.
    Bohus, D., Horvitz, E.: Models for Multiparty Engagement in Open-World Dialog. In: SIGDIAL 2009: the 10th Annual Meeting of the Special Interest Group in Discourse and Dialogue, Queen Mary University of London, 225–234 (2009)Google Scholar
  28. 28.
    DeVault, D., Sagae, K., Traum, D.: Can I finish? Learning when to respond to incremental interpretation results in interactive dialogue. In: The SIGDIAL 2009 Conference: The 10th Annual Meeting of the Special Interest Group on Discourse and Dialogue, London, UK, pp. 11–20 (2009)Google Scholar
  29. 29.
    Skantze, G., Gustafson, J.: Attention and Interaction Control in a Human-Human-Computer Dialogue Setting. In: The SIGDIAL 2009 Conference: The 10th Annual Meeting of the Special Interest Group on Discourse and Dialogue, London, UK, 310–313 (2009)Google Scholar
  30. 30.
    op den Akker, H., op den Akker, R.: Are You Being Addressed? - real-time addressee detection to support remote participants in hybrid meetings. In: The SIGDIAL 2009 Conference: The 10th Annual Meeting of the Special Interest Group on Discourse and Dialogue, London, UK, 21–28 (2009)Google Scholar
  31. 31.
    Crockett, K., et al.: Bullying and Debt: Developing Novel Applications of Dialogue Systems. In: Knowledge and Reasoning in Practical Dialogue Systems (IJCAI), 1–9. IJCAI, Pasadena (2009)Google Scholar
  32. 32.
    Cassell, J., et al.: More Than Just a Pretty Face: Conversational Protocols and the Affordances of Embodiment. Knowledge-Based Systems 14, 55–64 (2001)CrossRefGoogle Scholar
  33. 33.
    Robinson, S., et al.: What would you ask a CA? Observations of Human-Agent Dialogues in a Museum Setting. In: Language Resources and Evaluation Conference 2008, Marrakech, Morocco, 1125–1131 (2008)Google Scholar
  34. 34.
    Babu, S., et al.: “What Would You Like to Talk About?” An Evaluation of Social Conversations with a Virtual Receptionist. In: Gratch, J., Young, M., Aylett, R.S., Ballin, D., Olivier, P. (eds.) IVA 2006. LNCS (LNAI) 4133, 169–180. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  35. 35.
    Lance, B.J., Marsella, S.C.: A model of gaze for the purpose of emotional expression in virtual embodied agents. In: International Conference on Autonomous Agents, Estoril, Portugal, 199–206 (2008)Google Scholar
  36. 36.
    Gergle, D., Rosé, C.P., Kraut, R.E.: Modeling the Impact of Shared Visual Information on Collaborative Reference. In: The SIGCHI Conference on Human Factors in Computing Systems, 1543–1552 (2007)Google Scholar
  37. 37.
    Bickmore, T., Cassell, J.: ’How about this weather?’ Social Dialog with Embodied CAs. In: The American Association for Artificial Intelligence (AAAI) Fall Symposium on “Narrative Intelligence”, Cape Cod, MA, pp. 4–8 (2000)Google Scholar
  38. 38.
    Kopp, S., et al.: A Conversational Agent as Museum Guide – Design and Evaluation of a Real-World Application. In: Panayiotopoulos, T., Gratch, J., Aylett, R.S., Ballin, D., Olivier, P., Rist, T. (eds.) IVA 2005. LNCS (LNAI), vol. 3661, pp. 329–343. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  39. 39.
    Bevacqua, E., et al.: An expressive ECA showing complex emotions. In: AISB 2007 - Artificial and Ambient Intelligence, Newcastle University, Newcastle upon Tyne, UK (2007)Google Scholar
  40. 40.
    Walker, M.A., Hirschman, L., Aberdeen, J.: Evaluation for Darpa Communicator Spoken Dialogue Systems. In: Language Resources and Evaluation Conference, Athens, Greece (2000)Google Scholar
  41. 41.
    Walker, M.A., Passonneau, R., Boland, J.E.: Quantitative and Qualitative Evaluation of Darpa Communicator Spoken Dialogue Systems. In: The 39th Annual Meeting on Association for Computational Linguistics, Toulouse, France, pp. 515–522 (2001)Google Scholar
  42. 42.
    Walker, M.A., et al.: Speech-Plans: Generating Evaluative Responses in Spoken Dialogue. In: International Conference on Natural Language Generation, pp. 73–80 (2002)Google Scholar
  43. 43.
    Giraudo, E., Baggia, P.: EVALITA 2009: Loquendo Spoken Dialog System. In: Evaluation of NLP and Speech Tools for Italian EVALITA 2009, Reggio Emilia, Italy (2009)Google Scholar
  44. 44.
    Rigo, S., et al.: The 2009 UNITN EVALITA Italian Spoken Dialogue System. In: Evaluation of NLP and Speech Tools for Italian EVALITA 2009, Reggio Emilia, Italy (2009)Google Scholar
  45. 45.
    De Angeli, A.: Ethical implications of verbal disinhibition with CAs. PsychNology Journal 7(1), 49–57 (2009)Google Scholar
  46. 46.
    Kegel. Kegel - Oldest Harley Dealer, (cited March 23, 2010)
  47. 47.
    Hulstijn, H., et al.: Topics in schisma dialogues. In: Twente Workshop on Language Technology 11 (TWLT11), University of Twente (1996)Google Scholar
  48. 48.
    Andernach, T., et al.: Language Analysis for Dialogue Management in a Theatre Information & Booking System. In: 15th International Conference on Language Engineering, AI 1995, Montpellier, pp. 351–362 (1995)Google Scholar
  49. 49.
    Michie, D.: Return of the Imitation Game. Electronic Transactions in Artificial Intelligence 6(B), 205–220 (2001)Google Scholar
  50. 50.
    BBC. Tories criticise flu advice line 2009 06:45 GMT, Friday, July 24 (2009), 07:45 UK, (cited June 24, 2009)
  51. 51.
    Morge, M., Abdel-Naby, S., Beaufils, B.: Towards a dialectical approach for CAs in selling situations. In: The 9th International Conference on Autonomous Agents and Multiagent Systems, Toronto, Canada, pp. 127–144 (2010)Google Scholar
  52. 52.
    Lance, B., Marsella, S.: A Model of Gaze for the Purpose of Emotional Expression in Virtual Embodied Agents. In: 7th Int. Conf. on Autonomous Agents and Multiagent Systems (AAMAS 2008), Estoril, Portugal, pp. 199–206 (2008)Google Scholar
  53. 53.
    Plantec, P.: The Zen of Scripting Verbots (1998), (cited September 28, 2004)
  54. 54.
    Sammut, C.: Managing Context in a CA. Electronic Transactions in Artificial Intelligence 5(B), 191–201 (2001)Google Scholar
  55. 55.
    Dehaene, S., Naccache, L.: Towards a cognitive neuroscience of consciousness: basic evidence and a workspace framework. Cognition 79, 1–37 (2001)CrossRefGoogle Scholar
  56. 56.
    Forbes-Riley, K., Litman, D.: Designing and evaluating a wizarded uncertainty-adaptive spoken dialogue tutoring system. Computer Speech and Language, Special Issue on Evaluation 25(1), 105–126 (2011)Google Scholar
  57. 57.
    Walker, M.A., et al.: PARADISE: a framework for evaluating spoken dialogue agents. In: The 35th Annual Meeting of the Association for Computational Linguistics, Madrid, Spain, pp. 271–280 (1997)Google Scholar
  58. 58.
    Walker, M.A., et al.: Darpa communicator dialog travel planning systems: The June 2000 data collection. In: EUROSPEECH 2001 7th European Conference on Speech Communication and Technology 2nd INTERSPEECH Event, Aalborg, Denmark, pp. 1371–1374 (2001)Google Scholar
  59. 59.
    Fenton, N., Pfleeger, S.: Software Metrics: A Rigorous and Practical Approach. PWS (1998)Google Scholar
  60. 60.
    Kato, T., Matsushita, M., Kando, N.: Bridging Evaluations: Inspiration from Dialogue System Research. In: SIGIR 2010 33rd Annual International ACM SIGIR Conference, SIGIR, pp. 3–4 (2010)Google Scholar
  61. 61.
    Dethlefs, N., et al.: Evaluating Task Success in a Dialogue System for Indoor Navigation. In: SemDial 2010 14th Workshop on the Semantics and Pragmatics of Dialogue, pp. 143–146 (2010)Google Scholar
  62. 62.
    Lee, C., et al.: Recent Approaches to Dialog Management for Spoken Dialog Systems. Journal of Computing Science and Engineering 4(1), 1–22 (2010)Google Scholar
  63. 63.
    Foster, M.E., et al.: Evaluating Description and Reference Strategies in a Cooperative Human-Robot Dialogue System. In: The 21st International Joint Conference on Artificial Intelligence, Pasadena, California, USA, pp. 1818–1823 (2009)Google Scholar
  64. 64.
    Webb, N., et al.: Evaluating Human-Machine Conversation for Appropriateness. In: The 7th Conference on International Language Resources and Evaluation (LREC 2010), Valletta, Malta (2010)Google Scholar
  65. 65.
    Bouwman, G., Sturm, J., Boves, L.: Incorporating confidence measures in the Dutch train timetable information system developed in the ARISE project. In: ICASSP 1999, pp. 493–496 (1999)Google Scholar
  66. 66.
    Semeraro, G., et al.: Evaluation and Validation of a Conversational Agent Embodied in a Bookstore. In: Carbonell, N., Stephanidis, C. (eds.) UI4ALL 2002. LNCS, vol. 2615, pp. 360–371. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  67. 67.
    Andersen, V., et al.: A methodological approach for designing and evaluating intelligent applications for digital collections. Applied Artificial Intelligence 17(8-9), 745–771 (2003)CrossRefGoogle Scholar
  68. 68.
    Lamel, L., et al.: User evaluation of the MASK kiosk. Speech Communication 38(1), 131–139 (2002)CrossRefMATHGoogle Scholar
  69. 69.
    Ortony, A., Clore, G.L., Collins, A.: The Cognitive Structure of Emotions. Cambridge University Press, Cambridge (1990)Google Scholar
  70. 70.
    Yuan, X., Chee, Y.S.: Design and evaluation of Elva: an embodied tour guide in an interactive virtual art gallery. Computer Animation and Virtual Worlds 16(2), 109–119 (2005)CrossRefGoogle Scholar
  71. 71.
    McKevitt, P., Partridge, D., Wilks, Y.: Why machines should analyse intention in natural language dialogue. Int. J. Human-Computer Studies 51, 947–989 (1999)CrossRefGoogle Scholar
  72. 72.
    Le Bigot, L., Jamet, E., Rouet, J.-F.: Searching information with a natural language dialogue system: a comparison of spoken vs. written modalities. Applied Ergonomics 35, 557–564 (2004)CrossRefGoogle Scholar
  73. 73.
    Cassell, J., Vilhjálmsson, H.: Fully Embodied Conversational Avatars: Making Communicative Behaviors Autonomous. Autonomous Agents and Multi-Agent Systems 2(1), 45–64 (1999)CrossRefGoogle Scholar
  74. 74.
    Massaro, D.W., et al.: Developing and evaluating CAs. In: Cassell, J., et al. (eds.) Embodied CAs, pp. 286–318. MIT Press, Cambridge, MA (2000)Google Scholar
  75. 75.
    Cassell, J., Bickmore, T.: Negotiated Collusion: Modeling Social language and its Relationship Effects in Social Agents. User Modeling and User-Adapted Interaction 13, 89–132 (2003)CrossRefGoogle Scholar
  76. 76.
    Lamel, L., et al.: The LIMSI RailTel System: Field trial of a telephone service for rail travel information. Speech Communication 23(1-2), 67–82 (1997)CrossRefGoogle Scholar
  77. 77.
    Litman, D.J., Pan, S.: Designing and Evaluating an Adaptive Spoken Dialogue System. In: User Modeling and User-Adapted Interaction 12, 111–137 (2002)Google Scholar
  78. 78.
    Sanders, G.A., Scholtz, J.: Measurement and Evaluation of Embodied CAs. In: Cassell, J., et al. (eds.) Embodied CAs, MIT Press, Cambridge, MA (2000)Google Scholar
  79. 79.
    Bouwman, G., Hulstijn, J.: Dialog Strategy Redesign with Reliabilty Measures. In: 1st Int. Conf. on Language Resources and Evaluation, Granada, Spain, 191–198 (1998)Google Scholar
  80. 80.
    Spärck-Jones, K.: A Statistical Interpretation of Term Specificity and its Application in Retrieval. Journal of Documentation 28, 11–21 (1972)CrossRefGoogle Scholar
  81. 81.
    Salton, G., Wong, A., Yang, C.S.: A Vector Space Model for Automatic Indexing. Communications of the ACM 18(11), 613–620 (1975)CrossRefMATHGoogle Scholar
  82. 82.
    Li, Y., et al.: Sentence Similarity Based on Semantic Nets and Corpus Statistics. IEEE Transactions on Knowledge and Data Engineering 18(8), 1138–1150 (2006)CrossRefGoogle Scholar
  83. 83.
    Austin, J.L.: In: Urmson, J.O. (ed.) How to do things with Words: The William James Lectures delivered at Harvard University in 1955, 2nd edn., Harvard University Press, Cambridge, MA (1975)Google Scholar
  84. 84.
    Ferri, F., Grifoni, P., Paolozzi, S.: An Approach to Multimodal Input Interpretation in Human-Computer Interaction. In: The Nineteenth International Conference on Software Engineering Knowledge Engineering (SEKE 2007), Boston, MA, USA, pp. 664–669 (2007)Google Scholar
  85. 85.
    Tsatsaronis, G., Varlamis, I., Vazirgiannis, M.: Text Relatedness Based on a Word Thesaurus. Journal of Artificial Intelligence Research 37, 1–39 (2010)MATHGoogle Scholar
  86. 86.
    Min, F., Wenyin, L., Chen, W.: Answer Clustering and Fusion in a User-Interactive QA System. In: Second International Conference on Semantics, Knowledge and Grid, 41 (2006)Google Scholar
  87. 87.
    Gacitua-Decar, V., Pahl, C.: Automatic Business Process Pattern Matching for Enterprise Services Design. In: 2009 World Conference on Services - II, Bangalore, India, 111–118 (2009)Google Scholar
  88. 88.
    Huang, J.-J., Changt, S.-T., Hu, S.-Y.: Searching for Answers via Social Networks. In: 5th IEEE Consumer Communications and Networking Conference, CCNC 2008, Las Vegas, NV, 289–293 (2008)Google Scholar
  89. 89.
    Capuano, N., et al.: On-Demand Construction of Personalized Learning Experiences Using Semantic Web and Web 2.0 Techniques. In: Ninth IEEE International Conference on Advanced Learning Technologies, 484–488. IEEE Computer Society, Washington, DC (2009)CrossRefGoogle Scholar
  90. 90.
    Inkpen, D.: Semantic Similarity Knowledge and its Applications. Studia Universitatis Babes-Bolyai Informatica LII(1), 11–22 (2007)MathSciNetGoogle Scholar
  91. 91.
    Achananuparp, P., Hu, X., Shen, X.: The Evaluation of Sentence Similarity Measures. In: Song, I.-Y., Eder, J., Nguyen, T.M. (eds.) DaWaK 2008. LNCS 5182, 305–316. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  92. 92.
    O’Shea, K., Bandar, Z., Crockett, K.: Towards a New Generation of CAs Based on Sentence Similarity. Lecture Notes Electrical Engineering 39, 505–514 (2009)Google Scholar
  93. 93.
    O’Shea, K., Bandar, Z., Crockett, K.: A Novel Approach for Constructing CAs using Sentence Similarity Measures. In: Proceedings of the World Congress on Engineering WCE 2008, London, U.K., 321–326 (2008)Google Scholar
  94. 94.
    Liu, X., Zhou, Y., Zheng, R.: Sentence Similarity based on Dynamic Time Warping. In: International Conference on Semantic Computing, ICSC 2007, 250–256 (2007)Google Scholar
  95. 95.
    Li, Y., Bandar, Z., McLean, D.: An Approach for Measuring Semantic Similarity between Words Using Multiple Information Sources. IEEE Transactions on Knowledge and Data Engineering 15(4), 871–882 (2003)CrossRefGoogle Scholar
  96. 96.
    Resnik, P.: Semantic Similarity in a Taxonomy: An Information-Based Measure and its Application to Problems of Ambiguity in Natural Language. Journal of Artificial Intelligence Research 11, 95–130 (1999)MATHGoogle Scholar
  97. 97.
    O’Shea, J.: A Framework for Applying Short Text Semantic Similarity in Goal-Oriented CAs. In: Computing and Mathematics. Manchester Metropolitan University, Manchester (2010)Google Scholar
  98. 98.
    Landauer, T.K., Foltz, P.W., Laham, D.: An Introduction to Latent Semantic Analysis. Discourse Processes 25, 259–284 (1998)CrossRefGoogle Scholar
  99. 99.
    Laham, D.: Latent Semantic Analysis @ CU Boulder (1998), LSA website, (cited January 20, 2008)
  100. 100.
    Deerwester, S., et al.: Computer information retrieval using Latent Semantic Structure, U.S.P. Office. Bell Communications Research Inc., United States of America (1989)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • James O’Shea
    • 1
  • Zuhair Bandar
    • 1
  • Keeley Crockett
    • 1
  1. 1.School of Computing, Mathematics and Digital TechnologyManchester Metropolitan UniversityManchesterUnited Kingdom

Personalised recommendations