Evaluation of Question-Answering System About Conversational Agent’s Personality

  • Hiroaki Sugiyama
  • Toyomi Meguro
  • Ryuichiro Higashinaka
Part of the Lecture Notes in Electrical Engineering book series (LNEE, volume 427)


We develop a question-answering system for questions that ask about a conversational agent’s personality based on large-scale question-answer pairs created by hand. In casual dialogues, the speaker sometimes asks his conversation partner questions about favorites or experiences. Since this behavior also appears in conversational dialogues with a dialogue system, systems must be developed to respond to such questions. However, the effectiveness of personality-question-answering for conversational agents has not been investigated. Our user-machine chat experiments show that our question-answering system, which estimates appropriate answers with 60.7 % accuracy for the personality questions in our conversation corpus, significantly improves user’s subjective evaluations.


Conversational systems Question answering Agent’s personality 


  1. 1.
    Shibata, M., Nishiguchi, T., Tomiura, Y.: Dialog system for open-ended conversation using web documents. Informatica 33, 277–284 (2009),
  2. 2.
    Ritter, A., Cherry, C., Dolan, W.: Data-driven response generation in social media. In: Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pp. 583–593 (2011).
  3. 3.
    Wong, W., Cavedon, L., Thangarajah, J., Padgham, L.: Strategies for mixed-initiative conversation management using question-answer pairs. In: Proceedings of the 24th International Conference on Computational Linguistics. pp. 2821–2834 (2012).
  4. 4.
    Meguro, T., Higashinaka, R., Minami, Y., Dohsaka, K.: Controlling listening-oriented dialogue using partially observable markov decision processes. In: Proceedings of the 23rd International Conference on Computational Linguistics, pp. 761–769 (2010)Google Scholar
  5. 5.
    Higashinaka, R., Imamura, K., Meguro, T., Miyazaki, C., Kobayashi, N., Sugiyama, H., Hirano, T., Makino, T., Matsuo, Y.: Towards an open-domain conversational system fully based on natural language processing. In: Proceedings of the 25th International Conference on Computational Linguistics, pp. 928–939 (2014)Google Scholar
  6. 6.
    Bickmore, T., Cassell, J.: Relational agents: a model and implementation of building user trust. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 396–403 (2001).
  7. 7.
    Tidwell, L.C., Walther, J.B.: Computer-mediated communication effects on disclosure, impressions, and interpersonal evaluations. Hum. Commun. Res. 28(3), 317–348 (2002)CrossRefGoogle Scholar
  8. 8.
    Nisimura, R., Nishihara, Y., Tsurumi, R., Lee, A., Saruwatari, H., Shikano, K.: Takemaru-kun.: Speech-oriented information system for real world research platform. In: Proceedings of the First International Workshop on Language Understanding and Agents for Real World Interaction, pp. 70–78 (2003).
  9. 9.
    Weizenbaum, J.: ELIZA—a computer program for the study of natural language communication between man and machine. Commun. ACM 9(1), 36–45 (Jan 1966).
  10. 10.
    Caspi, A., Roberts, B.W., Shiner, R.L.: Personality development: stability and change. Ann. Rev. Psychol. 56, 453–484 (Jan 2005).
  11. 11.
    John, O.P., Srivastava, S.: The Big Five trait taxonomy: history, measurement, and theoretical perspectives. No. 510 (1999).,+Measurement,+and+Theoretical+Perspectives#0
  12. 12.
    Mairesse, F., Walker, M.: PERSONAGE: Personality generation for dialogue. In: Proceedings of the Annual Meeting of the Association For Computational Linguistics, pp. 496–503 (2007).
  13. 13.
    Batacharia, B., Levy, D., Catizone, R., Krotov, A., Wilks, Y.: CONVERSE: a conversational companion. Machine conversations, pp. 205–215 (1999).
  14. 14.
    Traum, D., Georgila, K., Artstein, R., Leuski, A.: Evaluating spoken dialogue processing for time-offset interaction. In: Proceedings of the 16th Annual SIGdial Meeting on Discourse and Dialogue, pp. 199–208 (2015)Google Scholar
  15. 15.
    Leuski, A., Traum, D.: Creating virtual human dialogue using information retrieval techniques. AI Mag. 32(2), 42–56 (2011).
  16. 16.
    Sugiyama, H., Meguro, T., Higashinaka, R.: Large-scale collection and analysis of personal question-answer pairs for conversational agents. In: Proceedings of Intelligent Virtual Agents, pp. 420–433 (2014)Google Scholar
  17. 17.
    Sugiyama, H., Meguro, T., Higashinaka, R., Minami, Y.: Open-domain utterance generation using phrase pairs based on dependency relations. In: Proceedings of Spoken Language Technology Workshop, pp. 60–65 (2014)Google Scholar
  18. 18.
    Turian, J., Ratinov, L., Bengio, Y.: Word representations: A simple and general method for semi-supervised learning. In: Proceedings of the 48th Annual Meeting of the Association for Computational Linquistics. pp. 384–394. July (2010)Google Scholar
  19. 19.
    Mikolov, T., Chen, K., Corrado, G., Dean, J.: Distributed representations of words and phrases and their compositionality. In: Proceedings of the 27th Annual Conference on Neural Information Processing Systems, pp. 1–9 (2013)Google Scholar

Copyright information

© Springer Science+Business Media Singapore 2017

Authors and Affiliations

  • Hiroaki Sugiyama
    • 1
  • Toyomi Meguro
    • 1
  • Ryuichiro Higashinaka
    • 1
    • 2
  1. 1.NTT Communication Science LaboratoriesKyotoJapan
  2. 2.NTT Media Intelligence LaboratoriesYokosuka-shiJapan

Personalised recommendations