Boosting a Rule-Based Chatbot Using Statistics and User Satisfaction Ratings

  • Octavia Efraim
  • Vladislav MaraevEmail author
  • João Rodrigues
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 789)


Using data from user-chatbot conversations where users have rated the answers as good or bad, we propose a more efficient alternative to a chatbot’s keyword-based answer retrieval heuristic. We test two neural network approaches to the near-duplicate question detection task as a first step towards a better answer retrieval method. A convolutional neural network architecture gives promising results on this difficult task.



This research is partly funded by the Regional Council of Brittany through an ARED grant. The present research was also partly supported by the CLARIN and ANI/3279/2016 grants. We are grateful to Telsi for providing the data.


  1. 1.
    Accorsi, P., Patel, N., Lopez, C., Panckhurst, R., Roche, M.: Seek & hide: anonymising a french sms corpus using natural language processing techniques. Lingvisticæ Investigationes 35(2), 163–180 (2012)Google Scholar
  2. 2.
    Afzal, N., Wang, Y., Liu, H.: MayoNLP at SemEval-2016 Task 1: semantic textual similarity based on lexical semantic net and deep learning semantic model. In: Proceedings of the 10th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT 2016, San Diego, CA, USA, 16–17 June 2016, pp. 674–679 (2016)Google Scholar
  3. 3.
    Baumeister, R.F., Bratslavsky, E., Finkenauer, C., Vohs, K.D.: Bad is stronger than good. Rev. Gen. Psychol. 5(4), 323 (2001)CrossRefGoogle Scholar
  4. 4.
    Bernhard, D., Gurevych, I.: Answering learners’ questions by retrieving question paraphrases from social Q&A sites. In: Proceedings of the Third Workshop on Innovative Use of NLP for Building Educational Applications, pp. 44–52. ACL (2008)Google Scholar
  5. 5.
    Bikel, D.M., Schwartz, R., Weischedel, R.M.: An algorithm that learns what’s in a name. Mach. Learn. 34(1), 211–231 (1999)CrossRefGoogle Scholar
  6. 6.
    Bogdanova, D., dos Santos, C.N., Barbosa, L., Zadrozny, B.: Detecting semantically equivalent questions in online user forums. In: Proceedings of the 19th Conference on Computational Natural Language Learning, CoNLL 2015, Beijing, China, 30–31 July 2015, pp. 123–131 (2015)Google Scholar
  7. 7.
    Denis, P., Sagot, B.: Coupling an annotated corpus and a lexicon for state-of-the-art pos tagging. Lang. Resour. Evaluation 46(4), 721–736 (2012)CrossRefGoogle Scholar
  8. 8.
    Feng, M., Xiang, B., Glass, M.R., Wang, L., Zhou, B.: Applying deep learning to answer selection: a study and an open task. In: 2015 IEEE Workshop on Automatic Speech Recognition and Understanding, ASRU 2015, Scottsdale, AZ, USA, 13–17 December 2015, pp. 813–820 (2015)Google Scholar
  9. 9.
    Goldberg, Y.: Neural Network Methods for Natural Language Processing. Morgan & Claypool, San Rafael (2017)Google Scholar
  10. 10.
    Higashinaka, R., Minami, Y., Dohsaka, K., Meguro, T.: Issues in predicting user satisfaction transitions in dialogues: individual differences, evaluation criteria, and prediction models. In: Lee, G.G., Mariani, J., Minker, W., Nakamura, S. (eds.) IWSDS 2010. LNCS (LNAI), vol. 6392, pp. 48–60. Springer, Heidelberg (2010). Scholar
  11. 11.
    Hogan, D., Leveling, J., Wang, H., Ferguson, P., Gurrin, C.: Dcu@fire 2011: SMS-based FAQ retrieval. In: 3rd Workshop of the Forum for Information Retrieval Evaluation, FIRE, pp. 2–4 (2011)Google Scholar
  12. 12.
    Hone, K.S., Graham, R.: Subjective assessment of speech-system interface usability. In: INTERSPEECH, pp. 2083–2086 (2001)Google Scholar
  13. 13.
    Jalbert, N., Weimer, W.: Automated duplicate detection for bug tracking systems. In: IEEE International Conference on Dependable Systems and Networks with FTCS and DCC (DSN 2008), pp. 52–61. IEEE (2008)Google Scholar
  14. 14.
    Jeon, J., Croft, W.B., Lee, J.H.: Finding semantically similar questions based on their answers. In: Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 617–618. ACM (2005)Google Scholar
  15. 15.
    Jijkoun, V., de Rijke, M.: Retrieving answers from frequently asked questions pages on the web. In: Proceedings of the 14th ACM International Conference on Information and Knowledge Management, pp. 76–83. ACM (2005)Google Scholar
  16. 16.
    Kim, Y.: Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882 (2014)
  17. 17.
    Kim, Y.: Convolutional neural networks for sentence classification. In: EMNLP, pp. 1746–1751. ACL (2014)Google Scholar
  18. 18.
    Liu, C.W., Lowe, R., Serban, I.V., Noseworthy, M., Charlin, L., Pineau, J.: How not to evaluate your dialogue system: an empirical study of unsupervised evaluation metrics for dialogue response generation. arXiv preprint arXiv:1603.08023 (2016)
  19. 19.
    Lowe, R., Serban, I.V., Noseworthy, M., Charlin, L., Pineau, J.: On the evaluation of dialogue systems with next utterance classification. arXiv preprint arXiv:1605.05414 (2016)
  20. 20.
    Malakasiotis, P., Androutsopoulos, I.: Learning textual entailment using SVMs and string similarity measures. In: Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, pp. 42–47. ACL (2007)Google Scholar
  21. 21.
    Manning, C.D., Raghavan, P., Schütze, H., et al.: Introduction to Information Retrieval, vol. 1. Cambridge University Press, Cambridge (2008)CrossRefGoogle Scholar
  22. 22.
    Muthmann, K., Petrova, A.: An automatic approach for identifying topical near-duplicate relations between questions from social media Q/A sites. In: Proceeding of WSDM 2014 Workshop: Web-Scale Classification: Classifying Big Data from the Web (2014)Google Scholar
  23. 23.
    Reitter, D., Moore, J.D.: Predicting success in dialogue. In: Proceedings of the 45th Annual Meeting of the ACL, ACL 2007, 23–30 June 2007, Prague, Czech Republic (2007)Google Scholar
  24. 24.
    Ritter, A., Cherry, C., Dolan, W.B.: Data-driven response generation in social media. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 583–593. ACL (2011)Google Scholar
  25. 25.
    Rodrigues, J.A., Saedi, C., Maraev, V., Silva, J., Branco, A.: Ways of asking and replying in duplicate question detection. In: Ide, N., Herbelot, A., Màrquez, L. (eds.) Proceedings of the 6th Joint Conference on Lexical and Computational Semantics, *SEM @ACM 2017, Vancouver, Canada, 3–4 August 2017, pp. 262–270. Association for Computational Linguistics (2017).
  26. 26.
    Seddah, D., Sagot, B., Candito, M., Mouilleron, V., Combet, V.: The French Social Media Bank: a treebank of noisy user generated content. In: 24th International Conference on Computational Linguistics, COLING 2012 (2012)Google Scholar
  27. 27.
    Vinyals, O., Le, Q.: A neural conversational model. arXiv preprint arXiv:1506.05869 (2015)
  28. 28.
    Walker, M., Langkilde, I., Wright, J., Gorin, A., Litman, D.: Learning to predict problematic situations in a spoken dialogue system: experiments with how may I help you? In: Proceedings of the 1st North American Chapter of the Association for Computational Linguistics Conference, pp. 210–217. Association for Computational Linguistics (2000)Google Scholar
  29. 29.
    Walker, M.A., Litman, D.J., Kamm, C.A., Abella, A.: PARADISE: A framework for evaluating spoken dialogue agents. In: Proceedings of the Eighth Conference on European Chapter of the Association for Computational Linguistics, pp. 271–280. ACL (1997)Google Scholar
  30. 30.
    Wu, Y., Zhang, Q., Huang, X.: Efficient near-duplicate detection for Q&A forum. In: Fifth International Joint Conference on Natural Language Processing, IJCNLP 2011, Chiang Mai, Thailand, 8–13 November 2011, pp. 1001–1009 (2011)Google Scholar
  31. 31.
    Xue, X., Jeon, J., Croft, W.B.: Retrieval models for question and answer archives. In: Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 475–482. ACM (2008)Google Scholar
  32. 32.
    Yin, W., Schütze, H., Xiang, B., Zhou, B.: ABCNN: attention-based convolutional neural network for modeling sentence pairs. arXiv preprint arXiv:1512.05193 (2015)

Copyright information

© Springer International Publishing AG 2018

Authors and Affiliations

  • Octavia Efraim
    • 1
  • Vladislav Maraev
    • 2
    Email author
  • João Rodrigues
    • 3
  1. 1.LIDILE EA3874University of Rennes 2RennesFrance
  2. 2.CLASPUniversity of GothenburgGothenburgSweden
  3. 3.Department of Informatics, Faculty of SciencesUniversity of LisbonLisbonPortugal

Personalised recommendations