Abstract
We implement a method for re-ranking top-10 results of a state-of-the-art question answering (QA) system. The goal of our re-ranking approach is to improve the answer selection given the user question and the top-10 candidates. We focus on improving deployed QA systems that do not allow re-training or when re-training comes at a high cost. Our re-ranking approach learns a similarity function using n-gram based features using the query, the answer and the initial system confidence as input. Our contributions are: (1) we generate a QA training corpus starting from 877 answers from the customer care domain of T-Mobile Austria, (2) we implement a state-of-the-art QA pipeline using neural sentence embeddings that encode queries in the same space than the answer index, and (3) we evaluate the QA pipeline and our re-ranking approach using a separately provided test set. The test set can be considered to be available after deployment of the system, e.g., based on feedback of users. Our results show that the system performance, in terms of top-n accuracy and the mean reciprocal rank, benefits from re-ranking using gradient boosted regression trees. On average, the mean reciprocal rank improves by \(9.15\%\).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
- 2.
- 3.
- 4.
- 5.
German 50k: https://github.com/hermitdave/FrequencyWords.
- 6.
We use default word tokenizer, Snowball stemmer and n-gram extraction of the nltk toolkit [3].
- 7.
We use the implementation for Jaccard distance and cosine similarity as found in the following Github gist: gaulinmp/similarity_example.ipynb.
References
Abadi M, Agarwal A, Barham P, Brevdo E, Chen Z, Citro C, Corrado GS, Davis A, Dean J, Devin M, Ghemawat S, Goodfellow I, Harp A, Irving G, Isard M, Jia Y, Jozefowicz R, Kaiser L, Kudlur M, Levenberg J, Mané D, Monga R, Moore S, Murray D, Olah C, Schuster M, Shlens J, Steiner B, Sutskever I, Talwar K, Tucker P, Vanhoucke V, Vasudevan V, Viégas F, Vinyals O, Warden P, Wattenberg M, Wicke M, Yu Y, Zheng X (2015) TensorFlow: large-scale machine learning on heterogeneous systems (2015). Software available from tensorflow.org
Bernstein MS, Teevan J, Dumais S, Liebling D, Horvitz E (2012) Direct answers for search queries in the long tail. In: Proceedings of the 2012 ACM annual conference on human factors in computing systems - CHI ’12, pp 237, New York, USA. ACM Press
Bird S, Klein E, Loper E (2009) Natural language processing with Python: analyzing text with the natural language toolkit. O’Reilly Media Inc, Sebastopol
Bocklisch T, Faulkner J, Pawlowski N, Nichol A (2017) Rasa: open source language understanding and dialogue management. 12
Buck C, Bulian J, Ciaramita M, Gesmundo A, Houlsby N, Gajewski W, Wang W (2017) Ask the right questions: active question reformulation with reinforcement learning. arXiv:1705.0
Chang JC, Kittur A, Hahn N (2016) Alloy: clustering with crowds and computation. In: Proceedings of the 2016 CHI conference on human factors in computing systems - CHI ’16, pp 3180–3191, New York, USA. ACM Press
Chen D, Fisch A, Weston J, Bordes A (2017) Reading wikipedia to answer open-domain questions. In: Proceedings of the 55th annual meeting of the association for computational linguistics (Volume 1: Long Papers), pp 1870–1879. ACL
Das A, Yenala H, Chinnakotla M, Shrivastava M (2016) Together we stand: siamese networks for similar question retrieval. In: Proceedings of the 54th annual meeting of the association for computational linguistics (Volume 1: Long Papers), vol 1, pp 378–387, Stroudsburg, PA, 2016. Association for Computational Linguistics
Dhingra B, Danish D, Rajagopal D (2018) Simple and effective semi-supervised question answering. In: Proceedings of the 2018 conference of the North American chapter of the association for computational linguistics: human language technologies, Volume 2 (Short Papers), pp 582–587. ACL
Eric M, Manning CD (2017) Key-value retrieval networks for task-oriented dialogue. In: Proceedings of the 18th annual SIGdial meeting on discourse and dialogue, pp 37–49, Saarbrücken, Germany. Association for Computational Linguistics
’Kenneth’ Huang T-H, Chang JC, Bigham JP (2018) Evorus: a crowd-powered conversational assistant built to automate itself over time. 1
Kenneth Huang T-H, Lasecki WS, Bigham JP (2015) Guardian: a crowd-powered spoken dialog system for web APIs. In: Third AAAI conference on human computation and crowdsourcing
Joshi M, Choi E, Weld DS, Zettlemoyer L (2017) TriviaQA: a large scale distantly supervised challenge dataset for reading comprehension. In: Barzilay R, Kan M-Y (eds) Proceedings of the 55th annual meeting of the association for computational linguistics, ACL 2017, Volume 1: Long Papers, pp 1601–1611. ACL
Mikolov T, Chen K, Corrado G, Dean J (2013) Efficient estimation of word representations in vector space. 1
Minaee S, Liu Z (2017) Automatic question-answering using a deep similarity neural network. 8
Oraby S, Gundecha P, Mahmud J, Bhuiyan M, Akkiraju R (2017) "How May I Help You?": modeling twitter customer service conversations using fine-grained dialogue acts. In: Proceedings of the 22nd international conference on intelligent user interfaces - IUI ’17, pp 343–355, New York, USA, 2017. ACM Press
Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, Blondel M, Prettenhofer P, Weiss R, Dubourg V, Vanderplas J, Passos A, Cournapeau D, Brucher M, Perrot M, Duchesnay E (2011) Scikit-learn: machine learning in Python. J Mach Learning Res 12:2825–2830
Rajpurkar P, Zhang J, Lopyrev K, Liang P (2016) Squad: 100,000+ questions for machine comprehension of text. In: Proceedings of the 2016 conference on empirical methods in natural language processing, pp 2383–2392. ACL
Savenkov D, Agichtein E (2016) CRQA: crowd-powered real-time automatic question answering system. In: Fourth AAAI conference on human computation and crowdsourcing
Serban IV, Sordoni A, Bengio Y, Courville A, Pineau J (2015) Building end-to-end dialogue systems using generative Hierarchical neural network models. 7
Serban IV, Lowe R, Henderson P, Charlin L, Pineau J (2015) A survey of available corpora for building data-driven dialogue systems. 12
Burr S (2010) Active learning literature survey. Univ Wisconsin Madison 52(55–66):11
Sonntag D (2008) On introspection, metacognitive control and augmented data mining live cycles
Sonntag D (2009) Introspection and adaptable model integration for dialogue-based question answering. In: IJCAI, pp 1549–1554
Sonntag D (2010) Ontologies and adaptivity in dialogue for question answering. Studies on the semantic web, 1st edn, vol 4. AKA and IOS Press, Heidelberg
Sonntag D, Engel R, Herzog G, Pfalzgraf A, Pfleger N, Romanelli M, Reithinger N (2007) Smartweb handheld - multimodal interaction with ontological knowledge bases and semantic web services. In: Artifical intelligence for human computing, ICMI 2006 and IJCAI 2007 international workshops, Banff, Canada, November 3, 2006, Hyderabad, India, January 6, 2007, Revised selected and invited papers, pp 272–295
Williams JD, Asadi K, Zweig G (2017) Hybrid code networks: practical and efficient end-to-end dialog control with supervised and reinforcement learning. In: Proceedings of the 55th annual meeting of the association for computational linguistics (Volume 1: Long Papers), vol 1, pp 665–677, Stroudsburg, PA, USA. Association for Computational Linguistics
Wu L, Fisch A, Chopra S, Adams K, Bordes A, Weston J (2017) Starspace: embed all the things! arXiv:1709.03856
Xue X, Jeon J, Croft WB (2008) Retrieval models for question and answer archives. In: Proceedings of the 31st annual international ACM SIGIR conference on research and development in information retrieval - SIGIR ’08, pp 475, New York, USA, 2008. ACM Press
Acknowledgements
This work is supported by the German Federal Ministry of Education and Research (BMBF) in the Software Campus project SciBot and by EIT Digital (H2020) in the ERICS project.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this chapter
Cite this chapter
Barz, M., Sonntag, D. (2021). Incremental Improvement of a Question Answering System by Re-ranking Answer Candidates Using Machine Learning. In: Marchi, E., Siniscalchi, S.M., Cumani, S., Salerno, V.M., Li, H. (eds) Increasing Naturalness and Flexibility in Spoken Dialogue Interaction. Lecture Notes in Electrical Engineering, vol 714. Springer, Singapore. https://doi.org/10.1007/978-981-15-9323-9_34
Download citation
DOI: https://doi.org/10.1007/978-981-15-9323-9_34
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-15-9322-2
Online ISBN: 978-981-15-9323-9
eBook Packages: EngineeringEngineering (R0)