Question Answering (QA) is a field of study addressed to develop automatic methods for answering questions expressed in natural language. Recently, the emergence of the new generation of intelligent assistants, such as Siri, Alexa, and Google Assistant, has intensified the importance of an effective and efficient QA system able to handle questions with different complexities. Regarding the type of question to be answered, QA systems have been divided into two sub-areas: (i) factoid questions that require a single fact – e.g., a name of a person or a date, and (ii) non-factoid questions that need a more complex answer – e.g., descriptions, opinions, or explanations. While factoid QA systems have overcome human performance on some benchmarks, automatic systems for answering non-factoid questions remain a challenge and an open research problem. This work provides an overview of recent research addressing non-factoid questions. It focuses on which methods have been applied in each task, the data sets available, challenges and limitations, and possible research directions. From a total of 455 recent studies, we selected 75 papers based on our quality control system and exclusion criteria for an in-depth analysis. This systematic review helped to answer what are the tasks and methods involved in non-factoid, what are the data sets available, what the limitations are, and what is the recommendations for future research.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Price includes VAT (USA)
Tax calculation will be finalised during checkout.
Stanford Question Answering Dataset (SQuAD) is a reading comprehension dataset (https://rajpurkar.github.io/SQuAD-explorer/)
Agichtein, E., Carmel, D., Pelleg, D., Pinter, Y., & Harman, D. (2015). Overview of the trec 2015 liveqa track.. In TREC.
Bae, K., & Ko, Y. (2019). Efficient question classification and retrieval using category information and word embedding on cQA services. Journal of Intelligent Information Systems, 53(1), 27–49. https://doi.org/10.1007/s10844-019-00556-x.
Bau, D., Liu, S., Wang, T., Zhu, J-Y, & Torralba, A. (2020). Rewriting a deep generative model. In A Vedaldi, H Bischof, T Brox, & J-M Frahm (Eds.) Computer Vision – ECCV 2020, pp 351–369. Springer International Publishing. Cham.
Ben Abacha, A, & Zweigenbaum, P. (2015). Means: A medical question-answering system combining nlp techniques and semantic web technologies. Information Processing & Management, 51 (5), 570–594. https://doi.org/10.1016/j.ipm.2015.04.006. https://www.sciencedirect.com/science/article/pii/S0306457315000515.
Bondarenko, A., Braslavski, P., Völske, M, Aly, R., Fröbe, M, Panchenko, A., Biemann, C., Stein, B., & Hagen, M. (2020). Comparative web search questions. In Proceedings of the 13th International Conference on Web Search and Data Mining, WSDM ’20, pp 52–60. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3336191.3371848.
Calijorne Soares, M.A., & Parreiras, F.S. (2020). A literature review on question answering techniques, paradigms and systems. Journal of King Saud University - Computer and Information Sciences, 32(6), 635–646. https://doi.org/10.1016/j.jksuci.2018.08.005.
Chali, Y., Hasan, S.A., & Mojahid, M. (2015). A reinforcement learning formulation to the complex question answering problem. Information Processing & Management, 51(3), 252–272. https://doi.org/10.1016/j.ipm.2015.01.002, https://www.sciencedirect.com/science/article/pii/S0306457315000035.
Cohen, D., Yang, L., & Croft, W.B. (2018). WikiPassageQA: A benchmark collection for research on non-factoid answer passage retrieval. 41st International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2018, pp 1165–1168. https://doi.org/10.1145/3209978.3210118.
Corbin, J., & Strauss, A. (2014). Basics of qualitative research: Techniques and procedures for developing grounded theory. Sage publications.
Cortes, E., Woloszyn, V., Binder, A., Himmelsbach, T., Barone, D., & Möller, S (May 2020). An empirical comparison of question classification methods for question answering systems. In Proceedings of the 12th Language Resources and Evaluation Conference, pp 5408–5416. European Language Resources Association, Marseille, France. https://www.aclweb.org/anthology/2020.lrec-1.665.
Denyer, D., & Tranfield, D. (2009). Producing a systematic review. Sage Publications Ltd, 671—689.
Dimitrakis, E., Sgontzos, K., & Tzitzikas, Y. (2019). A survey on question answering systems over linked data and documents. Journal of Intelligent Information Systems, 55, 233–259.
Dybå, T, & Dingsøyr, T (2008). Empirical studies of agile software development: A systematic review. Information and software technology, 50(9-10), 833–859.
Hazrina, S., Sharef, N.M., Ibrahim, H., Murad, M.A.A., & Noah, S.A.M. (2017). Review on the advancements of disambiguation in semantic question answering system. Information Processing & Management, 53 (1), 52–69. https://doi.org/10.1016/j.ipm.2016.06.006, https://www.sciencedirect.com/science/article/pii/S0306457316302102.
Hermjakob, U., Echihabi, A., & Marcu, D. (2002). Natural language based reformulation resource and web exploitation for question answering. In Proceedings of TREC, 11. Citeseer.
Higgins, J.P.T., Thomas, J., Chandler, J., Cumpston, M., Li, T., Page, M.J., & Welch, V.A. (2019). Cochrane handbook for systematic reviews of interventions. New York: John Wiley & Sons.
Joshi, M., Chen, D., Liu, Y., Weld, D.S., Zettlemoyer, L., & Levy, O. (2020). SpanBERT: Improving Pre-training by Representing and Predicting Spans. Transactions of the Association for Computational Linguistics, 8, 64–77. https://doi.org/10.1162/tacl_a_00300.
Khan, K.S., Kunz, R., Kleijnen, J., & Antes, G. (2003). Five steps to conducting a systematic review. Journal of the royal society of medicine, 96(3), 118–121.
Khushhal, S., Majid, A., Abbas, S.A., Nadeem, M.S.A., & Shah, S. (2020). Question retrieval using combined queries in community question answering. Journal of Intelligent Information Systems, 55, 307–327. https://doi.org/10.1007/s10844-020-00612-x.
Kodra, L., & Kajo, E. (2017). Question Answering Systems: A Review on Present Developments, Challenges and Trends. International Journal of Advanced Computer Science and Applications, 8(9), 217–224. https://doi.org/10.14569/ijacsa.2017.080931.
Kolomiyets, O., & Moens, M.F. (2011). A survey on question answering technology from an information retrieval perspective. Information Sciences, 181(24), 5412–5434. https://doi.org/10.1016/j.ins.2011.07.047.
Liu, Y., Yi, X., Chen, R., & Song, Y. (2016). A Survey on Frameworks and Methods of Question Answering. Proceedings - 2016 3rd International Conference on Information Science and Control Engineering, ICISCE 2016, pp 115–119. https://doi.org/10.1109/ICISCE.2016.35.
Malviya, M., & Soni, M. (2020). Question answering schemes: A review. International Journal of Scientific Research & Engineering Trends, 6(4), 2641–2648.
Mishra, A., & Jain, S.K. (2016). A survey on question answering systems with classification. Journal of King Saud University - Computer and Information Sciences, 28(3), 345–361. https://doi.org/10.1016/j.jksuci.2014.10.007.
Noraset, T., Lowphansirikul, L., & Tuarob, S. (2021). Wabiqa: A wikipedia-based thai question-answering system. Information Processing & Management, 58 (1), 102431. https://doi.org/10.1016/j.ipm.2020.102431.
Ouzzani, M., Hammady, H., Fedorowicz, Z., & Elmagarmid, A. (2016). Rayyan—a web and mobile app for systematic reviews. Systematic reviews, 5(1), 210.
Papadakis, M., & Tzitzikas, Y. (2015). Answering keyword queries through cached subqueries in best match retrieval models. Journal of Intelligent Information System, 44(1), 67–106. https://doi.org/10.1007/s10844-014-0330-7.
Seers, K. (2012). Qualitative data analysis. Evidence-based nursing, 15(1), 2–2.
Shah, A.A., Ravana, S.D., Hamid, S., & Ismail, M.A. (2019). Accuracy evaluation of methods and techniques in Web-based question answering systems: a survey. Knowledge and Information Systems, 58(3), 611–650. https://doi.org/10.1007/s10115-018-1203-0.
Shen, S., Dong, Z., Ye, J., Ma, L., Yao, Z., Gholami, A., Mahoney, M.W., & Keutzer, K. (2020). Q-bert: Hessian based ultra low precision quantization of bert. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05), 8815–8821. https://doi.org/10.1609/aaai.v34i05.6409, https://ojs.aaai.org/index.php/AAAI/article/view/6409.
Specia, L., Scarton, C., & Paetzold, G.H. (2018). Quality estimation for machine translation. Synthesis Lectures on Human Language Technologies, 11(1), 1–162.
Sultana, T., & Badugu, S. (2020). A review on different question answering system approaches. In S.C. Satapathy, K.S. Raju, K. Shyamala, D.R. Krishna, & M.N. Favorskaya (Eds.) Advances in Decision Sciences, Image Processing, Security and Computer Vision, pp 579–586. Springer International Publishing, Cham.
Surdeanu, M., Ciaramita, M., & Zaragoza, H. (2008). Learning to rank answers on large online QA collections. In ACL-08: HLT - 46th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference, pp 719–727.
Tranfield, D., Denyer, D., & Smart, P. (2003). Towards a methodology for developing evidence-informed management knowledge by means of systematic review. British journal of management, 14(3), 207–222.
Wang, W., Yang, N., Wei, F., Chang, B., & Zhou, M. (2017). Gated self-matching networks for reading comprehension and question answering. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp 189–198. Association for Computational Linguistics, Vancouver, Canada. https://www.aclweb.org/anthology/P17-1018.
Wu, Y., Hori, C., Kashioka, H., & Kawai, H. (2015). Leveraging social Q&A collections for improving complex question answering. Computer Speech and Language, 29(1), 1–19. https://doi.org/10.1016/j.csl.2014.06.001.
Yan, Z., & Zhou, J. (2015). Optimal answerer ranking for new questions in community question answering. Information Processing & Management, 51 (1), 163–178. https://doi.org/10.1016/j.ipm.2014.07.009.
Yang, L., Ai, Q., Spina, D., Chen, R.C., Pang, L., Bruce Croft, W., Guo, J., & Scholer, F. (2016). Beyond factoid QA: Effective methods for non-factoid answer sentence retrieval. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 9626, pp 115–128. Springer Verlag.
Yogish, D., Manjunath, T.N., & Hegadi, R.S. (2018). Survey on trends and methods of an intelligent answering system. International Conference on Electrical, Electronics, Communication Computer Technologies and Optimization Techniques, ICEECCOT 2017, 2018-Janua:346–353. https://doi.org/10.1109/ICEECCOT.2017.8284526.
Young, T., Hazarika, D., Poria, S., & Cambria, E. (2018). Recent trends in deep learning based natural language processing [review article]. IEEE Computational Intelligence Magazine, 13(3), 55–75. https://doi.org/10.1109/MCI.2018.2840738.
This study was financed in part by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - Brasil (CAPES) - Finance Code 001.
Consent to participate
There is the consent of all authors.
Consent for Publication
There is the consent of all authors.
Conflicts of Interest/Competing Interests
There are no conflicts or competing interests. We ensure that this manuscript has a novel contribution and has not been published or submitted to any other publisher before.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
This study was financed in part by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior -Brasil (CAPES) - Finance Code 001.
About this article
Cite this article
Cortes, E.G., Woloszyn, V., Barone, D. et al. A systematic review of question answering systems for non-factoid questions. J Intell Inf Syst (2021). https://doi.org/10.1007/s10844-021-00655-8
- Systematic review
- Non-factoid question
- Question answering