Skip to main content

A systematic review of question answering systems for non-factoid questions


Question Answering (QA) is a field of study addressed to develop automatic methods for answering questions expressed in natural language. Recently, the emergence of the new generation of intelligent assistants, such as Siri, Alexa, and Google Assistant, has intensified the importance of an effective and efficient QA system able to handle questions with different complexities. Regarding the type of question to be answered, QA systems have been divided into two sub-areas: (i) factoid questions that require a single fact – e.g., a name of a person or a date, and (ii) non-factoid questions that need a more complex answer – e.g., descriptions, opinions, or explanations. While factoid QA systems have overcome human performance on some benchmarks, automatic systems for answering non-factoid questions remain a challenge and an open research problem. This work provides an overview of recent research addressing non-factoid questions. It focuses on which methods have been applied in each task, the data sets available, challenges and limitations, and possible research directions. From a total of 455 recent studies, we selected 75 papers based on our quality control system and exclusion criteria for an in-depth analysis. This systematic review helped to answer what are the tasks and methods involved in non-factoid, what are the data sets available, what the limitations are, and what is the recommendations for future research.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3


  1. 1.

    Stanford Question Answering Dataset (SQuAD) is a reading comprehension dataset (


  1. Agichtein, E., Carmel, D., Pelleg, D., Pinter, Y., & Harman, D. (2015). Overview of the trec 2015 liveqa track.. In TREC.

  2. Bae, K., & Ko, Y. (2019). Efficient question classification and retrieval using category information and word embedding on cQA services. Journal of Intelligent Information Systems, 53(1), 27–49.

    Article  Google Scholar 

  3. Bau, D., Liu, S., Wang, T., Zhu, J-Y, & Torralba, A. (2020). Rewriting a deep generative model. In A Vedaldi, H Bischof, T Brox, & J-M Frahm (Eds.) Computer Vision – ECCV 2020, pp 351–369. Springer International Publishing. Cham.

  4. Ben Abacha, A, & Zweigenbaum, P. (2015). Means: A medical question-answering system combining nlp techniques and semantic web technologies. Information Processing & Management, 51 (5), 570–594.

    Article  Google Scholar 

  5. Bondarenko, A., Braslavski, P., Völske, M, Aly, R., Fröbe, M, Panchenko, A., Biemann, C., Stein, B., & Hagen, M. (2020). Comparative web search questions. In Proceedings of the 13th International Conference on Web Search and Data Mining, WSDM ’20, pp 52–60. Association for Computing Machinery, New York, NY, USA.

  6. Calijorne Soares, M.A., & Parreiras, F.S. (2020). A literature review on question answering techniques, paradigms and systems. Journal of King Saud University - Computer and Information Sciences, 32(6), 635–646.

    Article  Google Scholar 

  7. Chali, Y., Hasan, S.A., & Mojahid, M. (2015). A reinforcement learning formulation to the complex question answering problem. Information Processing & Management, 51(3), 252–272.,

    Article  Google Scholar 

  8. Cohen, D., Yang, L., & Croft, W.B. (2018). WikiPassageQA: A benchmark collection for research on non-factoid answer passage retrieval. 41st International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2018, pp 1165–1168.

  9. Corbin, J., & Strauss, A. (2014). Basics of qualitative research: Techniques and procedures for developing grounded theory. Sage publications.

  10. Cortes, E., Woloszyn, V., Binder, A., Himmelsbach, T., Barone, D., & Möller, S (May 2020). An empirical comparison of question classification methods for question answering systems. In Proceedings of the 12th Language Resources and Evaluation Conference, pp 5408–5416. European Language Resources Association, Marseille, France.

  11. Denyer, D., & Tranfield, D. (2009). Producing a systematic review. Sage Publications Ltd, 671—689.

  12. Dimitrakis, E., Sgontzos, K., & Tzitzikas, Y. (2019). A survey on question answering systems over linked data and documents. Journal of Intelligent Information Systems, 55, 233–259.

    Article  Google Scholar 

  13. Dybå, T, & Dingsøyr, T (2008). Empirical studies of agile software development: A systematic review. Information and software technology, 50(9-10), 833–859.

    Article  Google Scholar 

  14. Hazrina, S., Sharef, N.M., Ibrahim, H., Murad, M.A.A., & Noah, S.A.M. (2017). Review on the advancements of disambiguation in semantic question answering system. Information Processing & Management, 53 (1), 52–69.,

    Article  Google Scholar 

  15. Hermjakob, U., Echihabi, A., & Marcu, D. (2002). Natural language based reformulation resource and web exploitation for question answering. In Proceedings of TREC, 11. Citeseer.

  16. Higgins, J.P.T., Thomas, J., Chandler, J., Cumpston, M., Li, T., Page, M.J., & Welch, V.A. (2019). Cochrane handbook for systematic reviews of interventions. New York: John Wiley & Sons.

    Book  Google Scholar 

  17. Joshi, M., Chen, D., Liu, Y., Weld, D.S., Zettlemoyer, L., & Levy, O. (2020). SpanBERT: Improving Pre-training by Representing and Predicting Spans. Transactions of the Association for Computational Linguistics, 8, 64–77.

    Article  Google Scholar 

  18. Khan, K.S., Kunz, R., Kleijnen, J., & Antes, G. (2003). Five steps to conducting a systematic review. Journal of the royal society of medicine, 96(3), 118–121.

    Article  Google Scholar 

  19. Khushhal, S., Majid, A., Abbas, S.A., Nadeem, M.S.A., & Shah, S. (2020). Question retrieval using combined queries in community question answering. Journal of Intelligent Information Systems, 55, 307–327.

    Article  Google Scholar 

  20. Kodra, L., & Kajo, E. (2017). Question Answering Systems: A Review on Present Developments, Challenges and Trends. International Journal of Advanced Computer Science and Applications, 8(9), 217–224.

    Article  Google Scholar 

  21. Kolomiyets, O., & Moens, M.F. (2011). A survey on question answering technology from an information retrieval perspective. Information Sciences, 181(24), 5412–5434.

    MathSciNet  Article  Google Scholar 

  22. Liu, Y., Yi, X., Chen, R., & Song, Y. (2016). A Survey on Frameworks and Methods of Question Answering. Proceedings - 2016 3rd International Conference on Information Science and Control Engineering, ICISCE 2016, pp 115–119.

  23. Malviya, M., & Soni, M. (2020). Question answering schemes: A review. International Journal of Scientific Research & Engineering Trends, 6(4), 2641–2648.

    Google Scholar 

  24. Mishra, A., & Jain, S.K. (2016). A survey on question answering systems with classification. Journal of King Saud University - Computer and Information Sciences, 28(3), 345–361.

    Article  Google Scholar 

  25. Noraset, T., Lowphansirikul, L., & Tuarob, S. (2021). Wabiqa: A wikipedia-based thai question-answering system. Information Processing & Management, 58 (1), 102431.

    Article  Google Scholar 

  26. Ouzzani, M., Hammady, H., Fedorowicz, Z., & Elmagarmid, A. (2016). Rayyan—a web and mobile app for systematic reviews. Systematic reviews, 5(1), 210.

    Article  Google Scholar 

  27. Papadakis, M., & Tzitzikas, Y. (2015). Answering keyword queries through cached subqueries in best match retrieval models. Journal of Intelligent Information System, 44(1), 67–106.

    Article  Google Scholar 

  28. Seers, K. (2012). Qualitative data analysis. Evidence-based nursing, 15(1), 2–2.

    Article  Google Scholar 

  29. Shah, A.A., Ravana, S.D., Hamid, S., & Ismail, M.A. (2019). Accuracy evaluation of methods and techniques in Web-based question answering systems: a survey. Knowledge and Information Systems, 58(3), 611–650.

    Article  Google Scholar 

  30. Shen, S., Dong, Z., Ye, J., Ma, L., Yao, Z., Gholami, A., Mahoney, M.W., & Keutzer, K. (2020). Q-bert: Hessian based ultra low precision quantization of bert. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05), 8815–8821.,

    Article  Google Scholar 

  31. Specia, L., Scarton, C., & Paetzold, G.H. (2018). Quality estimation for machine translation. Synthesis Lectures on Human Language Technologies, 11(1), 1–162.

    Article  Google Scholar 

  32. Sultana, T., & Badugu, S. (2020). A review on different question answering system approaches. In S.C. Satapathy, K.S. Raju, K. Shyamala, D.R. Krishna, & M.N. Favorskaya (Eds.) Advances in Decision Sciences, Image Processing, Security and Computer Vision, pp 579–586. Springer International Publishing, Cham.

  33. Surdeanu, M., Ciaramita, M., & Zaragoza, H. (2008). Learning to rank answers on large online QA collections. In ACL-08: HLT - 46th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference, pp 719–727.

  34. Tranfield, D., Denyer, D., & Smart, P. (2003). Towards a methodology for developing evidence-informed management knowledge by means of systematic review. British journal of management, 14(3), 207–222.

    Article  Google Scholar 

  35. Wang, W., Yang, N., Wei, F., Chang, B., & Zhou, M. (2017). Gated self-matching networks for reading comprehension and question answering. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp 189–198. Association for Computational Linguistics, Vancouver, Canada.

  36. Wu, Y., Hori, C., Kashioka, H., & Kawai, H. (2015). Leveraging social Q&A collections for improving complex question answering. Computer Speech and Language, 29(1), 1–19.

    Article  Google Scholar 

  37. Yan, Z., & Zhou, J. (2015). Optimal answerer ranking for new questions in community question answering. Information Processing & Management, 51 (1), 163–178.

    Article  Google Scholar 

  38. Yang, L., Ai, Q., Spina, D., Chen, R.C., Pang, L., Bruce Croft, W., Guo, J., & Scholer, F. (2016). Beyond factoid QA: Effective methods for non-factoid answer sentence retrieval. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 9626, pp 115–128. Springer Verlag.

  39. Yogish, D., Manjunath, T.N., & Hegadi, R.S. (2018). Survey on trends and methods of an intelligent answering system. International Conference on Electrical, Electronics, Communication Computer Technologies and Optimization Techniques, ICEECCOT 2017, 2018-Janua:346–353.

  40. Young, T., Hazarika, D., Poria, S., & Cambria, E. (2018). Recent trends in deep learning based natural language processing [review article]. IEEE Computational Intelligence Magazine, 13(3), 55–75.

    Article  Google Scholar 

Download references


This study was financed in part by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - Brasil (CAPES) - Finance Code 001.

Author information




The idea of the systematic review on Non-factoid Question Answering came from the author Eduardo G. Cortes. All authors contributed to the study design. The survey of the analyzed studies and their analysis was carried out by Eduardo G. Cortes and Vinicius Woloszyn with Dante Barone’s supervision, Renata Vieira and Sebastian Möller. The first draft of the manuscript was written, evaluated, and corrected by all authors Eduardo G. Cortes, Vinicius Woloszyn, Dante Barone, Renata Vieira, and Sebastian Möller. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Eduardo Gabriel Cortes.

Ethics declarations

Consent to participate

There is the consent of all authors.

Consent for Publication

There is the consent of all authors.

Conflicts of Interest/Competing Interests

There are no conflicts or competing interests. We ensure that this manuscript has a novel contribution and has not been published or submitted to any other publisher before.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This study was financed in part by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior -Brasil (CAPES) - Finance Code 001.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Cortes, E.G., Woloszyn, V., Barone, D. et al. A systematic review of question answering systems for non-factoid questions. J Intell Inf Syst (2021).

Download citation


  • Systematic review
  • Non-factoid question
  • Question answering