Skip to main content
SpringerLink
Log in

Accuracy evaluation of methods and techniques in Web-based question answering systems: a survey

  • Regular Paper
  • Published:
Knowledge and Information Systems Aims and scope Submit manuscript

Abstract

Question answering (QA) systems answer the queries of users efficiently in the least amount of time. A researcher has to decide which among various methods and techniques available will be used to retrieve accurate answers when developing a QA system. This step creates an overhead before making a selection. The study highlights the methods and techniques that perform well in terms of the accuracy of answers provided. Nine Web-based question answering systems were consulted, and the applied methods and techniques evaluated on the basis of the percentage of questions correctly answered and the mean reciprocal rank evaluation measures. Results were discussed using three key stages involved in a QA system: answer extraction, scoring of answers, and answer aggregation. Results show some techniques have higher accuracy of answers than others. Not all methods in QA systems can improve the accuracy of answers individually, but the methods used in combination obtain greater effect. The results can be used to select methods and techniques optimal for producing highly accurate scores without spending time on benchmarking.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  1. Hirschman L, Gaizauskas R (2001) Natural language question answering: the view from here. Nat Lang Eng 7(04):275–300

    Article  Google Scholar 

  2. Gupta P, Gupta V (2012) A survey of text question answering techniques. Int J Comp Appl 53(4):1–8

    Google Scholar 

  3. McCallum A (2005) Information extraction: distilling structured data from unstructured text. Queue 3(9):48–57

    Article  Google Scholar 

  4. Ng JP, Kan M-Y (2010) QANUS: an open-source question-answering platform. Unfiled technical report: (Posted to cs-IR and cs-CL). Preprint arXiv:1501.00311. Accessed 14th June 2015

  5. Marriam-Webster Dictionary (2006) Merriam-Webster’s collegiate dictionary, 11th edn. Merriam-Webster, Springfield

    Google Scholar 

  6. Kolomiyets O, Moens M-F (2011) A survey on question answering technology from an information retrieval perspective. Inf Sci 181(24):5412–5434

    Article  MathSciNet  Google Scholar 

  7. Wang M (2006) A survey of answer extraction techniques in factoid question answering. Comput Linguist 1(1)

  8. Shim B, Ko Y, Seo J (2005) Extracting and utilizing of IS-A relation patterns for question answering systems. In: Lee G, Yamada A, Meng H, Myaeng S (eds) Information retrieval technology, vol 3689. Lecture notes in computer science. Springer, Heidelberg, pp 697–702. https://doi.org/10.1007/11562382_70

    Google Scholar 

  9. Li X, Roth D (2002) Learning question classifiers. In: Proceedings of the 19th international conference on computational linguistics. Association for Computational Linguistics, Taipei, 1072378, pp 1–7. https://doi.org/10.3115/1072228.1072378

  10. Molino P, Lops P, Semeraro G, de Gemmis M, Basile P (2015) Playing with knowledge: a virtual player for “Who Wants to Be a Millionaire?” that leverages question answering techniques. Artif Intell 222:157–181

    Article  Google Scholar 

  11. Abney S, Collins M, Singhal A (2000) Answer extraction. In: Proceedings of the sixth conference on Applied natural language processing, Seattle, Washington, 2000. Association for Computational Linguistics, 974188, pp 296–301. https://doi.org/10.3115/974147.974188

  12. Chen J, Diekema AR, Taffet MD, McCracken N, Ozgencil NE (2000) Question answering: CNLP at the TREC-10 question answering track

  13. Harabagiu SM, Moldovan DI, Paşca M, Mihalcea R, Surdeanu M, Bunescu R, Gîrju CR, Rus V, Morărescu P (2000) Falcon: boosting knowledge for answer engines

  14. Hovy EH, Gerber L, Hermjakob U, Junk M, Lin C-Y (2000) Question answering in Webclopedia. In: TREC, pp 53–56

  15. Hovy EH, Hermjakob U, Lin C-Y (2001) The use of external knowledge of factoid QA. In: Proceedings of the TREC-10 conference, NIST, Gaithersburg, MD, 2001, pp 166–174

  16. Pasca MA, Harabagiu SM (2001) High performance question/answering. In: Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, New Orleans, Louisiana, USA, 2001. ACM, 384025, pp 366–374. https://doi.org/10.1145/383952.384025

  17. Prager J, Chu-Carroll J, Brown EW, Czuba K (2006) Question answering by predictive annotation. In: Advances in open domain question answering. Springer, Berlin, pp 307–347

  18. Ferrucci D, Brown E, Chu-Carroll J, Fan J, Gondek D, Kalyanpur AA, Lally A, Murdock JW, Nyberg E, Prager J (2010) Building Watson: an overview of the DeepQA project. AI Mag 31(3):59–79

    Article  Google Scholar 

  19. Wu M, Marian A (2011) A framework for corroborating answers from multiple web sources. Inf Syst 36(2):431–449

    Article  Google Scholar 

  20. Yang H, Chua T-S (2003) QUALIFIER: question answering by lexical fabric and external resources. In: Copestake A, Hajič J (eds) Proceedings of the tenth conference on European chapter of the association for computational linguistics, Budapest, Hungary, 2003. Association for Computational Linguistics, 1067855, pp 363–370. https://doi.org/10.3115/1067807.1067855

  21. Wu M, Marian A (2007) Corroborating answers from multiple web sources. In: Proceedings of the 10th international workshop on the web and databases, ACM Special Interest Group on Management of Data, Beijing

  22. Kwok C, Etzioni O, Weld DS (2001) Scaling question answering to the web. ACM Trans Inf Syst 19(3):242–262

    Article  Google Scholar 

  23. Dumais S, Banko M, Brill E, Lin J, Ng A (2002) Web question answering: is more always better? In: Proceedings of the 25th annual international ACM SIGIR conference on research and development in information retrieval. ACM, Tampere, 564428, pp 291–298. https://doi.org/10.1145/564376.564428

  24. Yang H, Chua T-S (2002) The integration of lexical knowledge and external resources for question answering. In: Voorhees EM, Buckland LP (eds) Text retrieval conference. Department of Commerce, National Institute of Standards and Technology, Gaithersburg

  25. Zhang D, Lee WS A (2003) Web-based question answering system. In: Proceedings of the Singapore-MIT alliance annual symposium. Singapore-MIT Alliance, Singapore, pp 1–5

  26. Radev D, Fan W, Qi H, Wu H, Grewal A (2005) Probabilistic question answering on the web. J Am Soc Inform Sci Technol 56(6):571–583

    Article  Google Scholar 

  27. Liu Z-J, Wang X-L, Chen Q-C, Zhang Y-Y, Xiang Y (2014) A Chinese question answering system based on web search. In: International conference on machine learning and cybernetics. IEEE Computer Society, Lanzhou, pp 816–820. https://doi.org/10.1109/icmlc.2014.7009714

  28. Oh H-J, Ryu P-M, Kim H (2012) Which is the best? re-ranking answers merged from multiple web sources. J Emerg Technol Web Intell 4(1):35–42

    Google Scholar 

  29. Oh H-J, Yoon Y-C, Kim HK (2013) Finding more trustworthy answers: various trustworthiness factors in question answering. J Inf Sci 39(4):509–522

    Article  Google Scholar 

  30. Fader A, Zettlemoyer L, Etzioni O (2014) Open question answering over curated and extracted knowledge bases. In: Proceedings of the 20th ACM SIGKDD international conference on knowledge discovery and data mining. ACM, pp 1156–1165

  31. Mollá-Aliod D, Vicedo J-L (2010) Question answering. In: Indurkhya N, Damerau FJ (eds) Handbook of natural language processing, 2nd Edn. Chapman and Hall/CRC, pp 485–510

  32. Webber B, Webb N (2010) Question answering. In: Clark A, Fox C, Lappin S (eds) The handbook of computational linguistics and natural language processing, vol 1. Wiley-Blackwell, West Sussex, pp 630–654

  33. Allam AMN, Haggag MH (2012) The question answering systems: a survey. Int J Res Rev Inf Sci (IJRRIS) 2(3):211–221

    Google Scholar 

  34. Bouziane A, Bouchiha D, Doumi N, Malki M (2015) Question answering systems: survey and trends. Proc Comput Sci 73:366–375

    Article  Google Scholar 

  35. Lopez V, Uren V, Sabou M, Motta E (2011) Is question answering fit for the semantic web? a survey. Semant Web 2(2):125–155

    Google Scholar 

  36. Kalaivani S, Duraiswamy K (2012) Comparison of question answering systems based on ontology and semantic web in different environment. J Comput Sci 8(9):1407–1413

    Article  Google Scholar 

  37. Mollá D, Vicedo JL (2007) Question answering in restricted domains: an overview. Comput Linguist 33(1):41–61

    Article  Google Scholar 

  38. Simmons RF (1965) Answering English questions by computer: a survey. Commun ACM 8(1):53–70

    Article  Google Scholar 

  39. Green BF Jr, Wolf AK, Chomsky C, Laughery K (1961) Baseball: an automatic question-answerer. In: Papers presented at the May 9–11, 1961, western joint IRE-AIEE-ACM computer conference. ACM, pp 219–224

  40. Woods WA, Kaplan RM, Nash-Webber B (1972) The lunar sciences: natural language information system: final report. Bolt Beranek and Newman, Cambridge

    Google Scholar 

  41. Black FS (1964) A deductive question answering system. Harvard University, Cambridge

    Google Scholar 

  42. Copestake A, Jones KS (1990) Natural language interfaces to databases. Knowl Eng Rev 5(04):225–249

    Article  Google Scholar 

  43. Androutsopoulos I, Ritchie GD, Thanisch P (1995) Natural language interfaces to databases–an introduction. Nat Lang Eng 1(01):29–81

    Article  Google Scholar 

  44. Copestake A, Jones KS (1990) Natural language interfaces to databases. Knowl Eng Rev 5(4):225–249. https://doi.org/10.1017/S0269888900005476

    Article  Google Scholar 

  45. Pazos RRA, González BJJ, Aguirre LMA, Martínez FJA, Fraire HHJ (2013) Natural language interfaces to databases: an analysis of the state of the art. In: Castillo O, Melin P, Kacprzyk J (eds) Recent advances on hybrid intelligent systems. Studies in Computational Intelligence, vol 451. Springer, Berlin, Heidelberg, pp 463–480

  46. Nguyen A, Le H (2008) Natural language interface construction using semantic grammars. In: PRICAI 2008: trends in artificial intelligence, pp 728–739

  47. Owda M, Bandar Z, Crockett K (2007) Conversation-based natural language interface to relational databases. In: Proceedings of the 2007 IEEE/WIC/ACM international conferences on web intelligence and intelligent agent technology-workshops. IEEE Computer Society, pp 363–367

  48. Reddy V, Neumeier K, McFarlane J, Cothren J, Thompson CW (2007) Extending a natural language interface with geospatial queries. IEEE Internet Comput 11(6):82–85

    Article  Google Scholar 

  49. Popescu A-M, Armanasu A, Etzioni O, Ko D, Yates A (2004) Modern natural language interfaces to databases: composing statistical parsing with semantic tractability. In: Proceedings of the 20th international conference on computational linguistics. Association for Computational Linguistics, p 141

  50. Winograd T (1971) Procedures as a representation for data in a computer program for understanding natural language. DTIC document

  51. Shortliffe E (2012) Computer-based medical consultations: MYCIN, vol 2. Elsevier, Amsterdam

    Google Scholar 

  52. Tablan V, Damljanovic D, Bontcheva K (2008) A natural language query interface to structured information. In: The semantic web: research and applications, pp 361–375

  53. Damljanovic D, Agatonovic M, Cunningham H (2010) Natural language interfaces to ontologies: combining syntactic analysis and ontology-based lookup through the user interaction. In: The semantic web: research and applications, pp 106–120

  54. Kaufmann E, Bernstein A (2010) Evaluating the usability of natural language query languages and interfaces to semantic web knowledge bases. Web Semant Sci Serv Agents World Wide Web 8(4):377–393

    Article  Google Scholar 

  55. Brill E, Lin JJ, Banko M, Dumais ST, Ng AY (2001) Data-intensive question answering. In: Proceedings of the TREC-10 conference, NIST, Gaithersburg, MD, 2001, pp 183–189

  56. Clarke CL, Cormack GV, Lynam TR, Li C, McLearn G (2001) Web reinforced question answering (MultiTest experiments for TREC 2001). In: TREC

  57. Katz B (1997) Annotating the world wide web using natural language. In: Computer-assisted information searching on internet. Le Centre de Hautes Etudes Internationales D’informatique Documentaire, pp 136–155

  58. Voorhees EM (1999) The TREC-8 question answering track report. In: Trec, pp 77–82

  59. Alfonseca E, De Boni M, Jara-Valencia J-L, Manandhar S (2001) A prototype question answering system using syntactic and semantic information for answer retrieval. In: TREC

  60. Li X (2003) Syntactic features in question answering. In: Proceedings of the 26th annual international ACM SIGIR conference on research and development in information retrieval. ACM, pp 383–384

  61. Cui H, Sun R, Li K, Kan M-Y, Chua T-S (2005) Question answering passage retrieval using dependency relations. In: Proceedings of the 28th annual international ACM SIGIR conference on research and development in information retrieval. ACM, pp 400–407

  62. Kitchenham B (2004) Procedures for performing systematic reviews. Keele University and NICTA, Keele

    Google Scholar 

  63. Moldovan D, Clark C, Harabagiu S, Maiorano S (2003) Cogex: a logic prover for question answering. In: Proceedings of the 2003 conference of the North American chapter of the association for computational linguistics on human language technology—volume 1. Association for Computational Linguistics, pp 87–93

  64. Hovy E, Hermjakob U, Ravichandran D (2002) A question/answer typology with surface text patterns. In: Proceedings of the second international conference on human language technology research. Morgan Kaufmann Publishers Inc., pp 247–251

  65. Wu Y, Zhang R, Hu X, Kashioka H (2007) Learning unsupervised SVM classifier for answer selection in web question answering

  66. Brill E, Dumais S, Banko M (2002) An analysis of the AskMSR question-answering system. In: Proceedings of the ACL-02 conference on empirical methods in natural language processing—volume 10. Association for Computational Linguistics, pp 257–264

  67. Somasundaran S, Wilson T, Wiebe J, Stoyanov V (2007) QA with attitude: exploiting opinion type analysis for improving question answering in on-line discussions and the news. In: ICWSM

  68. Charniak E (2000) A maximum-entropy-inspired parser. In: Proceedings of the 1st North American chapter of the association for computational linguistics conference. Association for Computational Linguistics, pp 132–139

  69. Ittycheriah A, Franz M, Zhu W-J, Ratnaparkhi A, Mammone RJ (2000) IBM’s statistical question answering system. In: TREC

  70. Yang H, Chua T-S, Wang S, Koh C-K (2003) Structured use of external knowledge for event-based open domain question answering. In: Proceedings of the 26th annual international ACM SIGIR conference on research and development in information retrieval. ACM, pp 33–40

  71. Manning CD, Surdeanu M, Bauer J, Finkel J, Bethard SJ, McClosky D (2014) The stanford core NLP natural language processing toolkit. In: Proceedings of 52nd annual meeting of the association for computational linguistics: system demonstrations. Association for Computational Linguistics, Baltimore, June 22–27, 2014, pp 55–60

  72. Aggarwal S, Van Oostendorp H, Reddy YR, Indurkhya B (2014) Providing web credibility assessment support. In: Proceedings of the 2014 European conference on cognitive ergonomics. ACM, p 29

  73. Soubbotin MM, Soubbotin SM (2001) Patterns of potential answer expressions as clues to the right answers. In: TREC

  74. Ravichandran D, Hovy E (2002) Learning surface text patterns for a question answering system. In: Proceedings of the 40th annual meeting on association for computational linguistics. Association for Computational Linguistics, pp 41–47

  75. Shih C-W, Day M-Y, Tsai T-H, Jiang T-J, Wu C-W, Sung C-L, Chen Y-R, Wu S-H, Hsu W-L (2005) ASQA: academia sinica question answering system for NTCIR-5 CLQA. In: NTCIR-5 workshop, Tokyo, pp 202–208

  76. Barker K, Chaudhri VK, Chaw SY, Clark P, Fan J, Israel DJ, Mishra S, Porter BW, Romero P, Tecuci D (2004) A question-answering system for ap chemistry: assessing KR&R technologies. In: KR, pp 488–497

  77. Toutanova K, Manning CD (2000) Enriching the knowledge sources used in a maximum entropy part-of-speech tagger. In: Proceedings of the 2000 joint SIGDAT conference on empirical methods in natural language processing and very large corpora: held in conjunction with the 38th annual meeting of the association for computational linguistics—volume 13. Association for Computational Linguistics, pp 63–70

  78. Ittycheriah A, Franz M, Zhu W-J, Ratnaparkhi A, Mammone RJ (2001) Question answering using maximum entropy components. In: Proceedings of the second meeting of the North American chapter of the association for computational linguistics on language technologies. Association for Computational Linguistics, pp 1–7

Download references

Acknowledgements

This research was supported by UMRG Programme RP059A-17SBS from Universiti Malaya and the Ministry of Higher Education, Malaysia.

Author information

Authors and Affiliations

  1. Department of Computing, School of Electrical Engineering and Computer Science, National University of Science and Technology, Islamabad, Pakistan

    Asad Ali Shah

  2. Department of Information Systems, Faculty of Computer Science and Information Technology, University of Malaya, Kuala Lumpur, Malaysia

    Sri Devi Ravana, Suraya Hamid & Maizatul Akmar Ismail

Authors
  1. Asad Ali Shah
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sri Devi Ravana
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Suraya Hamid
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Maizatul Akmar Ismail
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sri Devi Ravana.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Shah, A.A., Ravana, S.D., Hamid, S. et al. Accuracy evaluation of methods and techniques in Web-based question answering systems: a survey. Knowl Inf Syst 58, 611–650 (2019). https://doi.org/10.1007/s10115-018-1203-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10115-018-1203-0

Keywords

Search

Navigation