Answer Extraction with Multiple Extraction Engines for Web-Based Question Answering

  • Hong Sun
  • Furu Wei
  • Ming Zhou
Part of the Communications in Computer and Information Science book series (CCIS, volume 496)


Answer Extraction of Web-based Question Answering aims to extract answers from snippets retrieved by search engines. Search results contain lots of noisy and incomplete texts, thus the task becomes more challenging comparing with traditional answer extraction upon off-line corpus. In this paper we discuss the important role of employing multiple extraction engines for Web-based Question Answering. Aggregating multiple engines could ease the negative effect from the noisy search results on single method. We adopt a Pruned Rank Aggregation method which performs pruning while aggregating candidate lists provided by multiple engines. It fully leverages redundancies within and across each list for reducing noises in candidate list without hurting answer recall. In addition, we rank the aggregated list with a Learning to Rank framework with similarity, redundancy, quality and search features. Experiment results on TREC data show that our method is effective for reducing noises in candidate list, and greatly helps to improve answer ranking results. Our method outperforms state-of-the-art answer extraction method, and is sufficient in dealing with the noisy search snippets for Web-based QA.


Web-based Question Answering Answer Extraction Rank Aggregation Learning to Rank 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Brill, E., Lin, J., Banko, M., Dumais, S., Ng, A.: Data-intensive question answering. In: TREC, pp. 393–400 (2001)Google Scholar
  2. 2.
    Yao, X., Van Durme, B., Callison-Burch, C., Clark, P.: Answer extraction as sequence tagging with tree edit distance. In: HLT-NAACL, pp. 858–867 (2013)Google Scholar
  3. 3.
    Severyn, A., Moschitti, A.: Automatic feature engineering for answer selection and extraction. In: EMNLP, pp. 458–467 (2013)Google Scholar
  4. 4.
    Sun, H., Duan, N., Duan, Y., Zhou, M.: Answer extraction from passage graph for question answering. In: Proceedings of the Twenty-Third International Joint Conference on Artificial Intelligence, pp. 2169–2175. AAAI Press (2013)Google Scholar
  5. 5.
    Xu, J., Licuanan, A., May, J., Miller, S., Weischedel, R.: Answer selection and confidence estimation. In: 2003 AAAI Symposium on New Directions in QA (2003)Google Scholar
  6. 6.
    Ravichandran, D., Ittycheriah, A., Roukos, S.: Automatic derivation of surface text patterns for a maximum entropy based question answering system. In: Proceedings of HLT-NAACL (2003)Google Scholar
  7. 7.
    Sasaki, Y.: Question answering as question-biased term extraction: A new approach toward multilingual qa. In: Proceedings of ACL, pp. 215–222 (2005)Google Scholar
  8. 8.
    Bunescu, R., Huang, Y.: Towards a general model of answer typing: Question focus identification. In: Proceedings of the 11th International Conference on Intelligent Text Processing and Computational Linguistics, RCS Volume, pp. 231–242 (2010)Google Scholar
  9. 9.
    Chu-Carroll, J., Fan, J.: Leveraging wikipedia characteristics for search and candidate generation in question answering. In: Proceedings of AAAI (2011)Google Scholar
  10. 10.
    Lin, J.: An exploration of the principles underlying redundancy-based factoid question answering. ACM Transactions on Information Systems 25(2), 6 (2007)CrossRefGoogle Scholar
  11. 11.
    Subbian, K., Melville, P.: Supervised rank aggregation for predicting influence in networks. arXiv preprint arXiv:1108.4801 (2011)Google Scholar
  12. 12.
    Agarwal, A., Raghavan, H., Subbian, K., Melville, P., Lawrence, R.D., Gondek, D.C., Fan, J.: Learning to rank for robust question answering. In: Proceedings of the 21st ACM International Conference on Information and Knowledge Management, pp. 833–842. ACM (2012)Google Scholar
  13. 13.
    Joachims, T.: Training linear svms in linear time. In: Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 217–226. ACM (2006)Google Scholar
  14. 14.
    Ferrucci, D., Brown, E., Chu-Carroll, J., Fan, J., Gondek, D., Kalyanpur, A.A., Lally, A., Murdock, J.W., Nyberg, E., Prager, J., et al.: Building watson: An overview of the deepqa project. AI Magazine 31(3), 59–79 (2010)Google Scholar
  15. 15.
    Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., Kuksa, P.P.: Natural language processing (almost) from scratch. Journal of Machine Learning Research 12, 2493–2537 (2011)MATHGoogle Scholar
  16. 16.
    Shi, S., Liu, X., Wen, J.R.: Pattern-based semantic class discovery with multi-membership support. In: Proceedings of the 17th ACM Conference on Information and Knowledge Management, pp. 1453–1454. ACM (2008)Google Scholar
  17. 17.
    Lafferty, J., McCallum, A., Pereira, F.C.: Conditional random fields: Probabilistic models for segmenting and labeling sequence data (2001)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  • Hong Sun
    • 1
  • Furu Wei
    • 2
  • Ming Zhou
    • 2
  1. 1.School of Computer Science and TechnologyTianjin UniversityTianjinChina
  2. 2.Microsoft Research AsiaBeijingChina

Personalised recommendations