A Comparison of Hard Filters and Soft Evidence for Answer Typing in Watson

  • Chris Welty
  • J. William Murdock
  • Aditya Kalyanpur
  • James Fan
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7650)


Questions often explicitly request a particular type of answer. One popular approach to answering natural language questions involves filtering candidate answers based on precompiled lists of instances of common answer types (e.g., countries, animals, foods, etc.). Such a strategy is poorly suited to an open domain in which there is an extremely broad range of types of answers, and the most frequently occurring types cover only a small fraction of all answers. In this paper we present an alternative approach called TyCor, that employs soft filtering of candidates using multiple strategies and sources. We find that TyCor significantly outperforms a single-source, single-strategy hard filtering approach, demonstrating both that multi-source multi-strategy outperforms a single source, single strategy, and that its fault tolerance yields significantly better performance than a hard filter.


Type Word Question Answering Answer Type Question Analysis Candidate Answer 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Hirschman, L., Gaizauskas, R.: Natural language question answering: the view from here. Nat. Lang. Eng. 7(4), 275–300 (2001)CrossRefGoogle Scholar
  2. 2.
    Ferrucci, D., Brown, E., Chu-Carroll, J., Fan, J., Gondek, D., Kalyanpur, A., Lally, A., Murdock, J.W., Nyberg, E., Prager, J., Schlaefer, N., Welty, C.: Building watson: An overview of the deepqa project. AI Magazine, 59–79 (2010)Google Scholar
  3. 3.
    Kaufmann, E., Bernstein, A., Fischer, L.: NLP-Reduce: A ”naive” but Domain-independent Natural Language Interface for Querying Ontologies (2007)Google Scholar
  4. 4.
    Lopez, V., Uren, V., Sabou, M., Motta, E.: Is question answering fit for the semantic web? a survey. Semantic Web? Interoperability, Usability, Applicability 2(2), 125–155 (2011)Google Scholar
  5. 5.
    Prager, J., Brown, E., Coden, A., Radev, D.: Question-answering by predictive annotation. In: Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2000, pp. 184–191. ACM, New York (2000)CrossRefGoogle Scholar
  6. 6.
    Miller, G.A.: Wordnet: a lexical database for english. Commun. ACM 38(11), 39–41 (1995)CrossRefGoogle Scholar
  7. 7.
    Pustejovsky, J.: Type coercion and lexical selection. In: Semantics and the Lexicon. Kluwer Academic Publishers, Dordrecht (1993)CrossRefGoogle Scholar
  8. 8.
    Lally, A., Prager, J.M., McCord, M.C., Boguraev, B.K., Patwardhan, S., Fan, J., Fodor, P., Chu-Carroll, J.: Question analysis: How watson reads a clue. IBM Journal of Research and Development 56(3.4), 2:1 –2:14 (2012)Google Scholar
  9. 9.
    Chu-Carroll, J., Fan, J., Boguraev, B.K., Carmel, D., Sheinwald, D., Welty, C.: Finding needles in the haystack: Search and candidate generation. IBM Journal of Research and Development 56(3.4), 6:1 –6:12 (2012)Google Scholar
  10. 10.
    Gondek, D., Lally, A., Kalyanpur, A., Murdock, J., Duboue, P., Zhang, L., Pan, Y., Qiu, Z., Welty, C.: Finding needles in the haystack: Search and candidate generation. IBM Journal of Research and Development 56(3.4), 14:1 –14:12 (2012)Google Scholar
  11. 11.
    Kalyanpur, A., Murdock, J.W., Fan, J., Welty, C.A.: Leveraging Community-Built Knowledge for Type Coercion in Question Answering. In: Aroyo, L., Welty, C., Alani, H., Taylor, J., Bernstein, A., Kagal, L., Noy, N., Blomqvist, E. (eds.) ISWC 2011, Part II. LNCS, vol. 7032, pp. 144–156. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  12. 12.
    Murdock, J.W., Kalyanpur, A., Welty, C., Fan, J., Ferrucci, D.A., Gondek, D.C., Zhang, L., Kanayama, H.: Typing candidate answers using type coercion. IBM Journal of Research and Development 56(3.4), 7:1 –7:13 (2012)Google Scholar
  13. 13.
    Hearst, M.A.: Automatic acquisition of hyponyms from large text corpora. In: Proceedings of the 14th Conference on Computational Linguistics, COLING 1992, vol. 2, pp. 539–545. Association for Computational Linguistics, Stroudsburg (1992)CrossRefGoogle Scholar
  14. 14.
    Suchanek, F.M., Kasneci, G., Weikum, G.: Yago: a core of semantic knowledge. In: Proceedings of the 16th International Conference on World Wide Web, WWW 2007, pp. 697–706. ACM, New York (2007)CrossRefGoogle Scholar
  15. 15.
    Fan, J., Kalyanpur, A., Gondek, D.C., Ferrucci, D.A.: Automatic knowledge extraction from documents. IBM Journal of Research and Development 56(3.4), 5:1 –5:10 (2012)Google Scholar
  16. 16.
    Voorhees, E. (ed.): Overview of the TREC 2006 Conference, Gaithersburg, MD (2006)Google Scholar
  17. 17.
    Schlobach, S., Ahn, D., de Rijke, M., Jijkoun, V.: Data-driven type checking in open domain question answering. J. Applied Logic 5(1), 121–143 (2007)CrossRefGoogle Scholar
  18. 18.
    Grappy, A., Grau, B.: Answer type validation in question answering systems. In: Adaptivity, Personalization and Fusion of Heterogeneous Information, RIAO 2010, Paris, France, France, Le Centre De Hautes Etudes Internationales D’Informatique Documentaire, pp. 9–15 (2010)Google Scholar
  19. 19.
    Aktolga, E., Allan, J., Smith, D.A.: Passage Reranking for Question Answering Using Syntactic Structures and Answer Types. In: Clough, P., Foley, C., Gurrin, C., Jones, G.J.F., Kraaij, W., Lee, H., Mudoch, V. (eds.) ECIR 2011. LNCS, vol. 6611, pp. 617–628. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  20. 20.
    Buscaldi, D., Rosso, P.: Mining Knowledge from Wikipedia from the question answering task. In: Proceedings of the 5th International Conference on Language Resources and Evaluation (LREC 2006), Genoa, Italy (2006)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Chris Welty
    • 1
  • J. William Murdock
    • 1
  • Aditya Kalyanpur
    • 1
  • James Fan
    • 1
  1. 1.IBM ResearchUSA

Personalised recommendations