FactBank: a corpus annotated with event factuality

  • Roser SauríEmail author
  • James Pustejovsky


Recent work in computational linguistics points out the need for systems to be sensitive to the veracity or factuality of events as mentioned in text; that is, to recognize whether events are presented as corresponding to actual situations in the world, situations that have not happened, or situations of uncertain interpretation. Event factuality is an important aspect of the representation of events in discourse, but the annotation of such information poses a representational challenge, largely because factuality is expressed through the interaction of numerous linguistic markers and constructions. Many of these markers are already encoded in existing corpora, albeit in a somewhat fragmented way. In this article, we present FactBank, a corpus annotated with information concerning the factuality of events. Its annotation has been carried out from a descriptive framework of factuality grounded on both theoretical findings and data analysis. FactBank is built on top of TimeBank, adding to it an additional level of semantic information.


Event factuality Modality Certainty Subjectivity analysis Corpus creation TimeBank 



We are very grateful to Marc Verhagen, Toni Badia, Lauri Karttunen, Rick Alterman, Sabine Bergler, Adam Meyers, and Silvia Pareti for their valuable comments and helpful discussion regarding this research. We also want to extend thanks to four anonymous reviewers for their constructive suggestions, which helped improve the original manuscript. All errors and mistakes are responsibility of the authors. This work is been supported by a grant to Prof. Pustejovsky, NAVAIR Contract No. N61339-06-C-0140.


  1. ACE (2008). ACE (Automatic Content Extraction) English annotation guidelines for relations (Version 6.0 – 2008.01.07 ed.). Linguistic Data Consortium.
  2. Aikhenvald, A. Y. (2004). Evidentiality. Oxford, England: Oxford University Press.Google Scholar
  3. Andreevskaia, A., & Bergler, S. (2006). Mining WordNet for fussy sentiment: Sentiment tag extraction from WordNet glosses. In Proceedings of the 11th conference of the European chapter of the Association for the Computational Linguistics, EACL-2006.Google Scholar
  4. Asher, N. (1993). Reference to abstract objects in English. Dordrecht, The Netherlands: Kluwer Academic Press.Google Scholar
  5. Bach, K., & Harnish, R. M. (1979). Linguistic communication and speech acts. Cambridge, Massachusetts, USA: The MIT Press.Google Scholar
  6. Baker, C. F., Fillmore, C. J., & Lowe, J. B. (1998). The Berkeley FrameNet project. In 17th International conference on computational linguistics (pp. 86–90).Google Scholar
  7. Bergler, S. (1992). Evidential analysis of reported speech. PhD thesis, Brandeis University.Google Scholar
  8. Bethard, S., Yu, H., Thornton, A., Hatzivassiloglou, V., & Jurafsky, D. (2004). Automatic extraction of opinion propositions and their holders. In Proceedings of AAAI spring symposium on exploring attitude and affect in text.Google Scholar
  9. Biber, D., & Finegan, E. (1989). Styles of stance in English: Lexical and grammatical marking of evidentiality and affect. Text, 9(1), 93–124.Google Scholar
  10. Carlson, L., Marcu, D., & Okurowski, M. E. (2003). Building a discourse-tagged corpus in the framework of Rhetorical Structure Theory. In J. v. Kuppevelt & R. W. Smith (Eds.), Current and new directions in discourse and dialogue. Springer.Google Scholar
  11. Chafe, W. (1986). Evidentiality in English conversation and academic writing. In W. Chafe & J. Nichols (Eds.), Evidentiality: The linguistic coding of epistemology. Norwood, New Jersey, USA: Ablex Publishing Corporation.Google Scholar
  12. Choi, Y., Cardie, C., Riloff, E., & Patwardhan, S. (2005). Identifying sources of opinions with conditional random fields and extraction patterns. In Proceedings of the HLT/EMNLP 2005. Vancouver, Canada.Google Scholar
  13. Clemen, G. (1997). The concept of hedging: Origins, approaches and definitions. In R. Markkanen & H. Schröder (Eds.), Hedging and discourse: Approaches to the analysis of a pragmatic phenomenon in academic texts (pp. 235–248). Berlin; New York: Walter de Gruyter.Google Scholar
  14. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 10, 37–46.CrossRefGoogle Scholar
  15. Condoravdi, C., Crouch, R., van den Berg, M., Everett, J., Stolle, R., Paiva, V., & Bobrow, D. (2001). Preventing existence. In Proceedings of the conference on formal ontologies in information systems (FOIS), Ogunquit, Maine, USA.Google Scholar
  16. Dave, K. (2003). Mining the peanut gallery: Opinion extraction and semantic classification of product reviews. In Proceedings of World Wide Web conference 2003.Google Scholar
  17. de Haan, F. (1997). The interaction of modality and negation: A typological study. New York, USA: Garland.Google Scholar
  18. de Haan, F. (2000). The relation between modality and evidentiality. In R. Müller & M. Reis (Eds.), Modalität und Modalverben im Deutschen. Hamburg, Germany: Helmut Buske Verlag.Google Scholar
  19. de Marneffe, M.-C., MacCartney, B., Grenager, T., Cer, D., Rafferty, A., & Manning, C. D. (2006a). Learning to distinguish valid textual entailments. In Second PASCAL RTE Challenge (RTE-2).Google Scholar
  20. de Marneffe, M.-C., MacCartney, B., & Manning, C. D. (2006b). Generating typed dependency parses from phrase structure parses. In Proceedings of LREC 2006.Google Scholar
  21. Di Eugenio, B., & Glass, M. (2004). The kappa statistic: a second look. Computational Linguistics, 30, 95–101.Google Scholar
  22. Dor, D. (1995). Representations, attitudes and factivity evaluations. An epistemically-based analysis of lexical selection. PhD thesis, Stanford University.Google Scholar
  23. Geurts, B. (1998). Presuppositions and anaphors in attitude contexts. Linguistics and Philosophy, 21, 545–601.CrossRefGoogle Scholar
  24. Givón, T. (1993). English grammar. A function-based introduction. Amsterdam, The Netherlands: John Benjamins.Google Scholar
  25. Glanzberg, M. (2003). Felicity and presupposition triggers. In University of Michigan Workshop in Philosophy and Linguistics. Michigan, USA.Google Scholar
  26. Halliday, M. A. K. (1994). An introduction to Functional Grammar (2nd ed.). London, England: Edward Arnold.Google Scholar
  27. Halliday, M. A. K., & Matthiessen, C. M. (2004). An introduction to Functional Grammar. London, England: Hodder Arnold.Google Scholar
  28. Hickl, A., & Bensley, J. (2007). A discourse commitment-based framework for recognizing textual entailment. In Proceedings of the workshop on textual entailment and paraphrasing (pp. 171–176). Prague, Czech Republic.Google Scholar
  29. Hooper, J. B. (1975). On assertive predicates. In J. Kimball (Ed.), Syntax and semantics, IV (pp. 91–124). New York, USA: Academic Press.Google Scholar
  30. Horn, L. R. (1972). On the semantic properties of logical operators in English. PhD thesis, UCLA. Distributed by the Indiana University Linguistics Club in 1976.Google Scholar
  31. Horn, L. R. (1989). A natural history of negation. Chicago, USA: University of Chicago Press.Google Scholar
  32. Huddleston, R. (1984). Introduction to the grammar of English. Cambridge, England: Cambridge University Press.Google Scholar
  33. Karttunen, L. (1970). Implicative verbs. Language, 47, 340–358.CrossRefGoogle Scholar
  34. Karttunen, L. (1973). Presuppositions of compound sentences. Linguistic Inquiry, 4(2), 169–193.Google Scholar
  35. Karttunen, L., & Zaenen, A. (2005). Veridicity. In G. Katz, J. Pustejovsky, & F. Schilder (Eds.), Dagstuhl seminar proceedings. Schloss Dagstuhl, Germany. Internationales Begegnungs- und Forschungszentrum (IBFI).Google Scholar
  36. Kiefer, F. (1987). On defining modality. Folia Linguistica, XXI, 67–94.CrossRefGoogle Scholar
  37. Kiparsky, P., & Kiparsky, C. (1970). Fact. In M. Bierwisch & K. E. Heidolph (Eds.), Progress in linguistics. A collection of papers (pp. 143–173). The Hague: Mouton.Google Scholar
  38. Koenig, J.-P., & Davis, A. R. (2001). Sublexical modality and the structure of lexical semantics. Linguistics and Philosophy, 24, 71–124.CrossRefGoogle Scholar
  39. Kratzer, A. (1991). Modality. In A. van Stechow & D. Wunderlich (Eds.), Semantik: Ein internationales Handbuch der zeitgenoessischen Forschung (pp. 639–650). Berlin, Germany: Walter de Gruyter.Google Scholar
  40. Light, M., Qiu, X. Y., & Srinivasan, P. (2004). The language of Bioscience: Facts, speculations, and statements in between. In BioLINK 2004: Linking biological literature, ontologies, and databases (pp. 17–24).Google Scholar
  41. Lyons, J. (1977). Semantics. Cambridge, England: Cambridge University Press.Google Scholar
  42. Martin, J. R., & White, P. R. R. (2005). Language of evaluation: Appraisal in English. Palgrave Macmillan.Google Scholar
  43. Meyers, A., Reeves, R., Macleod, C., Szekely, R., Zielinska, V., Young, B., & Grishman, R. (2004). The NomBank project: An interim report. In Proceedings of frontiers in corpus annotation workshop. HLT-NAACL.Google Scholar
  44. Miltsakaki, E., Prasad, R., Joshi, A., & Webber, B. (2004). The Penn Discourse TreeBank. In Proceedings of LREC 2004.Google Scholar
  45. Mushin, I. (2001). Evidentiality and epistemological stance. Amsterdam/Philadelphia: John Benjamin.Google Scholar
  46. Nairn, R., Condoravdi, C., & Karttunen, L. (2006). Computing relative polarity for textual inference. In Inference in Computational Semantics, ICoS-5.Google Scholar
  47. Palmer, F. R. (1986). Mood and modality. Cambridge, England: Cambridge University Press.Google Scholar
  48. Palmer, M., Gildea, D., & Kingsbury, P. (2005). The Proposition Bank: An annotated corpus of semantic roles. Computational Linguistics, 31(1), 71–105.Google Scholar
  49. Pang, B., & Lee, L. (2005). Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of the ACL, 115–124.Google Scholar
  50. Pang, B., Lee, L., & Vaithyanathan, S. (2002). Thumbs up? Sentiment classification using machine learning techniques. In Proceedings of the EMNLP 2002.Google Scholar
  51. Polanyi, L., & Zaenen, A. (2005). Contextual lexical valence shifters. In J. Shanahan, Y. Qu, & J. Wiebe (Eds.), Computing attitude and affect in text: Theories and applications. New York, NY, USA: Springer-Verlag.Google Scholar
  52. Pradhan, S., Hovy, E., Marcus, M., Palmer, M., Ramshaw, L., & Weischedel, R. (2007). OntoNotes: A unified relational semantic representation. In Proceedings of IEEE international conference on semantic computing, ICSC 2007 (pp. 517–526).Google Scholar
  53. Prasad, R., Dinesh, N., Lee, A., Joshi, A., & Webber, B. (2007). Attribution and its annotation in the Penn Discourse TreeBank. Traitement Automatique des Langues, 47(2), 43–63.Google Scholar
  54. Prasad, R., Dinesh, N., Lee, A., Robaldo, L., Joshi, A., & Webber, B. (2008). The Penn Discourse TreeBank 2.0. In Proceedings of LREC 2008, Marrakesh, Morocco.Google Scholar
  55. Pustejovsky, J., Castano, J., Ingria, R., Saurí, R., Gaizauskas, R., Setzer, A., & Katz, G. (2003). TimeML: Robust specification of event and temporal expressions in text. In IWCS-5, fifth international workshop on computational semantics.Google Scholar
  56. Pustejovsky, J., Verhagen, M., Saurí, R., Littman, J., Gaizauskas, R., Katz, G., Mani, I., Knippen, R., & Setzer, A. (2006). TimeBank 1.2. Linguistic Data Consortium (LDC). Philadelphia, PA.
  57. Pustejovsky, J., Knippen, B., Littman, J., & Saurí, R. (2005). Temporal and event information in natural language text. Language Resources and Evaluation, 39(2), 123–164.CrossRefGoogle Scholar
  58. Pustejovsky, J., & Rumshisky, A. (2008). Between chaos and structure: Interpreting lexical data through a theoretical lens. Special Issue of International Journal of Lexicography in Memory of John Sinclair, 21(3), 337–355.Google Scholar
  59. Quirk, R., Greenbaum, S., Leech, G., & Svartik, J. (1985). A comprehensive grammar of the English language. London, England: Longman.Google Scholar
  60. Read, J., Hope, D., & Carroll, J. (2007). Annotating expressions of appraisal in English. In Proceedings of the linguistic annotation workshop, Prague. Association for Computational Linguistics, ACL.Google Scholar
  61. Riloff, E., Wiebe, J., & Wilson, T. (2003). Learning subjective nouns using extraction pattern bootstrapping. In Proceedings of the 7th conference on natural language learning (CoNLL 2003).Google Scholar
  62. Rubin, V. L. (2006). Identifying certainty in texts. PhD thesis, Syracuse University.Google Scholar
  63. Rubin, V. L. (2007). Stating with certainty or stating with doubt: Intercoder reliability results for manual annotation of epistemically modalized statements. In Proceedings of the NAACL-HLT 2007.Google Scholar
  64. Rubin, V. L., Liddy, E. D., & Kando, N. (2005). Certainty identification in texts: Categorization model and manual tagging results. In J. Shanahan, Y. Qu, & J. Wiebe (Eds.), Computing attitude and affect in text: Theories and applications. New York, USA: Springer-Verlag.Google Scholar
  65. Saurí, R. (2008). A factuality profiler for eventualities in text. PhD thesis, Brandeis University.Google Scholar
  66. Saurí, R., & Pustejovsky, J. (2007). Determining modality and factuality for text entailment. In Proceedings of the first IEEE international conference on semantic computing, Irvine, CA, USA.Google Scholar
  67. Saurí, R., & Pustejovsky, J. (2008). From structure to interpretation: A double-layered annotation for event factuality. In Proceedings of the second linguistic annotation workshop (The LAW II). LREC 2008, Marrakesh, Morocco.Google Scholar
  68. Saurí, R., Verhagen, M., & Pustejovsky, J. (2006a). Annotating and recognizing event modality in text. In 19th International FLAIRS conference, FLAIRS 2006. The Florida Artificial Intelligence Research Society.Google Scholar
  69. Saurí, R., Verhagen, M., & Pustejovsky, J. (2006b). SlinkET: A partial modal parser for events. In Proceedings of LREC 2006, Genoa, Italy.Google Scholar
  70. Siegel, S., & Castellan, N. J. (1988). Nonparametric statistics for the behavioral sciences. Boston, MA, USA: McGraw Hill.Google Scholar
  71. Snow, R., & Vanderwende, L. (2006). Effectively using syntax for recognizing false entailment. In Proceedings of HLT-NAACL 2006.Google Scholar
  72. Stoyanov, V., & Cardie, C. (2008). Annotating topics of opinions. In Proceedings of LREC 2008, Marrakech, Morocco. ELDA.Google Scholar
  73. Tatu, M., & Moldovan, D. (2005). A semantic approach to recognizing textual entailment. In Proceedings of HLT/EMNLP (pp. 371–378).Google Scholar
  74. Turney, P. D. (2002). Thumbs up or thumbs down? Semantic orientation applied to unsupervised classification of reviews. In Proceedings of the 40th ACL, 417–424.Google Scholar
  75. Van Valin, R. D., & LaPolla, R. J. (1997). Syntax. Structure, meaning and function. Cambridge, England: Cambridge University Press.Google Scholar
  76. Verhagen, M., Stubbs, A., & Pustejovsky, J. (2007). Combining independent syntactic and semantic annotation schemes. In Proceedings of the Linguistic Annotation Workshop (pp. 109–112). ACL. Prague, Czech Republic.Google Scholar
  77. Waugh, L. R. (1995). Reported speech in journalistic discourse: The relation of function and text. Text, 15(1), 129–173.Google Scholar
  78. Wiebe, J., Wilson, T., & Cardie, C. (2005). Annotating expressions of opinions and emotions in language. Language Resources and Evaluation, 39(2), 165–210.CrossRefGoogle Scholar
  79. Wiebe, J. M. (2000). Learning subjective adjectives from corpora. In Proceedings of the 17th National Conference on Artificial Intelligence (AAAI 2000).Google Scholar
  80. Wierzbicka, A. (1987). English speech act verbs. A semantic dictionary. Sydney, Australia: Academic Press.Google Scholar
  81. Wilson, T., Hoffmann, P., Somasundaran, S., Kessler, J., Wiebe, J., Choi, Y., Cardie, C., Riloff, E., & Patwardhan, S. (2005). OpinionFinder: A system for subjectivity analysis. In Proceedings of the HLT/EMNLP 2005 Demonstration Abstracts (pp. 34–35). Vancouver, Canada.Google Scholar
  82. Wilson, T., Wiebe, J., & Hwa, R. (2004). Just how mad are you? Finding strong and weak opinion clauses. In Proceedings of the 19th National Conference on Artificial Intelligence (AAAI 2004).Google Scholar
  83. Wolf, F., & Gibson, E. (2005). Representing discourse coherence: A corpus-based analysis. Computational Linguistics, 31(2), 249–287.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media B.V. 2009

Authors and Affiliations

  1. 1.Laboratory for Linguistics and Computation, Computer Science DepartmentBrandeis UniversityWalthamUSA

Personalised recommendations