Language Resources and Evaluation

, Volume 41, Issue 1, pp 91–115 | Cite as

TimeBank evolution as a community resource for TimeML parsing

  • Branimir BoguraevEmail author
  • James Pustejovsky
  • Rie Ando
  • Marc Verhagen
Original Paper


TimeBank is the only reference corpus for TimeML, an expressive language for annotating complex temporal information. It is a rich resource for a broad range of research into various aspects of the expression of time and temporally related events. This paper traces the development of TimeBank from its initial—and somewhat noisy—version (1.1) to a substantially revised release (1.2), now available via the Linguistic Data Consortium. The development path is motivated by the encouraging empirical results of TimeML-compliant annotators developed on the basis of TimeBank 1.1, and is informed by a detailed study of the characteristics of that initial release, which guides a clean-up process turning TimeBank 1.2 into a consistent and robust community resource.


TimeML TimeBank Corpus analysis Temporal information extraction 



A Markup Language for Time


Time Expression


Linguistic Data Consortium


Information Extraction


Inter-Annotator Agreement



This work was supported in part by the ARDA NIMD and AQUAINT programs, PNWD-SW-6059 and NBCHC040027-MOD-0003.


  1. Allen, J. (1983). Maintaining knowledge about temporal intervals. Communications of the ACM, 26(11), 832–843.Google Scholar
  2. Boguraev, B., & Ando, R. K. (2005a). TimeBank-driven TimeML analysis. In: G. Katz, J. Pustejovsky, & F. Schilder (Eds.), International Workshop on Annotating, Extracting, and Reasoning with Time. Dagstuhl, Germany.Google Scholar
  3. Boguraev, B., & Ando, R. K. (2005b). TimeML-compliant text analysis for temporal reasoning. In: Nineteenth International Joint Conference on Artificial Intelligence (IJCAI-05). Edinburgh, Scotland.Google Scholar
  4. Ferro, L. (2001). tides: Instruction manual for the annotation of temporal expressions. Technical Report MTR 01W0000046V01, The MITRE Corporation.Google Scholar
  5. Fikes, R., Jenkins, J., & Frank, G. (2003). JTP: A system architecture and component library for hybrid reasoning. Technical Report KSL-03-01, Knowledge Systems Laboratory, Stanford University.Google Scholar
  6. Gaizauskas, R., Harkema, H., Hepple, M., & Setzer, A. (2006). Task-oriented extraction of temporal information: The case of clinical narratives. In: A. Montanari, J. Pustejovsky, & P. Revesz (Eds.), TIME 2006: International Symposium on Temporal Representation and Reasoning. Budapest, Hungary.Google Scholar
  7. Han, B., & Lavie, A. (2004). A framework for resolution of time in natural language. TALIP Special Issue on Spatial and Temporal Information Processing, 3(1), 11–35.Google Scholar
  8. Hirschman, L., Robinson, P., Burger, J., & Vilain, M. (1998). Automatic coreference: The role of annotated training data. In: AAAI 1998 Spring Symposium on Applying Machine Learning to Discourse Processing. Stanford, USA, pp. 1419–1422.Google Scholar
  9. Hobbs, J., & Pan, F. (2004). An ontology of time for the semantic web. TALIP Special Issue on Spatial and Temporal Information Processing, 3(1), 66–85.Google Scholar
  10. Hobbs, J., & Pustejovsky, J. (2004). Annotating and reasoning about time and events. In: AAAI Spring Symposium on Logical Formalizations of Commonsense Reasoning. Stanford, CA.Google Scholar
  11. Lee, K., Pustejovsky, J., & Boguraev, B. (2006). Towards an international standard for annotating temporal information. In: Third International Conference on Terminology, Standardization and Technology Transfer. Beijing, China.Google Scholar
  12. Mani, I., Wellner, B., Verhagen, M., Lee, C. M., & Pustejovsky, J. (2006). Machine Learning of Temporal Relations. In: Proceedings of the 44th Annual Meeting of the Association for Computational Linguistics. Sydney, Australia.Google Scholar
  13. Pustejovsky, J., Castaño, J., Ingria, R., Saurí, R., Gaizauskas, R., Setzer, A., Katz, G., & Radev, D. (2003a).TimeML: Robust specification of event and temporal expressions in text. In: AAI Spring Symposium on New Directions in Question-Answering (Working Papers). Stanford, CA, pp. 28–34.Google Scholar
  14. Pustejovsky, J., Hanks, P., Saurí, R., See, A., Gaizauskas, R., Setzer, A., Radev, D., Sundheim, B., Day, D., Ferro, L., & Lazo, M. (2003b). The TimeBank corpus. In: T. McEnery (Ed.), Corpus Linguistics (pp. 47–656). Lancaster.Google Scholar
  15. Pustejovsky, J., Knippen, R., Littman, J., & Saurí, R. (2005). Temporal and event information in natural language text. Language Resources and Evaluation 39(2–3), 123–164.CrossRefGoogle Scholar
  16. Pustejovsky, J., Mani, I., Bélanger, L., Boguraev, B., Knippen, B., Littman, J., Rumshisky, A., See, A.,Symonenko, S., Guilder, J. V., Guilder, L. V., Verhagen, M., & Ingria, R. (2003c). Graphical Annotation Kit for TimeML’. Technical report, TANGO (TimeML ANnotation Graphical Organizer) Workshop Version 1.4, <> [date of citation: 2005-06-20]
  17. Saurí, R., Littman, J., Knippen, B., Gaizauskas, R., Setzer, A., & Pustejovsky, J. (2005). TimeML Annotation Guidelines, Version 1.2.1’. Technical report, TERQAS Workshop/Linguistic Data Consortium. <> [date of citation: 2006-07-16].
  18. Setzer, A. (2001). Temporal information in newswire articles: An annotation scheme and corpus study. Ph.D. thesis, University of Sheffield, Sheffield, UK.Google Scholar
  19. Verhagen, M. (2005). Temporal closure in an annotation environment. Language Resources and Evaluation, 39(2–3), 211–241.CrossRefGoogle Scholar
  20. Verhagen, M., Mani, I., Sauri, R., Littman, J., Knippen, R., Jang, S. B., Rumshisky, A., Phillips, J., & Pustejovsky, J. (2005). Automating Temporal Annotation with tarsqi’. In: 43rd Annual Meeting of the Association for Computational Linguistics (ACL-05). Ann Arbor, Michigan, Poster/Demo.Google Scholar

Copyright information

© Springer Science+Business Media B.V. 2007

Authors and Affiliations

  • Branimir Boguraev
    • 1
  • James Pustejovsky
    • 2
  • Rie Ando
    • 1
  • Marc Verhagen
    • 2
  1. 1.IBM T.J. Watson Research CenterHawthorneUSA
  2. 2.Brandeis UniversityWalthamUSA

Personalised recommendations