Skip to main content
SpringerLink
Log in

Ongoing Efforts: Toward Behaviour-Based Corpus Evaluation

  • Chapter
  • First Online:
Handbook of Linguistic Annotation

  • 2075 Accesses

Abstract

This chapter describes our recent attempts to explore a methodology for evaluating annotated corpora through analysing annotator behaviour during annotation. We first describe the details of an experiment for collecting annotator behaviour during annotating predicate argument relations in Japanese texts. In this experiment, every annotation tool operation and annotator eye gaze were collected from three annotators. We discuss the relationship between the collected data and the annotation agreement between multiple annotators, in which two types of disagreement are distinguished: explicit annotation disagreement (EAD) and missing annotation disagreement (MAD). We further report the preliminary results of an attempt for detecting missing disagreement by analysing the collected data. The chapter concludes with some remarks for future research directions.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 349.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 449.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 449.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    We follow the wording by Russo and Leclerc [28].

  2. 2.

    A dwell is a collection of one or several fixations within a certain area of interest, a segment in our case.

  3. 3.

    The other symbols, L, R, U denote fixation positions relative to the target predicate, and + and - denote temporal offsets from the working period on the target predicate. The details are explained in Sect. 4.2.

  4. 4.

    See Chap. 10 in this volume ([1]) for detailed discussion on inter-annotator agreement.

  5. 5.

    The table shows just agreement between two annotators, apart from the gold annotation.

References

  1. Artstein, R.: Inter-annotator agreement. In: Ide, N., Pustejovsky, J. (eds.) Handbook of Linguistic Annotation, Chap. 10. Springer, Berlin (2017)

    Google Scholar 

  2. Artstein, R., Poesio, M.: Inter-coder agreement for computational linguistics. Comput. Linguist. 34(4), 555–596 (2008)

    Article  Google Scholar 

  3. Bednarik, R., Tukiainen, M.: Temporal eye-tracking data: evolution of debugging strategies with multiple representations. In: Proceedings of the 2008 symposium on Eye tracking research & applications (ETRA ’08), pp. 99–102 (2008). doi:10.1145/1344471.1344497

  4. Carletta, J.: Assessing agreement on classification tasks: the kappa statistic. Comput. Linguist. 22(2), 249–254 (1996)

    Google Scholar 

  5. Chen, Z.: From data mining to behavior mining. Int. J. Inf. Technol. Decis. Mak. 5(4), 703–711 (2006)

    Article  Google Scholar 

  6. Chou, W.C., Tsai, R.T.H., Su, Y.S., Ku, W., Sung, T.Y., Hsu, W.L.: A semi-automatic method for annotating a biomedical proposition bank. In: Proceedings of the Workshop on Frontiers in Linguistically Annotated Corpora, pp. 5–12 (2006)

    Google Scholar 

  7. Conati, C., Merten, C.: Eye-tracking for user modeling in exploratory learning environments: an empirical evaluation. Knowl. Based Syst. 20(6), 557–574 (2007). doi:10.1016/j.knosys.2007.04.010

    Article  Google Scholar 

  8. Duchowski, A.T.: A breadth-first survey of eye-tracking applications. Behav. Res. Methods Instrum. Comput. 34(4), 455–470 (2002)

    Article  Google Scholar 

  9. Ericsson, K., Simon, H.A.: Protocol Analysis – Verbal Reports as Data –. The MIT Press, Cambridge (1984)

    Google Scholar 

  10. Fort, K., François, C., Galibert, O., Ghribi, M.: Analyzing the impact of prevalence on the evaluation of a manual annotation campaign. In: Proceedings of the Eigth International Conference on Language Resources and Evaluation (LREC 2012), pp. 1474–1480 (2012)

    Google Scholar 

  11. Gidlöf, K., Wallin, A., Dewhurst, R., Holmqvist, K.: Using eye tracking to trace a cognitive process: gaze behaviour during decision making in a natural environment. J. Eye Mov. Res. 6(1), 1–14 (2013)

    Google Scholar 

  12. Griffin, Z.M., Bock, K.: What the eyes say about speaking. Psychol. Sci. 11(4), 274–279 (2000)

    Article  Google Scholar 

  13. Just, M.A., Carpenter, P.A.: A theory of reading: from eye fixations to comprehension. Psychol. Rev. 87(4), 329–354 (1980)

    Article  Google Scholar 

  14. Just, M.A., Carpenter, P.A.: Cognitive coordinate systems: accounts of mental rotation and individual differences in spatial ability. Psychol. Rev. 92(2), 137–172 (1985)

    Article  Google Scholar 

  15. Kaplan, D., Iida, R., Nishina, K., Tokunaga, T.: Slate - a tool for creating and maintaining annotated corpora. J. Lang. Technol. Comput. Linguist. 26(2), 89–101 (2012)

    Google Scholar 

  16. Lenzi, V.B., Moretti, G., Sprugnoli, R.: CAT: the CELCT annotation tool. In: Proceedings of the Eigth International Conference on Language Resources and Evaluation (LREC 2012), pp. 333–338 (2012)

    Google Scholar 

  17. Maekawa, K., Yamazaki, M., Maruyama, T., Yamaguchi, M., Ogura, H., Kashino, W., Ogiso, T., Koiso, H., Den, Y.: Design, compilation, and preliminary analyses of balanced corpus of contemporary written Japanese. In: Proceedings of the Eigth International Conference on Language Resources and Evaluation (LREC 2010), pp. 1483–1486 (2010)

    Google Scholar 

  18. Malcolm, G.L., Henderson, J.M.: The effects of target template specificity on visual search in real-world scenes: evidence from eye movements. J. Vis. 9(11), 8:1–13 (2009). doi:10.1167/9.11.8

    Article  Google Scholar 

  19. Marcińczuk, M., Kocoń, J., Broda, B.: Inforex – a web-based tool for text corpus management and semantic annotation. In: Proceedings of the Eigth International Conference on Language Resources and Evaluation (LREC 2012), pp. 224–230 (2012)

    Google Scholar 

  20. Marcus, M.P., Santorini, B., Marcinkiewicz, M.A.: Building a large annotated corpus of english: the Penn Treebank. Comput. Linguist. 19(2), 313–330 (1993)

    Google Scholar 

  21. Mitsuda, K., Iida, R., Tokunaga, T.: Detecting missing annotation disagreement using eye gaze information. In: Proceedings of the 11th Workshop on Asian Language Resources, pp. 19–26 (2013)

    Google Scholar 

  22. Passonneau, R.: Measuring agreement on set-valued items (MASI) for semantic and pragmatic annotation. In: Proceedings of the International Conference on Language Resources and Evaluation (LREC 2006), pp. 831–836 (2006)

    Google Scholar 

  23. Pei, J., Han, J., Mortazavi-Asl, B., Pinto, H., Chen, Q., Dayal, U., Hsu, M.C.: PrefixSpan: Mining sequential patterns efficiently by prefix-projected pattern growth. In: Proceedings of the 17th International Conference on Data Engineering (ICDE ’01), pp. 215–224 (2001). doi:10.1109/ICDE.2001.914830

  24. Rehbein, I., Ruppenhofer, J., Sporleder, C.: Is it worth the effort? assessing the benefits of partial automatic pre-labeling for frame-semantic annotation. Lang. Resour. Eval. 46(1), 1–23 (2012). doi:10.1007/s10579-011-9170-z

    Article  Google Scholar 

  25. Richardson, D.C., Dale, R., Spivey, M.J.: Eye movements in language and cognition: a brief introduction. In: Gonzalez-Marquez, M., Mittelberg, I., Coulson, S., Spivey, M.J. (eds.) Methods in Cognitive Linguistics, pp. 323–344. John Benjamins, Amsterdam (2007)

    Google Scholar 

  26. Romero, C., Ventura, S.: Educational data mining: a review of the state of the art. IEEE Trans.Syst. Man Cybern. PartC Appl. Rev. 40(6), 601–618 (2010). doi:10.1109/TSMCC.2010.2053532

    Article  Google Scholar 

  27. Rosengrant, D.: Gaze scribing in physics problem solving. In: Proceedings of the 2010 Symposium on Eye Tracking Research & Applications (ETRA ’10), pp. 45–48 (2010). doi:10.1145/1743666.1743676

  28. Russo, J.E., Leclerc, F.: An eye-fixation analysis of choice processes for consumer nondurables. J. Consum. Res. 21(2), 274–290 (1994)

    Article  Google Scholar 

  29. Salvucci, D.D., Goldberg, J.H.: Identifying fixations and saccades in eye-tracking protocols. In: Proceedings of the 2000 Symposium on Eye Tracking Research & Applications (ETRA ’00), pp. 71–78 (2000). doi:10.1145/355017.355028

  30. Stede, M., Huang, C.R.: Inter-operability and reusability: the science of annotation. Lang. Resour. Eva. 46(1), 91–94 (2012). doi:10.1007/s10579-011-9164-x

    Article  Google Scholar 

  31. Tokunaga, T., Iida, R., Mitsuda, K.: Annotation for annotation – toward eliciting implicit linguistic knowledge through annotation –. In: Proceedings of the 9th Joint ISO - ACL SIGSEM Workshop on Interoperable Semantic Annotation (ISA-9), pp. 79–83 (2013)

    Google Scholar 

  32. Tomanek, K., Hahn, U., Lohmann, S., Ziegler, J.: A cognitive cost model of annotations based on eye-tracking data. In: Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics (ACL 2010), pp. 1158–1167 (2010). http://www.aclweb.org/anthology/P10-1118

  33. Vapnik, V.N.: Statistical Learning Theory. Adaptive and Learning Systems for Signal Processing Communications, and control. Wiley, New York (1998)

    Google Scholar 

  34. Voutilainen, A.: Improving corpus annotation productivity: a method and experiment with interactive tagging. In: Proceedings of the Eigth International Conference on Language Resources and Evaluation (LREC 2012), pp. 2097–2102 (2012)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Tokyo Institute of Technology, 2-12-1 W8-73 Meguro, Ôokayama, Tokyo, Japan

    Takenobu Tokunaga

Authors
  1. Takenobu Tokunaga
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Takenobu Tokunaga .

Editor information

Editors and Affiliations

  1. Department of Computer Science, Vassar College, Poughkeepsie, New York, USA

    Nancy Ide

  2. Department of Computer Science, Volen Center for Complex Systems, Brandeis University, Waltham, Massachusetts, USA

    James Pustejovsky

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer Science+Business Media Dordrecht

About this chapter

Cite this chapter

Tokunaga, T. (2017). Ongoing Efforts: Toward Behaviour-Based Corpus Evaluation. In: Ide, N., Pustejovsky, J. (eds) Handbook of Linguistic Annotation. Springer, Dordrecht. https://doi.org/10.1007/978-94-024-0881-2_12

Download citation

  • DOI: https://doi.org/10.1007/978-94-024-0881-2_12

  • Published:

  • Publisher Name: Springer, Dordrecht

  • Print ISBN: 978-94-024-0879-9

  • Online ISBN: 978-94-024-0881-2

  • eBook Packages: Social SciencesSocial Sciences (R0)

Publish with us

Policies and ethics

Search

Navigation