Skip to main content
Log in

The GUM corpus: creating multilayer resources in the classroom

  • Original Paper
  • Published:
Language Resources and Evaluation Aims and scope Submit manuscript

Abstract

This paper presents the methodology, design principles and detailed evaluation of a new freely available multilayer corpus, collected and edited via classroom annotation using collaborative software. After briefly discussing corpus design for open, extensible corpora, five classroom annotation projects are presented, covering structural markup in TEI XML, multiple part of speech tagging, constituent and dependency parsing, information structural and coreference annotation, and Rhetorical Structure Theory analysis. Layers are inspected for annotation quality and together they coalesce to form a richly annotated corpus that can be used to study the interactions between different levels of linguistic description. The evaluation gives an indication of the expected quality of a corpus created by students with relatively little training. A multifactorial example study on lexical NP coreference likelihood is also presented, which illustrates some applications of the corpus. The results of this project show that high quality, richly annotated resources can be created effectively as part of a linguistics curriculum, opening new possibilities not just for research, but also for corpora in linguistics pedagogy.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Notes

  1. Another alternative to student participation is crowdsourcing over platforms such as Amazon Mechanical Turk or CrowdFlower (see Sabou et al. 2014 for an overview of recent projects and some best practices). Here individuals with minimal or no training can carry out relatively simple tasks on a large scale. However the costs involved need to be covered, which is difficult to sustain for an open-ended corpus, and some more complex annotations, such as syntactic analysis, are difficult to find qualified persons to do. It is possible that the unavailability of crowdsourcing and other resources for older languages has contributed to the popularity of classroom annotation or ‘class-sourcing’ in these domains (I’m indebted to an anonymous reviewer for pointing the latter term out).

  2. These numbers represent the first round of documents from GUM, collected in 2014; at the time of writing, a second round is being processed which contains over 21,500 tokens from the 2015 iteration of the same course, bringing the total up to about 44,000 tokens (see more details below).

  3. The motivational effect of choosing one’s own text is similar to Computer Assisted Language Learning tools that allow learners to work on a text of their own choosing in the target language, often from the Web (see the REAP project, http://boston.lti.cs.cmu.edu/reap/ and VIEW, http://sifnos.sfs.uni-tuebingen.de/VIEW/). I thank an anonymous reviewer for pointing this out.

  4. Each of 21 students enrolled in the class selected a single text for annotation throughout the class. In one unusual case, a text which turned out to be too short after segmentation was supplemented by a second text of a similar length from the same genre. Three further texts were contributed, two by the instructor, and one by the teaching assistant; these were left out of the evaluation below. The course was open to both undergraduate and graduate students, but graduate students represented the majority of participants.

  5. The second round of data in 2015 adds 29 further documents from the same text types. See the corpus website for the latest data.

  6. ANNIS is an open source browser based platform for accessing multilayer corpora, originally developed at Potsdam University and currently in development at Humboldt University in Berlin and Georgetown University; see http://corpus-tools.org/annis for more information.

  7. An anonymous reviewer has commented on consultation of the instructor as a possible source of skewing for annotator accuracy. While it is true that some errors were certainly averted by discussion with the instructor during class, it is conversely very much not the case that there was time for the instructor or TA to advise students on the individual details of much of their annotations, given the size of the class and the corpus. Notwithstanding the degree to which the instructor or TA were able to directly reduce error rates, the data below should be taken as an evaluation of the quality of a ‘class-sourced’ corpus, which as we will see, contains errors nonetheless.

  8. Main additions in the extended tag set are special tags for the verbs be (VB*) and have (VH*) versus lexical verbs (VV*), more tags for punctuation, and a special tag for the word that as a complementizer (IN/that). Especially the more fine-grained use of punctuation tags was useful for data from the Web, since there is a wide range of different symbols and functions. We also added a second layer of POS annotations using the UCREL CLAWS5 tag set (Garside and Smith 1997), to allow for comparison and combined searching. This layer has not yet been corrected or evaluated, but is available in the corpus online.

  9. Spreadsheet validation has been replaced in the most recent iteration of the course in favor of a Perl script which simultaneously controls XML markup guidelines and gives more verbose error messages. Students are instructed to use validation before submitting assignments to ensure that tag names are formally possible, thereby ruling out typos.

  10. Data from the one annotator working on two shorter texts has been collapsed into one group, here and below.

  11. Other elements were produced in keeping with TEI markup, including hierarchical divs for sections and subsections, but these were discarded from the merged corpus before it was indexed for searching, as they turned out to be rather inconsistent across genres semantically.

  12. Even for binary sentiment analysis (negative/positive), Hsueh et al. (2009: 30) report gold standard average accuracy of 0.974 for expert annotators, as opposed to 0.714 for Mechanical Turk annotators with minimal training. Although an evaluation attempting syntactic analysis via Mechanical Turk or Crowdflower remains outstanding, it is doubtful that complete parses could be obtained in this way, even when using many concurrent annotators. Simpler subtasks however, such as PP attachment disambiguation in new genres, have been shown to be possible using some semi-automatic support and quintuple annotation of the same data via Mechanical Turk (Jha et al. 2010), so that using crowdsourcing to improve specific aspects of parses in a multilayer corpus is very much a possibility.

  13. The decision to correct parser output, rather than annotate from scratch, was largely motivated by time constraints. Gerdes (2013: 94) suggests that student annotation of dependencies from scratch can achieve a rather low 79 % average attachment accuracy for French, also supporting the idea that parser correction is a good strategy. Although it would have been possible to use a native dependency parser, using constituent parses as a basis is known to produce good results (see Cer et al. 2010; I thank an anonymous reviewer for pointing this out), and also has the added advantage of representing constituent and dependency structure side by side in the merged corpus.

  14. The reason for counting labeling errors only for correctly attached dependencies is that an incorrectly attached, but otherwise correct label is not seen as correctly ascertaining the function of the word in the sentence. Note that the numbers for the parser and students mean slightly different things: parser errors are those caught by either students or instructors, while student performance indicates how much correction is still required after the student pass.

  15. Similar accuracy has been achieved working from scratch by allowing 4–5 student annotators to work on the same sentences and taking a majority vote (see Gerdes 2013: 95 for collaborative F-scores of 0.91–0.92, working on French). However quadruple annotation in the context of GUM would have meant a very substantial reduction in corpus size, and also in the variability of data discussed in class.

  16. Especially as opposed to blogs, chat or Twitter data. This confirms the criticalness of creating manually annotated gold standards for Web genres and new social media for domain adaptation of automatic parsing; see Silveira et al. (2014).

  17. For a very similar result in an annotation experiment with 14 students using two different interfaces doing coreference in somewhat different data (including Twitter), see Jiang et al. (2013), who report an F-Score of 0.83 in coreference pairs for people and locations, using the better of the two interfaces. Jiang et al. also remark on high human precision but low recall, the opposite of current NLP.

  18. For the latest iteration of GUM, a new browser-based interface called rstWeb has been developed, which is being used for annotating RST online and facilitating collaborative annotation and adjudication over a server. See http://corpling.uis.georgetown.edu/rstweb/info/ for details.

  19. ‘Style 1’, which consists of immediately building relations for incrementally segmented texts, proved slower, matching Carlson et al.’s notion that it is less suitable for texts of substantial length (GUM documents average 64.12 EDUs, somewhat above the RST Discourse Treebank with 56.59 EDUs, cf. Carlson et al. 2001: 7).

  20. In a similar vein, Recasens et al. (2013) attempt to characterize singleton phrases, i.e. phrases that do not have an antecedent in the text. This study is complementary to their findings (see below).

  21. Adding pronouns to the evaluation results in accuracy of 74.14 %, and makes the interaction between referent length and proper noun status insignificant (possible interference from the inherent shortness of pronouns). This brings results in line with the precision score (72.2) reported in Recasens et al. (2013), however in my opinion mixing evaluation on pronouns and nominals obscures the theoretical issue somewhat, except in the context of in situ evaluation within a coreference resolution system, which is the goal of Recasens et al.’s paper.

  22. The latter is admittedly due almost entirely to one document dealing with the cultivation of Basil, though the latest iteration of GUM is set to introduce a rather similar document on growing cactuses.

References

  • Anderson, A. H., Bader, M., Bard, E. G., Boyle, E., Doherty, G., Garrod, S., et al. (1991). The HCRC map task corpus. Language and Speech, 34, 351–366.

    Article  Google Scholar 

  • Biber, D. (1993). Representativeness in corpus design. Literary and Linguistic Computing, 8(4), 243–257.

    Article  Google Scholar 

  • Blackwell, C., & Martin, T. R. (2009). Technology, collaboration, and undergraduate research. Digital Humanities Quarterly, 3(1). http://digitalhumanities.org/dhq/vol/003/1/000024/000024.html.

  • Burnard, L., & Bauman, S. (2008). TEI P5: Guidelines for electronic text encoding and interchange. Technical report. http://www.tei-c.org/Guidelines/P5/.

  • Calhoun, S., Carletta, J., Brenier, J., Mayo, N., Jurafsky, D., Steedman, M., & Beaver, D. (2010). The NXT-format Switchboard Corpus: A rich resource for investigating the syntax, semantics, pragmatics and prosody of dialogue. Language Resources and Evaluation, 44(4), 387–419.

    Article  Google Scholar 

  • Carlson, L., Marcu, D., & Okurowski, M. E. (2001). Building a discourse-tagged corpus in the framework of rhetorical structure theory. In Proceedings of 2nd SIGDIAL workshop on discourse and dialogue, Eurospeech 2001 (pp. 1–10). Aalborg, Denmark.

  • Cer, D., de Marneffe, M.-C., Jurafsky, D., & Manning, C. D. (2010). Parsing to stanford dependencies: Trade-offs between speed and accuracy. In 7th International conference on language resources and evaluation (LREC 2010) (pp. 1628–1632). Valletta, Malta.

  • Chambers, N., & Jurafsky, D. (2009). Unsupervised learning of narrative schemas and their participants. In Proceedings of the 47th annual meeting of the ACL and the 4th IJCNLP of the AFNLP (pp. 602–610). Suntec, Singapore.

  • Chen, D., & Manning, C. D. (2014). A fast and accurate dependency parser using neural networks. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) (pp. 740–750). Doha, Qatar.

  • Crowdy, S. (1993). Spoken corpus design. Literary and Linguistic Computing, 8, 259–265.

    Article  Google Scholar 

  • de Marneffe, M.-C., Dozat, T., Silveira, N., Haverinen, K., Ginter, F., Nivre, J., & Manning, C. D. (2014). Universal Stanford dependencies: A cross-linguistic typology. In Proceedings of 9th international conference on language resources and evaluation (LREC 2014) (pp. 4585–4592). Reykjavík, Iceland.

  • de Marneffe, M.-C., & Manning, C. D. (2013). Stanford typed dependencies manual. Stanford University, Technical Report.

  • Dipper, S., Götze, M., & Skopeteas, S. (Eds.) (2007). Information structure in cross-linguistic corpora: annotation guidelines for phonology, morphology, syntax, semantics, and information structure. Interdisciplinary Studies on Information Structure, Working papers of the SFB 632, 7.

  • Durrett, G., & Klein, D. (2013). Easy victories and uphill battles in coreference resolution. In Proceedings of the conference on empirical methods in natural language processing (EMNLP 2013). Seattle: ACL.

  • Garside, R., & Smith, N. (1997). A hybrid grammatical tagger: CLAWS4. In R. Garside, G. Leech, & A. McEnery (Eds.), Corpus annotation: Linguistic information from computer text corpora (pp. 102–121). London: Longman.

    Google Scholar 

  • Gerdes, K. (2013). Collaborative dependency annotation. In Proceedings of the second international conference on dependency linguistics (DepLing 2013) (pp. 88–97). Prague.

  • Givón, T. (Ed.). (1983). Topic continuity in discourse. A quantitative cross-language study (Typlological Studies in Language 3). Amsterdam: John Benjamins.

    Google Scholar 

  • Godfrey, J. J., Holliman, E. C., & McDaniel, J. (1992). SWITCHBOARD: Telephone speech corpus for research and development. In Proceedings of ICASSP-92 (pp. 517–520). San Francisco, CA.

  • Grosz, B. J., Joshi, A. K., & Weinstein, S. (1995). Centering: A framework for modeling the local coherence of discourse. Computational Linguistics, 21(2), 203–225.

    Google Scholar 

  • Haug, D. T., Eckhoff, H. M., Majer, M., & Welo, E. (2009). Breaking down and putting back together: Analysis and synthesis of New Testament Greek. Journal of Greek Linguistics, 9(1), 56–92.

    Google Scholar 

  • Hirschman, L., Robinson, P., Burger, J. D., & Vilain, M. B. (1998). Automating coreference: The role of annotated training data. AAAI, Technical Report SS-98-01. http://www.aaai.org/Papers/Symposia/Spring/1998/SS-98-01/SS98-01-018.pdf.

  • Hovy, E., Marcus, M., Palmer, M., Ramshaw, L., & Weischedel, R. (2006). OntoNotes: The 90 % solution. In Proceedings of the human language technology conference of the NAACL, companion volume: Short Papers (pp. 57–60). New York: Association for Computational Linguistics.

  • Hsueh, P.-Y., Melville, P., & Sindhwani, V. (2009). Data quality from crowdsourcing: A study of annotation selection criteria. In Proceedings of the NAACL HLT workshop on active learning for natural language processing (pp. 27–35). Boulder, CO.

  • Hunston, S. (2008). Collection strategies and design decisions. In A. Lüdeling & M. Kytö (Eds.), Corpus linguistics. An international handbook (pp. 154–168). Berlin: Mouton de Gruyter.

    Google Scholar 

  • Ide, N., Baker, C., Fellbaum, C., & Passonneau, R. (2010). The manually annotated sub-corpus: A community resource for and by the people. In Proceedings of the 48th annual meeting of the association for computational linguistics (pp. 68–73). Uppsala, Sweden.

  • Jha, M., Andreas, J., Thadani, K., Rosenthal, S., & McKeown, K. (2010). Corpus creation for new genres: A crowdsourced approach to PP attachment. In Proceedings of the NAACL HLT 2010 workshop on creating speech and language data with Amazon’s mechanical turk (pp. 13–20). Los Angeles, CA.

  • Jiang, L., Wang, Y., Hoffart, J., & Weikum, G. (2013). Crowdsourced entity markup. In Proceedings of the 1st international workshop on crowdsourcing the semantic web (pp. 59–68). Sydney.

  • Krause, T., Lüdeling, A., Odebrecht, C., & Zeldes, A. (2012). Multiple tokenizations in a Diachronic Corpus. In Exploring ancient languages through Corpora. Oslo.

  • Krause, T., & Zeldes, A. (2014). ANNIS3: A new architecture for generic corpus query and visualization. Digital Scholarship in the Humanities. http://dsh.oxfordjournals.org/content/digitalsh/early/2014/12/02/llc.fqu057.full.pdf.

  • Kučera, H., & Francis, W. N. (1967). Computational analysis of present-day English. Providence: Brown University Press.

    Google Scholar 

  • Lee, H., Chang, A., Peirsman, Y., Chambers, N., Surdeanu, M., & Jurafsky, D. (2013). Deterministic coreference resolution based on entity-centric, precision-ranked rules. Computational Linguistics, 39(4), 885–916.

    Article  Google Scholar 

  • Lüdeling, A., Doolittle, S., Hirschmann, H., Schmidt, K., & Walter, M. (2008). Das Lernerkorpus Falko. Deutsch als Fremdsprache, 2, 67–73.

    Google Scholar 

  • Lüdeling, A., Evert, S., & Baroni, M. (2007). Using web data for linguistic purposes. In M. Hundt, N. Nesselhauf, & C. Biewer (Eds.), Corpus linguistics and the web (Language and computers—studies in practical linguistics 59) (pp. 7–24). Amsterdam: Rodopi.

  • Lüdeling, A., Ritz, J., Stede, M., & Zeldes, A. (2016). Corpus linguistics and information structure research. In Féry, C., & Ichihara, S. (Eds.), The Oxford handbook of information structure. Oxford: Oxford University Press.

  • Lyons, J. (1977). Semantics. Cambridge: Cambridge University Press.

    Book  Google Scholar 

  • Mann, W. C., & Thompson, S. A. (1988). Rhetorical structure theory: Toward a functional theory of text organization. Text, 8(3), 243–281.

    Article  Google Scholar 

  • Manning, C. D., Surdeanu, M., Bauer, J., Finkel, J., Bethard, S. J., & McClosky, D. (2014). The Stanford CoreNLP natural language processing toolkit. In Proceedings of 52nd annual meeting of the association for computational linguistics: System demonstrations (pp. 55–60). Baltimore, MD.

  • Marcu, D., Amorrortu, E., & Romera, M. (1999). Experiments in constructing a corpus of discourse trees. In Proceedings of the ACL workshop towards standards and tools for discourse tagging (pp. 48–57). College Park, MD.

  • Marcus, M. P., Santorini, B., & Marcinkiewicz, M. A. (1993). Building a large annotated corpus of English: The Penn Treebank. Special Issue on Using Large Corpora, Computational Linguistics, 19(2), 313–330.

    Google Scholar 

  • Mitchell, A., Strassel, S., Przybocki, M., Davis, J., Doddington, G., Grishman, R., Meyers, A., Brunstein, A., Ferro, L., & Sundheim, B. (2003). ACE-2 Version 1.0. Linguistic Data Consortium, Technical Report LDC2003T11, Philadelphia.

  • Nissim, M. (2006). Learning information status of discourse entities. In Proceedings of the 2006 conference on empirical methods in natural language processing (EMNLP 2006) (pp. 94–102). Sydney, Australia.

  • O’Donnell, M. (2000). RSTTool 2.4—A markup tool for rhetorical structure theory. In Proceedings of the international natural language generation conference (INLG’2000) (pp. 253–256). Mitzpe Ramon, Israel.

  • Paul, D. B., & Baker, J. M. (1992). The design for the Wall Street Journal-based CSR corpus. In Proceedings of the workshop on speech and natural language, HLT ‘91 (pp. 357–362). Stroudsburg, PA: ACL.

  • Ragheb, M., & Dickinson, M. (2013). Inter-annotator Agreement for Dependency Annotation of Learner Language. In Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications (pp. 169–179). Atlanta, GA.

  • Recasens, M., de Marneffe, M.-C., & Potts, C. (2013). The life and death of discourse entities: Identifying singleton mentions. In Proceedings of NAACL 2013 (pp. 627–633). Atlanta, GA.

  • Redeker, G., Berzlánovich, I., van der Vliet, N., Bouma, G., & Egg, M. (2012). Multi-layer discourse annotation of a Dutch text corpus. In N. Calzolari, K. Choukri, T. Declerck, M. U. Doǧan, B. Maegaard, J. Mariani, A. Moreno, J. Odijk, & S. Piperidis (Eds.), Proceedings of the eight international conference on language resources and evaluation (LREC’12) (pp. 2820–2825). Istanbul: ELRA.

    Google Scholar 

  • Reppen, R. (2010). Building a corpus: What are the basics? In A. O’Keeffe & M. McCarthy (Eds.), The Routledge handbook of corpus linguistics (pp. 31–38). London: Routledge.

    Google Scholar 

  • Reznicek, M., Lüdeling, A., Krummes, C., Schwantuschke, F., Walter, M., Schmidt, K., Hirschmann, H., & Andreas, T. (2012). Das Falko-Handbuch. Korpusaufbau und Annotationen. Humboldt-Universität zu Berlin, Technical Report Version 2.01, Berlin.

  • Riester, A., Killmann, L., Lorenz, D., & Portz, M. (2007). Richtlinien zur Annotation von Gegebenheit und Kontrast in Projekt A1. Draft version, November 2007. SFB 732, University of Stuttgart, Technical Report, Stuttgart.

  • Ritz, J. (2010). Using tf-idf-related measures for determining the anaphoricity of noun phrases. In Proceedings of KONVENS 2010 (pp. 85–92). Saarbrücken.

  • Ritz, J., Dipper, S., & Götze, M. (2008). Annotation of information structure: An evaluation across different types of texts. In N. Calzolari, K. Choukri, B. Maegaard, J. Mariani, J. Odjik, S. Piperidis, & D. Tapias (Eds.), Proceedings of the 6th international conference on language resources and evaluation (LREC-2008) (pp. 2137–2142). Marrakech.

  • Sabou, M., Bontcheva, K., Derczynski, L., & Scharl, A. (2014). Corpus annotation through crowdsourcing: Towards best practice guidelines. In N. Calzolari, K. Choukri, T. Declerck, H. Loftsson, B. Maegaard, J. Mariani, A. Moreno, J. Odijk, & S. Piperidis (Eds.), Proceedings of the ninth international conference on language resources and evaluation (LREC’14). Reykjavik: ELRA.

    Google Scholar 

  • Santorini, B. (1990). Part-of-speech tagging guidelines for the Penn Treebank Project (3rd Revision). University of Pennsylvania, Technical Report.

  • Schmid, H. (1994). Probabilistic part-of-speech tagging using decision trees. In Proceedings of the conference on new methods in language processing (pp. 44–49). Manchester.

  • Silveira, N., Dozat, T., de Marneffe, M.-C., Bowman, S. R., Connor, M., Bauery, J., & Manning, C. D. (2014). A gold standard dependency corpus for English. In Proceedings of the ninth international conference on language resources and evaluation (LREC-2014) (pp. 2897–2904). Reykjavik, Iceland.

  • Sinclair, J. (2004). Trust the text. London: Routledge.

    Google Scholar 

  • Snow, R., O’Connor, B., Jurafsky, D., & Ng, A. (2008). Cheap and fast—But is it good? Evaluating non-expert annotations for natural language tasks. In Proceedings of the 2008 conference on empirical methods in natural language processing (EMNLP 2008) (pp. 254–263). Honolulu, HI.

  • Socher, R., Bauer, J., Manning, C. D., & Ng, A. Y. (2013). Parsing with compositional vector grammars. In Proceedings of the 51st annual meeting of the association for computational linguistics (pp. 455–465). Sofia, Bulgaria.

  • Stede, M. (2004). The Potsdam commentary corpus. In Webber, B., & Byron, D. K. (Eds.), Proceeding of the ACL-04 workshop on discourse annotation (pp. 96–102). Barcelona, Spain.

  • Stede, M. (2008). Disambiguating rhetorical structure. Research on Language and Computation, 6(3), 311–332.

  • Stede, M., & Neumann, A. (2014). Potsdam commentary corpus 2.0: Annotation for discourse research. In Proceedings of the language resources and evaluation conference (LREC ‘14) (pp. 925–929). Reykjavik.

  • Taboada, M., & Mann, W. C. (2006). Rhetorical structure theory: Looking back and moving ahead. Discourse Studies, 8, 423–459.

    Article  Google Scholar 

  • Telljohann, H., Hinrichs, E. W., Kübler, S., Zinsmeister, H., & Beck, K. (2012). Stylebook for the Tübingen Treebank of Written German (TüBa-D/Z). Seminar für Sprachwissenschaft, Universität Tübingen, Technical Report.

  • Weischedel, R., Pradhan, S., Ramshaw, L., Kaufman, J., Franchini, M., El-Bachouti, M., Xue, N., Palmer, M., Hwang, J. D., Bonial, C., Choi, J., Mansouri, A., Foster, M., Hawwary, A.-A., Marcus, M., Taylor, A., Greenberg, C., Hovy, E., Belvin, R., & Houston, A. (2012). OntoNotes Release 5.0. Linguistic Data Consortium, Philadelphia, Technical Report.

  • Yimam, S. M., Gurevych, I., Castilho, R. Eckart de, & Biemann, C. (2013). WebAnno: A flexible, web-based and visually supported system for distributed annotations. In Proceedings of the 51st annual meeting of the association for computational linguistics (pp. 1–6). Sofia, Bulgaria.

Download references

Acknowledgments

I would like to thank the participants, past, present and future, of the course LING-367 ‘Computational Corpus Linguistics’, for their contributions to the corpus described in this paper. Special thanks are due to Dan Simonson for his help in preparing the data. For a current list of contributors and a link to the course syllabus, please see http://corpling.uis.georgetown.edu/gum. I am also grateful for very helpful suggestions from Aurelie Herbelot, Anke Lüdeling, Mark Sicoli, Manfred Stede, the editors, and three anonymous reviewers; the usual disclaimers apply.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Amir Zeldes.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zeldes, A. The GUM corpus: creating multilayer resources in the classroom. Lang Resources & Evaluation 51, 581–612 (2017). https://doi.org/10.1007/s10579-016-9343-x

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10579-016-9343-x

Keywords

Navigation