Advertisement

Language Resources and Evaluation

, Volume 49, Issue 3, pp 659–683 | Cite as

CityU corpus of essay drafts of English language learners: a corpus of textual revision in second language writing

  • John Lee
  • Chak Yan Yeung
  • Amir Zeldes
  • Marc Reznicek
  • Anke Lüdeling
  • Jonathan Webster
Original Paper

Abstract

Learner corpora consist of texts produced by non-native speakers. In addition to these texts, some learner corpora also contain error annotations, which can reveal common errors made by language learners, and provide training material for automatic error correction. We present a novel type of error-annotated learner corpus containing sequences of revised essay drafts written by non-native speakers of English. Sentences in these drafts are annotated with comments by language tutors, and are aligned to sentences in subsequent drafts. We describe the compilation process of our corpus, present its encoding in TEI XML, and report agreement levels on the error annotations. Further, we demonstrate the potential of the corpus to facilitate research on textual revision in L2 writing, by conducting a case study on verb tenses using ANNIS, a corpus search and visualization platform.

Keywords

Learner corpus Textual revision Feedback English as a second language Multi-layer corpus annotation Corpus search and visualization 

Notes

Acknowledgments

The work described in this article was supported by a grant from the Germany / Hong Kong Joint Research Scheme sponsored by the Research Grants Council of Hong Kong and the German Academic Exchange Service (Reference No. G_HK013/11).

References

  1. Andreu Andrés, M. A., Guardiola, A. A., Matarredona, M. B., MacDonald, P., Fleta, B. M., & Pérez Sabater, C. (2010). Analysing EFL learner output in the MiLC Project: An error * it’s, but which tag? In M. C. Campoy-Cubillo, B. Bellés-Fortuño, & M. L. Gea-Valor (Eds.), Corpus-based approaches to English language teaching (pp. 167–179). London: Continuum.Google Scholar
  2. Ashwell, T. (2000). Patterns of teacher response to student writing in a multiple-draft composition classroom: Is content feedback followed by form feedback the best method? Journal of Second Language Writing, 9(3), 227–257.CrossRefGoogle Scholar
  3. Barzilay, R., & Elhadad, N. (2003). Sentence Alignment for Monolingual Comparable Corpora. In Proceedings of the 2003 conference on empirical methods in natural language processing. Sapporo, Japan, pp. 25–32.Google Scholar
  4. Biber, D., Nekrasova, T., & Horn, B. (2011). The effectiveness of feedback for L1-English and L2-writing development: A meta-analysis. TOEFL iBT research report.Google Scholar
  5. Bitchener, J., & Ferris, D. R. (2012). Written corrective feedback in Second Language Acquisition and Writing. New York, NY: Routledge.Google Scholar
  6. Burstein, J., Chodorow, M., & Leacock, C. (2004). Automated essay evaluation: The criterion online writing service. AI Magazine, 25(3), 27–36.Google Scholar
  7. Chandler, J. (2003). The efficacy of various kinds of error feedback for improvement in the accuracy and fluency of L2 student writing. Journal of Second Language Writing, 12(3), 267–296.CrossRefGoogle Scholar
  8. Dahlmeier, D., & Ng, H. T. (2011). Grammatical error correction with alternating structure optimization. Proceedings of the 49th annual meeting of The Association for Computational Linguistics (pp. 915–923). Stroudsburg, PA: ACL.Google Scholar
  9. Dahlmeier, D., Ng, H. T., & Wu, S. M. (2013). Building a large annotated corpus of learner English: The NUS corpus of learner English. In Proceedings of the Eighth workshop on innovative use of NLP for building educational applications, 22–31.Google Scholar
  10. Dale, R., & Kilgarriff, A. (2011). Helping our own: The HOO 2011 pilot shared task. In Proceedings of the 13th European Workshop on Natural Language Generation (ENLG). Nancy, France, 242–249.Google Scholar
  11. Dipper, S. (2005). XML-based stand-off representation and exploitation of multi-level linguistic annotation. In Proceedings of Berliner XML Tage 2005 (BXML 2005). Berlin, Germany, 39–50.Google Scholar
  12. Eriksson, A., Finnegan, D., Kauppinen, A., Wiktorsson, M., Wärnsby, A., & Withers, P. (2012). MUCH: The Malmö University-Chalmers Corpus of Academic Writing as a Process. In Proceedings of the 10th teaching and language corpora conference.Google Scholar
  13. Fathman, A. K. & Whalley, E. (1990). Teacher response to student writing: Focus on form versus content. In Kroll, B. (ed.) Second language writing: Research insights for the classroom, pp. 178–190.Google Scholar
  14. Ferris, D. R. (1997). The influence of teacher commentary on student revision. TESOL Quarterly, 31(2), 315–339.CrossRefGoogle Scholar
  15. Ferris, D. R. (2006). Does error feedback help student writers? New evidence on the short-and long-term effects of written error correction. In K. Hyland & F. Hyland (Eds.), Feedback in second language writing: Contexts and issues (pp. 81–104). Cambridge: Cambridge University Press.CrossRefGoogle Scholar
  16. Ferris, D. R., & Roberts, B. (2001). Error feedback in L2 writing classes: How explicit does it need to be? Journal of Second Language Writing, 10, 161–184.CrossRefGoogle Scholar
  17. Foster, J., Wagner, J., & van Genabith, J. (2008). Adapting a WSJ-trained parser to grammatically noisy text. In Proceedings of ACL. Google Scholar
  18. Graham, S., & Perin, D. (2007). A meta-analysis of writing instruction for adolescent students. Journal of Educational Psychology, 99(3), 445–476.CrossRefGoogle Scholar
  19. Granger, S. (1999). Use of tenses by advanced EFL learners: Evidence from error-tagged computer corpus. In H. Hasselgård (Ed.), Out of Corpora—Studies in Honour of Stig Johansson (pp. 191–202). Amsterdam, Atlanta: Rodopi.Google Scholar
  20. Granger, S. (2004). Computer learner corpus research: Current status and future prospects. Language and Computers, 23, 123–145.Google Scholar
  21. Granger, S. (2008). Learner corpora. In A. Lüdeling & M. Kyto (Eds.), Corpus linguistics: An international handbook (Vol. 1). Berlin: Mouton de Gruyter.Google Scholar
  22. Granger, S., Dagneaux, E., Meunier, F., & Paquot, M. (2009). The international corpus of learner English. Version 2. Handbook and CD-ROM. Louvain-la-Neuve: Presses universitaires de Louvain.Google Scholar
  23. Han, N.-R., Chodorow, M., & Leacock, C. (2006). Detecting errors in English article usage by non-native speakers. Natural Language Engineering, 12(2), 115–129.CrossRefGoogle Scholar
  24. Ide, N., Bonhomme, P., & Romary, L. (2000). XCES: An XML-based encoding standard for linguistic corpora. Proceedings of the second international language resources and evaluation conference (pp. 825–830). Paris: ELRA.Google Scholar
  25. Krause, T. & Zeldes, A. (2014). ANNIS3: A new architecture for generic corpus query and visualization. To appear in Literary and Linguistic Computing. http://dsh.oxfordjournals.org/content/early/2014/12/02/llc.fqu057
  26. Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159–174.CrossRefGoogle Scholar
  27. Lee, J., & Seneff, S. (2008). An analysis of grammatical errors in nonnative speech in English. In Proceedings of the IEEE Workshop on Spoken Language Technology 2008. pp. 89–92.Google Scholar
  28. Lee, J., Tetreault, J., & Chodorow, M. (2009). Human evaluation of article and noun number usage: Influences of context and construction variability. In Proceedings of the Third Linguistic Annotation Workshop, pp. 60–63.Google Scholar
  29. Lipnevich, A. A., & Smith, J. K. (2009). “I really need feedback to learn:” Students’ perspectives on the effectiveness of the differential feedback messages. Educational Assessment, Evaluation and Accountability, 21(4), 347–367.CrossRefGoogle Scholar
  30. Lüdeling, A., Doolittle, S., Hirschmann, H., Schmidt, K., & Walter, M. (2008). Das Lernerkorpus Falko. Deutsch als Fremdsprache, 2, 67–73.Google Scholar
  31. Lüdeling, A., Walter, M., Kroymann, E., & Adolphs, P. (2005). Multi-level Error Annotation in Learner Corpora. In Proceedings of Corpus Linguistics 2005. Birmingham.Google Scholar
  32. Lüdeling, A., & Hirschmann, H. (to appear). Error Annotation. In Granger, S., Gilquin, G., & Meunier, F. (eds.), The Cambridge Handbook of Learner Corpus Research. Cambridge: Cambridge University Press.Google Scholar
  33. Marcus, M. P., Santorini, B., & Marcinkiewicz, M. A. (1993). Building a large annotated corpus of English: The Penn Treebank. Special Issue on Using Large Corpora, Computational Linguistics, 19(2), 313–330.Google Scholar
  34. Nagata, R., Whittaker, E., & Sheinman, V. (2011). Creating a manually error-tagged and shallow-parsed learner corpus. Proceedings of the 49th annual meeting of the association for computational linguistics (pp. 1210–1219). Stroudsburg, PA: ACL.Google Scholar
  35. Nesi, H., Sharpling, G., & Ganobcsik-Williams, L. (2004). Student papers across the curriculum: designing and developing a corpus of British student writing. Computers and Composition, 21(4), 439–450.CrossRefGoogle Scholar
  36. Nguyen, N. L. T. & Miyao, Y. (2013). Alignment-based annotation of proofreading texts toward professional writing assistance. In Proceedings of the international joint conference on natural language processing, pp. 753–759.Google Scholar
  37. Nicholls, D. (2003). The Cambridge learner corpus: Error coding and analysis for lexicography and ELT. In Proceedings of the corpus linguistics 2003 conference.Google Scholar
  38. Paulus, T. M. (1999). The effect of peer and teacher feedback on student writing. Journal of Second Language Writing, 8(3), 265–289.CrossRefGoogle Scholar
  39. Polio, C., & Fleck, C. (1998). “If I only had more time:” ESL learners’ changes in linguistic accuracy on essay revisions. Journal of Second Language Writing, 7(1), 43–68.CrossRefGoogle Scholar
  40. Reznicek, M., Lüdeling, A., & Hirschmann, H. (2013). Competing target hypotheses in the Falko corpus: A flexible multi-layer corpus architecture. In A. Díaz-Negrillo, N. Ballier, & P. Thompson (Eds.), Automatic treatment and analysis of learner corpus data (pp. 101–124). Amsterdam: John Benjamins.CrossRefGoogle Scholar
  41. Rosen, A., Hana, J., Stindlova, B., & Feldman, A. (2014). Evaluating and automating the annotation of a learner corpus. Language Resources and Evaluation, 48, 65–92.CrossRefGoogle Scholar
  42. Rozovskaya, A., & Roth, D. (2010). Annotating ESL errors: Challenges and rewards. In: Proceedings of NAACL’10 workshop on innovative use of NLP for building educational applications.Google Scholar
  43. Russell, J., & Spada, N. (2006). The effectiveness of corrective feedback for the acquisition of L2 grammar: A meta-analysis of the research. In J. Norris & L. Ortega (Eds.), Synthesizing research on language learning and teaching (Language learning and language teaching 13) (pp. 133–164). Amsterdam and Philadelphia: John Benjamins.Google Scholar
  44. Shemtov, H. (1993). Text Alignment in a Tool for Translating Revised Documents. Proceedings of the sixth conference on European chapter of the Association for Computational Linguistics (EACL-93) (pp. 449–453). Stroudsburg, PA: ACL.CrossRefGoogle Scholar
  45. Snover, M., Dorr, B., Schwartz, R., Micciulla, L., & Makhoul, J. (2006). A study of translation edit rate with targeted human annotation. In Proceedings of the 7th conference of the association for machine translation in the Americas. Cambridge, MA, pp. 223–231.Google Scholar
  46. Tetreault, J. R., & Chodorow, M. (2008). Native judgments of non-native usage: Experiments in preposition error detection. In Proceedings of the workshop on human judgements in computational linguistics, pp. 24–32.Google Scholar
  47. Toutanova, K., Klein, D., Manning, C. D., & Singer, Y. (2003). Feature-rich part-of-speech tagging with a cyclic dependency network. Proceedings of the 2003 conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology (NAACL-HLT 2003) (pp. 252–259). Stroudsburg, PA: ACL.Google Scholar
  48. Toutanova, K., & Manning, C. D. (2000). Enriching the knowledge sources used in a maximum entropy part-of-speech tagger. In Proceedings of the 2000 joint SIGDAT conference on empirical methods in natural language processing and very large corpora. Hong Kong, pp. 63–70.Google Scholar
  49. Truscott, J. (1996). The case against grammar correction in L2 writing classes. Language Learning, 46(2), 327–369.CrossRefGoogle Scholar
  50. Truscott, J., & Hsu, A. Y.-P. (2008). Error correction, revision, and learning. Journal of Second Language Writing, 17(4), 292–305.CrossRefGoogle Scholar
  51. Webster, J., Chan, A., & Lee, J. (2011). Introducing an online language learning environment and its corpus of tertiary student writing. Asia Pacific World, 2(2), 44–65.CrossRefGoogle Scholar
  52. Wible, D., Kuo, C.-H., Chien, F.-L., Liu, A., & Tsao, N.-L. (2001). A web-based EFL writing environment: Integrating information for learners, teachers, and researchers. Computers & Education, 37(3–4), 297–315.CrossRefGoogle Scholar
  53. Zeldes, A., Ritz, J., Lüdeling, A., & Chiarcos, C. (2009). ANNIS: A search tool for multi-layer annotated corpora. In Proceedings of corpus linguistics 2009. Liverpool, UK.Google Scholar
  54. Zipser, F., & Romary, L. (2010). A model oriented approach to the mapping of annotation formats using standards. Proceedings of the workshop on language resource and language technology standards, LREC-2010 (pp. 7–18). Malta: Valletta.Google Scholar

Copyright information

© Springer Science+Business Media Dordrecht 2015

Authors and Affiliations

  • John Lee
    • 1
  • Chak Yan Yeung
    • 1
  • Amir Zeldes
    • 2
  • Marc Reznicek
    • 3
  • Anke Lüdeling
    • 4
  • Jonathan Webster
    • 1
  1. 1.City University of Hong KongKowloonHong Kong
  2. 2.Georgetown UniversityWashingtonUSA
  3. 3.Universidad Complutense de MadridMadridSpain
  4. 4.Humboldt-Universität zu BerlinBerlinGermany

Personalised recommendations