Abstract
This paper investigates the suitability of state-of-the-art natural language processing (NLP) tools for parsing the spoken language of second language learners of English. The task of parsing spoken learner-language is important to the domains of automated language assessment (ALA) and computer-assisted language learning (CALL). Due to the non-canonical nature of spoken language (containing filled pauses, non-standard grammatical variations, hesitations and other disfluencies) and compounded by a lack of available training data, spoken language parsing has been a challenge for standard NLP tools. Recently the Redshift parser (Honnibal et al. In: Proceedings of CoNLL (2013)) has been shown to be successful in identifying grammatical relations and certain disfluencies in native speaker spoken language, returning unlabelled dependency accuracy of 90.5% and a disfluency F-measure of 84.1% (Honnibal & Johnson: TACL 2, 131-142 (2014)). We investigate how this parser handles spoken data from learners of English at various proficiency levels. Firstly, we find that Redshift’s parsing accuracy on non-native speech data is comparable to Honnibal & Johnson’s results, with 91.1% of dependency relations correctly identified. However, disfluency detection is markedly down, with an F-measure of just 47.8%. We attempt to explain why this should be, and investigate the effect of proficiency level on parsing accuracy. We relate our findings to the use of NLP technology for CALL and ALA applications.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Ballesteros, M., Nivre, J.: Going to the roots of dependency parsing. Computational Linguistics 39(1) (2013)
Biber, D.: Dimensions of register variation: a cross-linguistic comparison. Cambridge University Press, Cambridge (1995)
Brazil, D.: A grammar of speech. Oxford University Press, Oxford (1995)
Bresnan, J.: Lexical-Functional Syntax. Blackwell, Oxford (2001)
Briscoe, T., Carroll, J., Watson, R.: The second release of the RASP System. In: Proceedings of the COLING/ACL 2006 Interactive Presentations Session. Association for Computational Linguistics (2006)
Buchholz, S., Marsi, E.: CoNLL-X shared task on multilingual dependency parsing. In: Proceedings of the 10th Conference on Computational Natural Language Learning (CoNLL-X). Association for Computational Linguistics (2006)
Caines, A., Bentz, C., Graham, C., Polzehl, T., Buttery, P.: Crowdsourcing a multi-lingual speech corpus: recording, transcription, and natural language processing. In: Proceedings of INTERSPEECH 2015. International Speech Communication Association (2015)
Caines, A., Buttery, P.: The effect of disfluencies and learner errors on the parsing of spoken learner language. In: Proceedings of the First Joint Workshop on Statistical Parsing of Morphologically Rich Languages and Syntactic Analysis of Non-Canonical Languages (2014)
Carter, R., McCarthy, M.: Spoken Grammar: where are we and where are we going? Applied Linguistics (in press)
Charniak, E., Johnson, M.: Edit detection and parsing for transcribed speech. In: Proceedings of the 2nd Meeting of the North American Chapter of the Association for Computational Linguistics (NAACL). Association for Computational Linguistics (2001)
Council of Europe: Common European Framework of Reference for Languages. Cambridge University Press, Cambridge (2001)
Foster, J., Çetinoǧlu, Ö., Wagner, J., Roux, J.L., Nivre, J., Hogan, D., van Genabith, J.: From news to comment: resources and benchmarks for parsing the language of Web 2.0. In: Proceedings of the 5th International Joint Conference on Natural Language Processing (IJCNLP). Association for Computational Linguistics (2011)
Godfrey, J., Holliman, E.: Switchboard-1 Release 2 LDC97S62. DVD (1993)
Godfrey, J.J., Holliman, E.C., McDaniel, J.: SWITCHBOARD: telephone speech corpus for research and development. In: Proceedings of Acoustics, Speech, and Signal Processing (ICASSP 1992). IEEE (1992)
Honnibal, M., Goldberg, Y., Johnson, M.: A non-monotonic arc-eager transition system for dependency parsing. In: Proceedings of the Seventh Conference on Computational Natural Language Learning. Association for Computational Linguistics (2013)
Honnibal, M., Johnson, M.: Joint incremental disfluency detection and dependency parsing. Transactions of the Association for Computational Linguistics 2, 131–142 (2014)
Hovy, D., Plank, B., Søgaard, A.: When POS data sets don’t add up: combatting sample bias. In: Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC). European Language Resources Association (2014)
Johnson, M., Charniak, E.: A TAG-based noisy channel model of speech repairs. In: Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL). Association for Computational Linguistics (2004)
Klein, D., Manning, C.D.: Accurate unlexicalized parsing. In: Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics (ACL). Association for Computational Linguistics (2003)
Kübler, S., McDonald, R., Nivre, J.: Dependency parsing. Morgan & Claypool Publishers, Synthesis Lectures on Human Language Technologies (2009)
Lease, M., Charniak, E.: Parsing biomedical literature. In: Dale, R., Wong, K.-F., Su, J., Kwong, O.Y. (eds.) IJCNLP 2005. LNCS (LNAI), vol. 3651, pp. 58–69. Springer, Heidelberg (2005)
de Marneffe, M.C., MacCartney, B., Manning, C.D.: Generating typed dependency parses from phrase structure parses. In: Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC). European Language Resources Association (2006)
de Marneffe, M.C., Manning, C.D.: The Stanford typed dependencies representation. In: Proceedings of the COLING Workshop on Cross-framework and Cross-domain Parser Evaluation (2008)
Mikheev, A.: Text segmentation. In: Mitkov, R. (ed.) The Oxford Handbook of Computational Linguistics. Oxford University Press, Oxford (2005)
Nivre, J.: Algorithms for deterministic incremental dependency parsing. Computational Linguistics 34(4), 513–553 (2008)
Qian, X., Liu, Y.: Disfluency detection using multi-step stacked learning. In: Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT). Association for Computational Linguistics (2013)
Rasooli, M.S., Tetreault, J.: Non-monotonic parsing of Fluent umm I Mean disfluent sentences. In: Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2014). Association for Computational Linguistics (2014)
Rimell, L., Clark, S.: Porting a lexicalized-grammar parser to the biomedical domain. Journal of Biomedical Informatics 42, 852–865 (2009)
Shriberg, E.: Preliminaries to a theory of speech disfluencies. Ph.D. thesis, University of California, Berkeley (1994)
Taylor, A., Marcus, M., Santorini, B.: The Penn Treebank: An Overview (2003)
Zhang, Y., Clark, S.: Syntactic processing using the generalized perceptron and beam search. Computational Linguistics 37(1), 105–151 (2011)
Zhang, Y., Nivre, J.: Transition-based dependency parsing with rich non-local features. In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (ACL-HLT). Association for Computational Linguistics (2011)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this paper
Cite this paper
Moore, R., Caines, A., Graham, C., Buttery, P. (2015). Incremental Dependency Parsing and Disfluency Detection in Spoken Learner English. In: Král, P., Matoušek, V. (eds) Text, Speech, and Dialogue. TSD 2015. Lecture Notes in Computer Science(), vol 9302. Springer, Cham. https://doi.org/10.1007/978-3-319-24033-6_53
Download citation
DOI: https://doi.org/10.1007/978-3-319-24033-6_53
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-24032-9
Online ISBN: 978-3-319-24033-6
eBook Packages: Computer ScienceComputer Science (R0)