Measuring Intelligibility of Japanese Learner English

  • Emi Izumi
  • Kiyotaka Uchimoto
  • Hitoshi Isahara
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4139)


Although pursuing accuracy is important in language learning or teaching, knowing what types of errors interfere with communication and what types do not would be more beneficial for efficiently enhancing communicative competence. Language learners could be greatly helped by a system that detected errors in learner language and automatically measured their effect on intelligibility. In this paper, we reported our attempt, based on machine learning, to measure the intelligibility of learner language. In the learning process, the system referred to the BLEU and NIST scores between the learners’ original sentences and their back translation (or corrected sentences), the log-probability of the parse, sentence length, and error types (manually or automatically assigned) as a key feature. We found that the system can distinguish between intelligible sentences and others (unnatural and unintelligible) rather successfully, but still has a lot of difficulties in distinguishing the three levels of intelligibility.


Machine Translation Sentence Length Proficiency Level Communicative Competence Original Sentence 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Briscoe, E., Carroll, J.: Robust Accurate Statistical Annotation of General Text. In: Proceedings of the Third International Conference on Language Resources and Evaluation, Las Palmas, Gran Canaria, pp. 1499–1504 (2002)Google Scholar
  2. 2.
    Burnstein, J.: The E-rater scoring engine: Automated Essay Scoring with Natural Language Processing. In: Shermis, M.D., Burnstein, J. (eds.) Automated Essay Scoring: A Cross-Disciplinary Perspective, pp. 113–122. Lawrence Erlbaum Associates, Inc., New Jersey (2003)Google Scholar
  3. 3.
    Canale, M., Swain, M.: Theoretical Bases of Communicative Approach to Second Language Teaching and Testing. Applied Linguistics 1, 1–47 (1980)CrossRefGoogle Scholar
  4. 4.
    Ellis, R.: Task-based Language Learning and Teaching. Oxford University Press, Oxford (2003)Google Scholar
  5. 5.
    Izumi, E., Saiga, T., Uchimoto, K., Supnithi, T., Isahara, H.: Automatic Error Detection in the Japanese Learners English Spoken Data. In: Companion Volume of the Proceedings of the 41st Annual Meeting of the Association of Computational Linguistics (ACL 2003), Japan, pp. 145–148 (2003)Google Scholar
  6. 6.
    Izumi, E., Uchimoto, K., Isahara, H.: Standard Speaking Test (SST) Speech Corpus of Japanese Learners’ English and Automatic Detection of Learners’ Errors. International Computer Achieves of Modern and Medieval English (ICAME) Journal 28, 31–48 (2004)Google Scholar
  7. 7.
    Joachims, T.: Text Categorization with Support Vector Machines. In: Nédellec, C., Rouveirol, C. (eds.) ECML 1998. LNCS, vol. 1398, pp. 137–142. Springer, Heidelberg (1998)CrossRefGoogle Scholar
  8. 8.
    Kudo, T., Matsumoto, Y.: Chunking with Support Vector Machines. In: Proceedings of the 2nd Meeting of the North American Chapter of the Association of Computational Linguistics, Pittsburgh, pp. 192–199 (2001)Google Scholar
  9. 9.
    Larkey, L.S.: Automatic Essay Grading Using Text Categorization Techniques. In: Proceedings of the 21st Annual International Conference on Research and Development in Information Retrieval (SIGIR 1998), Australia, pp. 90–95 (1998)Google Scholar
  10. 10.
    NIST: Automatic Evaluation of Machine Translation Quality Using N-gram Co-Occurrence Statistics, Philadelphia (2002)Google Scholar
  11. 11.
    Page, E.B.: Computer Grading of Student Prose, Using Modern Concepts and Software. Journal of Experimental Education 62, 127–142 (1994)CrossRefGoogle Scholar
  12. 12.
    Page, E.B., Petersen, N.S.: The Computer Moves into Essay Grading: Updating the Ancient Test. Phi Delta Kappan 76(7), 561–565 (1995)Google Scholar
  13. 13.
    Papineni, K., Roukos, S., Ward, T., Zhu, W.: BLEU: A Method for Automatic Evaluation of Machine Translation. In: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL), USA, pp. 311–318 (2002)Google Scholar
  14. 14.
    Shermis, M.D., Burnstein, J. (eds.): Automated Essay Scoring: A Cross-Disciplinary Perspective, pp. xiii–xvi. Lawrence Erlbaum Associates, Inc., New Jersey (2003)Google Scholar
  15. 15.
    Skehan, P.: A Cognitive Approach to Language Learning. Oxford University Press, Oxford (1998)Google Scholar
  16. 16.
    Uchimoto, K., Hayashida, N., Ishida, T., Isahara, H.: Automatic Rating of Machine Translatability. In: Proceedings of the MT Summit X, Thailand, pp. 235–242 (2005)Google Scholar
  17. 17.
    Williams, R.: Automated Essay Grading: An Evaluation of Four Conceptual Models. In: Proceedings of the 10th Annual Teaching Learning Forum, Curtin University of Technology, Perth, pp. 7–9 (2001),

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Emi Izumi
    • 1
  • Kiyotaka Uchimoto
    • 1
  • Hitoshi Isahara
    • 1
  1. 1.Computational Linguistics GroupNational Institute of Information and Communications TechnologyKyotoJapan

Personalised recommendations