Advertisement

ReaderBench Learns Dutch: Building a Comprehensive Automated Essay Scoring System for Dutch Language

  • Mihai DascaluEmail author
  • Wim Westera
  • Stefan Ruseti
  • Stefan Trausan-Matu
  • Hub Kurvers
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10331)

Abstract

Automated Essay Scoring has gained a wider applicability and usage with the integration of advanced Natural Language Processing techniques which enabled in-depth analyses of discourse in order capture the specificities of written texts. In this paper, we introduce a novel Automatic Essay Scoring method for Dutch language, built within the Readerbench framework, which encompasses a wide range of textual complexity indices, as well as an automated segmentation approach. Our method was evaluated on a corpus of 173 technical reports automatically split into sections and subsections, thus forming a hierarchical structure on which textual complexity indices were subsequently applied. The stepwise regression model explained 30.5% of the variance in students’ scores, while a Discriminant Function Analysis predicted with substantial accuracy (75.1%) whether they are high or low performance students.

Keywords

Automated Essay Scoring Textual complexity assessment Academic performance ReaderBench framework Dutch semantic models 

Notes

Acknowledgments

This work was partially funded by the 644187 EC H2020 Realising an Applied Gaming Eco-system (RAGE) project, by the FP7 208-212578 LTfLL project, as well as by University Politehnica of Bucharest through the “Excellence Research Grants” Program UPB–GEX 12/26.09.2016.

References

  1. 1.
    Landauer, T.K., Dumais, S.T.: A solution to Plato’s problem: the Latent Semantic Analysis theory of acquisition, induction and representation of knowledge. Psychol. Rev. 104(2), 211–240 (1997)CrossRefGoogle Scholar
  2. 2.
    Miller, T.: Essay assessment with Latent Semantic Analysis. J. Educ. Comput. Res. 29(4), 495–512 (2003)CrossRefGoogle Scholar
  3. 3.
    Blei, D.M., Ng, A.Y., Jordan, M.I.: Latent Dirichlet Allocation. J. Mach. Learn. Res. 3(4–5), 993–1022 (2003)zbMATHGoogle Scholar
  4. 4.
    Crossley, S.A., McNamara, D.S.: Text coherence and judgments of essay quality: models of quality and coherence. In: 33rd Annual Conference of the Cognitive Science Society, pp. 1236–1231. Cognitive Science Society, Boston (2011)Google Scholar
  5. 5.
    Nelson, J., Perfetti, C., Liben, D., Liben, M.: Measures of Text Difficulty: Testing their Predictive Value for Grade Levels and Student Performance. Council of Chief State School Officers, Washington, DC (2012)Google Scholar
  6. 6.
    Dascalu, M.: Analyzing Discourse and text complexity for learning and collaborating, Studies in Computational Intelligence, vol. 534. Springer, Cham (2014)CrossRefGoogle Scholar
  7. 7.
    Dascalu, M., Dessus, P., Bianco, M., Trausan-Matu, S., Nardy, A.: Mining texts, learner productions and strategies with ReaderBench. In: Peña-Ayala, A. (ed.) Educational Data Mining. SCI, vol. 524, pp. 345–377. Springer, Cham (2014). doi: 10.1007/978-3-319-02738-8_13 CrossRefGoogle Scholar
  8. 8.
    Dascalu, M., McNamara, D.S., Trausan-Matu, S., Stavarache, L.L., Allen, L.K.: Cohesion network analysis of CSCL participation. Behavior Research Methods, PP. 1–16 (2017)Google Scholar
  9. 9.
    Dascalu, M., Trausan-Matu, S., McNamara, D.S., Dessus, P.: ReaderBench – automated evaluation of collaboration based on cohesion and dialogism. Int. J. Comput. Support. Collaborative Learn. 10(4), 395–423 (2015)CrossRefGoogle Scholar
  10. 10.
    Bakhtin, M.M.: The dialogic imagination: four essays. The University of Texas Press, Austin (1981)Google Scholar
  11. 11.
    Dascalu, M., Allen, K.A., McNamara, D.S., Trausan-Matu, S., Crossley, S.A.: Modeling comprehension processes via automated analyses of dialogism. In: 39th Annual Meeting of the Cognitive Science Society (CogSci 2017). Cognitive Science Society, London (2017, in Press)Google Scholar
  12. 12.
    Duyck, W., Desmet, T., Verbeke, L.P., Brysbaert, M.: WordGen: A tool for word selection and nonword generation in Dutch, English, German, and French. Behav. Res. Methods 36(3), 488–499 (2004)CrossRefGoogle Scholar
  13. 13.
    National Governors Association Center for Best Practices & Council of Chief State School Officers: Common Core State Standards. Authors, Washington D.C. (2010)Google Scholar
  14. 14.
    Powers, D.E., Burstein, J., Chodorow, M., Fowles, M.E., Kukich, K.: Stumping e-rater®: Challenging the Validity of Automated Essay Scoring. Educational Testing Service, Princeton (2001)Google Scholar
  15. 15.
    McNamara, D.S., Graesser, A.C., Louwerse, M.M.: Sources of text difficulty: Across the ages and genres. In: Sabatini, J.P., Albro, E., O’Reilly, T. (eds.) Measuring up: Advances in How we Assess Reading Ability, pp. 89–116. R&L Education, Lanham (2012)Google Scholar
  16. 16.
    Williams, R., Dreher, H.: Automatically grading essays with Markit©. J. Issues Informing Sci. Inform. Technol. 1, 693–700 (2004)CrossRefGoogle Scholar
  17. 17.
    Elliot, S.: IntelliMetric: from here to validity. In: Shermis, M.D., Burstein, J.C. (eds.) Automated Essay Scoring: A Cross Disciplinary Approach, pp. 71–86. Lawrence Erlbaum Associates, Mahwah (2003)Google Scholar
  18. 18.
    Crossley, S.A., Allen, L.K., McNamara, D.S.: The Writing Pal: a writing strategy tutor. In: Crossley, S.A., McNamara, D.S. (eds.) Handbook on Educational Technologies for Literacy. Taylor & Francis, Routledge, New York (in press)Google Scholar
  19. 19.
    McNamara, D.S., Crossley, S.A., Roscoe, R., Allen, L.K., Dai, J.: A hierarchical classification approach to automated essay scoring. Assessing Writ. 23, 35–59 (2015)CrossRefGoogle Scholar
  20. 20.
    Pander Maat, H.L.W., Kraf, R.L., van den Bosch, A., van Gompel, M., Kleijn, S., Sanders, T.J.M., van der Sloot, K.: T-Scan: a new tool for analyzing Dutch text. Comput. Linguist. Neth. J. 4, 53–74 (2014)Google Scholar
  21. 21.
    Graesser, A.C., McNamara, D.S., Louwerse, M.M., Cai, Z.: Coh-Metrix: Analysis of text on cohesion and language. Behav. Res. Methods Instrum. Comput. 36(2), 193–202 (2004)CrossRefGoogle Scholar
  22. 22.
    Graesser, A.C., McNamara, D.S., Kulikowich, J.M.: Coh-Metrix: Providing multilevel analyses of text characteristics. Educ. Res. 40(5), 223–234 (2011)CrossRefGoogle Scholar
  23. 23.
    McNamara, D.S., Graesser, A.C., McCarthy, P., Cai, Z.: Automated Evaluation of Text and Discourse with Coh-Metrix. Cambridge University Press, Cambridge (2014)CrossRefGoogle Scholar
  24. 24.
    Kraf, R., Lentz, L., Pander Maat, H.: Drie Nederlandse instrumenten voor het automatisch voorspellen van begrijpelijkheid. Een klein consumentenonderzoek. Tijdschift voor Taalbeheersing 33(3), 249–265 (2011)Google Scholar
  25. 25.
    CGN Consortium: e-Lex, lexicale databank (lexical database). Instituut voor Nederlandse Taal, Leiden, the Netherlands (2017)Google Scholar
  26. 26.
    Owen, S., Anil, R., Dunning, T., Friedman, E.: Mahout in Action. Manning Publications Co., Greenwich (2011)Google Scholar
  27. 27.
    McCallum, A.K.: MALLET: A Machine Learning for Language Toolkit (2002). http://mallet.cs.umass.edu/
  28. 28.
    Blei, D.M.: Probabilistic topic models. Commun. ACM 55(4), 77–84 (2012)CrossRefGoogle Scholar
  29. 29.
    Galley, M., McKeown, K.: Improving word sense disambiguation in lexical chaining. In: 18th International Joint Conference on Artificial Intelligence (IJCAI 2003), pp. 1486–1488. Morgan Kaufmann Publishers, Inc., Acapulco (2003)Google Scholar
  30. 30.
    Budanitsky, A., Hirst, G.: Evaluating WordNet-based measures of lexical semantic relatedness. Comput. Linguist. 32(1), 13–47 (2006)CrossRefzbMATHGoogle Scholar
  31. 31.
    Trausan-Matu, S., Dascalu, M., Dessus, P.: Textual complexity and discourse structure in computer-supported collaborative learning. In: Cerri, Stefano A., Clancey, William J., Papadourakis, G., Panourgia, K. (eds.) ITS 2012. LNCS, vol. 7315, pp. 352–357. Springer, Heidelberg (2012). doi: 10.1007/978-3-642-30950-2_46 CrossRefGoogle Scholar
  32. 32.
    Wresch, W.: The imminence of grading essays by computer—25 years later. Comput. Compos. 10(2), 45–58 (1993)CrossRefGoogle Scholar
  33. 33.
    Shannon, C.E.: Prediction and entropy of printed English. Bell Syst. Tech. J. 30, 50–64 (1951)CrossRefzbMATHGoogle Scholar
  34. 34.
    Gervasi, V., Ambriola, V.: Quantitative assessment of textual complexity. In: Barbaresi, M.L. (ed.) Complexity in Language and Text, pp. 197–228. Plus, Pisa, Italy (2002)Google Scholar
  35. 35.
    van Dijk, T.A., Kintsch, W.: Strategies of Discourse Comprehension. Academic Press, New York (1983)Google Scholar
  36. 36.
    Dascalu, M., Dessus, P., Trausan-Matu, Ş., Bianco, M., Nardy, A.: ReaderBench, an environment for analyzing text complexity and reading strategies. In: Lane, H.Chad, Yacef, K., Mostow, J., Pavlik, P. (eds.) AIED 2013. LNCS, vol. 7926, pp. 379–388. Springer, Heidelberg (2013). doi: 10.1007/978-3-642-39112-5_39 CrossRefGoogle Scholar
  37. 37.
    Manning, C.D., Schütze, H.: Foundations of statistical Natural Language Processing. MIT Press, Cambridge (1999)zbMATHGoogle Scholar
  38. 38.
    van der Vliet, H.: The Referentiebestand Nederlands as a multi-purpose lexical database. Int. J. Lexicogr. 20(3), 239–257 (2007)CrossRefGoogle Scholar
  39. 39.
    Miller, G.A.: WordNet: a lexical database for English. Commun. ACM 38(11), 39–41 (1995)CrossRefGoogle Scholar
  40. 40.
    Zijlstra, H., van Meerveld, T., van Middendorp, H., Pennebaker, J.W., Geenen, R.: De Nederlandse versie van de Linguistic Inquiry and Word Count (LIWC), een gecomputeriseerd tekstanalyseprogramma [Dutch version of the Linguistic Inquiry and Word Count (LIWC), a computerized text analysis program]. Gedrag & Gezondheid 32, 273–283 (2004)Google Scholar
  41. 41.
    Westera, W., Nadolski, N., Hummel, H.: Serious gaming analytics: what students’ log files tell us about gaming and learning. Int. J. Serious Games 1(2), 35–50 (2014)CrossRefGoogle Scholar
  42. 42.
    Klecka, W.R.: Discriminant Analysis. Quantitative Applications in the Social Sciences Series, vol. 19. Sage Publications, Thousand Oaks (1980)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Mihai Dascalu
    • 1
    • 2
    Email author
  • Wim Westera
    • 3
  • Stefan Ruseti
    • 1
  • Stefan Trausan-Matu
    • 1
    • 2
  • Hub Kurvers
    • 3
  1. 1.Faculty of Automatic Control and ComputersUniversity “Politehnica” of BucharestBucharestRomania
  2. 2.Academy of Romanian ScientistsBucharestRomania
  3. 3.Open University of the NetherlandsHeerlenThe Netherlands

Personalised recommendations