Letting the Genie Out of the Lamp: Using Natural Language Processing Tools to Predict Math Performance

  • Scott Crossley
  • Victor Kostyuk
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10318)


This study examines links between natural language processing and its application in math education. Specifically, the study examines language production and math success in an on-line, blended learning math program. Unlike previous studies that have relied on correlational analyses between linguistic knowledge tests and standardized math tests or compared math success between proficient and non-proficient speakers of English, this study examines the linguistic features of students’ language production while e-mailing a virtual pedagogical agent. In addition, the study examines a number of non-linguistic features such as grade and objective met within the program. The findings indicate that linguistic features related to the use of standardized language use explain around 8% of math success. These linguistic features outperform non-linguistic features.


Natural language processing Online tutoring systems Math education Text analytics 



This research was supported in part by NSF 1623730. Opinions, conclusions, or recommendations do not necessarily reflect the views of the NSF.


  1. 1.
    LeFevre, J.A., Fast, L., Skwarchuk, S.L., Smith-Chant, B.L., Bisanz, J., Kamawar, D., Penner-Wilger, M.: Pathways to math: longitudinal predictors of performance. Child Dev. 81(6), 1753–1767 (2010). doi: 10.1111/j.1467-8624.2010.01508.x CrossRefGoogle Scholar
  2. 2.
    Vukovic, R.K., Lesaux, N.K.: The relationship between linguistic skills and arithmetic knowledge. Learn. Individ. Diff. 23, 87–91 (2013). doi: 10.1016/j.lindif.2012.10.007 CrossRefGoogle Scholar
  3. 3.
    Martiniello, M.: Language and the performance of English-language learners in math word problems. Harv. Educ. Rev. 78(2), 333–368 (2008)CrossRefGoogle Scholar
  4. 4.
    Adams, T.L.: Reading math: more than words can say. Read. Teach. 56(8), 786–795 (2003)Google Scholar
  5. 5.
    MacGregor, M., Price, E.: An exploration of aspects of language proficiency and algebra learning. J. Res. Math. Educ. 30, 449–467 (1999). doi: 10.2307/749709 CrossRefGoogle Scholar
  6. 6.
    Alt, M., Arizmendi, G.D., Beal, C.R.: The relationship between math and language: academic implications for children with specific language impairment and English language learners. Lang. Speech Hear. Serv. Sch. 45(3), 220–233 (2014). doi: 10.1044/2014_LSHSS-13-0003 CrossRefGoogle Scholar
  7. 7.
    Hampden-Thompson, G., Mulligan, G., Kinukawa, A., Halle, T.: Math Achievement of Language-Minority Students During the Elementary Years. U.S. Department of Education, National Center for Education Statistics, Washington, DC (2008)Google Scholar
  8. 8.
    Martiniello, M.: Linguistic complexity, schematic representations, and differential item functioning for English language learners in math tests. Educ. Assess. 14(3–4), 160–179 (2009). doi: 10.1080/10627190903422906 CrossRefGoogle Scholar
  9. 9.
    Crossley, S.A., Liu, R., McNamara, D.: Predicting math performance using natural language processing tools. In: Proceedings of the 7th International Learning Analytics and Knowledge (LAK) Conference. ACM, New York (2017)Google Scholar
  10. 10.
    Hernandez, F.: The relationship between reading and math achievement of middle school students as measured by the Texas assessment of knowledge and skills. Doctoral dissertation (2013)Google Scholar
  11. 11.
    Cummins, J.: Linguistic interdependence and the educational development of bilingual children. Rev. Educ. Res. 49(2), 222–251 (1979)CrossRefGoogle Scholar
  12. 12.
    Ardasheva, Y., Tretter, T.R., Kinny, M.: English language learners and academic achievement: revisiting the threshold hypothesis. Lang. Learn. 62(3), 769–812 (2012). doi: 10.1111/j.1467-9922.2011.00652.x CrossRefGoogle Scholar
  13. 13.
    Mosqueda, E., Maldonado, S.I.: The effects of English language proficiency and curricular pathways: Latina/os’ math achievement in secondary schools. Equity Excell. Educ. 46(2), 202–219 (2013). doi: 10.1080/10665684.2013.780647 CrossRefGoogle Scholar
  14. 14.
    Wang, J., Goldschmidt, P.: Opportunity to learn, language proficiency, and immigrant status effects on math achievement. J. Educ. Res. 93(2), 101–111 (1999). doi: 10.1080/00220679909597634 CrossRefGoogle Scholar
  15. 15.
    Chen, F., Chalhoub-Deville, M.: Differential and long-term language impact on math. Lang. Test. 33(4), 577–605 (2015)CrossRefGoogle Scholar
  16. 16.
    Khachatryan, G.A., Romashov, A.V., Khachatryan, A.R., Gaudino, S.J., Khachatryan, J.M., Guarian, K.R., Yufa, N.V.: Reasoning Mind Genie 2: an intelligent tutoring system as a vehicle for international transfer of instructional methods in mathematics. Int. J. Artif. Intell. Educ. 24(3), 333–382 (2014). doi: 10.1007/s40593-014-0019-7 CrossRefGoogle Scholar
  17. 17.
    Kyle, K., Crossley, S.A.: Automatically assessing lexical sophistication: indices, tools, findings, and application. TESOL Q. 49(4), 757–786 (2015). doi: 10.1002/tesq.194 CrossRefGoogle Scholar
  18. 18.
    Crossley, S.A., Kyle, K., McNamara, D.S.: The tool for the automatic analysis of text cohesion (TAACO): automatic assessment of local, global, and text cohesion. Behav. Res. Methods 48(4), 1227–1237 (2016)CrossRefGoogle Scholar
  19. 19.
    Crossley, S.A., Kyle, K., McNamara, D.S.: Sentiment analysis and social cognition engine (SEANCE): an automatic tool for sentiment, social cognition, and social order analysis. Behav. Res. Methods (in press). doi: 10.3758/s13428-016-0743-z
  20. 20.
    Thorndike, E.L., Lorge, I.: The Teacher’s Wordbook of 30,000 Words. Teachers College: Bureau of Publications: Columbia University, New York (1944)Google Scholar
  21. 21.
    Kučera, H., Francis, N.: Computational Analysis of Present-Day American English. Brown University Press, Providence (1967)Google Scholar
  22. 22.
    Brown, G.D.: A frequency count of 190,000 words in the London-Lund Corpus of English conversation. Behav. Res. Methods Instr. Comput. 16(6), 502–532 (1984). doi: 10.3758/BF03200836 CrossRefGoogle Scholar
  23. 23.
    Brysbaert, M., New, B.: Subtlexus: American word frequencies (2009).
  24. 24.
    The British National Corpus, Version 3 (BNC XML Edition) 2007 Distributed by Oxford University Computing Services on Behalf of the BNC Consortium.
  25. 25.
    Davies, M.: The 385+ million word Corpus of Contemporary American English (1990–2008+): design, architecture, and linguistic insights. Int. J. Corpus Linguist. 14, 159–190 (2009). doi: 10.1075/ijcl.14.2.02dav CrossRefGoogle Scholar
  26. 26.
    Coltheart, M.: The MRC psycholinguistic database. Q. J. Exp. Psychol. 33(4), 497–505 (1981)CrossRefGoogle Scholar
  27. 27.
    Kuperman, V., Stadthagen-Gonzalez, H., Brysbaert, M.: Age-of-acquisition ratings for 30,000 English words. Behav. Res. Methods 44(4), 978–990 (2012). doi: 10.3758/s13428-012-0210-4 CrossRefGoogle Scholar
  28. 28.
    Brysbaert, M., Warriner, A.B., Kuperman, V.: Concreteness ratings for 40 thousand generally known English word lemmas. Behav. Res. Methods 46(3), 904–911 (2014). doi: 10.3758/s13428-013-0403-5 CrossRefGoogle Scholar
  29. 29.
    Miller, G.A.: WordNet: a lexical database for English. Commun. ACM 38(11), 39–41 (1995). doi: 10.1145/219717.219748 CrossRefGoogle Scholar
  30. 30.
    Kiss, G.R., Armstrong, C., Milroy, R., Piper, J.: An associative thesaurus of English and its computer analysis. In: Aitken, A.J., Bailey, R.W., Hamilton-Smith, N. (eds.) The Computer and Literary Studies, pp. 153–165. Edinburgh University Press, Edinburgh (1973)Google Scholar
  31. 31.
    Nelson, D.L., McEvoy, C. L., Schreiber, T.A.: The University of South Florida word association, rhyme, and word fragment norms (1998).
  32. 32.
    Balota, D.A., Yap, M.J., Cortese, M.J., Hutchison, K.I., Kessler, B., Loftis, B., et al.: The English lexicon project. Behav. Res. Methods 39, 445–459 (2007)CrossRefGoogle Scholar
  33. 33.
    Bird, S., Klein, E., Loper, E.: Natural Language Processing with Python. O’Reilly Media, Inc., Sebastopol (2009)zbMATHGoogle Scholar
  34. 34.
    R Core Team: R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. (2014).
  35. 35.
    Grömping, U.: Relative importance for linear regression in R: the package relaimpo. J. Stat. Softw. 1(1) (2006)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.Applied Linguistics and ESLGeorgia State UniversityAtlantaUSA
  2. 2.Reasoning MindHoustonUSA

Personalised recommendations