The effect of language modification of mathematics story problems on problem-solving in online homework

  • Candace WalkingtonEmail author
  • Virginia Clinton
  • Anthony Sparks


Students’ grasp of the non-mathematical language in a mathematics story problem—such as vocabulary and syntax—may have an important effect on their problem-solving, and this may be particularly true for students with weaker language skills. However, little experimental research has examined which individual language features influence students’ performance while solving problems—much research has been correlational or has combined language features together. In the present study, we manipulated six different language features of algebra story problems—number of sentences, pronouns, word concreteness, word hypernymy, consistency of sentences, and problem topic—and examined how systematically varying readability demands impacts student performance. We examined both accuracy and response time measures, using an assignment for learning linear functions in the ASSISTments online problem-solving environment. We found little evidence that individual language features have a considerable effect on mathematics word problem solving performance for a general population of students. However, sentence consistency reduced response time and problems about motion or travel had shorter response times than problems about business or work. In addition, it appears students may benefit or be harmed by language modifications depending on their familiarity with ASSISTments. Implications for the role of language in math word problems are discussed.


Readability Word problems Reading demands Math problems 



We would like to thank Neil Heffernan, Cristina Heffernan, and Korinn Ostrow for their assistance in setting up this study. We also thank NSF for their support in making ASSISTments available to conduct this research, through NSF Cyberinfrastructure Award 1440753, SI2-SSE: Adding Research Accounts to the ASSISTments Platform: Helping Researchers do Randomized Controlled Studies with Thousands of Students.


  1. Abedi, J., & Lord, C. (2001). The language factor in mathematics tests. Applied Measurement in Education, 14, 219–234. Scholar
  2. Abedi, J., Lord, C., Hofstetter, C., & Baker, E. (2000). Impact of accommodation strategies on English language learners’ test performance. Educational Measurement: Issues and Practice, 19(3), 16–26. Scholar
  3. Anand, P. G., & Ross, S. M. (1987). Using computer-assisted instruction to personalize arithmetic materials for elementary school children. Journal of Educational Psychology, 79(1), 72–78. Scholar
  4. Baker, R. S., Corbett, A. T., & Koedinger, K. R. (2007). The difficulty factors approach to the design of lessons in intelligent tutor curricula. International Journal of Artificial Intelligence in Education, 17(4), 341–369.Google Scholar
  5. Baker, R. S., de Carvalho, A. M., Raspat, J., Aleven, V., Corbett, A. T., & Koedinger, K. R. (2009). Educational software features that encourage and discourage “gaming the system.” Proceedings of the 14th International Conference on Artificial Intelligence in Education, 475–482.
  6. Bates, D., Maechler, M., Bolker, B., & Walker, S. (2014). lme4: Linear mixed-effects models using Eigen and S4. R package version 1.1-7.Google Scholar
  7. Beal, C. R., & Galan, F. C. (2015). Math word problem solving by English learners and English primary students in an intelligent tutoring system. International Journal of Learning Technology, 10, 170–184. Scholar
  8. Bernacki, M. L., & Walkington, C. (2018). The role of situational interest in personalized learning. Journal of Educational Psychology, 110(6), 864–881. Scholar
  9. Chen, O., Kalyuga, S., & Sweller, J. (2017). The expertise reversal effect is a variant of the more general element interactivity effect. Educational Psychology Review, 29(2), 393–405. Scholar
  10. Chipman, S. F., Marshall, S. P., & Scott, P. A. (1991). Content effects on word problem performance: A possible source of test bias? American Educational Research Journal, 28(4), 897–915. Scholar
  11. Clinton, V., Basaraba, D. L., & Walkington, C. (2018). English learners and mathematical word problem solving: A systematic review. In D. L. Baker, D. L. Basaraba, & C. Richards-Tutor (Eds.), Second language acquisition: Methods, perspectives and challenges (pp. 171–208). New York: Nova Science Publishers.Google Scholar
  12. Clinton, V., Cooper, J. L., Michaelis, J., Alibali, M. W., & Nathan, M. J. (2017). Revising visuals based on instructional design principles: Effects on cognitive load and learning. In C. Was, F. J. Sansosti, & B. J. Morris (Eds.), Eye-tracking technology applications in educational research (pp. 195–218). Hershey, PA: IGI Global.Google Scholar
  13. Clinton, V., & van den Broek, P. (2012). Interest, inferences, and learning from texts. Learning and Individual Differences, 22, 650–663. Scholar
  14. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.Google Scholar
  15. Crossley, S. A., & McNamara, D. S. (2012). Predicting second language writing proficiency: The roles of cohesion and linguistic sophistication. Journal of Research in Reading, 35(2), 115–135. Scholar
  16. Crossley, S. A., Skalicky, S., Dascalu, M., McNamara, D. S., & Kyle, K. (2017). Predicting text comprehension, processing, and familiarity in adult readers: New approaches to readability formulas. Discourse Processes, 54(5–6), 340–359. Scholar
  17. Cummins, D. D., Kintsch, W., Reusser, K., & Weimer, R. (1988). The role of understanding in solving word problems. Cognitive Psychology, 20, 405–438. Scholar
  18. d’Ailly, H. H., Simpson, J., & MacKinnon, G. E. (1997). Where should “you” go in a math compare problem? Journal of Educational Psychology, 89(3), 562–567. Scholar
  19. Daroczy, G., Wolska, M., Meurers, W. D., & Nuerk, H. C. (2015). Word problems: A review of linguistic and numerical factors contributing to their difficulty. Frontiers in Psychology, 6, 348–361. Scholar
  20. Davis-Dorsey, J., Ross, S. M., & Morrison, G. R. (1991). The role of rewording and context personalization in the solving of mathematical word problems. Journal of Educational Psychology, 83(1), 61–68. Scholar
  21. De Jong, T. (2010). Cognitive load theory, educational research, and instructional design: some food for thought. Instructional Science, 38(2), 105–134.Google Scholar
  22. Doddannara, L. S., Gowda, S. M., Baker, R. S., Gowda, S. M., & De Carvalho, A. M. (2013). Exploring the relationships between design, students’ affective states, and disengaged behaviors within an ITS. In International Conference on Artificial Intelligence in Education (pp. 31–40). Springer, Berlin, Heidelberg.
  23. Duffy, S. A., & Rayner, K. (1990). Eye movements and anaphor resolution: Effects of antecedent typicality and distance. Language and Speech, 33(2), 103–119. Scholar
  24. Duran, N. D., Bellissens, C., Taylor, R. S., & McNamara, D. S. (2007). Quantifying text difficulty with automated indices of cohesion and semantics. In Proceedings of the Annual Meeting of the Cognitive Science Society, Volume 29 (pp. 233–238). Cognitive Science Society, Austin, TX.Google Scholar
  25. Fang, Z. (2012). Language correlates of disciplinary literacy. Topics in Language Disorders, 32(1), 19–34. Scholar
  26. Fliessbach, K., Weis, S., Klaver, P., Elger, C. E., & Weber, B. (2006). The effect of word concreteness on recognition memory. NeuroImage, 32(3), 1413–1421. Scholar
  27. Fuchs, L. S., Fuchs, D., Compton, D. L., Hamlett, C. L., & Wang, A. Y. (2015). Is word-problem solving a form of text comprehension? Scientific Studies of Reading, 19(3), 204–223. Scholar
  28. Gernsbacher, M. A. (1989). Mechanisms that improve referential access. Cognition, 32(2), 99–156. Scholar
  29. Gillmor, S., Poggio, J., & Embretson, S. (2015). Effects of reducing the cognitive load of mathematics test items on student performance. Numeracy, 8(1), 1–18. Scholar
  30. Graesser, A. C., & McNamara, D. S. (2011). Computational analyses of multilevel discourse comprehension. Topics in Cognitive Science, 3(2), 371–398. Scholar
  31. Graesser, A. C., McNamara, D. S., & Louwerse, M. M. (2003). What do readers need to learn in order to process coherence relations in narrative and expository text. In A. P. Sweet & C. E. Snow (Eds.), Rethinking reading comprehension (pp. 82–98). New York: Guilford.Google Scholar
  32. Haag, N., Heppt, B., Stanat, P., Kuhl, P., & Pant, H. A. (2013). Second language learners’ performance in mathematics: Disentangling the effects of academic language features. Learning and Instruction, 28, 24–34. Scholar
  33. Haag, N., Roppelt, A., & Heppt, B. (2015). Effects of mathematics items’ language demands for language minority students: Do they differ between grades? Learning and Individual Differences, 42, 70–76. Scholar
  34. Harp, S. F., & Mayer, R. E. (1998). How seductive details do their damage: A theory of cognitive interest in science learning. Journal of Educational Psychology, 90(3), 414–434. Scholar
  35. Heffernan, N. T., & Heffernan, C. L. (2014). The ASSISTments ecosystem: Building a platform that brings scientists and teachers together for minimally invasive research on human learning and teaching. International Journal of Artificial Intelligence in Education, 24(4), 470–497. Scholar
  36. Hembree, R. (1992). Experiments and relational studies in problem solving: A meta-analysis. Journal for Research in Mathematics Education, 23(3), 242–273. Scholar
  37. Hidi, S. E. (1995). A reexamination of the role of attention in learning from text. Educational Psychology Review, 7(4), 323–350. Scholar
  38. Hidi, S., & Renninger, K. A. (2006). The four-phase model of interest development. Educational Psychologist, 41(2), 111–127. Scholar
  39. Jerman, M., & Rees, R. (1972). Predicting the relative difficulty of verbal arithmetic problems. Educational Studies in Mathematics, 4(3), 306–323. Scholar
  40. Kane, M. T. (2001). Current concerns in validity theory. Journal of Educational Measurement, 38(4), 319–342. Scholar
  41. Kintsch, W. (1998). Comprehension: A paradigm for cognition. Cambridge, UK: Cambridge University Press.Google Scholar
  42. Kintsch, W., & Greeno, J. G. (1985). Understanding and solving word arithmetic problems. Psychological Review, 92(1), 109–129. Scholar
  43. Kintsch, W., Welsch, D., Schmalhofer, F., & Zimny, S. (1990). Sentence memory: A theoretical analysis. Journal of Memory and Language, 29(2), 133–159. Scholar
  44. Koedinger, K., & Nathan, M. J. (2004). The real story behind story problems: Effects of representations on quantitative reasoning. Journal of the Learning Sciences, 13(2), 129–164. Scholar
  45. Korbach, A., Brünken, R., & Park, B. (2018). Differentiating different types of cognitive load: A comparison of different measures. Educational Psychology Review, 30(2), 503–529. Scholar
  46. Leroy, G., Endicott, J. E., Kauchak, D., Mouradi, O., & Just, M. (2013). User evaluation of the effects of a text simplification algorithm using term familiarity on perception, understanding, learning, and information retention. Journal of Medical Internet Research, 15(7), e144. Scholar
  47. Loftus, E. F., & Suppes, P. (1972). Structural variables that determine problem-solving difficulty in computer assisted instruction. Journal of Educational Psychology, 63(6), 531. Scholar
  48. Martiniello, M. (2009). Linguistic complexity, schematic representations, and differential item functioning for English language learners in math tests. Educational Assessment, 14(3–4), 160–179. Scholar
  49. Mavilidi, M. F., Hoogerheide, V., & Paas, F. (2014). A quick and easy strategy to reduce test anxiety and enhance test performance. Applied Cognitive Psychology, 28(5), 720–726. Scholar
  50. Mayer, R. E., Fennell, S., Farmer, L., & Campbell, J. (2004). A personalization effect in multimedia learning: Students learn better when words are in conversational style rather than formal style. Journal of Educational Psychology, 96(2), 389–395. Scholar
  51. McNamara, D. S., Graesser, A. C., McCarthy, P. M., & Cai, Z. (2014). Automated evaluation of text and discourse with Coh-Metrix. Cambridge: Cambridge University Press. Scholar
  52. Mullis, I. V., Martin, M. O., & Foy, P. (2013). The impact of reading ability on TIMSS mathematics and science achievement at the fourth grade: An analysis by item reading demands. In M. O. Martin & I. V. S. Mullis (Eds.), TIMSS and PIRLS 2011: Relationships among reading, mathematics, and science achievement at the fourth grade: Implications for early learning (pp. 67–108). Chestnut Hill, MA: TIMSS & PIRLS International Study Center, Lynch School of Education, Boston College and IEA.Google Scholar
  53. Nagy, W., & Townsend, D. (2012). Words as tools: Learning academic vocabulary as language acquisition. Reading Research Quarterly, 47(1), 91–108. Scholar
  54. Nathan, M. J., Kintsch, W., & Young, E. (1992). A theory of algebra-word-problem comprehension and its implications for the design of learning environments. Cognition and Instruction, 9, 329–389. Scholar
  55. Nathan, M. J., & Koedinger, K. R. (2000). An investigation of teachers’ beliefs of students’ algebra development. Cognition and Instruction, 18(2), 209–237. Scholar
  56. Ngu, B. H., Phan, H. P., Hong, K. S., & Usop, H. (2016). Reducing intrinsic cognitive load in percentage change problems: The equation approach. Learning and Individual Differences, 51, 81–90. Scholar
  57. Noble, T., Kachchaf, R. R., & Rosebery, A. S. (2018). Perspectives from research on the linguistic features of mathematics and science test items and the performance of English Learners. In D. L. Baker & D.L. Basaraba, & C. Richards-Tutor (Eds.), Second language acquisition: Methods, perspectives and challenges (pp. 209–236). Nova Science Publishers.Google Scholar
  58. Paas, F., Renkl, A., & Sweller, J. (2003). Cognitive load theory and instructional design: Recent developments. Educational Psychologist, 38(1), 1–4. Scholar
  59. Phan, H. P., Ngu, B. H., & Yeung, A. S. (2017). Achieving optimal best: Instructional efficiency and the use of cognitive load theory in mathematical problem solving. Educational Psychology Review, 29(4), 667–692. Scholar
  60. Raghubar, K. P., Barnes, M. A., & Hecht, S. A. (2010). Working memory and mathematics: A review of developmental, individual difference, and cognitive approaches. Learning and Individual Differences, 20(2), 110–122. Scholar
  61. Sadoski, M. (2001). Resolving the effects of concreteness on interest, comprehension, and learning important ideas from text. Educational Psychology Review, 13(3), 263–281. Scholar
  62. Sadoski, M., Goetz, E. T., & Avila, E. (1995). Concreteness effects in text recall: Dual coding or context availability? Reading Research Quarterly, 30(2), 278–288. Scholar
  63. Sadoski, M., Goetz, E. T., & Rodriguez, M. (2000). Engaging texts: Effects of concreteness on comprehensibility, interest, and recall in four text types. Journal of Educational Psychology, 92(1), 85–95. Scholar
  64. Scheiter, K., Gerjets, P., & Catrambone, R. (2006). Making the abstract concrete: Visualizing mathematical solution procedures. Computers in Human Behavior, 22(1), 9–25. Scholar
  65. Schmidt, R. A., & Bjork, R. A. (1992). New conceptualizations of practice: Common principles in three paradigms suggest new concepts for training. Psychological Science, 3(4), 207–218. Scholar
  66. Schraw, G., & Lehman, S. (2001). Situational interest: A review of the literature and directions for future research. Educational Psychology Review, 13(1), 23–52. Scholar
  67. Searle, B. W., Lorton, P., & Suppes, P. (1974). Structural variables affecting CAI performance on arithmetic word problems of disadvantaged and deaf students. Educational Studies in Mathematics, 5(1), 371–384. Scholar
  68. Shaftel, J., Belton-Kocher, E., Glasnapp, D., & Poggio, J. (2006). The impact of language characteristics in mathematics test items on the performance of English language learners and students with disabilities. Educational Assessment, 11, 105–126. Scholar
  69. Suppes, P., Loftus, E. F., & Jerman, M. (1969). Problem-solving on a computer-based teletype. Educational Studies in Mathematics, 2(1), 1–15. Scholar
  70. Sweller, J. (1994). Cognitive load theory, learning difficulty, and instructional design. Learning and Instruction, 4(4), 295–312. Scholar
  71. Sweller, J. (2016). Cognitive load theory, evolutionary educational psychology, and instructional design. In D. Geary & D. Berch (Eds.), Evolutionary perspectives on child development and education (pp. 291–306). Cham: Springer. Scholar
  72. Sweller, J., Ayres, P. L., Kalyuga, S., & Chandler, P. A. (2003). The expertise reversal effect. Educational Psychologist, 38(1), 23–31. Scholar
  73. Sweller, J., Ayres, P., & Kalyuga, S. (2011). Measuring cognitive load. In Cognitive Load Theory: Explorations in the Learning Sciences, Instructional Systems and Performance Technologies, Vol 1 (pp. 237–242). Springer, New York.
  74. Sweller, J., & Chandler, P. (1991). Evidence for cognitive load theory. Cognition and Instruction, 8(4), 351–362. Scholar
  75. Sweller, J., van Merrienboer, J., & Paas, F. (1998). Cognitive architecture and instructional design. Educational Psychology Review, 10, 251–296.Google Scholar
  76. Van Gog, T., Paas, F., & Van Merriënboer, J. J. (2006). Effects of process-oriented worked examples on troubleshooting transfer performance. Learning and Instruction, 16(2), 154–164. Scholar
  77. Vilenius-Tuohimaa, P. M., Aunola, K., & Nurmi, J. E. (2008). The association between mathematical word problems and reading comprehension. Educational Psychology, 28(4), 409–426. Scholar
  78. Walkington, C. A. (2013). Using adaptive learning technologies to personalize instruction to student interests: The impact of relevant contexts on performance and learning outcomes. Journal of Educational Psychology, 105(4), 932–945. Scholar
  79. Walkington, C., & Bernacki, M. (2014). Motivating students by “personalizing” learning around individual interests: A consideration of theory, design, and implementation issues. In S. Karabenick & T. Urdan (Eds.), Advances in motivation and achievement (Vol. 18, pp. 139–176). Bingley: Emerald Group Publishing. Scholar
  80. Walkington, C., & Bernacki, M. (2018). Personalizing algebra to students’ interests: How student funds of knowledge moderate outcomes. International Journal of Artificial Intelligence in Education. Scholar
  81. Walkington, C., Clinton, V., Ritter, S. N., & Nathan, M. J. (2015). How readability and topic incidence relate to performance on mathematics story problems in computer-based curricula. Journal of Educational Psychology, 107, 1051–1074. Scholar
  82. Walkington, C., Clinton, V., & Shivraj, P. (2018). How readability factors are differentially associated with performance for students of different backgrounds when solving mathematics word problems. American Educational Research Journal, 55(2), 362–414. Scholar
  83. Walkington, C., & Maull, K. (2011). Exploring the assistance dilemma: The case of context personalization. In L. Carlson, C. Hölscher, & T. Shipley (Eds.), Proceedings of the 33rd Annual Conference of the Cognitive Science Society (pp. 90–95). Boston, MA: Cognitive Science Society.Google Scholar
  84. Walkington, C., Sherman, M., & Petrosino, A. (2012). ‘Playing the game’ of story problems: Coordinating situation-based reasoning with algebraic representation. Journal of Mathematical Behavior, 31(2), 174–195.Google Scholar
  85. Wang, A. Y., Fuchs, L. S., & Fuchs, D. (2016). Cognitive and linguistic predictors of mathematical word problems with and without irrelevant information. Learning and Individual Differences, 52, 79–87. Scholar
  86. Wolf, M. K., & Leon, S. (2009). An investigation of the language demands in content assessments for English language learners. Educational Assessment, 14(3–4), 139–159. Scholar
  87. Wu, M. (2010). Comparing the similarities and differences of PISA 2003 and TIMSS (OECD Working Papers No. 32). Paris: OECD Publishing.
  88. Young, J. W. (2009). A framework for test validity research on content assessments taken by English language learners. Educational Assessment, 14(3–4), 122–138. Scholar

Copyright information

© Springer Nature B.V. 2019

Authors and Affiliations

  • Candace Walkington
    • 1
    Email author
  • Virginia Clinton
    • 2
  • Anthony Sparks
    • 1
  1. 1.Department of Teaching and LearningSouthern Methodist UniversityDallasUSA
  2. 2.Department of Education, Health, and Behavior StudiesUniversity of North DakotaGrand ForksUSA

Personalised recommendations