Advertisement

Reading and Writing

, Volume 31, Issue 4, pp 813–833 | Cite as

Investigating the validity of two widely used quantitative text tools

  • James W. Cunningham
  • Elfrieda H. Hiebert
  • Heidi Anne Mesmer
Article
First Online:

Abstract

In recent years, readability formulas have gained new prominence as a basis for selecting texts for learning and assessment. Variables that quantitative tools count (e.g., word frequency, sentence length) provide valid measures of text complexity insofar as they accurately predict representative and high-quality criteria. The longstanding consensus of text researchers has been that such criteria will measure readers’ comprehension of sample texts. This study used Bormuth’s (1969) rigorously developed criterion measure to investigate two of today’s most widely used quantitative text tools—the Lexile Framework and the Flesch–Kincaid Grade-Level formula. Correlations between the two tools’ complexity scores and Bormuth’s measured difficulties of criterion passages were only moderately high in light of the literature and new high stakes uses for such tools. These correlations declined a small amount when passages from the University grade band of use were removed. The ability of these tools to predict measured text difficulties within any single grade band below University was low. Analyses showed that word complexity made a larger contribution relative to sentence complexity when each tool’s predictors were regressed on the Bormuth criterion rather than their original criteria. When the criterion was texts’ grade band of use instead of mean cloze scores, neither tool classified texts well and errors disproportionally placed texts from higher grade bands into lower ones. Results suggest these two text tools may lack adequate validity for their current uses in educational settings.

Keywords

Quantitative text analysis Readability Text complexity Text difficulty 
This is a preview of subscription content, log in to check access.

References

  1. Bormuth, J. R. (1969). Development of readability analyses (Final Report, Project No. 7-0052, Contract No. OEC-3-7-070052-0326). Retrieved from ERIC database. (ED029166).Google Scholar
  2. Bormuth, J. R. (1971). Development of standards of readability: Toward a rational criterion of passage performance (Final Report, Project No. 9-0237, Contract No. OEG-0-9-230237-4125(010)). Retrieved from ERIC database. (ED054233).Google Scholar
  3. Briggs, D. C. (2013). Measuring growth with vertical scales. Journal of Educational Measurement, 50, 204–226.  https://doi.org/10.1111/jedm.12011.CrossRefGoogle Scholar
  4. Carroll, J. B., Davies, P., & Richman, B. (1971). The American Heritage Word Frequency Book. Boston, MA: Houghton Mifflin.Google Scholar
  5. Coleman, E. B. (1965). ON understanding prose: Some determiners of its complexity (NSF Final Report GB-2604). Washington, DC: National Science Foundation.Google Scholar
  6. Cunningham, J. W., & Mesmer, H. A. (2014). Quantitative measurement of text difficulty: What’s the use? The Elementary School Journal, 115, 255–269.CrossRefGoogle Scholar
  7. Dale, E., & Tyler, R. W. (1934). A study of the factors influencing the difficulty of reading materials for adults of limited reading ability. The Library Quarterly: Information, Community, Policy, 4, 384–412.CrossRefGoogle Scholar
  8. Dolch, E. W. (1948). Graded reading difficulty. Problems in Reading (pp. 229–255). Champaign, IL: The Garrard Press.Google Scholar
  9. Domingue, B. (2014). Evaluating the equal-interval hypothesis with test score scales. Psychometrika, 79, 1–19.CrossRefGoogle Scholar
  10. Dunn, L. M., & Dunn, L. M. (1981). Peabody Picture Vocabulary Test, Revised: Forms L and M. Circle Pines, MN: American Guidance Service.Google Scholar
  11. Dunn, L. M., & Markwardt, F. C. (1970). Peabody Individual Achievement Test. Circle Pines, MN: American Guidance Service.Google Scholar
  12. Flesch, R. (1948). A new readability yardstick. Journal of Applied Psychology, 32, 221–233.CrossRefGoogle Scholar
  13. Gamson, D. A., Lu, X., & Eckert, S. A. (2013). Challenging the research base of the common core state standards a historical reanalysis of text complexity. Educational Researcher, 42, 381–391.CrossRefGoogle Scholar
  14. Gray, W. S., & Leary, B. E. (1935). What Makes a Book Readable: With Special Reference to Adults of Limited Reading Ability. Chicago, IL: University of Chicago Press.Google Scholar
  15. Keenan, J. M., Betjemann, R. S., & Olson, R. K. (2008). Reading comprehension tests vary in the skills they assess: Differential dependence on decoding and oral comprehension. Scientific Studies of Reading, 12, 281–300.CrossRefGoogle Scholar
  16. Kincaid, J. P., Fishburne Jr, R. P., Rogers, R. L., & Chissom, B. S. (1975). Derivation of new readability formulas (automated readability index, fog count and Flesch reading ease formula) for navy enlisted personnel (No. RBR-8-75). Millington TN: Naval Technical Training Command Research Branch.Google Scholar
  17. Kintsch, W. (1988). The role of knowledge in discourse comprehension: A construction-integration model. Psychological Review, 95, 163–182.CrossRefGoogle Scholar
  18. Klare, G. R. (1963). The Measurement of Readability. Ames, IA: Iowa State University Press.Google Scholar
  19. Klare, G. R. (1974). Assessing readability. Reading Research Quarterly, 10, 62–102.CrossRefGoogle Scholar
  20. Klare, G. R. (1984). Readability. In P. D. Pearson, R. Barr, M. L. Kamil, & P. Mosenthal (Eds.), Handbook of Reading Research (Vol. 1, pp. 681–744). New York, NY: Longman.Google Scholar
  21. Koslin, B. I., Zeno, S., & Koslin, S. (1987). The DRP: An Effective Measure in Reading. New York, NY: College Entrance Examination Board.Google Scholar
  22. Lively, B. A., & Pressey, S. L. (1923). A method for measuring the vocabulary burden of textbooks. Educational Administration and Supervision, 9, 389–398.Google Scholar
  23. Lorge, I. (1939). Predicting reading difficulty of selections for children. Elementary English Review, 16, 229–233.Google Scholar
  24. McCall, W. A., & Crabbs, L. M. (1925, 1950, 1961). Standard Test Lessons in Reading: Teacher’s Manual for All Books. New York, NY: Teachers College Press.Google Scholar
  25. McNamara, D. S., & Kintsch, W. (1996). Learning from text: Effect of prior knowledge and text coherence. Discourse Processes, 22, 247–288.CrossRefGoogle Scholar
  26. Mesmer, H. A. E. (2008). Tools for matching readers to texts: Research-based practices. New York, NY: Guilford Press.Google Scholar
  27. Mesmer, H. A., Cunningham, J. W., & Hiebert, E. H. (2012). Toward a theoretical model of text complexity for the early grades: Learning from the past, anticipating the future. Reading Research Quarterly, 47, 235–258.CrossRefGoogle Scholar
  28. Metametrics. (2017a). Statewide assessments. Retrieved from https://lexile.com/about-lexile/How-to-get-lexile-measures/states/.
  29. Metametrics. (2017b). State consortia. Retrieved from https://lexile.com/using-lexile/lexile-measures-and-the-ccssi/state-consortia/.
  30. Miller, L. R. (1975). Predictive powers of multiple-choice and cloze-derived readability formulas. Reading Improvement, 12, 52–58.Google Scholar
  31. Muijselaar, M. L., Kendeou, P., de Jong, P. F., & van den Broek, P. W. (2017). What does the CBM-Maze test measure? Scientific Studies of Reading, 21, 120–132.  https://doi.org/10.1080/10888438.2016.1263994.CrossRefGoogle Scholar
  32. Muijselaar, M. M. L., Swart, N. M., Steenbeek-Planting, E. G., Droop, M., Verhoeven, L., & de Jong, P. F. (2017). The dimensions of reading comprehension in Dutch children: Is differentiation by text and question type necessary? Journal of Educational Psychology, 109(1), 70–83.  https://doi.org/10.1037/edu0000120.CrossRefGoogle Scholar
  33. National Governors Association Center for Best Practices & Council of Chief State School Officers. (2010a). Common Core State Standards for English language arts & literacy in history/social studies, science, and technical subjects, Appendix A. Washington, DC: Author. Retrieved from www.corestandards.org/assets/Appendix_A.pdf.
  34. National Governors Association Center for Best Practices & Council of Chief State School Officers. (2010b). Common Core State Standards for English language arts and literacy in history/social studies, science, and technical subjects. Washington, DC: Authors. Retrieved from www.corestandards.org/assets/CCSSI_ELA%20Standards.pdf.
  35. Nelson, J., Perfetti, C., Liben, D., & Liben, M. (2012). Measures of Text Difficulty: TESTING their Predictive Value for Grade Levels and Student Performance. New York, NY: Student Achievement Partners.Google Scholar
  36. Ouellette, G. P. (2006). What’s meaning got to do with it: The role of vocabulary in word reading and reading comprehension. Journal of Educational Psychology, 98, 554.CrossRefGoogle Scholar
  37. PARCC (n.d.). ELA test specifications documents: Understanding summative assessment design. Retrieved from: http://www.parcconline.org/assessments/test-design/ela-literacy/ela-performance-level-descriptors.
  38. Pearson, P. D., & Hiebert, E. H. (2014). The state of the field: Qualitative analyses of text complexity. The Elementary School Journal, 115, 161–183.CrossRefGoogle Scholar
  39. Rodriguez, N., & Hansen, L. H. (1975). Performance of readability formulas under conditions of restricted ability level and restricted difficulty of materials. Journal of Experimental Education, 44, 8–14.Google Scholar
  40. Shanahan, T., Kamil, M. L., & Tobin, A. W. (1982). Cloze as a measure of intersentential comprehension. Reading Research Quarterly, 17, 229–255.CrossRefGoogle Scholar
  41. Smarter Balanced (November 17, 2016). Smarter Balanced, MetaMetrics partner to provide more specific info about students’ reading abilities. Retrieved from: http://www.smarterbalanced.org/smarter-balanced-metametrics-partner-provide-specific-info-students-reading-abilities/.
  42. Spache, G. (1953). A new readability formula for primary grade reading materials. The Elementary School Journal, 53, 410–413.CrossRefGoogle Scholar
  43. Stenner, A. J. (1996). Measuring reading comprehension with the Lexile Framework. Paper presented at the North American Conference Adolescent/Adult Literacy (4th, Washington, DC, February). Retrieved from ERIC database (ED435977).Google Scholar
  44. Stenner, A. J., & Burdick, D. S. (1997). The objective measurement of reading comprehension: In response to technical questions raised by the California Department of Education Technical Study Group. Washington, DC: IES/ERIC. Retrieved from ERIC database (ED435978).Google Scholar
  45. Stenner, A. J., & Fisher, W. P., Jr. (2013). Metrological traceability in the social sciences: A model from reading measurement. Journal of Physics: Conference Series, 459, 1–6. Retrieved from http://iopscience.iop.org/1742-6596/459/1/012025.
  46. Stenner, A. J., Smith, M., & Burdick, D. S. (1983). Toward a theory of construct definition. Journal of Educational Measurement, 20, 305–316.CrossRefGoogle Scholar
  47. Stevens, K. C. (1980). Readability formulae and McCall-Crabbs standard test lessons in reading. The Reading Teacher, 33, 413–415.Google Scholar
  48. Taylor, W. L. (1953). “Cloze procedure”: A new tool for measuring readability. Journalism Bulletin, 30, 415–433.CrossRefGoogle Scholar
  49. Vogel, M., & Washburne, C. (1928). An objective method of determining grade placement of children’s reading material. The Elementary School Journal, 28, 373–381.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media B.V., part of Springer Nature 2018

Authors and Affiliations

  • James W. Cunningham
    • 1
  • Elfrieda H. Hiebert
    • 2
  • Heidi Anne Mesmer
    • 3
  1. 1.University of North Carolina at Chapel HillChapel HillUSA
  2. 2.TextProjectSanta CruzUSA
  3. 3.Virginia TechBlacksburgUSA

Personalised recommendations

Cite article

Buy options