Skip to main content
SpringerLink
Log in

Performance variations across reading comprehension assessments: Examining the unique contributions of text, activity, and reader

  • Published:
Reading and Writing Aims and scope Submit manuscript

Abstract

These studies examined the contribution of text, activity, and reader to variance in reading comprehension test scores. Study 1 focused on multiple-choice and open-ended item responses, whereas Study 2 examined retell. Both studies included 79 fourth-grade students (age M = 9.72; SD = .34). Each student read six passages from the Qualitative Reading Inventory-Fifth Edition (QRI-5) and completed comprehension assessments of varying response format (open-ended questions, multiple choice, and retell). Measures of cognitive capacity, language knowledge, learning motivation, and word reading fluency were also administered. In Study 1, item-response crossed random effects models revealed statistically significant differences between open-ended question and multiple-choice response formats, and three covariates significantly predicted reading comprehension test scores: (a) attentive behavior, (b) language knowledge, and (c) working memory. Further exploratory analyses identified two-way interactions: (a) Response Format × Attentive Behavior, and (b) Response Format × Language Knowledge. In Study 2, crossed random effects models revealed two statistically significant predictors of retell scores: (a) text genre, and (b) language knowledge. Findings suggest different response format activities may contribute to variance in reading comprehension tests scores, and this test property may further interact with text as well as reader abilities.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

References

  • Akaike, H. (1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control,19, 716–723.

    Article  Google Scholar 

  • Andreassen, R., & Bråten, I. (2010). Examining the prediction of reading comprehension on different multiple-choice tests. Journal of Research in Reading,33, 263–283. https://doi.org/10.1111/j.1467-9817.2009.01413.x.

    Article  Google Scholar 

  • Andreassen, R., & Bråten, I. (2011). Implementation and effects of explicit reading comprehension instruction in fifth-grade classrooms. Learning & Instruction,21(4), 520–537. https://doi.org/10.1016/j.learninstruc.2010.08.003.

    Article  Google Scholar 

  • Arrington, C. N., Kulesz, P. A., Francis, D. J., Fletcher, J. M., & Barnes, M. A. (2014). The contribution of attentive control and working memory to reading comprehension and decoding. Scientific Studies of Reading,18, 325–346. https://doi.org/10.1080/10888438.2014.902461.

    Article  Google Scholar 

  • Best, R. M., Floyd, R. G., & McNamara, D. S. (2008). Differential competencies contributing to children’s comprehension of narrative and expository texts. Reading Psychology,29, 137–164.

    Article  Google Scholar 

  • Brame, C. (2013). Writing good multiple choice test questions. Retrieved from http://cft.vanderbilt.edu/guides-sub-pages/writing-good-multiple-choice-test-questions/. Accessed 24 July 2014.

  • Cain, K., Oakhill, J., & Bryant, P. (2004). Children’s reading comprehension ability: Concurrent prediction by working memory, verbal ability, and component skills. Journal of Educational Psychology,96, 31–42.

    Article  Google Scholar 

  • Campbell, J. R. (2005). Single instrument, multiple measures: Considering the use of multiple item formats to assess reading comprehension. In S. G. Paris & S. A. Stahl (Eds.), Children’s reading comprehension and assessment (pp. 347–368). Mahwah: Lawrence Erlbaum.

    Google Scholar 

  • Carretti, B., Borella, E., Cornoldi, C., & De Beni, R. (2009). Role of working memory in explaining the performance of individuals with specific reading comprehension difficulties: A meta-analysis. Learning and Individual Differences,19, 246–251.

    Article  Google Scholar 

  • Center for Teaching Excellence, Virginia Commonwealth University. (2013). Writing multiple-choice questions. Retrieved from http://www.vcu.edu/cte/resources/nfrg/12_03_writing_MCQs.htm. Accessed 24 July 2014.

  • Cho, S.-J., Partchev, I., & De Boeck, P. (2012). Parameter estimation of multiple item response profile model. British Journal of Mathematical and Statistical Psychology,65, 438–466.

    Article  Google Scholar 

  • Christopher, M. E., Miyake, A., Keenan, J. M., Pennington, B., DeFries, J. C., Wadsworth, S. J., et al. (2012). Predicting word reading and comprehension with executive function and speed measures across development: A latent variable analysis. Journal of Experimental Psychology: General,141, 470–488. https://doi.org/10.1037/a0027375.

    Article  Google Scholar 

  • Cirino, P. T. (2012). Student contextual learning scales. Houston, TX: Author.

    Google Scholar 

  • Clemens, N. H., Davis, J. L., Simmons, L. E., Oslund, E. L., & Simmons, D. C. (2015). Interpreting secondary students’ performance on a timed, multiple-choice reading comprehension assessment: The prevalence and impact of non-attempted items. Journal of Psychoeducational Assessment,33, 154–165. https://doi.org/10.1177/0734282914547493.

    Article  Google Scholar 

  • Coh-metrix. (2017). Coh-Metrix 3.0. Retrieved from http://www.cohmetrix.com/.

  • Collins, A. A., Lindström, E. R., & Compton, D. L. (2018). Comparing students with and without reading difficulties on reading comprehension assessments: A meta-analysis. Journal of Learning Disabilities, 51, 108–123. https://doi.org/10.1177/0022219417704636.

  • Cutting, L. E., Materek, A., Cole, C. A. S., Levine, T. M., & Mahone, E. M. (2009). Effects of fluency, oral language, and executive function on reading comprehension performance. Annals of Dyslexia,59, 34–54. https://doi.org/10.1007/s11881-009-0022-0.

    Article  Google Scholar 

  • Cutting, L. E., & Scarborough, H. S. (2006). Prediction of reading comprehension: Relative contributions of word recognition, language proficiency, and other cognitive skills can depend on how comprehension is measured. Scientific Studies of Reading,10, 277–299. https://doi.org/10.1207/s1532799xssr1003_5.

    Article  Google Scholar 

  • De Boeck, P. (2008). Random item IRT models. Psychometrika, 73, 533–559. https://doi.org/10.1007/s11336-008-9092-x.

  • Eason, S. H., Goldberg, L. F., Young, K. M., Geist, M. C., & Cutting, L. E. (2012). Reader-text interactions: How differential text and question types influence cognitive skills needed for reading comprehension. Journal of Educational Psychology,104, 515–528.

    Article  Google Scholar 

  • Francis, D. J., Fletcher, J. M., Catts, H. W., & Tomblin, J. B. (2005). Dimensions affecting the assessment of reading comprehension. In S. G. Paris & S. A. Stahl (Eds.), Children’s reading comprehension and assessment (pp. 369–394). Mahwah: Lawrence Erlbaum.

    Google Scholar 

  • García, J. R., & Cain, K. (2013). Decoding and reading comprehension: A meta-analysis to identify which reader and assessment characteristics influence the strength of the relationship in English. Review of Educational Research,84, 1–38. https://doi.org/10.3102/0034654313499616.

  • Gough, P. B., & Tunmer, W. E. (1986). Decoding, reading, and reading disability. Remedial and Special Education,7(1), 6–10.

    Article  Google Scholar 

  • Haladyna, T. M. (1999). Developing and validating multiple-choice test items (2nd ed.). Mahwah: Lawrence Erlbaum.

    Google Scholar 

  • Hua, A. N., & Keenan, J. M. (2014). The role of text memory in inferencing and in comprehension deficits. Scientific Studies of Reading,6, 415–431. https://doi.org/10.1080/10888438.2014.926906.

    Article  Google Scholar 

  • Janssen, R., Schepers, J., & Peres, D. (2004). Models with item and item group predictors. In P. D. Boeck & M. Wilson (Eds.), Explanatory item response models: A generalized and non-linear approach (pp. 189–212). New York: Springer.

    Chapter  Google Scholar 

  • Jenkins, J. R., Johnson, E., & Hileman, J. (2004). When is reading also writing: Sources of individual differences on the new reading performance assessments. Scientific Studies of Reading,8, 125–151. https://doi.org/10.1207/s1532799xssr0802_2.

    Article  Google Scholar 

  • Johnston, A. M., Barnes, M. A., & Desrochers, A. (2008). Reading comprehension: Developmental processes, individual differences, and interventions. Canadian Psychology,49, 125–132. https://doi.org/10.1037/0708-5591.49.2.125.

    Article  Google Scholar 

  • Kearns, D. M., Steacy, L. M., Compton, D. L., Gilbert, J. K., Goodwin, A. P., Cho, E., Lindstrom, E. R., & Collins, A. A. (2016). Modeling polymorphemic word recognition: Exploring differences among children with early-emerging and late-emerging word reading difficulty. Journal of Learning Disabilities, 49, 368–394. https://doi.org/10.1177/0022219414554229.

  • Keenan, J. M. (2013). Assessment of reading comprehension. In C. A. Stone, E. R. Silliman, B. J. Ehren, & G. P. Wallach (Eds.), Handbook of language and literacy: Development and disorders (2nd ed., pp. 469–484). New York: Guilford Press.

    Google Scholar 

  • Keenan, J. M., Betjemann, R. S., & Olson, R. K. (2008). Reading comprehension tests vary in the skills they assess: Differential dependence on decoding and oral comprehension. Scientific Studies of Reading,12, 281–300. https://doi.org/10.1080/10888430802132279

    Article  Google Scholar 

  • Keenan, J. M., & Meenan, C. E. (2014). Test differences in diagnosing reading comprehension deficits. Journal of Learning Disabilities,47, 125–135. https://doi.org/10.1177/0022219412439326.

    Article  Google Scholar 

  • Kendeou, P., Papadopoulos, T. C., & Spanoudis, G. (2012). Processing demands of reading comprehension tests in young readers. Learning and Instruction,22, 354–367.

    Article  Google Scholar 

  • Kieffer, M. J., Vukovic, R. K., & Berry, D. (2013). Roles of attention shifting and inhibitory control in fourth-grade reading comprehension. Reading Research Quarterly,48, 333–348. https://doi.org/10.1002/rrq.54.

    Article  Google Scholar 

  • Kintsch, W., & Kintsch, E. (2005). Comprehension. In S. G. Paris & S. A. Stahl (Eds.), Children’s reading comprehension and assessment (pp. 71–92). Mahwah: Lawrence Erlbaum Associates Inc.

    Google Scholar 

  • Kulesz, P. A., Francis, D. J., Barnes, M. A., & Fletcher, J. M. (2016). The influence of properties of the test and their interactions with reader characteristics on reading comprehension: An explanatory item response study. Journal of Educational Psychology,108, 1078–1097. https://doi.org/10.1037/edu0000126.

    Article  Google Scholar 

  • Leslie, L., & Caldwell, J. (2011). Qualitative reading inventory (5th ed.). Boston: Pearson Education Inc.

    Google Scholar 

  • McCallum, R. S., Sharp, S., Bell, S. M., & George, T. (2004). Silent versus oral reading comprehension and efficiency. Psychology in the Schools,41, 241–246. https://doi.org/10.1002/pits.10152.

    Article  Google Scholar 

  • McGrew, K. S., Schrank, F. A., & Woodcock, R. W. (2007). Woodcock–Johnson III normative update: Technical manual. Rolling Meadows: Riverside Publishing.

    Google Scholar 

  • McNamara, D. S., & Kintsch, W. (1996). Learning from texts: Effects of prior knowledge and text coherence. Discourse Processes,22, 247–288. https://doi.org/10.1080/01638539609544975.

    Article  Google Scholar 

  • McNamara, D. S., Ozuru, Y., & Floyd, R. G. (2011). Comprehension challenges in the fourth grade: The roles of text cohesion, text genre, and readers’ prior knowledge. International Electronic Journal of Elementary Education,4, 229–257.

    Google Scholar 

  • Miller, A. C., Davis, N., Gilbert, J. K., Cho, S.-J., Toste, J. R., Street, J., & Cutting, L. E. (2014). Novel approaches to examine passage, student, and question effects on reading comprehension. Learning Disabilities Research & Practice, 29, 25–35. https://doi.org/10.1111/ldrp.12027.

    Article  Google Scholar 

  • Miller, S. D., & Smith, D. E. P. (1989). Relations among oral reading, silent reading and listening comprehension of students at differing competency levels. Reading Research and Instruction,29, 73–84. https://doi.org/10.1080/19388079009558006.

    Article  Google Scholar 

  • Nation, K., & Snowling, M. (1997). Assessing reading difficulties: The validity and utility of current measures of reading skill. British Journal of Educational Psychology,67, 359–370.

    Article  Google Scholar 

  • Olson, M. W. (1985). Text type and reader ability: The effects on paraphrase and text-based inference questions. Journal of Reading Behavior,17, 199–214.

    Article  Google Scholar 

  • Pearson, P. D., & Hamm, D. N. (2005). The assessment of reading comprehension: A review of practices—Past, present, and future. In S. G. Paris & S. A. Stahl (Eds.), Children’s reading comprehension and assessment (pp. 13–69). Mahwah: Lawrence Erlbaum.

    Google Scholar 

  • Perfetti, C. A., Landi, N., & Oakhill, J. (2007). The acquisition of reading comprehension skill. In M. J. Snowling & C. Hulme (Eds.), The science of reading: A handbook (pp. 228–247). Malden: Blackwell.

    Google Scholar 

  • Pickering, S., & Gathercole, S. (2001). Working memory test battery for children. London: Pearson.

    Google Scholar 

  • RAND Reading Study Group. (2002). Reading for understanding: Toward an R&D program in reading comprehension. Santa Monica, CA: RAND.

  • Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods. Thousand Oaks: SAGE.

    Google Scholar 

  • Reardon, S. F., Kalogrides, D., Fahle, E. M., Podolsky, A., & Zárate, R. C. (2018). The relationship between test item format and gender achievement gaps on math and ELA tests in fourth and eighth grades. Educational Researcher,47, 284–294. https://doi.org/10.3102/0013189X18762105.

    Article  Google Scholar 

  • Reed, D. K., & Vaughn, S. (2012). Retell as an indicator of reading comprehension. Scientific Studies of Reading,16, 187–217.

    Article  Google Scholar 

  • Schwarz, G. (1978). Estimating the dimension of a model. The Annals of Statistics,6, 461–464.

    Article  Google Scholar 

  • Sesma, H. W., Mahone, E. M., Levine, T., Eason, S. H., & Cutting, L. E. (2009). The contribution of executive skills to reading comprehension. Child Neuropsychology,15, 232–246. https://doi.org/10.1080/09297040802220029.

    Article  Google Scholar 

  • Snijders, T. A. B., & Bosker, R. J. (2011). Multilevel analysis: An introduction to basic and advanced multilevel modeling. Thousand Oaks: SAGE.

    Google Scholar 

  • Spear-Swerling, L. (2004). Fourth graders performance on a state-mandated assessment involving two different measures of reading comprehension. Reading Psychology,25, 121–148. https://doi.org/10.1080/02702710490435727.

    Article  Google Scholar 

  • Spencer, M., Gilmour, A. F., Miller, A. C., Emerson, A. M., Saha, N. M., & Cutting, L. E. (2019). Understanding the influence of text complexity and question type on reading outcomes. Reading and Writing: An Interdisciplinary Journal,32, 603–637. https://doi.org/10.1007/s11145-018-9883-0.

    Article  Google Scholar 

  • Stram, D. O., & Lee, J. W. (1994). Variance components testing in the longitudinal mixed effects model. Biometrics,50, 1171–1177.

    Article  Google Scholar 

  • Stroud, K. C., & Reynolds, C. R. (2006). School motivation and learning strategies inventory. Los Angeles: Western Psychological Services.

    Google Scholar 

  • Swanson, J. M., Schuck, S., Porter, M. M., Carlson, C., Hartman, C. A., Sergeant, J. A. … Wigal, T. (2012). Categorical and dimensional definitions and evaluations of symptoms of ADHD: History of the SNAP and the SWAN rating scales. The International Journal of Educational and Psychological Assessment, 10(1), 51–70.

  • The Psychological Corporation. (1999). Wechsler abbreviated scale of intelligence. San Antonio: The Psychological Corporation.

    Google Scholar 

  • Tighe, E. L., & Schatschneider, C. (2013). A dominance analysis approach to determining predictor importance in third, seventh, and tenth grade reading comprehension skills. Reading and Writing: An Interdisciplinary Journal,27, 101–127. https://doi.org/10.1007/s11145-013-9435-6.

    Article  Google Scholar 

  • Torgesen, J. K., Wagner, R. K., & Rashotte, C. A. (2012). Test of word reading efficiency (2nd ed.). Austin: PRO-ED Inc.

    Google Scholar 

  • Van den Noortgate, W., De Boeck, P., & Meulders, M. (2003). Cross-classification multilevel logistic models in psychometrics. Journal of Educational and Behavioral Statistics,28, 369–386.

    Article  Google Scholar 

  • Walker, R. (2017). The effect of response format and presentation conditions on comprehension assessments for students with and without a reading disability (Doctoral dissertation). Retrieved from ProQuest Dissertations & Theses Global. (10601892).

  • Willcutt, E. G., Pennington, B. F., Olson, R. K., Chhabildas, N., & Hulslander, J. (2005). Neuropsychological analyses of comorbidity between reading disability and attention deficit hyperactivity disorder: In search of the common deficit. Developmental Neuropsychology,27, 35–78.

    Article  Google Scholar 

  • Woodcock, R. W., McGrew, K. S., & Mather, N. (2001). Woodcock–Johnson III tests of achievement. Rolling Meadows: Riverside Publishing.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Curriculum and Instruction, Texas State University, San Marcos, TX, 78666, USA

    Alyson A. Collins

  2. Department of Psychology, Florida Center for Reading Research, Florida State University, Tallahassee, USA

    Donald L. Compton

  3. Department of Education and Human Services, Lehigh University, Bethlehem, USA

    Esther R. Lindström

  4. Nashville, TN, USA

    Jennifer K. Gilbert

Authors
  1. Alyson A. Collins
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Donald L. Compton
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Esther R. Lindström
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jennifer K. Gilbert
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Alyson A. Collins.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Collins, A.A., Compton, D.L., Lindström, E.R. et al. Performance variations across reading comprehension assessments: Examining the unique contributions of text, activity, and reader. Read Writ 33, 605–634 (2020). https://doi.org/10.1007/s11145-019-09972-5

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11145-019-09972-5

Keywords

Search

Navigation