Measuring strategy use in context with multiple-choice items

Abstract

A number of authors have presented data that challenge the validity of self-report of strategy use or choice of strategy. We created a multiple-choice measure of students’ strategy use based on the work of Kozminsky, E., and Kozminsky, L. (2001), and tested it with three samples as part of a series of studies testing the fit of the DIME model of reading comprehension. One study was conducted at the high school level (N = 175) and two at the undergraduate level (N = 185 and 737). Over the three studies with three different samples, we found good evidence for the internal consistency reliability and concurrent validity of this type of measure. Commonality analysis suggested that strategy use mainly makes a shared contribution to comprehension with other predictors, especially inference, and to some extent vocabulary, background knowledge, and word reading. The measure was relatively easy to construct and easy to administer to large numbers of students, and showed much higher evidence of concurrent validity than self-ratings of frequency of use of strategies.

This is a preview of subscription content, log in to check access.

Fig. 1

Notes

  1. 1.

    Needless to say, reading researchers also need high-quality measures of reading comprehension. To date, most comprehension measures have been atheoretical—large pools of items are tested, and a subset of questions showing good reliability, predictive validity with school achievement, and other good psychometric qualities are retained (queryPellegrino, Chudowski, & Glaser, 2001).

References

  1. Alexander, P. A., & Judy, J. E. (1988). The interaction of domain-specific and strategic knowledge in academic performance. Review of Educational Research, 58(4), 375–404.

    Google Scholar 

  2. Azevedo, R., Moos, D. C., Greene, J. A., Winters, F. I., & Cromley, J. C. (2008a). Why is externally-regulated learning more effective than self-regulated learning with hypermedia? Educational Technology Research and Development, 56(1), 45–72.

    Article  Google Scholar 

  3. Azevedo, R., Witherspoon, A. M., Graesser, A., McNamara, D., Rus, V., Cai, Z., Lintean, M., & Siler, E. (2008b). MetaTutor: An adaptive hypermedia system for training and fostering self-regulated learning about complex science topics. Paper presented at a Symposium on ITSs with agents at the annual meeting of the Society for Computers in Psychology, Chicago, IL

  4. Baker, L. (1989). Metacognition, comprehension monitoring, and the adult reader. Educational Psychology Review, 1(1), 3–38.

    Article  Google Scholar 

  5. Baker, L., & Cerro, L. C. (2000). Assessing metacognition in children and adults. In G. Schraw & J. C. Impara (Eds.), Issues in the measurement of metacognition (pp. 99–146). Lincoln: Buros Institute.

    Google Scholar 

  6. Best, R. M., Rowe, M., Ozuru, Y., & McNamara, D. S. (2005). Deep-level comprehension of science texts: The role of the reader and the text. Topics in Language Disorders, 25, 62–80.

    Article  Google Scholar 

  7. Brand-Gruwel, S., Aarnoutse, C., & Van Den Bos, K. P. (1998). Improving text comprehension strategies in reading and listening settings. Learning and Instruction, 8, 63–81.

    Article  Google Scholar 

  8. Cain, K., Oakhill, J., & Bryant, P. (2004). Children’s reading comprehension ability: Concurrent prediction by working memory, verbal ability, and component skills. Journal of Educational Psychology, 96(1), 31–42.

    Article  Google Scholar 

  9. Calvo, M. (2005). Relative contribution of vocabulary knowledge and working memory span to elaborative inferences in reading. Learning and Individual Differences, 15(1), 53–65.

    Article  Google Scholar 

  10. Campbell, N. A., & Reece, J. B. (2001). Biology (6th ed.). San Francisco: Benjamin Cummings.

    Google Scholar 

  11. Cromley, J. G., & Azevedo, R. (2006). Self-report of reading comprehension strategies: What are we measuring? Metacognition and Learning, 1(3), 229–247.

    Article  Google Scholar 

  12. Cromley, J. G., & Azevedo, R. (2007). Testing and refining the direct and inferential mediation model of reading comprehension. Journal of Educational Psychology, 99(2), 311–325. doi:10.1037/0022-0663.99.2.311.

    Google Scholar 

  13. Cromley, J. G., & Snyder, L. E. (2007). Testing the fit of the DIME model of reading comprehension with undergraduate students. Paper presented at the annual conference of the American Educational Research Association, Chicago, IL.

  14. Cromley, J. G., Snyder-Hogan, L. E., & Luciw-Dubas, U. A. (2010a). Reading comprehension of scientific text: A domain-specific test of the direct and inferential mediation model of reading comprehension. Journal of Educational Psychology, 102(3), 687–700. doi:10.1037/a001945.

    Google Scholar 

  15. Cromley, J. G., Snyder-Hogan, L. E., & Luciw-Dubas, U. A. (2010b). Cognitive activities in complex science text and diagrams. Contemporary Educational Psychology, 35, 59–74. doi:10.1016/j.cedpsych.2009.10.002.

    Google Scholar 

  16. Evans, J. E., Floyd, R. G., McGrew, K. S., & Leforgee, M. H. (2001). The relations between measures of Cattell-Horn-Carroll (CHC) cognitive abilities and reading achievement during childhood and adolescence. School Psychology Review, 31(2), 246–262.

    Google Scholar 

  17. Glaser, C., & Brunstein, J. (2007). Improving fourth-grade students’ composition skills: Effects of strategy instruction and self-regulation procedures. Journal of Educational Psychology, 99(2), 297–310.

    Article  Google Scholar 

  18. Gonzalez, J. E., & Uhing, B. M. (2008). Home literacy environments and young Hispanic children’s English and Spanish oral language: A communality analysis. Journal of Early Intervention, 30(2), 116–139. doi:10.1177/1053815107313858.

    Article  Google Scholar 

  19. Graesser, A. (2007). An introduction to strategic reading comprehension. In D. S. McNamara (Ed.), Reading comprehension strategies: Theories, interventions, and technologies. Mahwah: Lawrence Erlbaum Associates.

    Google Scholar 

  20. Hadwin, A., Winne, P., Stockley, D., Nesbit, J., & Woszczyna, C. (2001). Context moderates students’ self-reports about how they study. Journal of Educational Psychology, 93(3), 477–487.

    Article  Google Scholar 

  21. Hadwin, A. F., Nesbit, J. C., Jamieson-Noel, D., Code, J., & Winne, P. H. (2007). Examining trace data to explore self-regulated learning. Metacognition and Learning, 2(2), 107–124. doi:10.1007/s11409-007-9016-7.

    Article  Google Scholar 

  22. Hannon, B., & Daneman, M. (2001). A new tool for measuring and understanding individual differences in the component processes of reading comprehension. Journal of Educational Psychology, 93(1), 103–128.

    Article  Google Scholar 

  23. Hannus, M., & Hyona, J. (1999). Utilization of illustrations during learning of science textbook passages among low- and high-ability children. Contemporary Educational Psychology, 24(2), 95–123.

    Article  Google Scholar 

  24. Hine, D. C. (1994). Hine sight: Black women and the re-construction of American history. NY: Carlson Publications.

    Google Scholar 

  25. Johnson-Glenberg, M. C. (2005). Web-based training of metacognitive strategies for text comprehension: Focus on poor comprehenders. Reading and Writing, 18, 755–786.

    Article  Google Scholar 

  26. Katz, S., Blackburn, A. B., & Lautenschlager, G. J. (1991). Answering reading comprehension items without passages on the SAT when items are quasi-randomized. Educational and Psychological Measurement, 51(3), 747–754.

    Article  Google Scholar 

  27. Kintsch, W. (1998). Comprehension: A paradigm for cognition. Cambridge, England: Cambridge University Press.

  28. Kozminsky, E., & Kozminsky, L. (2001). How do general knowledge and reading strategies ability relate to reading comprehension of high school students at different educational levels? Journal of Research in Reading, 24(2), 187–204.

    Article  Google Scholar 

  29. Leslie, L., & Caldwell, J. (2000). Qualitative reading inventory III. New York: Longman.

    Google Scholar 

  30. MacGinitie, W. H., MacGinitie, R. K., Maria, K., & Dreyer, L. G. (2001). Gates-MacGinitie reading tests, Level 7/9, Form S (4th ed.). Itasca: Riverside.

    Google Scholar 

  31. MacGinitie, W. H., MacGinitie, R. K., Maria, K., & Dreyer, L. G. (2002). Technical report for the fourth edition, Gates-MacGinitie reading tests. Itasca: Riverside.

    Google Scholar 

  32. McKeown, M. G., Beck, I. L., & Blake, R. G. K. (2009). Rethinking Comprehension Instruction: Comparing strategies and content instructional approaches. Reading Research Quarterly, 44(3), 218–253.

    Google Scholar 

  33. McNamara, D. S., & Magliano, J. P. (2009). Towards a comprehensive model of comprehension. In B. Ross (Ed.), The psychology of learning and motivation (51st ed., pp. 297–384). NY: Elsevier Science.

    Google Scholar 

  34. McNamara, D. S., O’Reilly, T. P., Best, R. M., & Ozuru, Y. (2006). Improving adolescent students’ reading comprehension with iSTART. Journal of Educational Computing Research, 34(2), 147–171.

    Article  Google Scholar 

  35. McNamara, D. S., de Vega, M., & O’Reilly, T. (2007). Comprehension skill, inference making, and the role of knowledge. In F. Schmalhofer & C.A. Perfetti (Eds.), Higher level language processes in the brain: Inference and comprehension processes (pp. 233–251). Mahwah, NJ: Erlbaum.

  36. Mokhtari, K., & Reichard, C. A. (2002). Assessing students’ metacognitive awareness of reading strategies. Journal of Educational Psychology, 94(2), 249–259.

    Article  Google Scholar 

  37. Naceur, A., & Schiefele, U. (2005). Motivation and learning—The role of interest in construction of representation of text and long-term retention: Inter- and intraindividual analyses. European Journal of Psychology of Education, 20(2), 155–170.

    Article  Google Scholar 

  38. NICHD. (2000). Teaching children to read: An evidence-based assessment of the scientific research literature on reading and its implications for reading instruction. Washington, DC: NICHD.

  39. Oakhill, J., & Yuill, N. (1996). Higher order factors in comprehension disability: Processes and remediation. In C. Cornoldi & J. Oakhill (Eds.), Reading comprehension difficulties: Processes and intervention (pp. 69–92). Mahwah: Erlbaum.

    Google Scholar 

  40. OECD. (2000). Manual for the PISA 2000 database. Paris: OECD.

    Google Scholar 

  41. Richardson, J. T. E. (2004). Methodological issues in questionnaire-based research on student learning in higher education. Educational Psychology Review, 16(4), 347–358.

    Google Scholar 

  42. Samuelstuen, M., & Bråten, I. (2007). Examining the validity of self-reports on scales measuring students’ strategic processing. The British Journal of Educational Psychology, 77(2), 351–378.

    Article  Google Scholar 

  43. Schmalhofer, F., McDaniel, M. A., & Keefe, D. (2002). A unified model for predictive and bridging inferences. Discourse Processes, 33(2), 105–132.

    Google Scholar 

  44. Schmitt, M. (1990). A questionnaire to measure children’s awareness of strategic reading process. The Reading Teacher, 49, 454–461.

    Google Scholar 

  45. Shorris, E. (1997). On the uses of a liberal education II. As a weapon in the hands of the restless poor. Harper‘s Magazine, 295, 50–59.

    Google Scholar 

  46. Starr, C., & McMillan, B. (2001). Human biology (4th ed.). Pacific Grove: Wadsworth Group.

    Google Scholar 

  47. Stromso, H. I., Braten, I., & Samuelstuen, M. S. (2003). Students’ strategic use of multiple sources during expository text reading: A longitudinal think-aloud study. Cognition and Instruction, 21(2), 113–147.

    Article  Google Scholar 

  48. VanSledright, B. A., & Frankes, L. (2000). Concept- and strategic-knowledge development in historical study: Reading American history: A comparative exploration in two fourth-grade classrooms. Cognition and Instruction, 18(2), 239–283.

    Article  Google Scholar 

  49. Veenman, M. V. J. (2005). The assessment of metacognitive skills: What can be learned from multi-method designs? In B. Moschner & C. Artelt (Eds.), Lernstrategien und Metakognition: Implikationen für Forschung und Praxis (pp. 75–97). Berlin: Waxmann.

    Google Scholar 

  50. Willson, V. L., & Rupley, W. H. (1997). A structural equation model for reading comprehension based on background, phonemic, and strategy knowledge. Scientific Studies of Reading, 1(1), 45–63.

    Article  Google Scholar 

  51. Winne, P. H., & Nesbit, J. C. (2009). Supporting self-regulated learning with cognitive tools. In D. J. Hacker, J. Dunlosky, & A. C. Graesser (Eds.), Handbook of metacognition in education (pp. 259–277). Mahwah: Lawrence Erlbaum Associates.

    Google Scholar 

  52. Woodcock, R. W. (1997). Woodcock diagnostic reading battery. Itasca: Riverside.

    Google Scholar 

  53. Zientek, L. R., & Thompson, B. (2009). Matrix summaries improve research reports: Secondary analyses using published literature. Educational Researcher, 38(5), 343–352. doi:10.3102/0013189X09339056.

    Article  Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Jennifer Cromley.

Appendix

Appendix

Strategy measure item stems

Study 1

Which of the following is most likely to follow this passage? (used twice)

Which of the following would be most useful to know in order to understand the passage?

Which of the following is the best summary of the paragraph? (used twice)

What would be the best strategy for finding the date the Stamp Act was passed?

Which of the following questions could NOT be answered from the passage? (used twice)

Which of the following sentences could most easily be omitted from the paragraph without changing its meaning?

Which sentence would it make the most sense to underline or highlight?

Study 2

Which of the following is most likely to follow this passage? (used twice)

Which of the following would be most useful to know in order to understand the passage? (used twice)

Which of the following is the best summary of the paragraph? (used twice)

Which of the following questions could be answered from the passage? (used twice)

The main idea of this passage is that (used twice)

Study 3

Which of the following is most likely to follow this passage? (used twice)

Which of the following would be most useful to know in order to understand the passage? (used twice)

Which of the following is the best summary of the paragraph? (used twice)

Which of the following questions could [NOT] be answered from the passage? (used three times)

The diagram below that best represents the second paragraph is

The label “secreted antibodies” in Figure 8.12 means that

The most effective way to take notes on this paragraph would be [text structure]

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Cromley, J., Azevedo, R. Measuring strategy use in context with multiple-choice items. Metacognition Learning 6, 155–177 (2011). https://doi.org/10.1007/s11409-011-9070-z

Download citation

Keywords

  • Strategy use
  • Comprehension
  • Validity
  • Reliability
  • Measurement