Skip to main content
SpringerLink
Log in

Measuring strategy use in context with multiple-choice items

  • Published:
Metacognition and Learning Aims and scope Submit manuscript

Abstract

A number of authors have presented data that challenge the validity of self-report of strategy use or choice of strategy. We created a multiple-choice measure of students’ strategy use based on the work of Kozminsky, E., and Kozminsky, L. (2001), and tested it with three samples as part of a series of studies testing the fit of the DIME model of reading comprehension. One study was conducted at the high school level (N = 175) and two at the undergraduate level (N = 185 and 737). Over the three studies with three different samples, we found good evidence for the internal consistency reliability and concurrent validity of this type of measure. Commonality analysis suggested that strategy use mainly makes a shared contribution to comprehension with other predictors, especially inference, and to some extent vocabulary, background knowledge, and word reading. The measure was relatively easy to construct and easy to administer to large numbers of students, and showed much higher evidence of concurrent validity than self-ratings of frequency of use of strategies.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

Notes

  1. Needless to say, reading researchers also need high-quality measures of reading comprehension. To date, most comprehension measures have been atheoretical—large pools of items are tested, and a subset of questions showing good reliability, predictive validity with school achievement, and other good psychometric qualities are retained (queryPellegrino, Chudowski, & Glaser, 2001).

References

  • Alexander, P. A., & Judy, J. E. (1988). The interaction of domain-specific and strategic knowledge in academic performance. Review of Educational Research, 58(4), 375–404.

    Google Scholar 

  • Azevedo, R., Moos, D. C., Greene, J. A., Winters, F. I., & Cromley, J. C. (2008a). Why is externally-regulated learning more effective than self-regulated learning with hypermedia? Educational Technology Research and Development, 56(1), 45–72.

    Article  Google Scholar 

  • Azevedo, R., Witherspoon, A. M., Graesser, A., McNamara, D., Rus, V., Cai, Z., Lintean, M., & Siler, E. (2008b). MetaTutor: An adaptive hypermedia system for training and fostering self-regulated learning about complex science topics. Paper presented at a Symposium on ITSs with agents at the annual meeting of the Society for Computers in Psychology, Chicago, IL

  • Baker, L. (1989). Metacognition, comprehension monitoring, and the adult reader. Educational Psychology Review, 1(1), 3–38.

    Article  Google Scholar 

  • Baker, L., & Cerro, L. C. (2000). Assessing metacognition in children and adults. In G. Schraw & J. C. Impara (Eds.), Issues in the measurement of metacognition (pp. 99–146). Lincoln: Buros Institute.

    Google Scholar 

  • Best, R. M., Rowe, M., Ozuru, Y., & McNamara, D. S. (2005). Deep-level comprehension of science texts: The role of the reader and the text. Topics in Language Disorders, 25, 62–80.

    Article  Google Scholar 

  • Brand-Gruwel, S., Aarnoutse, C., & Van Den Bos, K. P. (1998). Improving text comprehension strategies in reading and listening settings. Learning and Instruction, 8, 63–81.

    Article  Google Scholar 

  • Cain, K., Oakhill, J., & Bryant, P. (2004). Children’s reading comprehension ability: Concurrent prediction by working memory, verbal ability, and component skills. Journal of Educational Psychology, 96(1), 31–42.

    Article  Google Scholar 

  • Calvo, M. (2005). Relative contribution of vocabulary knowledge and working memory span to elaborative inferences in reading. Learning and Individual Differences, 15(1), 53–65.

    Article  Google Scholar 

  • Campbell, N. A., & Reece, J. B. (2001). Biology (6th ed.). San Francisco: Benjamin Cummings.

    Google Scholar 

  • Cromley, J. G., & Azevedo, R. (2006). Self-report of reading comprehension strategies: What are we measuring? Metacognition and Learning, 1(3), 229–247.

    Article  Google Scholar 

  • Cromley, J. G., & Azevedo, R. (2007). Testing and refining the direct and inferential mediation model of reading comprehension. Journal of Educational Psychology, 99(2), 311–325. doi:10.1037/0022-0663.99.2.311.

    Google Scholar 

  • Cromley, J. G., & Snyder, L. E. (2007). Testing the fit of the DIME model of reading comprehension with undergraduate students. Paper presented at the annual conference of the American Educational Research Association, Chicago, IL.

  • Cromley, J. G., Snyder-Hogan, L. E., & Luciw-Dubas, U. A. (2010a). Reading comprehension of scientific text: A domain-specific test of the direct and inferential mediation model of reading comprehension. Journal of Educational Psychology, 102(3), 687–700. doi:10.1037/a001945.

    Google Scholar 

  • Cromley, J. G., Snyder-Hogan, L. E., & Luciw-Dubas, U. A. (2010b). Cognitive activities in complex science text and diagrams. Contemporary Educational Psychology, 35, 59–74. doi:10.1016/j.cedpsych.2009.10.002.

    Google Scholar 

  • Evans, J. E., Floyd, R. G., McGrew, K. S., & Leforgee, M. H. (2001). The relations between measures of Cattell-Horn-Carroll (CHC) cognitive abilities and reading achievement during childhood and adolescence. School Psychology Review, 31(2), 246–262.

    Google Scholar 

  • Glaser, C., & Brunstein, J. (2007). Improving fourth-grade students’ composition skills: Effects of strategy instruction and self-regulation procedures. Journal of Educational Psychology, 99(2), 297–310.

    Article  Google Scholar 

  • Gonzalez, J. E., & Uhing, B. M. (2008). Home literacy environments and young Hispanic children’s English and Spanish oral language: A communality analysis. Journal of Early Intervention, 30(2), 116–139. doi:10.1177/1053815107313858.

    Article  Google Scholar 

  • Graesser, A. (2007). An introduction to strategic reading comprehension. In D. S. McNamara (Ed.), Reading comprehension strategies: Theories, interventions, and technologies. Mahwah: Lawrence Erlbaum Associates.

    Google Scholar 

  • Hadwin, A., Winne, P., Stockley, D., Nesbit, J., & Woszczyna, C. (2001). Context moderates students’ self-reports about how they study. Journal of Educational Psychology, 93(3), 477–487.

    Article  Google Scholar 

  • Hadwin, A. F., Nesbit, J. C., Jamieson-Noel, D., Code, J., & Winne, P. H. (2007). Examining trace data to explore self-regulated learning. Metacognition and Learning, 2(2), 107–124. doi:10.1007/s11409-007-9016-7.

    Article  Google Scholar 

  • Hannon, B., & Daneman, M. (2001). A new tool for measuring and understanding individual differences in the component processes of reading comprehension. Journal of Educational Psychology, 93(1), 103–128.

    Article  Google Scholar 

  • Hannus, M., & Hyona, J. (1999). Utilization of illustrations during learning of science textbook passages among low- and high-ability children. Contemporary Educational Psychology, 24(2), 95–123.

    Article  Google Scholar 

  • Hine, D. C. (1994). Hine sight: Black women and the re-construction of American history. NY: Carlson Publications.

    Google Scholar 

  • Johnson-Glenberg, M. C. (2005). Web-based training of metacognitive strategies for text comprehension: Focus on poor comprehenders. Reading and Writing, 18, 755–786.

    Article  Google Scholar 

  • Katz, S., Blackburn, A. B., & Lautenschlager, G. J. (1991). Answering reading comprehension items without passages on the SAT when items are quasi-randomized. Educational and Psychological Measurement, 51(3), 747–754.

    Article  Google Scholar 

  • Kintsch, W. (1998). Comprehension: A paradigm for cognition. Cambridge, England: Cambridge University Press.

  • Kozminsky, E., & Kozminsky, L. (2001). How do general knowledge and reading strategies ability relate to reading comprehension of high school students at different educational levels? Journal of Research in Reading, 24(2), 187–204.

    Article  Google Scholar 

  • Leslie, L., & Caldwell, J. (2000). Qualitative reading inventory III. New York: Longman.

    Google Scholar 

  • MacGinitie, W. H., MacGinitie, R. K., Maria, K., & Dreyer, L. G. (2001). Gates-MacGinitie reading tests, Level 7/9, Form S (4th ed.). Itasca: Riverside.

    Google Scholar 

  • MacGinitie, W. H., MacGinitie, R. K., Maria, K., & Dreyer, L. G. (2002). Technical report for the fourth edition, Gates-MacGinitie reading tests. Itasca: Riverside.

    Google Scholar 

  • McKeown, M. G., Beck, I. L., & Blake, R. G. K. (2009). Rethinking Comprehension Instruction: Comparing strategies and content instructional approaches. Reading Research Quarterly, 44(3), 218–253.

    Google Scholar 

  • McNamara, D. S., & Magliano, J. P. (2009). Towards a comprehensive model of comprehension. In B. Ross (Ed.), The psychology of learning and motivation (51st ed., pp. 297–384). NY: Elsevier Science.

    Chapter  Google Scholar 

  • McNamara, D. S., O’Reilly, T. P., Best, R. M., & Ozuru, Y. (2006). Improving adolescent students’ reading comprehension with iSTART. Journal of Educational Computing Research, 34(2), 147–171.

    Article  Google Scholar 

  • McNamara, D. S., de Vega, M., & O’Reilly, T. (2007). Comprehension skill, inference making, and the role of knowledge. In F. Schmalhofer & C.A. Perfetti (Eds.), Higher level language processes in the brain: Inference and comprehension processes (pp. 233–251). Mahwah, NJ: Erlbaum.

  • Mokhtari, K., & Reichard, C. A. (2002). Assessing students’ metacognitive awareness of reading strategies. Journal of Educational Psychology, 94(2), 249–259.

    Article  Google Scholar 

  • Naceur, A., & Schiefele, U. (2005). Motivation and learning—The role of interest in construction of representation of text and long-term retention: Inter- and intraindividual analyses. European Journal of Psychology of Education, 20(2), 155–170.

    Article  Google Scholar 

  • NICHD. (2000). Teaching children to read: An evidence-based assessment of the scientific research literature on reading and its implications for reading instruction. Washington, DC: NICHD.

  • Oakhill, J., & Yuill, N. (1996). Higher order factors in comprehension disability: Processes and remediation. In C. Cornoldi & J. Oakhill (Eds.), Reading comprehension difficulties: Processes and intervention (pp. 69–92). Mahwah: Erlbaum.

    Google Scholar 

  • OECD. (2000). Manual for the PISA 2000 database. Paris: OECD.

    Google Scholar 

  • Richardson, J. T. E. (2004). Methodological issues in questionnaire-based research on student learning in higher education. Educational Psychology Review, 16(4), 347–358.

    Google Scholar 

  • Samuelstuen, M., & Bråten, I. (2007). Examining the validity of self-reports on scales measuring students’ strategic processing. The British Journal of Educational Psychology, 77(2), 351–378.

    Article  Google Scholar 

  • Schmalhofer, F., McDaniel, M. A., & Keefe, D. (2002). A unified model for predictive and bridging inferences. Discourse Processes, 33(2), 105–132.

    Google Scholar 

  • Schmitt, M. (1990). A questionnaire to measure children’s awareness of strategic reading process. The Reading Teacher, 49, 454–461.

    Google Scholar 

  • Shorris, E. (1997). On the uses of a liberal education II. As a weapon in the hands of the restless poor. Harper‘s Magazine, 295, 50–59.

    Google Scholar 

  • Starr, C., & McMillan, B. (2001). Human biology (4th ed.). Pacific Grove: Wadsworth Group.

    Google Scholar 

  • Stromso, H. I., Braten, I., & Samuelstuen, M. S. (2003). Students’ strategic use of multiple sources during expository text reading: A longitudinal think-aloud study. Cognition and Instruction, 21(2), 113–147.

    Article  Google Scholar 

  • VanSledright, B. A., & Frankes, L. (2000). Concept- and strategic-knowledge development in historical study: Reading American history: A comparative exploration in two fourth-grade classrooms. Cognition and Instruction, 18(2), 239–283.

    Article  Google Scholar 

  • Veenman, M. V. J. (2005). The assessment of metacognitive skills: What can be learned from multi-method designs? In B. Moschner & C. Artelt (Eds.), Lernstrategien und Metakognition: Implikationen für Forschung und Praxis (pp. 75–97). Berlin: Waxmann.

    Google Scholar 

  • Willson, V. L., & Rupley, W. H. (1997). A structural equation model for reading comprehension based on background, phonemic, and strategy knowledge. Scientific Studies of Reading, 1(1), 45–63.

    Article  Google Scholar 

  • Winne, P. H., & Nesbit, J. C. (2009). Supporting self-regulated learning with cognitive tools. In D. J. Hacker, J. Dunlosky, & A. C. Graesser (Eds.), Handbook of metacognition in education (pp. 259–277). Mahwah: Lawrence Erlbaum Associates.

    Google Scholar 

  • Woodcock, R. W. (1997). Woodcock diagnostic reading battery. Itasca: Riverside.

    Google Scholar 

  • Zientek, L. R., & Thompson, B. (2009). Matrix summaries improve research reports: Secondary analyses using published literature. Educational Researcher, 38(5), 343–352. doi:10.3102/0013189X09339056.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Psychological Studies in Education, Temple University, Ritter Annex 201, 1301 Cecil B. Moore Avenue, Philadelphia, 19122-6091, PA, USA

    Jennifer Cromley

  2. Department of Educational and Counselling Psychology (ECP), Laboratory for the Study of Metacognition and Advanced Learning Technologies, McGill University, 3700 McTavish Street, Room 549, Montreal, Québec, H3A 1Y2, Canada

    Roger Azevedo

Authors
  1. Jennifer Cromley
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Roger Azevedo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jennifer Cromley.

Appendix

Appendix

Strategy measure item stems

Study 1

Which of the following is most likely to follow this passage? (used twice)

Which of the following would be most useful to know in order to understand the passage?

Which of the following is the best summary of the paragraph? (used twice)

What would be the best strategy for finding the date the Stamp Act was passed?

Which of the following questions could NOT be answered from the passage? (used twice)

Which of the following sentences could most easily be omitted from the paragraph without changing its meaning?

Which sentence would it make the most sense to underline or highlight?

Study 2

Which of the following is most likely to follow this passage? (used twice)

Which of the following would be most useful to know in order to understand the passage? (used twice)

Which of the following is the best summary of the paragraph? (used twice)

Which of the following questions could be answered from the passage? (used twice)

The main idea of this passage is that (used twice)

Study 3

Which of the following is most likely to follow this passage? (used twice)

Which of the following would be most useful to know in order to understand the passage? (used twice)

Which of the following is the best summary of the paragraph? (used twice)

Which of the following questions could [NOT] be answered from the passage? (used three times)

The diagram below that best represents the second paragraph is

The label “secreted antibodies” in Figure 8.12 means that

The most effective way to take notes on this paragraph would be [text structure]

Rights and permissions

Reprints and permissions

About this article

Cite this article

Cromley, J., Azevedo, R. Measuring strategy use in context with multiple-choice items. Metacognition Learning 6, 155–177 (2011). https://doi.org/10.1007/s11409-011-9070-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11409-011-9070-z

Keywords

Search

Navigation