Self-report of reading comprehension strategies: What are we measuring?

Abstract

Proficient readers engage in a wide range of cognitive and metacognitive strategies, and both developmental and classroom intervention researchers are in need of high-quality measures of strategy use. Several researchers have recently called into question the validity of the most common type of measures of strategy use in reading, self-report or introspective measures (i.e., the participant must report on his or her cognitive activity while not actually engaged in the activity). We administered three parallel strategy use measures to a sample of 30 ninth-grade students: a prospective self-report measure, a concurrent multiple-choice measure which required students to apply the strategies to specific passages, and a text on which we asked students to think aloud. We also collected two measures of reading comprehension—a standardized measure and free recall scores. Consistent with Veenman’s (2005) conclusions based on a literature review, the concurrent multiple-choice and think-aloud data were both significantly correlated with both of the comprehension scores and with each other, whereas the prospective self-report data had non-significant correlations with all of the other measures. We conclude by recommending concurrent measures for researchers who wish to study strategy use in reading comprehension.

This is a preview of subscription content, log in to check access.

References

  1. Afflerbach, P. (2000). Verbal reports and protocol analysis. In M. L. Kamil, P. B. Mosenthal, P. D. Pearson, & R. Barr (Eds.), Handbook of reading research, vol. 3 (pp. 163–179). Mahwah, NJ: Erlbaum.

    Google Scholar 

  2. Alexander, P. A., & Judy, J. E. (1988). The interaction of domain-specific and strategic knowledge in academic performance. Review of Educational Research, 58(4), 375–404.

    Article  Google Scholar 

  3. Alexander, P. A., Murphy, P. K., Woods, B. S., Duhon, K. E., & Parker, D. (1997). College instruction and concomitant changes in students’ knowledge, interest, and strategy use: A study of domain learning. Contemporary Educational Psychology, 22(2), 125–146.

    Article  Google Scholar 

  4. Alfassi, M. (2004). Reading to learn: Effects of combined strategy instruction on high school students. Journal of Educational Research, 97(4), 171–184.

    Google Scholar 

  5. Artelt, C., Schiefele, U., & Schneider, S. (2001). Predictors of reading literacy. European Journal of Psychology of Education, 16(3), 262–383.

    Article  Google Scholar 

  6. Azevedo, R., & Cromley, J. G. (2004). Does training on self-regulated learning facilitate students’ learning with hypermedia? Journal of Educational Psychology, 96(3), 523–535.

    Article  Google Scholar 

  7. Azevedo, R., Cromley, J. G., & Seibert, D. (2004). Does adaptive scaffolding facilitate students’ ability to regulate their learning with hypermedia? Contemporary Educational Psychology, 29, 344–370.

    Article  Google Scholar 

  8. Baker, L., & Cerro, L. C. (2000). Assessing metacognition in children and adults. In G. Schraw & J. C. Impara (Eds.), Issues in the measurement of metacognition (pp. 99–146). Lincoln, NE: Buros Institute.

    Google Scholar 

  9. Bråten, I., & Samuelstuen, M. S. (2004). Does the influence of reading purpose on reports of strategic text processing depend on students’ topic knowledge? Journal of Educational Psychology, 96(2), 324–336.

    Article  Google Scholar 

  10. Brown, A. L., & Day, J. D. (1983). Macrorules for summarizing texts: The development of expertise. Journal of Verbal Learning and Verbal Behavior, 22(1), 1–14.

    Article  Google Scholar 

  11. Brown, R., Pressley, M., Van Meter, P., & Schuder, T. (1996). A quasi-experimental validation of transactional strategies instruction with low-achieving second-grade readers. Journal of Educational Psychology, 88(1), 18–37.

    Article  Google Scholar 

  12. Crain-Thoreson, C., Lippman, M. Z., & McClendon-Magnuson, D. (1997). Windows on comprehension: Reading comprehension processes as revealed by two think-aloud procedures. Journal of Educational Psychology, 89(4), 579–591.

    Article  Google Scholar 

  13. Cromley, J. G. (2005). Reading comprehension component processes in early adolescence.Dissertation, University of Maryland, College Park, MD.

  14. Cromley, J. G., & Azevedo, R. (2004a). Testing the fit of three models of reading comprehension with 9th grade students. Poster presented at the annual meeting of the American Educational Research Association, San Diego, CA.

  15. Cromley, J. G., & Azevedo, R. (2004b). Using think-aloud data to illuminate a model of high school reading comprehension. Poster presented at the annual meeting of the American Educational Research Association, San Diego, CA.

  16. Cromley, J. G., & Azevedo, R. (2005). Testing the fit of four variations of the Inferential Mediation model of reading comprehension. Paper presented at the annual meeting of the American Educational Research Association, Montreal, Quebec, CA.

  17. Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis: Verbal reports as data (revised edition). Cambridge, MA: Bradford.

    Google Scholar 

  18. Ericsson, K. A., & Simon, H. A. (1998). How to study thinking in everyday life: Contrasting think-aloud protocols with descriptions and explanations of thinking. Mind, Culture, & Activity, 5(3), 178–186.

    Article  Google Scholar 

  19. Garcia, G. E. (1991). Factors influencing the English reading test performance of Spanish-speaking Hispanic children. Reading Research Quarterly, 25(4), 371–392.

    Article  Google Scholar 

  20. Garner, R. (1987). Metacognition and reading comprehension. Norwood, NJ: Ablex.

    Google Scholar 

  21. Hacker, D. J. (1998). Self-regulated comprehension during normal reading. In D. J. Hacker, J. Dunlosky, & A. C. Graesser (Eds.), Metacognition in educational theory and practice (pp. 165–191). Mahwah, NJ: Erlbaum.

    Google Scholar 

  22. Hadwin, A., Winne, P., Stockley, D., Nesbit, J., & Woszczyna, C. (2001). Context moderates students’ self-reports about how they study. Journal of Educational Psychology, 93(3), 477–487.

    Article  Google Scholar 

  23. Kozminsky, E., & Kozminsky, L. (2001). How do general knowledge and reading strategies ability relate to reading comprehension of high school students at different educational levels? Journal of Research in Reading, 24(2), 187–204.

    Article  Google Scholar 

  24. MacGinitie, W. H., MacGinitie, R. K., Maria, K., & Dreyer, L. G. (2001). Gates–MacGinitie reading tests, Level 7/9, Form S (4th ed.). Itasca, IL: Riverside.

    Google Scholar 

  25. Magliano, J. P., & Graesser, A. C. (1991). A three-pronged method for studying inference generation in literary text. Poetics, 20, 193–232.

    Article  Google Scholar 

  26. McNamara, D. S. (2004). SERT: Self-Explanation Reading Training. Discourse Processes, 38(1), 1–30.

    Article  Google Scholar 

  27. Mokhtari, K., & Reichard, C. A. (2002). Assessing students’ metacognitive awareness of reading strategies. Journal of Educational Psychology, 94(2), 249–259.

    Article  Google Scholar 

  28. National Reading Panel. (2000). Teaching children to read: An evidence-based assessment of the scientific research literature on reading and its implications for reading instruction. Washington, DC: NICHD.

    Google Scholar 

  29. Nokes, J. D., & Dole, J. A. (2004). Helping adolescent readers through explicit strategy instruction. In T. L. Jetton & J. A. Dole (Eds.), Adolescent literacy research and practice (pp. 162–182). NY: Guilford.

    Google Scholar 

  30. Palincsar, A. S., & Brown, A. L. (1984). Reciprocal teaching of comprehension-fostering and comprehension-monitoring activities. Cognition and Instruction, 2, 117–175.

    Google Scholar 

  31. Paris, S. G., Cross, D. R., & Lipson, M. Y. (1984). Informed strategies for learning: A program to improve children’s reading awareness and comprehension. Journal of Educational Psychology, 76(6), 1239–1252.

    Article  Google Scholar 

  32. Paris, S. G., Wasik, B. A., & Turner, J. C. (1991). The development of strategic readers. In R. Barr, M. L. Kamil, P. B. Mosenthal, & P. D. Pearson (Eds.), Handbook of reading research, volume II (pp. 609–640). New York: Longman.

    Google Scholar 

  33. Pressley, M., & Afflerbach, P. (1995). Verbal protocols of reading: The nature of constructively responsive reading. Hillsdale, NJ: Erlbaum.

    Google Scholar 

  34. Pressley, M., & Hilden, K. (2004). Verbal protocols of reading. In N. K. Duke & M. H. Mallette (Eds.), Literacy research methodologies (pp. 308–321). NY: Guilford.

    Google Scholar 

  35. Rinehart, S. D., Stahl, S. A., & Erickson, L. G. (1986). Some effects of summarization training on reading and studying. Reading Research Quarterly, 21(4), 422–438.

    Article  Google Scholar 

  36. Rogers, T. (1991). Students as literary critics: The interpretive experiences, beliefs, and processes of ninth-grade students. Journal of Reading Behavior, 23(4), 391–423.

    Google Scholar 

  37. Roller, C. M. (1986). Overcoming underwriting and other textbook sins. Social Education, 50(1), 56–57.

    Google Scholar 

  38. Smith, R. S. W., Roehrig, A. D., & Guan, Q. (2006, April). Validation of a tool to assess the metacognitive awareness of reading strategies within adolescent and adult readers. Paper presented at the annual meeting of the American Educational Research Association, San Francisco, CA.

  39. Sperling, R. A., Howard, B. C., Miller, L. A., & Murphy, C. (2002). Measures of children’s knowledge and regulation of cognition. Contemporary Educational Psychology, 27, 51–79.

    Article  Google Scholar 

  40. Sperling, R. A., Howard, B. C., Staley, R., & DuBois, N. (2004). Metacognition and self-regulated learning constructs. Educational Research and Evaluation, 10(2), 117–139.

    Article  Google Scholar 

  41. Stamp Act. (2004). Britannica Student Encyclopedia. Downloaded from the World Wide Web from http://www.search.eb.com/.

  42. Veenman, M. V. J. (2005). The assessment of metacognitive skills: What can be learned from multi-method designs? In B. Moschner & C. Artelt (Eds.), Lernstrategien und Metakognition: Implikationen für Forschung und Praxis (pp. 75–97). Berlin: Waxmann.

    Google Scholar 

  43. Veenman, M. V. J., Elshout, J. J., & Groen, M. G. M. (1993). Thinking aloud: Does it affect regulatory processes in learning. Tijdschrift voor Onderwijsresearch, 18, 322–330.

    Google Scholar 

  44. Viola, H. J., Wheatley, H., & Hart, D. (1998). Why we remember. Menlo Park, CA: Addison-Wesley.

    Google Scholar 

  45. Wyatt, D., Pressley, M., El-Dinary, P. B., Stein, S., Evans, P., & Brown, R. (1993). Comprehension strategies, worth and credibility monitoring, and evaluations: Cold and hot cognition when experts read professional articles that are important to them. Learning and Individual Differences, 5(1), 49–72.

    Article  Google Scholar 

Download references

Acknowledgments

This research was supported by funding from an AERA/Spencer Pre-Dissertation Fellowship and a Spencer Dissertation Fellowship to the first author and by funding from the National Science Foundation (REC#0133346) awarded to the second author. Portions of this research were previously presented at the Annual Meeting of the American Educational Research Association, April 15, 2005, Montreal, Quebec, Canada. We thank Dr. Sheila Barron, Jeffrey Greene, Dr. Andrew Ho, Dr. Lorena Llosa, and Dr. Jean-Pierre Verhaege for comments that improved the quality of this paper.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Jennifer G. Cromley.

Appendices

Appendix A

Major topics, sub-topics, and supporting details used for scoring the free recalls

Major topics (4 points each) Sub-topics (2 points each) Supporting details (1 point each)
To fund army   French and Indian War
British
debt
need money
protect
colonists
10,000 soldiers
Indians
Sugar Act halved/dropped [tax] 1760s
duties/tax [on molasses]  
molasses rum
  West Indies
enforce writs of assistance
non-British goods
smuggling
bribing
British judges
guilty until proven innocent
No taxation without representation no representatives Parliament
   1,000s of miles away
Stamp Act buy tax stamps [not just special stamp] tax stamp
on paper legal documents
dice
playing cards
newspapers
affected more people  
disagreements in Parliament Barre
Townshend
children
fled
planted
Stamp Act Congress nine colonies
New York
unite
petition
repeal
Protests/dislike/rebel violent
windows
Sons of Violence
threaten/own hands
effigy
Oliver
burn [effigy or paper]
hang [effigy]
[CT] tax collector
started to bury
resigned
boycotts
women
cloth
English goods
made own
Sons and Daughters of Liberty

Appendix B

Sample items from the self-reported strategy use and multiple-choice strategy use measures

Self-reported Strategy Use (MARSI; Mokhtari & Reichard, 2002, p. 259)

1:

‘I never or almost never do this.’

2:

‘I do this only occasionally.’

3:

‘I sometimes do this.’ (about 50% of the time)

4:

‘I usually do this.’

5:

‘I always or almost always do this.’

“6. I summarize what I read to reflect on important information in the text 1 2 3 4 5 ”

Multiple-choice strategy use

A Stamp Act Congress, representing nine colonies, met in New York City on Oct. 7, 1765. The congress declared that only the colonial assemblies should tax the colonists. The congress also petitioned the king and Parliament for repeal of the objectionable measures. When the stamped papers began to arrive, mobs seized them or forced the ships’ captains to take them back to England.

  1. 15.

    Which of the following is the best summary of the passage?

    1. I.

      The Stamp Act Congress met in 1765

    2. J.

      Mobs seized stamped papers from English ships

    3. K.

      The Stamp Act Congress met and asked Parliament to repeal the Act

    4. L.

      Nine colonies met and petitioned the King

Many wealthy merchants favored stopping all business that required the use of stamped papers. This, they said, would be perfectly legal. They also argued that it would so seriously interfere with the business of British merchants that Parliament would be forced to repeal the law.

  1. 23.

    Which sentence would it make the most sense to underline or highlight?

    1. A.

      Many wealthy merchants favored stopping all business...

    2. B.

      This, they said, would be...

    3. C.

      They also argued that it would so seriously interfere...

    4. D.

      All of the above.

Appendix C

Strategy codes for the think-aloud measure (Adapted from Azevedo & Cromley, 2004)

Code Definition Example
Accurate
HYP+ Any hypothesis or prediction about events to follow in the text (could be an accurate, sensible, or even insensible hypothesis/prediction). “They might start, like, charging people for the soldiers.”
INF+ Makes an accurate within-text inference. The word “because” always signals an inference, but not all inferences include a causal word. “Because they had a lot of goods to sell.”
PKA+ Activates prior knowledge that is both accurate and relevant. “But didn’t Britain like force them to take it? Yeah.”
SQ+ Participant poses him/herself a question that might potentially be answered by the forthcoming text; not a rhetorical question; need not include a question word; not a misunderstanding of text that came before; not a statement of lack of understanding; not PKA-. “I’m thinking how big is the army then if they needed 10,000 soldiers just for North America.”
SUM+ Accurately summarizes (note that part of a summary can be accurate and part can be inaccurate—2 codes). “to make people pay for their own defenses.”
Inaccurate
INF- Makes an inaccurate within-text inference. “So they were doing illegal acts in order to raise money.”
PKA- Activates inaccurate and/or irrelevant prior knowledge or states that he/she lacks background knowledge. “That rum was like prohibited, like in those times.”
SUM- Inaccurately summarizes (note that part of a summary can be inaccurate and part can be accurate—2 codes). Could misunderstand or over-generalize in the summary; not PKA-. “They were selling their things.”

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Cromley, J.G., Azevedo, R. Self-report of reading comprehension strategies: What are we measuring?. Metacognition Learning 1, 229–247 (2006). https://doi.org/10.1007/s11409-006-9002-5

Download citation

Keywords

  • Measurement
  • Self-report
  • Reading
  • Comprehension
  • Strategies