Self-report of reading comprehension strategies: What are we measuring?

Abstract

Proficient readers engage in a wide range of cognitive and metacognitive strategies, and both developmental and classroom intervention researchers are in need of high-quality measures of strategy use. Several researchers have recently called into question the validity of the most common type of measures of strategy use in reading, self-report or introspective measures (i.e., the participant must report on his or her cognitive activity while not actually engaged in the activity). We administered three parallel strategy use measures to a sample of 30 ninth-grade students: a prospective self-report measure, a concurrent multiple-choice measure which required students to apply the strategies to specific passages, and a text on which we asked students to think aloud. We also collected two measures of reading comprehension—a standardized measure and free recall scores. Consistent with Veenman’s (2005) conclusions based on a literature review, the concurrent multiple-choice and think-aloud data were both significantly correlated with both of the comprehension scores and with each other, whereas the prospective self-report data had non-significant correlations with all of the other measures. We conclude by recommending concurrent measures for researchers who wish to study strategy use in reading comprehension.

Appendices

Appendix A

Major topics, sub-topics, and supporting details used for scoring the free recalls

Major topics (4 points each)	Sub-topics (2 points each)	Supporting details (1 point each)
To fund army		French and Indian War
		British
		debt
		need money
		protect
		colonists
		10,000 soldiers
		Indians
Sugar Act	halved/dropped [tax]	1760s
	duties/tax [on molasses]
	molasses	rum
		West Indies
	enforce	writs of assistance
		non-British goods
		smuggling
		bribing
		British judges
		guilty until proven innocent
No taxation without representation	no representatives	Parliament
		1,000s of miles away
Stamp Act	buy tax stamps [not just special stamp]	tax stamp
	on paper	legal documents
		dice
		playing cards
		newspapers
	affected more people
	disagreements in Parliament	Barre
		Townshend
		children
		fled
		planted
	Stamp Act Congress	nine colonies
		New York
		unite
		petition
		repeal
	Protests/dislike/rebel	violent
		windows
		Sons of Violence
		threaten/own hands
		effigy
		Oliver
		burn [effigy or paper]
		hang [effigy]
		[CT] tax collector
		started to bury
		resigned
		boycotts
		women
		cloth
		English goods
		made own
		Sons and Daughters of Liberty

Appendix B

Sample items from the self-reported strategy use and multiple-choice strategy use measures

Self-reported Strategy Use (MARSI; Mokhtari & Reichard, 2002, p. 259)

1:

‘I never or almost never do this.’

2:

‘I do this only occasionally.’

3:

‘I sometimes do this.’ (about 50% of the time)

4:

‘I usually do this.’

5:

‘I always or almost always do this.’

“6. I summarize what I read to reflect on important information in the text 1 2 3 4 5 ”

Multiple-choice strategy use

A Stamp Act Congress, representing nine colonies, met in New York City on Oct. 7, 1765. The congress declared that only the colonial assemblies should tax the colonists. The congress also petitioned the king and Parliament for repeal of the objectionable measures. When the stamped papers began to arrive, mobs seized them or forced the ships’ captains to take them back to England.

• 15. Which of the following is the best summary of the passage?

  • I. The Stamp Act Congress met in 1765

  • J. Mobs seized stamped papers from English ships

  • K. The Stamp Act Congress met and asked Parliament to repeal the Act

  • L. Nine colonies met and petitioned the King

Many wealthy merchants favored stopping all business that required the use of stamped papers. This, they said, would be perfectly legal. They also argued that it would so seriously interfere with the business of British merchants that Parliament would be forced to repeal the law.

• 23. Which sentence would it make the most sense to underline or highlight?

  • A. Many wealthy merchants favored stopping all business...

  • B. This, they said, would be...

  • C. They also argued that it would so seriously interfere...

  • D. All of the above.

Appendix C

Strategy codes for the think-aloud measure (Adapted from Azevedo & Cromley, 2004)

Code	Definition	Example
Accurate
HYP+	Any hypothesis or prediction about events to follow in the text (could be an accurate, sensible, or even insensible hypothesis/prediction).	“They might start, like, charging people for the soldiers.”
INF+	Makes an accurate within-text inference. The word “because” always signals an inference, but not all inferences include a causal word.	“Because they had a lot of goods to sell.”
PKA+	Activates prior knowledge that is both accurate and relevant.	“But didn’t Britain like force them to take it? Yeah.”
SQ+	Participant poses him/herself a question that might potentially be answered by the forthcoming text; not a rhetorical question; need not include a question word; not a misunderstanding of text that came before; not a statement of lack of understanding; not PKA-.	“I’m thinking how big is the army then if they needed 10,000 soldiers just for North America.”
SUM+	Accurately summarizes (note that part of a summary can be accurate and part can be inaccurate—2 codes).	“to make people pay for their own defenses.”
Inaccurate
INF-	Makes an inaccurate within-text inference.	“So they were doing illegal acts in order to raise money.”
PKA-	Activates inaccurate and/or irrelevant prior knowledge or states that he/she lacks background knowledge.	“That rum was like prohibited, like in those times.”
SUM-	Inaccurately summarizes (note that part of a summary can be inaccurate and part can be accurate—2 codes). Could misunderstand or over-generalize in the summary; not PKA-.	“They were selling their things.”