Multiple-choice pretesting potentiates learning of related information
Although the testing effect has received a substantial amount of empirical attention, such research has largely focused on the effects of tests given after study. The present research examines the effect of using tests prior to study (i.e., as pretests), focusing particularly on how pretesting influences the subsequent learning of information that is not itself pretested but that is related to the pretested information. In Experiment 1, we found that multiple-choice pretesting was better for the learning of such related information than was cued-recall pretesting or a pre-fact-study control condition. In Experiment 2, we found that the increased learning of non-pretested related information following multiple-choice testing could not be attributed to increased time allocated to that information during subsequent study. Last, in Experiment 3, we showed that the benefits of multiple-choice pretesting over cued-recall pretesting for the learning of related information persist over 48 hours, thus demonstrating the promise of multiple-choice pretesting to potentiate learning in educational contexts. A possible explanation for the observed benefits of multiple-choice pretesting for enhancing the effectiveness with which related nontested information is learned during subsequent study is discussed.
KeywordsPretesting Testing effects Learning Multiple choice Test-potentiated learning
That pretesting can potentiate the learning of pretested information during subsequent study has been demonstrated for a variety of materials and across a variety of methodologies (e.g., Arnold & McDermott, 2013; Kornell, Hays, & Bjork, 2009). This finding has clear applications for learning in educational contexts (Anderson & Biddle, 1975; Kane & Anderson, 1978; Little & Bjork, 2011; Pressley, Tanenbaum, McDaniel, & Wood, 1990; Richland, Kornell, & Kao; 2009; Rothkopf, 1966). Relatively little work, however, has examined whether such pretesting can also potentiate the learning of information that was not itself pretested, or instead, whether the learning of such information might be impaired by such pretesting. This question is particularly pertinent to conditions in which such non-pretested material is related to, and thereby potentially confusable or competitive with, the specifically pretested information.
The issue raised previously is important because students are often presented with large amounts of related and thus potentially confusable information (e.g., anatomy, geography, history courses), and they are sometimes pretested on subsets of that information (e.g., with “clicker” questions presented at the beginning of a lecture), followed by more comprehensive exams that would actually count toward their grades and that would include the testing of non-pretested related information as well. Of particular interest in the present research was whether, and if so how, pretesting might improve the learning of such related, and thus possibly competitive, non-pretested information. The present research examines such possible effects for multiple-choice pretests in comparison to cued-recall pretests. We see a comparison of these two types of testing formats to be particularly critical, given the already extensive use of multiple-choice testing in large classrooms and its increasing use as a tool for learning with classroom response systems (e.g., “clickers”; Buhay, Best, & McGuire, 2010; Lantz & Stawiski, 2014; Mayer et al., 2009; Smith et al., 2009), with such tests often given before instruction (Glass, Brill, & Ingate, 2008).
Previously, Little and Bjork (2011) demonstrated that answering multiple-choice questions prior to study can improve one’s ability to learn such competitive non-pretested related information compared to no-pretest control conditions—when answers to related questions had appeared as incorrect alternatives on the pretest questions. In their Experiment 3, participants read passages (e.g., about Yellowstone National Park and the planet Saturn), with one preceded by multiple-choice questions (e.g., What geyser is thought to be the oldest in the world? a. Old Faithful, b. Steamboat Geyser, c. Castle Geyser, d. Daisy Geyser) and one preceded by the study of facts containing all of the information in the questions, including the incorrect alternatives (a fact-study control condition; e.g., Castle Geyser is thought to be the oldest geyser in the world, not Old Faithful, Steamboat Geyser, or Daisy Geyser). When later asked the name of the tallest (rather than the oldest) geyser, participants were better able to access the correct answer, Steamboat Geyser, when they had answered a multiple-choice pretest question for which Castle Geyser was the correct answer than when they had studied the comparable fact beforehand, demonstrating that the benefit depends upon more than simply preexposure to the correct answers during a prestudy activity. How multiple-choice pretest questions induce such enhanced learning of related information and whether their benefits would surpass those provided by cued-recall pretest questions are issues that we address in the present research.
Cued-recall tests are generally heralded for their ability to serve as effective learning events (e.g., Foos & Fisher, 1988), and indeed, most extant research has shown cued-recall tests to be more effective than multiple-choice tests for improving later retention of the tested material—both when tests serve as posttests and, critically for the present purposes, when they serve as pretests (e.g., Anderson & Biddle, 1975; Hamaker, 1986). In addition, smaller benefits for the retention of information related to tested information have also been shown, but there are some notable exceptions with regard to cued-recall testing (e.g., retrieval-induced forgetting observed with educational materials; Little, Storm, & Bjork, 2011) and, furthermore, the types of relationships that have been examined are limited. In Hamaker’s review, for example, in which he reported the finding of greater benefits from cued-recall testing versus multiple-choice testing on retention of related information, such “related” items were almost exclusively of a noncompetitive nature—like rewording of questions from initial test to later test and definition questions on an initial test with concept questions on a later test. The type of competitive relationship between tested and nontested information explored in the present research—namely, the situation in which an answer to one question might compete with the answer to a related question—has yet to be systematically examined with different pretest formats (but see Little, Bjork, Bjork, & Angello, 2012, for an examination with posttest formats).
The relationship between pretested and non-pretested information and the format of the pretest questions with respect to this relationship is important because pretesting may not always be expected to potentiate learning of related information. To the extent, for example, that pretesting focuses attention toward the subsequent encoding of the specific pretested information, it might draw attention away from the encoding of other information that does not remind the learner of a pretest question (Frase, 1967; Hamaker, 1986; Reynolds & Anderson, 1982; Reynolds, Standiford, & Anderson, 1979), and we postulate that such might be the case when pretested and non-pretested information is related in a competitive manner and the relationship is not established and/or brought to mind in the pretest question. By contrast, the multiple-choice format, we would argue, can serve to make the relationship between pretested and competitive non-pretested information explicit. When, for example, answers to competitive non-pretested questions serve as incorrect alternatives on a multiple-choice pretest question, learners may be alerted to the need to learn (or at least primed to learn) information pertaining to the question and all the alternatives. That is, in contrast to a cued-recall pretest question (on which competitive information is not presented), a multiple-choice pretest question (on which the answers to potential competitive questions are presented as incorrect alternatives) might direct attention more broadly—toward the processing, not only of the information explicitly addressed by the pretest question but also to the processing of information having to do with the incorrect alternatives, with a consequent increase in performance on questions about that competitive information should they appear on a later test. Following from the example presented previously (Little & Bjork, 2011), a multiple-choice question in which four different geysers serve as alternatives may increase the likelihood that learners attend to information about all of those geysers when later encountered, whereas a cued-recall question may direct attention just to the geyser that was addressed by the question (e.g., the oldest geyser).
Furthermore, a commonsensical hypothesis ensuing from this attention-direction notion would be that any improvements in performance would be associated with increased time spent attending to that information (both for pretested information and for information pertaining to the incorrect competitors), but this is not a foregone conclusion. First, with respect to the explicitly pretested information, learners may not spend more time on such information when it is encountered during subsequent study, especially if feedback had been provided on the pretest. Learners will have had practice with such information—perhaps making its subsequent encoding seem more fluent—and thus may not feel it necessary to allocate additional time to it (Little & McDaniel, 2015). But what should the assumption be regarding non-pretested information? Unless by chance it is something already known to the participant, such information should not be learned or encoded during the taking of the pretest but would have to be acquired through subsequent reading of the passage. On the one hand, it seems reasonable to assume that any performance benefits occurring for such information would be associated with increased time spent processing that information when encountered during subsequent reading of the passage. On the other hand, and more interestingly, perhaps the nature of the multiple-choice pretesting format invokes a type of processing during the attempt to select an answer such that the pertinent nontested information can then be learned more efficiently when encountered during subsequent study. We explored these possibilities in the present Experiment 2.
The present research
Our overarching goal in the present research was to compare the benefits afforded by multiple-choice pretests to those afforded by cued-recall pretests and to assess the extent to which any observed benefits might be afforded by allocation of time during study. Additionally, we hoped to replicate the finding of Little and Bjork (2011) that showed multiple-choice pretesting to be better than a pre-fact-study control condition for the learning of related information and to assess the possible contribution of study time allocation to this difference. Toward this goal, we conducted three experiments, and in all of these experiments we defined related information in terms of its competitiveness to the pretested material. In Experiment 1, we compared effects of multiple-choice pretesting to cued-recall pretesting (as well as pre-fact-study activities, serving as no-pretest control conditions, similar to Little & Bjork, 2011; see also, Richland et al., 2009). In Experiment 2, we assessed how learners allocate their study time during reading of a subsequent passage following pretests (multiple choice vs. cued recall) or the study of facts. In Experiment 3, we examined the effects of multiple-choice pretesting as compared to cued-recall pretesting over both a 5-min and 2-day delay. Additionally, Experiments 1 and 2 employed pretests with feedback, whereas Experiment 3 employed pretests without feedback. Feedback was necessary in Experiments 1 and 2 owing to our goal to directly compare effects of pretesting to fact study prior to study of the to-be-learned passages.
Participants and design
This design allowed us to compare (a) multiple-choice pretesting to cued-recall pretesting with a between-subjects manipulation and (b) multiple-choice pretesting to fact study (which controlled for exposure to the alternatives) with a within-subjects manipulation. These were our main comparisons.
The passages were about the planet Saturn, Yellowstone National Park, and stimulant drugs, each containing about 700 words and with 10 pairs of facts for each passage. The two facts in each pair were semantically related in that they could be used to construct pairs of multiple-choice questions that tested the same topic (e.g., geysers) and had the same four alternatives (e.g., Old Faithful, Steamboat Geyser, Castle Geyser, and Daisy Geyser) but different correct answers (e.g., What is the tallest geyser in Yellowstone National Park? Answer: Steamboat Geyser; What geyser is thought to be the oldest in the world? Answer: Castle Geyser). The Yellowstone passage and multiple-choice questions are shown in the Appendix.
When a passage was to be given a multiple-choice pretest, one question from each pair appeared on the pretest, with the remaining question from each pair then appearing as a related non-pretested question on the delayed final cued-recall test, with assignment of questions to the pretest or as related questions on the final cued-recall test counterbalanced across participants. When a passage was to be given a cued-recall pretest, the same procedure for creating the pretests and the delayed final cued-recall test was used, but the questions presented on the pretest appeared without the alternatives. When a passage was to be preceded by the study of facts with competitors, the questions were changed to statements that included the items that would have been presented as incorrect alternatives for the corresponding multiple-choice question (e.g., Castle Geyser is thought to be the oldest geyser in the world, not Old Faithful, Steamboat Geyser, or Daisy Geyser). Facts without competitors were the same except for the absence of the alternatives (e.g., Castle Geyser is thought to be the oldest geyser in the world).
All participants read three passages, with two of the passages immediately preceded by a pre-reading activity: either a pretest or a fact-study activity, as shown in Fig. 1. The pre-reading activities for the random half assigned to the multiple-choice condition were a 10-question multiple-choice pretest and the study of 10 facts, including competitors. The pre-reading activities for the half assigned to the cued-recall condition were a 10-question cued-recall pretest and the study of 10 facts without competitors.
For the passage that was pretested, participants were given 20 s to answer each question. After typing their answer, they continued to view the question until the 20 s elapsed. After each question, participants received feedback (i.e., correct answer presented below the question) for 4 s. Immediately after the pretest, participants were given 6 min to read the pretested passage. For the passage preceded by fact study, participants were presented with 10 facts for 24 s each and told to spend the full 24 s thinking about the fact. Immediately thereafter, participants were given 6 min to read the passage. For the passage not preceded by a pretest or fact-study activity, participants received 10 min to read the passage, with the extra 4 min compensating for the time taken for the pretest or fact-study activities, and thus controlling for time on task. For each passage, participants were told that if they finished reading early, they should spend the remainder of the time studying, like they would for a class.
Finally, after a 5-min retention interval during which they played Tetris, participants received a 60-item final cued-recall test (20 items each for the pretest, fact-study, and extended-reading conditions), with questions presented one at a time on the computer screen. Of the 20 final-test questions about the pretested passage, half were identical to the 10 questions on the initial pretest (except now appearing as cued-recall questions in the multiple-choice pretest condition), while the other half consisted of the corresponding related questions for that passage that had not appeared on the pretest. Of the 20 final-test questions about the passage preceded by fact study, 10 items tested information central to each fact and 10 items tested information related to each fact. (That is, questions for a given passage were the same on the final cued-recall test whether that passage had appeared in one of the pretest conditions or the pre-fact-study condition.) Of the 20 final-test questions about the extended-reading passage, all came from the set of 10-paired questions constructed for that passage, and none had been previously exposed.
Because we considered assessing possible effects on the retention of non-pretested related information to be our most important research goal, non-pretested questions from the pretest conditions and questions pertaining to the competitive alternatives presented in the fact-study conditions were included in the first half of the final test along with the comparison questions from the extended-reading condition. This ordering would reduce output interference from the stronger pretested or prestudied information, whose corresponding questions were tested in the second half of the final test, along with the corresponding extended-reading questions (see Roediger & Schmidt, 1980; Smith, 1971). The order in which passages were presented, the condition (pretest, fact study, or extended reading) to which each passage was assigned, and which 10-item question set and 10-item fact set were presented during the pretest or fact-study conditions were counterbalanced across participants.
Results and discussion
Overall, participants chose the correct answer to 28% (SD = 6%) of the multiple-choice questions on the multiple-choice pretests, which was not significantly different from chance (25%), t(23) = 1.03, p = .31, and they produced the correct answer to 7% (SD = 10%) of the cued-recall questions on the pretest.
Final-test performance for related items
A 2 (pre-reading activity: multiple-choice/facts with alternatives, cued-recall/facts without alternatives) × 3 (item type: pretest related, fact-study related, extended reading) ANOVA examining performance for related information revealed an interaction between pre-reading activity and item type, F(2, 45) = 6.27, p < .01, ηp2 = .22, as well as a main effect of item type, F(2, 45) = 8.79, p < .001, ηp2 = .28. With respect to our main question of interest as to whether multiple-choice pretesting would improve learning of related information more than would cued-recall pretesting, a planned independent-samples t test confirmed that items related to multiple-choice pretest questions (M = 50%, SE = 4%) were recalled at a higher rate than were items related to cued-recall pretest questions (M = 37%, SE = 4%), t(46) = 2.29, p =.03, d = 0.66. Additionally, with respect to our second main question of interest as to whether our findings would replicate those of Little and Bjork (2011), showing that multiple-choice pretests improve learning as compared to a control condition in which statements with alternatives were presented for prestudy, a planned pairwise t test also confirmed that higher performance was obtained in the multiple-choice pretest condition than in the comparable fact-study condition (M = 36%, SE = 4%), t(23) = 3.40, p < .01, d = 0.69. As in that previous study, this result demonstrates that simple exposure of alternatives—including the correct answer to what will be a related question on the final test—is not what leads to the increased learning of related information following competitive multiple-choice pretests. Finally, performance was marginally higher for items related to multiple-choice questions than for questions from the extended-reading condition (M = 42%, SE = 4%), t(23) = 1.99, p = .06.
In contrast, participants’ performance on questions related to cued-recall pretest questions did not differ from their performance on questions related to facts (M = 33%, SE = 4%), t(23) = 1.31, p = .20, and was worse than performance for corresponding items in the extended-reading condition (M = 50%, SE = 5%), t(23) = 3.05, p < .001.
Final-test performance for pretested items and previously studied facts
In assessing these effects with planned pairwise t tests, we found that—in contrast to the general assumption of cued-recall testing being more beneficial than multiple-choice testing for retention—correct performance for items pretested as cued-recall questions (M = 75%, SE = 3%) did not significantly differ from that for items pretested as multiple-choice questions (M = 73%, SE = 5%), t(46) = 0.30, p = .76, but correct performance on either was better than performance on corresponding items in the extended-reading condition (M = 43%, SE = 4%), t(23) = 5.74, p < .01, and M = 42%, SE = 4%, t(23) = 6.92, p < .01, for multiple-choice and cued-recall conditions, respectively. Overall, pretesting (M = 74%, SE = 3%) improved performance more than fact study (M = 67%, SE = 3%), t(47) = 2.05, p < .05.
The overall pattern of results obtained in Experiment 1 suggests that cued-recall and multiple-choice pretests differentially affect later recall of competitive related information. Specifically, as indicated by performance on related items in the two types of pretest conditions, competitive related information appears to be learned less well as a consequence of taking a cued-recall pretest than as a consequence of taking a multiple-choice pretest with competitive incorrect alternatives, even when the final test utilizes a cued-recall format. This result suggests that being exposed to incorrect alternatives associated with related information in the context of a multiple-choice question induces processing that improves later learning of that related information beyond that provided by a cued-recall question. It is not exposure of the alternatives alone, however, that induces deep processing leading to improved performance: Studying facts with the alternatives provided does not provide the same benefit, a finding first shown by Little and Bjork (2011) and replicated in the present experiment.
What processing differences during subsequent reading of passages preceded by a multiple-choice pretest versus a cued-recall pretest or the study of facts might produce these differences in final-exam performance? One possibility is that learners spend more time studying such competitive information following a multiple-choice pretest than following a cued-recall pretest or the study of facts. A more interesting possibility, however, is that the enhanced performance for these items on the later test occurs—not because learners spend more time on that critical information but rather because they are able to process such information more efficiently. Perhaps these competitive incorrect alternatives receive deep processing during pretesting, which then facilitates the learning of information pertaining to them when it is encountered during subsequent reading of the passage.
To gain some leverage on this issue, we examined how long learners spent reading sentences containing information pertaining to the pretested (or prestudied) information and information related to that pretested (or prestudied) information in Experiment 2. The basic design employed was the same as that of Experiment 1 except that reading time was self-paced sentence by sentence. Accordingly, while we expected to replicate the basic patterns of performance observed in the pretesting and fact-study conditions of Experiment 1, we thought the performance in the reading-only condition of Experiment 2 might be lower than that of the reading-only (extended-reading) condition of Experiment 1 because we did not expect participants to spend four additional minutes reading this passage—although they were free to do so.
Seventy individuals in the Washington University in St. Louis community participated for either pay or partial credit in a psychology course.
Design, materials, and procedure
The materials were the same as those used in Experiment 1, and the design and procedure were the same except that participants read the passages, sentence by sentence, at their own pace. Reading times for each sentence—the main data of interest in this experiment—were recorded.
Results and discussion
Overall, participants chose the correct answer to 31% (SD = 14%) of the multiple-choice questions on the multiple-choice pretests, which was significantly different from chance (25%), t(35) = 2.77, p = .01, and they produced the correct answer to 13% (SD = 13%) of the cued-recall questions on the pretest.
Final-test performance for related items
As we expected, given the self-paced presentation procedure used in Experiment 2, participants spent less time on the passage in the reading-only condition in Experiment 2 (M = 4.8 min, SD = 1.6) than what was allocated toward extended reading in Experiment 1. Thus, as we expected, performance in the reading-only condition in the present experiment was numerically lower than that in the extended-reading condition in Experiment 1. With this exception, however, the basic pattern of results—that is, for pretested and pre-fact-study passages—was consistent across Experiments 1 and 2.
Performance on the final cued-recall test following cued-recall pretesting (M = 41%, SE = 3%) was not better than performance following the comparable fact study (M = 44%, SE = 3%), t(34) = 0.79, p = .44, or following reading only (M = 41%, SE = 3%), t(34) = 0.08, p = .94.
Final-test performance for pretested items and previously studied facts
Correct performance for items pretested as cued-recall questions (M = 77%, SE = 3%) was marginally greater than that for items pretested as multiple-choice questions (M = 69%, SE = 3%), t(68) = 1.91, p = .06. The cued-recall pretest provided a performance advantage compared to studying facts (M = 62%, SE = 3%), t(34) = 4.17, p < .001, d = 0.70, but a multiple-choice pretest did not (M = 66%, SE = 4%), t(34) = 0.65, p = 0.44. Performance for both types of questions, however, was reliably better than for the corresponding questions in the reading-only condition (M = 37%, SE = 3%), t(34) = 7.48, p < .001, d = 1.26, in the multiple-choice condition; (M = 41%, SE = 3%), t(34) = 9.33, p < .001, d = 1.58, in the cued-recall condition.
Given the overall replication of the performance patterns observed in Experiment 1, we next analyzed the results directly pertinent to our main question of interest in Experiment 2: reading-time allocation to the critical sentences (i.e., those containing information pertaining to the pretested or prestudied information or containing information related to the pretested or prestudied information). Of key interest was whether individuals allocated more or less time to these critical sentences when a passage had been pretested or preceded by fact study versus when it had not. As previously discussed, the straightforward prediction would be that the better performance shown for related information in the multiple-choice pretest condition as compared to the cued-recall pretest condition or the comparable fact-study condition would be associated with increased time allocation; a more interesting possibility, however, was that better performance would not be driven by increased time allocation but by more effective or efficient processing of related information in the multiple-choice pretest condition.
A 2 (pre-reading activity: multiple-choice/facts with alternatives, cued-recall/facts without alternatives) × 3 (item type: pretest related, fact-study related, reading only) ANOVA indicated a marginal interaction between pre-reading activity and item type on reading time allocated to related items, F(2, 67) = 2.74, p = .07, ηp2 = .08. Planned paired-samples t tests showed that following a multiple-choice pretest, individuals spent less time (in ms) reading sentences containing information pertaining to the non-pretested related question (M = 5,858, SE = 241) than they spent reading comparable sentences from the reading-only passage (M = 6,539, SE = 385), t(34) = 2.20, p = .04, d = 0.41, and comparable sentences from the fact-study passage (M = 6,613, SE = 405), t(34) = 2.18, p = .04, d = 0.40. Following a cued-recall pretest, however, time to read sentences containing information pertaining to the non-pretested related question (M = 6,309, SE = 380) did not differ from time spent reading comparable sentences from the reading-only passage (M = 6,426, SE = 411), t(34) = 0.40, p = .69, or from the fact-study passage (M = 6,031, SE = 315), t(34) = 1.01, p = .32. The overall difference between the time spent in the multiple-choice condition and the cued-recall condition was not reliable, however, t(68) = 1.00, p = .32. Although the interaction of pre-reading activity and item type on reading times did not reach statistical significance, and a reliable difference in overall reading times did not emerge between the two pretest conditions, the intriguing result is that the highest performance on the final test (i.e., that related to multiple-choice pretesting) was associated with the least numerical amount of time spent in study.
Pretested and prestudied items
A 2 (pre-reading activity: multiple-choice/facts with alternatives, cued-recall/facts without alternatives) × 3 (item type: pretested, fact-study control, reading only) ANOVA did not indicate an interaction between pre-reading activity and item type on reading time allocated to pretested or prestudied items, F(2, 67) = 0.04, p = .96. A main effect of item type, however, did emerge, F(2, 67) = 10.57, p < .001, ηp2 = .24. Paired-samples t tests confirmed that individuals spent less time reading sentences containing information that had been previously tested than reading comparable sentences from a passage that had not been preceded by a pre-reading activity of any type in both the multiple-choice condition (M = 5 400, SE = 208 vs. M = 6,398, SE = 387), t(34) = 2.80, p < .01, d = 0.47, and the cued-recall condition (M = 5,367, SE = 280 vs. M = 6,335, SE = 370), t(34) = 3.27, p < .01, d = 0.55. They also spent significantly less time, compared to the reading-only condition, on sentences with information pertaining to facts with competitors (M = 5,478, SE = 306 vs. M = 6,398, SE = 387), t(34) = 3.37, p < .01, d = 0.57, and on facts without competitors (M = 5,423, SE = 272 vs. M = 6,335, SE = 370), t(34) = 2.50, p = .02, d = 0.42. Preexposure in any form reduced the time participants spent during subsequent study on that directly preexposed information.
The goal of Experiment 2 was to gain further insight into how pretesting might direct subsequent study, especially for related information, and whether time spent studying would relate to later test performance. One clear possibility was that testing would direct learners’ attention toward tested material: Learners would spend more time processing this material, thus leading to its enhanced retrieval on a later exam. A more interesting possibility was that testing would enhance later recall performance without learners having to spend more time on this information during subsequent study. Experiment 2 provides evidence consistent with this second possibility, and its results are especially intriguing in terms of retention of related information. Although participants could have learned the specific pretested information before study—that is, while taking the pretest—their learning of any specific information associated with incorrect alternatives would seem unlikely. They could learn, however, that such information may be important to attend to later, and the results suggest that they then learned such information more efficiently. That is, following a multiple-choice pretest, learners spent less time on related sentences (compared to that spent on corresponding sentences in the reading-only or pre-fact-study conditions), but (based on their later test performance) they learned such information better, at least compared to the fact-study control condition. Although following a multiple-choice pretest, learners did not spend significantly less time on sentences containing related information than they did following a cued-recall pretest, the efficacy of learning that information was clearly better in the case of the multiple-choice pretest than in the case of the cued-recall pretest, as indicated by the significantly better performance for such items on the final cued-recall test.
How is related information being processed more efficiently?
On the one hand, it seems reasonable to assume that choosing incorrect answers would lead to interference and potentially the persistence of incorrect associations (see, e.g., research on the effect of choosing an incorrect photograph on later eyewitness identification; Gorenstein & Ellsworth, 1980). On the other hand, it also seems reasonable to assume that because the pretest is followed by learning (and not preceded by it), participants are likely to seek out corrections to their errors (knowing that they have likely made many), and thus incorrect alternatives are likely to garner more attention and processing during subsequent passage reading when they have been previously presented as part of a multiple-choice pretest question than as part of a prestudied fact. Furthermore, it also seems reasonable to assume that an incorrect alternative and its associated information would be processed more deeply during subsequent reading when it had been previously selected as the answer to the pretest question than when it had not been. The reasoning for this conjecture is that when learners do not already have sufficient knowledge to select the correct answer, they may turn to some other basis for selecting an answer that may involve deep processing. Perhaps, for example, they select the one that seems most pleasant to them or the one that is most relevant to their own personal lives—two types of processing that have been shown to improve subsequent recall of information even when subjects are not expecting a later test (see Craik & Tulving, 1975; Packman & Battig, 1978). Processing that leads to enhanced encoding of a given alternative may then lead to more efficient and effective learning of information pertaining to that alternative when encountered later in the subsequently studied passage.
Percentage of times chosen on pretesta
Final test correct performance | chosen on pretest
Final test correct performance | not chosen on pretestb
3 (5 min)
3 (48 hr)
One might also wonder how the selection of an incorrect alternative influenced the time spent studying information pertaining to that alternative during subsequent reading. Thus, for Experiment 2, we also analyzed the reading times associated with correct performance, as a function of whether participants chose the incorrect alternative on the initial test that would later become the answer to the related question on the final test versus when they chose another alternative on the initial test. When participants incorrectly chose an alternative that would later become the answer to the related question and then answered that related question correctly, the average reading time (M = 4,376, SE = 620) was shorter than when participants chose any other alternative and then answered the question correctly (M = 5,865, SE = 310), t(34) = 2.34, p = .03. This finding is consistent with the idea that selection of an answer made it easier for participants to learn the information related to that choice when it was later encountered in their reading of the passage; when another answer had been selected, participants could still answer the related question correctly, but their learning of the information necessary to do so was associated with more time to process it when encountered in their subsequent reading of the passage. The results thus far are consistent with the notion that answering multiple-choice questions leads to processing of competitive incorrect alternatives that is deeper than that occurring when such alternatives are provided within facts. Furthermore, this benefit for the learning of related information is especially clear for the item that is chosen, both in terms of the level of performance and the efficiency of learning. Neither cued-recall pretesting nor exposure to the alternatives in facts directs reading in such an effective manner.
The results of Experiments 1 and 2 demonstrate a potentially unique benefit of using multiple-choice pretests to promote learning in educational contexts. Namely, not only are multiple-choice pretests about as beneficial as cued-recall pretests or studying comparable facts for the retention of the pretested information, they are actually more effective than these other activities for the learning of competitive related information. This latter advantage of multiple-choice pretests would, in particular, seem to be a valuable asset for their fruitful application to educational settings. After all, it seems likely that most instructors would typically want to ask similar, but not identical, questions on a later exam to assess whether students learned concepts more generally rather than simply being able to remember specific answers to previously tested questions.
In educational contexts, however, delays between learning activities and a final test of the to-be-learned information are often longer than those we used in the present Experiments 1 and 2. Thus, in Experiment 3, we used a modified procedure to assess whether benefits of pretesting would persist across longer delays and, in particular, whether the ability of multiple-choice pretesting to foster retention of nontested related information would persist, as compared to cued-recall pretesting. If the performance benefits observed for previously tested and related information in Experiments 1 and 2 resulted as the consequence of deep processing, then we should expect such benefits to persist over time.
In the modified procedure of Experiment 3, participants were first given pretests for two different passages, with one being a multiple-choice pretest and the other being a cued-recall pretest. Then, participants read three passages; two of the passages had received a pretest and the third had not received a pretest, thus serving as a reading-only baseline. A final cued-recall test was then administered for information presented in each of the three passages at either a short (5 min) or long (48 hrs) delay to assess the persistence of pretesting effects on retention for both pretested and non-pretested related information.
Participants and design
Seventy-two individuals in the Washington University in St. Louis community participated for either pay or partial credit in a psychology course and were all recruited for a two-session study. All participants took two pretests (one multiple choice and one cued recall), followed by the reading of three passages. On the final cued-recall test, participants answered pretested questions and related non-pretested questions from the two pretested passages, and nontested questions from the passage not receiving an initial pretest. We randomly manipulated whether participants were given the final test after a 5-min or 48-hour delay.
Materials and procedure
The materials were the same as those used in Experiments 1 and 2, and the procedure was the same except for the following differences. First, participants took two pretests before reading any passages: one consisting of multiple-choice questions for one of the three to-be-presented passages and the other consisting of cued-recall questions for another one of the three to-be-presented passages. Importantly, and unlike the procedures of Experiments 1 and 2 (but like the procedure used by Little & Bjork, 2011, in their Experiment 1), the pretests did not provide feedback. After administration of the two pretests, all three passages were read in immediate succession and for 6 min each. Which passages were pretested or not pretested, the type of pretest received, and the order of the two pretests were counterbalanced across participants. After a 5-min distractor task, participants randomly assigned to the 5-min delay condition were given a final cued-recall test for both pretested and non-pretested related information. Those participants randomly assigned to the 24-hour delay condition were dismissed after the 5-min distractor task and asked to return in 2 days.
As in the present Experiments 1 and 2, non-pretested related questions and the corresponding questions from the reading-only passage appeared in the first half of the final cued-recall test, whereas pretested questions and their corresponding questions from the reading-only passage appeared in the second half of the final cued-recall test.
Results and discussion
For the multiple-choice pretest, participants chose the correct answer to 34% (SD = 15%) and 33% (SD = 15%) of the questions in the 5-min delay condition and the 48-hour delay condition, respectively, with this level of performance being greater than chance performance (25%), t(71) = 4.81, p < .01. For the cued-recall pretest, participants provided the correct answer to 10% (SD = 9%) and 9% (SD = 11%) of the questions in the 5-min delay condition and the 48-hour delay condition, respectively.
Final-test performance for related items
Final-test performance for pretested items
A 2 (delay: short, long) × 3 (pre-reading activity: multiple-choice pretest, cued-recall pretest, no pretest/reading only) ANOVA did not indicate an interaction between delay and pre-reading activity on retention of pretested information, F(2, 69) = 1.36, p = 0.26, although there were main effects of both pre-reading activity, F(2, 69) = 44.20, p < .001, ηp2 = .56, and delay, F(1, 70) = 31.11, p < .001, ηp2= .31. Indeed, as confirmed by planned-comparison tests, overall final-test performance for information pretested with either a multiple-choice test (M = 57%, SE = 3%) or a cued-recall test (M = 52%, SE = 2%) was better than that for corresponding information from the reading-only passage (M = 34%, SE = 2%), t(71) = 8.49, p < .01, d = 1.00, and t(71) = 7.94, p < .01, d = 0.94, respectively.
The pattern of results observed in Experiment 3 demonstrates that a pretesting benefit for the retention of pretested information (as compared to non-pretested information) can persist over a delay that would be considered more educationally realistic (i.e., 48 hrs) and, additionally, in the case of a multiple-choice pretest, that a pretesting benefit for the retention of related information can also persist over such a delay. In evaluating this outcome, it is important to note that although we presented participants with the same textual materials as those used in Experiments 1 and 2, we introduced changes to the design and procedure of Experiment 3 that could have been expected to impact the previously observed benefits of pretesting. First, in Experiment 3, all participants received both a multiple-choice pretest and a cued-recall test, which they took before presentation of any of the to-be-studied passages. Thus, whereas in Experiments 1 and 2 a given passage was always presented for study immediately after participants had taken the relevant pretest, a given passage was unlikely to be presented for study immediately after participants had taken the relevant pretest in Experiment 3. In fact, for some participants, as much as 20 minutes of filled activity (e.g., taking another pretest, reading other passages) may have occurred between their taking of a given pretest and their opportunity to read the relevant passage—a procedural difference that would seem likely to diminish the participants’ ability to direct their attention to previously tested or related information during the reading of the relevant passage. This procedure also served to directly compare effects of cued-recall pretests to multiple-choice pretests in the same participants and without specific fact-study conditions being tied to those pretest conditions. Second, in Experiment 3, we did not provide feedback during the pretests, which also might be expected to produce a smaller pretesting benefit for tested information. Despite these changes, however, a pretesting benefit emerged in Experiment 3, which we believe underscores the strength and persistence of such effects.
Another procedural change made in Experiment 3 that we believe to be important, as it speaks to the generality of the pretesting effect, was our change from using an extended-reading condition (i.e., in Experiment 1, the time allotted for studying the reading-only passage was the sum of the time allowed for the study of a pretested passage plus the time taken for administrating its pretest) to giving all passages the same amount of time for reading. Our use of an extended-reading condition in Experiment 1 models the situation in which students might elect to spend a given amount of time for their study activities (or instructors have allowed only a fixed amount of classroom time for such activities), and, thus, taking a pretest before reading a passage would take away from the time they have available for reading/studying the passage. Students, however, may not always follow such total-time study routines. They might, for example, sometimes elect to engage in extra study activities, such as taking a pretest, in addition to the total time they spend studying or reading a passage or chapter.
Increasingly in classrooms, instructors are using pretests given with the help of classroom response systems, such as clickers, with the goal of increasing attention and learning in their courses (Buhay et al., 2010; Glass et al., 2008; Lantz & Stawiski, 2014; Mayer et al., 2009; Smith et al., 2009). Our findings suggest that using multiple-choice questions for such purposes would make the subsequent learning of related competitive information more effective than would the use of cued-recall questions (Experiments 1–3) or even studying facts beforehand (Experiments 1 and 2), and that this benefit of multiple-choice pretesting over cued-recall pretesting persists over a delay of 48 hours and even when feedback is not provided on the pretest (Experiment 3).1 Furthermore, our findings from Experiment 2 provide evidence that the enhanced recall of related information observed on a later test is not associated with increased time devoted to the study of that information, suggesting that, following a multiple-choice pretest, information pertaining to the incorrect alternatives on a previously given multiple-choice test is learned more efficiently. Finally, it should be noted that this benefit of a multiple-choice pretest occurs even though the format of the later test is cued recall.
How does multiple-choice pretesting (compared to cued-recall pretesting or studying facts) benefit the retention of related information?
As discussed in the Results and Discussion section for Experiment 2, the present results are consistent with the notion that multiple-choice pretesting can direct attention more broadly during subsequent study than can cued-recall pretesting—both toward information pertaining to the question at hand and toward information pertaining to the incorrect alternatives. It, of course, may not be the case that all multiple-choice questions presented in pretests would serve as effective learning tools in this way. Rather, they may need to possess certain qualities in order to do so. The alternatives, for example, may need to be competitive. This quality has been shown to be necessary for the learning of related, nontested information to be enhanced in prior research by Little and Bjork (2015; see also, Little et al., 2012) when multiple-choice tests are given following study rather than prior to study. The mechanism proposed for the enhanced retention of related, nontested information observed by these investigators, however, does not readily apply to the present situation of giving multiple-choice pretests, as it involves the assumption that test takers think back to the previously studied passage in search of information related to the alternatives to help them in their selection of the correct answer. By so doing, information about the competitive incorrect alternatives can be retrieved and thereby strengthened in addition to information related to the specifically correct answer. Then, if such nontested information becomes the bases for related questions on the final cued-recall test, the test taker’s ability to answer such questions will be enhanced. In the present situation, however, when the multiple-choice tests are presented before study, accessing of information contained in the passage is not possible during the taking of the test. Thus, in light of this inherent difference in the pretest versus the posttest situation, it remains unclear whether the incorrect alternatives in multiple-choice pretests questions need to be competitive in order to produce the enhanced processing of information related to them during the test takers’ subsequent reading of the passage, and the resolution of this issue will require future research.
For the time being, however, our results are consistent with the notion that presenting competitive information as multiple-choice alternatives leads to deeper processing of that information (particularly with respect to the item that is selected) than does presenting competitive information within a fact. We argue that learners, although not able to successfully answer most questions, may bring to mind any information that they can—whether entirely relevant to the to-be-studied information or not—in order to help them decide upon an answer. They might, for example, in trying to answer a question asking for the name of the oldest geyser in Yellowstone Park, turn to semantic knowledge (e.g., Old Faithful has “old” in its name—thus, maybe it is the oldest) or they might choose an answer for reasons that are arbitrary yet helpful for memory (e.g., selecting the most pleasant one or the one that is most relevant to their own personal lives; Craik & Tulving, 1975; Packman & Battig, 1978). Then, when participants see these items in the text, they will be more likely to (a) notice them and (b) encode them, perhaps using the information that they had previously used to select them as mediator(s) for their encoding (see, e.g., Pyc & Rawson, 2010, and also Kornell, Hays, & Bjork, 2009; Richland et al., 2009, for related arguments about why pretesting might improve learning of pretested information). It is possible, however, that other entirely different mechanisms also play a role in enhancing the learning of related information when encountered during subsequent study, and this possibility should be examined in future research.
Multiple-choice pretesting and the retention of pretested information
In the present research, performance on the final cued-recall test did not differ for questions about information that was either directly tested in a multiple-choice pretest question or studied as a fact before subsequent passage reading. This finding is intriguing because, in the present context, one might have expected that later performance on questions for information previously studied as a fact would be better than that for information previously tested in a multiple-choice question. To clarify: Our fact-study activity essentially provided participants with 24 seconds of access to the question and correct answer prior to their being asked that same question on the final test. In contrast, in our multiple-choice pretesting activity, participants were presented with the correct answer for only 4 seconds after spending 20 seconds considering the question and, typically, choosing an incorrect answer for it. Nonetheless, the ability of participants to recall such tested information was not impaired in the later final cued-recall test compared to their ability to recall the same information when it had, instead, appeared in the fact-study condition. That lack of a difference between later recall of previously tested and previously studied information, combined with the finding that participants’ ability to answer questions about related information was significantly better in the multiple-choice pretest condition than in the fact-study condition, would seem to suggest that trying to answer a multiple-choice question about to-be-studied information, even if one cannot do so correctly, would function as a more powerful way to potentiate overall future learning than simply being preexposed to that information in the form of facts. Additionally, this pattern of results would seem to call for a reconsideration of claims made by proponents of errorless learning, who would argue that the making of errors should be avoided during learning activities, with the assumption that such errors will tend to persist and thereby impair future correct learning (e.g., Guthrie, 1952; Skinner, 1958). We also see the pattern of results obtained in the present Experiments 1 and 2 to be consistent with findings observed by Kornell et al. (2009; see also, Carpenter, 2011; Grimaldi & Karpicke, 2012; Hays, Kornell, & Bjork 2013; Huelser & Metcalfe, 2012; Knight, Ball, Brewer, DeWitt, & Marsh, 2012; Kornell, 2014; Vaughn & Rawson, 2012; Yue et al. 2015) in their research on test-potentiated learning.
Trying to answer multiple-choice pretest questions—at least ones constructed with competitive alternatives, as was used in the present research—can provide unique benefits to the learning of subsequently studied related and competitive information that outweighs that of other activities, such as cued-recall pretests and/or prestudy of to-be-encountered facts. This unique advantage of multiple-choice pretests may well come about owing to the operation of two simultaneous but different processes stemming from the taking of such tests. First, the answering of multiple-choice questions with competitive alternatives induces deeper processing of the information contained in the question than what results from simple exposure to that competitive information (e.g., in the form of facts with competitors), and second, it directs attention more broadly (but without requiring more time) during subsequent reading than does answering cued-recall pretest questions. From an educational perspective, then, the present results suggest an effective, efficient, and practical way for instructors to enhance their students learning: giving them multiple-choice pretests containing competitive alternatives before their study of potentially confusable topics.
This research was supported by a Collaborative Activity Award from the James S. McDonnell Foundation (Grant 29192G). We thank Robert Bjork, Nate Kornell, Barbara Knowlton, and members of CogFog for helpful insights. Experiment 1 appears as part of the dissertation of Jeri L. Little.
- Anderson, R. C., & Biddle, W. B. (1975). On asking people questions about what they are reading. In G. H. Bower (Ed.), The psychology of learning and motivation (Vol. 9, pp. 89–132). New York, NY: Academic Press.Google Scholar
- Carpenter, S. K. (2011). Semantic information activated during retrieval contributes to later retention: Support for the mediator effectiveness hypothesis of the testing effect. Journal of Experimental Psychology: Learning, Memory, & Cognition, 37, 1547–1552.Google Scholar
- Guthrie, E. R. (1952). The psychology of learning. Oxford, UK: Harper.Google Scholar
- Little, J. L., & Bjork, E. L. (2011). Pretesting with multiple-choice questions facilitates learning. In L. Carlson, C. Hölscher, & T. Shipley (Eds.), Proceedings of the 33rd annual conference of the Cognitive Science Society (pp. 294–299). Austin, TX: Cognitive Science Society.Google Scholar
- Roediger, H. L., & Schmidt, S. R. (1980). Output interference in the recall of categorized and paired-associate lists. Journal of Experimental Psychology: Human Learning and Memory, 6, 91–105.Google Scholar
- Soderstrom, N., & Bjork, E. L. (2015). Using pretesting to enhance learning in the classroom. Manuscript in preparation.Google Scholar