Advertisement

Memory & Cognition

, Volume 45, Issue 5, pp 745–754 | Cite as

The effects of context in item-based directed forgetting: Evidence for “one-shot” context storage

  • Nicole Burgess
  • William E. HockleyEmail author
  • Kathleen L. Hourihan
Article

Abstract

The effects of context on item-based directed forgetting were assessed. Study words were presented against different background pictures and were followed by a cue to remember (R) or forget (F) the target item. The effects of incidental and intentional encoding of context on recognition of the study words were examined in Experiments 1 and 2. Recognition memory for the picture contexts was assessed in Experiments 3a and 3b. Recognition was greater for R-cued compared to F-cued targets, demonstrating an effect of directed forgetting. In contrast, no directed forgetting effect was seen for the background pictures. An effect of context-dependent recognition was seen in Experiments 1 and 2, such that the hit rate and the false-alarm rate were greater for items tested in an old compared to a novel context. An effect of context-dependent discrimination was also observed in Experiment 2 as the hit rate was greater for targets shown in their same old study context compared to a different old context. The effects of context and directed forgetting did not interact. The results are consistent with Malmberg and Shiffrin’s (Journal of Experimental Psychology: Learning, Memory, and Cognition, 31, 322–336, 2005) “one-shot” context storage hypothesis that assumes that a fixed amount of context is stored in the first 1 to 2 s of the presentation of the study item. The effects of context are independent of item-based directed forgetting because context is encoded prior to the R or F cue, and the differential processing of target information that gives rise to the directed forgetting effect occurs after the cue.

Keywords

Context dependent recognition Context dependent discrimination Directed forgetting effect Picture recognition 

It has long been appreciated by researchers who study animal cognition and human memory that reinstating the learning context at the time of retrieval benefits memory, whereas changes in context from study to test can have an adverse effect (Carr, 1925; Dulsky, 1935; for reviews, see Hockley & Bancroft, 2015; Smith & Vela, 2001; Spear, 1978). In tests of recall, an old context provides a source of cues to guide retrieval (Godden & Baddeley, 1975; Smith, 1979, 1988). In tests of recognition, it has been argued that context can influence both the familiarity and recollection of the target item.

Murnane, Phelps, and Malmberg (1999) distinguished between two patterns of effects seen in examinations of context on recognition memory. In such studies, a target item can be tested in its old studied context (same old context), in an old context that was associated with a different study item (a rearranged old context), or a novel context. Similarly, a distractor can be tested in the old context of a study item or a novel context. Context-dependent recognition is seen as an increase in the hit and false-alarm rates of items tested in either same old or rearranged old context conditions compared to a novel context without a difference in overall discrimination between the context conditions. This type of context effect is generally assumed to be due to the familiarity of the old context increasing the familiarity of the test item (see also Dalton, 1993; Hockley, 2008; Hockley, Bancroft, & Bryant, 2012; Murnane & Phelps, 1993, 1994, 1995; Vakil, Raz, & Levy, 2007). Context-dependent discrimination, in contrast, is evidenced when the hit rate is greater for targets tested in their same old study context compared to a rearranged old context, and overall discrimination is therefore greater in the same old context condition. This comparison of same old and rearranged old contexts equates the familiarity of the old contexts, and the discrimination advantage is assumed to reflect memory for an association between the target and the encoding context (see also Gruppuso, Lindsay, & Masson, 2007; Hockley, 2008; Macken, 2002; Reder et al., 2013; Vakil et al., 2007).

This study was designed to determine if reinstating the study context at test would attenuate item-based directed forgetting. In this version of the directed forgetting procedure, a cue follows each study item, indicating whether participants should try to remember (R cue) or try to forget (F cue) that item. Typically, presentation order of the cued study items is random. A directed forgetting effect is demonstrated by better memory performance for R-cued compared to F-cued items (see Johnson, 1994, and MacLeod, 1998, for reviews).

Current explanations of item-based directed forgetting are based on the differential encoding of R and F items during the study phase. In the selective rehearsal account, items are maintained in short-term or working memory until the cue is presented (Basden & Basden, 1998; Basden, Basden, & Gargan, 1993). Items followed by an R cue become the focus of more elaborative rehearsal, whereas items followed by an F cue are not rehearsed and undergo passive decay. In the attentional inhibition account, it is argued that mechanisms are activated that serve to withdraw attention away from the representations of F-cued items in working memory (Zacks & Hasher, 1994; Zacks, Radvincky, & Hasher, 1996). This attentional inhibition process is seen by some to function passively (e.g., Zacks et al., 1996) whereas others view it as an active and effortful process (e.g., Engle, Conway, Tuholski, & Shisler, 1995; Fawcett & Taylor, 2008, 2010, 2012; Hourihan & Taylor, 2006). Diverting attention and processing away from F-cued items also becomes less successful with increasing cue delay (Hourihan & Taylor, 2006).

There is also debate as to whether item-based directed forgetting reflects processes that operate only at encoding, or at both encoding and retrieval. Studies using electrophysiological measures have reported differences in cortical activity between R-cued and F-cued items, both at encoding and retrieval (e.g., Nowicka, Jednórog, Wypych, & Marchewka, 2009; Ullsperger, Mecklinger, & Muller, 2000; van Hooff, Whitaker, & Ford, 2009; van Hooff, & Ford, 2011). Multinomial processing modeling based on recall also suggests that item-based directed forgetting costs are due to both reduced storage and retrieval (Rummel, Marevic, & Kuhlman, 2016).

In the present investigation, study words were presented against different background pictures of natural landscapes, and these presentations were either followed by an R or an F cue. At test, targets were shown in their same old contexts, rearranged old contexts, or in a new (blank colored screen) context in order to evaluate both context-dependent recognition and context-dependent discrimination for R and F items. We chose to manipulate context by using background pictures because this manipulation has been shown to produce relatively robust effects of context (e.g., Gruppuso et al., 2007; Hockley, 2008; Hockley et al., 2012; Reder et al., 2013). In addition, item-based directed forgetting effects have been found for a wide variety of visual stimuli, including line drawings (Basden & Basden, 1996; Goernert, Widner, & Otani, 2007; Quinlan, Taylor, & Fawcett, 2010), abstract visual symbols (Hourihan, Ozubko, & MacLeod, 2009), and photographs of scenes (Hauswald & Kissler, 2008), although the magnitude of the directed forgetting effect tends to be reduced for line drawings and photographs compared to words. We also used a blank screen as the new context to heighten the contrast between the old and new context conditions. This manipulation has been shown to produce context effects in both tests of yes–no (Hockley, 2008) and forced-choice recognition (Hockley et al., 2012, Exp. 2).

We expected to find a directed-forgetting effect such that recognition performance would be greater for R-cued compared to F-cued targets, given the robustness of this effect. We also anticipated that we would see effects of context-dependent recognition and context-dependent discrimination for R-cued items, consistent with previous demonstrations of these effects in studies of intentional learning. Of greater interest was whether or not effects of context would be observed for to-be-forgotten items. Two possibilities seemed most likely. The first was based on the assumption that the directed forgetting effect involves the differential processing of both the target item and the context information. Following an R cue, elaborative rehearsal of the target item could also increase or strengthen the encoding of the context information. Participants may even be motivated to elaboratively rehearse both the target item and the context in those situations where the context information is a relevant aspect of the encoding task, or when they have been given explicit instructions to pay attention to the context (e.g., Gruppuso et al., 2007; Hockley, 2008; Murnane et al., 1999; Reder et al., 2013; Vakil et al, 2007, and Experiment 2 of this study). In contrast, further encoding of both the target item and the context should be curtailed following an F cue. If this is the case, then the directed forgetting effect would be seen in both greater hit rates and greater effects of context for R-cued items compared to F-cued items. The differential processing of the picture contexts following R and F cues may be similar to the item-based directed forgetting effects that have been found for such visual stimuli as line drawings (Quinlan et al., 2010), abstract visual symbols (Hourihan et al., 2009), and photographs of scenes (Hauswald & Kissler, 2008).

The second possibility was based on Malmberg and Shiffrin’s (2005) “one-shot” hypothesis of context storage. According to this hypothesis, a fixed amount of context is encoded in the first 1 to 2 s of presentation of the study item. Malmberg and Shiffrin showed that increases in study time, or different orienting tasks, do not lead to additional increases in the amount of contextual information than what was stored in the first 1 to 2 s. In the present study, the target and context were presented for 2 s followed by the presentation of the R or F cue for 3 s. Based on the one-shot hypothesis, context information would be fully encoded in the initial presentation of the target item before the cue was presented, and no further contextual information would be stored after presentation of the R or F cue. The encoding of the context would therefore be the same for both R-cued and F-cued items, and similar effects of the reinstatement of the context at test would be expected. In this case, the directed-forgetting effect would be limited to the enhanced encoding and memory of the target items.

To test between these two possibilities, the effects of context were examined for item-based directed forgetting in Experiment 1 when encoding of context was incidental and in Experiment 2 when the encoding of context was intentional. In addition, recognition memory for the context pictures presented with R- and F-cued items was directly assessed in Experiments 3a and 3b.

Experiment 1: Incidental encoding of context

In Experiment 1, participants were instructed to remember only the study words that were followed by an R cue, and that they did not have to remember the background pictures. Therefore, to the extent that participants followed these instructions, any encoding of the background picture contexts was incidental.

Method

Participants

The subjects in each experiment were male and female Wilfrid Laurier University undergraduate students who volunteered to participate in return for course credit. All experimental procedures were approved by the Laurier Research Ethics Board, and informed consent was obtained from each student. The number of participants planned for each experiment was 28 to 32. The specific number tested was based on the number of participants who signed up and arrived for their appointment on the final day of testing. This range was selected based on previous studies of the effects of context (Hockley, 2008) and item-based-directed forgetting (Bancroft, Hockley, & Farquhar, 2013) where the number of participants in each experiment varied from 24 to 33, with a mean of 30. Twenty-eight students participated in Experiment 1.

Apparatus and stimuli

There were 164 words selected from the MRC Psycholinguistic Database (Wilson, 1988). The words were all common nouns, five to eight letters in length, with a Thorndike–Lorge word frequency range of 20–120, and ratings of concreteness, imagability, and meaningfulness according to Colorado norms, between 250 and 600 on scales that range from 100 to 700. Eighty-four photographs of natural landscape scenes were selected from the Internet and used for old-context backgrounds in the study and test lists. The pictures were chosen to be distinctive, interesting, and memorable. The new context introduced at test was a blank pale yellow screen. The words were presented in size 48 Times New Roman uppercase black font in a white text box in the centre of the background. The cues indicating whether each study presentation should be remembered or be forgotten were a green R or a red F, respectively, presented in 200 size Times New Roman font on a plain white background. At test, participants used the N key on the keyboard to indicate a new recognition response and the O key to signify an old response.

Procedure

Presentation and response recording were controlled by SuperLab 4.0 (Cendrus Corp.) software run on a computer equipped with a 17-inch colour monitor housed in an individual cubicle. The study list consisted of 84 words presented against different landscape picture backgrounds for 2 s. The first and last two presentations served as primacy and recency buffers and were not tested. The critical 80 presentations were presented in a different random order for each participant. Each presentation was followed by an R or an F cue for 3 s. The word–picture pairs and the R and F cues were counterbalanced across the participants.

Participants were told that they would be presented with a series of words on different picture backgrounds. They were instructed that after each word–picture pair was presented, they would be shown either an R cue indicating that they should remember the word or an F cue indicating that they should try to forget the word. Participants were also informed that they did not need to remember the picture backgrounds.

The test list consisted of 160 presentations, including the critical 80 old words from the study list and 80 new words. There were nine different test conditions. Of the 40 old R-cued words, 10 were presented with the same studied picture background (same old context), 10 were presented with a different old picture background (rearranged old context), and 20 were presented in the new context condition. For the rearranged old context test pairs, an R-cued item was always paired with an R-cued context and an F-cued item with an F-cued context. The 40 old F-cued words were tested in the same three context conditions. Twenty of the new test words were presented with a studied picture background that had been followed by an R cue, 20 new words were presented with a studied picture background that had been followed by an F cue, and 40 new words were tested in the new context condition. The order of test conditions was random, with a different random order for each participant. The presentation rate was subject paced, with the next test presentation appearing after each recognition response.

The instructions given to participants before the test phase were as follows: “You will now be presented with another list of words. If you DO NOT recognize the word from the study list, click the N key for NEW. If you DO recognize the word from the study list, click the O key to represent OLD. If you recognize the word, respond with OLD even if you were told to forget the word.” No mention was made of the picture backgrounds. Participants initiated the test list when they were ready to begin.

Results and discussion

The mean proportions of correct old responses (hits) are shown in the top portion of Table 1. Separate analyses of the hit rates were conducted to examine context-dependent recognition and context-dependent discrimination. A 2 (cue: R vs. F) × 2 (context: old rearranged vs. novel) repeated-measures analysis of variance (ANOVA) revealed significant effects of context-dependent recognition and directed forgetting. The hit rates were greater in a rearranged old context compared to the novel context, F(1, 27) = 5.133, MSE = .022, ηp 2 = .16, p = .032, and greater for R-cued than for F-cued targets, F(1, 27) = 31.19, MSE = .023, ηp 2 = .536, p < .001. The interaction between these main effects did not approach significance, F(1, 27) < 1.
Table 1

Mean hit rates with standard errors in parentheses for each cue and context condition of Experiments 1 and 2

 

Context

Same old

Rearranged old

Novel

Mean

Experiment 1

 R-cued

.75 (.03)

.69 (.05)

.63 (.04)

.69 (.03)

 F-cued

.54 (.04)

.53 (.04)

.46 (.05)

.51 (.04)

 Mean

.65 (.03)

.61 (.04)

.55 (.04)

.60 (.03)

Experiment 2

 R-cued

.83 (.03)

.76 (.03)

.70 (.03)

.76 (.03)

 F-cued

.65 (.04)

.60 (.04)

.53 (.04)

.59 (.03)

 Mean

.74 (.03)

.68 (.03)

.61 (.03)

.68 (.03)

There was no effect of context-dependent discrimination. A 2 (cue: R vs. F) × 2 (context: old intact vs. rearranged) repeated-measures ANOVA showed only a main effect of directed forgetting; hit rates were greater for R-cued than for F-cued targets, F(1, 27) = 35.26, MSE = .027, ηp 2 = .566, p < .001. The difference between intact and rearranged old contexts, F(1, 27) = 1.556, MSE = .025, ηp 2 = .054, p = .23, and the Cue × Context interaction, F(1, 27) = 1.052, MSE = .014, ηp 2 = .037, p = .31, were not significant.

The mean proportions of incorrect old responses (false alarms) did not differ between new tests shown in an R-cued context (M = .21, SE = .03) compared to an F-cued context (M = .19, SE = .03), t(27) = .837, p = .41, Cohen’s d = 0.160. However, the false-alarm rates were greater in both R- and F-cued old contexts compared to the novel context condition (M = .15, SE = .02), t(27) = 2.415, p = .023, Cohen’s d = 0.482, and t(27) = 2.102, p = .045, Cohen’s d = 0.347, respectively.

Given the difference in false-alarm rates between the old and new context conditions, corrected recognition scores (hit rate minus false-alarm rate) were calculated to assess discrimination performance. These mean scores for each context condition are shown in the top half of Table 2. A 2 (cue: R vs. F) × 3 (context: intact vs. rearranged vs. novel) repeated-measures ANOVA based on corrected recognition scores showed only a main effect of directed forgetting. Discrimination was greater for R-cued than for F-cued targets, F(1, 27) = 25.48, MSE = .044, ηp 2 = .486, p < .001. The main effect of context, F(1, 27) = 1.597, MSE = .013, ηp 2 = .056, p = .212, and the Cue × Context interaction, F(1, 27) < 1, did not approach significance.
Table 2

Corrected recognition scores (hit rate minus false-alarm rate) for each cue and context condition of Experiments 1 and 2

 

Context

Same old

Rearranged old

Novel

Mean

Experiment 1

 R-cued

.54 (.04)

.48 (.05)

.47 (.04)

.50 (.04)

 F-cued

.35 (.04)

.33 (.04)

.31 (.04)

.33 (.03)

 Mean

.44 (.03)

.40 (.04)

.39 (.03)

.41 (.03)

Experiment 2

 R-cued

.48 (.04)

.41 (.04)

.40 (.04)

.43 (.03)

 F-cued

.29 (.04)

.25 (.03)

.21 (.04)

.25 (.03)

 Mean

.39 (.03)

.33 (.03)

.31 (.03)

.34 (.03)

Standard errors of the means are given in parentheses

The results of Experiment 1 are relatively straightforward. First, a robust directed forgetting effect was seen as recognition performance was greater for targets cued to be remembered compared to targets cued to be forgotten. Second, an effect of context-dependent recognition was observed for both to-be-remembered and to-be-forgotten items. The hit and false-alarm rates were both greater for test words presented in an old context compared to a new context. Third, the effects of directed forgetting and context-dependent recognition did not statistically interact, suggesting they could be independent effects. Finally, there was no effect of context-dependent discrimination as there was no significant difference in hit rates between the same old and rearranged old context conditions, and there was no effect of context on overall discrimination performance in the analysis of corrected recognition scores.

Participants in Experiment 1 were not instructed to remember the background pictures at study, and therefore encoding of the picture contexts would be largely incidental. Previous research has shown that incidental encoding of the context is usually sufficient to produce context-dependent recognition (e.g., Hockley, 2008; Hockley et al., 2012; Murnane & Phelps, 1993, 1994, 1995), but when participants are encouraged to attend to the context or associate the target items with their context, context-dependent discrimination is more likely to be evidenced (e.g., Gruppuso et al., 2007; Hockley, 2008; Murnane et al., 1999; Reder et al., 2013; Vakil et al., 2007). Experiment 2 was designed to examine the effects of context on item-based directed forgetting when the encoding of context was intentional.

Experiment 2: Intentional encoding of context

The procedure of Experiment 2 was very similar to Experiment 1, except that participants were advised that it may be helpful to also remember the background pictures for the words that they were cued to remember. Participants were also asked at test to indicate for the words they recognized as old whether they were cued to be remembered or cued to be forgotten at study (see Thompson, Fawcett, & Taylor, 2011). This was done to see if context also affects memory for the instructional cue that follows the item-in-context presentation.

Method

Participants

Thirty-two students participated in Experiment 2.

Apparatus and stimuli

The words, pictures, and cues were identical to those used in Experiment 1. Participants were also tested in the same location, with the same apparatus that was used in Experiment 1. Participants used the N key to make a new recognition decision, and used the R or F keys to indicate if they thought an old word had been cued to be remembered or forgotten.

Procedure

The procedure was identical to Experiment 1, with two exceptions. First, the study instruction that participants did not need to remember the background pictures was changed to: “For the words that you are instructed to remember (R), it may be helpful to also remember the background picture that went with that word.” Second, the test instructions were also changed with the following addition: “If you DO recognize the word from the study list, there are two options that indicate that you recognize this as an old word. If you remember the word being presented with an F (forget) cue, click the F key. If you remember the word being presented with R (remember) cue, click the R key. If you recognize the word as old but do not remember if it was presented with an F or R cue, please make your best guess.”

Results and discussion

The mean proportions of hits summed over R and F responses are given in the bottom half of Table 1. Replicating Experiment 1, the results of Experiment 2 showed an effect of directed forgetting and an effect of context dependent recognition. A 2 (cue: R vs. F) × 2 (context: rearranged vs. new) repeated-measures ANOVA revealed that hit rates were greater for R-cued than for F-cued targets, F(1, 31) = 47.37, MSE = .019, ηp 2 = .604, p < .001, and greater in a rearranged old context compared to the novel context, F(1, 31) = 7.57, MSE = .016, ηp 2 = .196, p = .010. The interaction between these variables again did not approach significance, F(1, 27) < 1.

In contrast to Experiment 1, an effect of context dependent discrimination was observed for both to-be-remembered and to-be-forgotten items. A 2 (cue: R vs. F) × 2 (context: intact vs. rearranged) repeated-measures ANOVA based on hit rates showed the expected advantage of R-cued versus F-cued targets, F(1, 31) = 28.13, MSE = .035, ηp 2 = .476, p < .001. The hit rate was also greater when tested in an intact old context compared to a rearranged old context, F(1, 31) = 6.513, MSE = .019, ηp 2 = .174, p = .016. These main effects did not interact, F(1, 31) < 1.

As found in Experiment 1, the false-alarm rates did not differ between new tests shown in an R-cued context (M = .37, SE = .04) compared to an F-cued context (M = .35, SE = .04), t(31) = .910, p = .370, Cohen’s d = 0.165, but false-alarm rates were greater in R-cued and F-cued old contexts compared to the novel context (M = .30, SE = .04), t(31) = 2.93, p = .006, Cohen’s d = 0.519, and t(31) = 2.42, p = .022, Cohen’s d = 0.430, respectively.

Given the difference in false-alarm rates between the old and new context conditions, corrected recognition scores were again calculated to assess overall discrimination performance. The means of these scores are presented in Table 2. A 2 (cue: R vs. F) × 3 (context: intact vs. rearranged vs. new) repeated-measures ANOVA based on these scores showed a similar pattern of results as the analysis of hit rates. There was a main effect of directed forgetting, F(1, 31) = 46.323, MSE = .029, ηp 2 = .599, p < .001, and context, F(2, 62) = 4.461, MSE = .021, ηp 2 = .126, p = .015. These variables did not interact, F(1, 31) < 1. To explore the main effect of context, two further ANOVAs were conducted. A 2 (cue: R vs. F) × 2 (context: rearranged vs. new) repeated-measures ANOVA revealed only a main effect of cue, F(1, 31) = 51.30, MSE = .018, ηp 2 = .6238, p < .001. The main effect of context and the Cue × Context interaction were not reliable (Fs < 1). A 2 (cue: R vs. F) x 2 (context: intact vs. rearranged) ANOVA revealed main effects of the cue, F(1, 31) = 24.65, MSE = .035, ηp 2 = .443, p < .001, and context, F(1, 31) = 4.904, MSE = .018, ηp 2 = .137, p = .034. The Cue × Context interaction was not reliable, F(1, 31) < 1. This pattern of results demonstrates an effect of context-dependent discrimination for both items cued to be remembered and items cued to be forgotten.

Proportions of R and F old responses

The mean individual proportions of R and F responses made for correct old responses for each context condition are shown in Table 3. Inspection of Table 3 shows that the proportion of correct R responses was greater than the proportion of incorrect F responses for R-cued target items, and the reverse was the case for F-cued target items. Thus, participants could discriminate the cue that was associated with the target items at study to a reasonable degree. It should also be noted, however, that the overall proportion of incorrect F responses to new test items (M = .27, SE = .04) was significantly greater than the proportion of incorrect R responses (M = .07, SE = .01), t(31) = 5.51, p < .001, Cohen’s d = 1.168. This indicates that participants were more likely to make an F response when they were unsure or guessing, and the following comparisons of R and F responses for old tests items, and the interpretation of F responses more generally should be treated with some caution.
Table 3

Mean proportions of R and F old responses for each directed forgetting cue and context condition of Experiment 2. Standard errors of the means are given in parentheses

 

Context

Same old

Rearranged old

Novel

Mean

R responses

 R-cued

.58 (.03)

.46 (.03)

.45 (.03)

.49 (.03)

 F-cued

.14 (.03)

.16 (.03)

.14 (.02)

.15 (.02)

 Mean

.36 (.03)

.31 (.02)

.29 (.02)

.32 (.03)

F responses

 R-cued

.26 (.03)

.30 (.03)

.26 (.02)

.27 (.02)

 F-cued

.51 (.04)

.43 (.04)

.39 (.04)

.45 (.03)

 Mean

.38 (.02)

.37 (.03)

.33 (.03)

.36 (.02)

A 2 (response: R vs. F) × 2 (cue: R vs. F) × 3 (context: old vs. rearranged vs. new) repeated-measures ANOVA was conducted to compare the two types of correct old responses. Old responses were greater for R-cued than for F-cued targets, F(1, 31) = 48.48, MSE = .015, ηp 2 = .610, p < .001, and greater in the old contexts than the new context, F(2, 62) = 12.30, MSE = .010, ηp 2 = .284, p < .001. The main effect of R versus F old responses was not reliable, F(1, 31) = 1.994, MSE = .073, ηp 2 = .060, p = .168, but the interactions between response type and cue, F(1, 31) = 81.06, MSE = .080, ηp 2 = .723, p <.001, and between response type, cue, and context, F(2, 62) = 7.462, MSE = .021, ηp 2 = .194, p = .001, were significant.

To explore the significant interactions involving response type, separate 2 (cue: R vs. F) × 3 (context: old vs. rearranged vs. new) ANOVAs were conducted for each R and F response type. The analysis of R responses revealed old responses were greater for R-cued than F-cued targets, F(1, 31) = 108.16, MSE = .053, ηp 2 = .777, p < .001, and greater in the old than the new contexts, F(2, 62) = 6.623, MSE = .011, ηp 2 = .176, p = .002. The interaction between these variables was also significant, F(2, 62) = 7.066, MSE = .013, ηp 2 = .186, p = .002. Paired-sample t tests showed that the proportion of old R responses was greater for R-cued old items in the same old context than a new context, t(31) = 4.344, p < .001, Cohen’s d = 0.785, and greater in the same old context than a rearranged old context, t(31) = 2.991, p = .005, Cohen’s d = 0.529, but did not differ in rearranged and new contexts, t(31) = 0.476, p = .637, Cohen’s d = 0. The proportion of R responses did not differ for F-cued targets in any context condition (all ps > .38). Therefore, reinstating the old context served to not only increase the overall hit rate, but to also increase the proportion of correct R responses.

The same ANOVA for F responses revealed a similar pattern of results. Old responses were greater for F-cued than for R-cued targets, F(1, 31) = 34.14, MSE = .041, ηp 2 = .524, p < .001, and greater in the old than the new contexts, F(2, 62) = 3.31, MSE = .017, ηp 2 = .096, p = .043. The interaction between these variables was again significant, F(2, 62) = 4.231, MSE = .016, ηp 2 = .120, p = .019. Paired-sample t tests showed that the proportion of old F responses were greater for F-cued old items in the same old context than a new context, t(31) = 2.759, p = .010, Cohen’s d = 0.489, and greater in the same old than a rearranged old context, t(31) = 2.354, p = .025, Cohen’s d = 0.418, but did not differ in rearranged old and new contexts, t(31) = 1.262, p = .216, Cohen’s d = 0.233. The proportion of F responses did not differ for R-cued targets in any context condition (all ps > .14).

Considered together, the analyses of R and F responses for old items show that participants had a reasonable ability to discriminate R-cued from F-cued targets at test, and this discrimination was greater when the study context was reinstated at test.

The results of Experiments 1 and 2 demonstrated a robust effect of item-based directed forgetting. Both experiments also showed an effect of context dependent recognition, as the hit and false-alarm rates were both greater in old compared to new contexts. Experiment 2 also revealed an effect of context dependent discrimination when context was encoded intentionally. Interestingly, the effects of directed forgetting and context did not interact statistically, suggesting they may be independent effects.

Two possible effects of directed forgetting on the encoding and reinstatement of context at test were considered in the introduction. The first possibility was that the differential processing that follows R and F cues would affect memory for both the target items and their context in a similar fashion, with the result that the effects of context would be reduced for F compared to R items. The second possibility, based on Malmberg and Shiffrin’s (2005) one-shot context storage hypothesis, was that the encoding and resulting influence of context at test would be the same for both R-cued and F-cued items because all context information would be encoded prior to the presentation of the cue. The findings of Experiments 1 and 2 indicate that context has a similar effect on recognition of R-cued and F-cued target items and therefore support the one-shot hypothesis.

In Experiments 1 and 2, the encoding of context information can only be inferred from the effects of reinstating the context at test. Experiments 3a and 3b were designed to directly assess the effects of directed forgetting on the encoding of context information by testing recognition memory for the background pictures for R-cued and F-cued presentations.

Experiments 3a and 3b

In Experiments 3a and 3b, recognition memory for the background pictures was examined following the same study phase as in Experiment 2. In Experiment 3a, participants also reported whether the pictures they recognized were associated with an R or an F cue at study. As this discrimination proved difficult for participants, in Experiment 3b participants were only asked to give yes–no recognition responses for the pictures.

Method

Participants

There were 28 participants in Experiment 3a and 27 in Experiment 3b.

Apparatus and stimuli

The same apparatus and stimulus materials were used as in the previous experiments, except that an additional 80 pictures were added to the picture pool to accommodate the addition of new pictures in the test phase. In Experiment 3a, participants used the N key for new responses, and the R and F keys for R-cued and F-cued old responses. In Experiment 3b, participants used the N for new responses and the O key for old responses.

Procedure

The study list and instructions were identical to the study phase of Experiment 2. The test phase was changed to consist only of pictures, 80 from the study phase and 80 new ones. The test instructions for Experiment 3a were: “You were told to remember the words, but you will actually be tested on the backgrounds. If you DO NOT recognize the picture from the study list, click the N key for NEW. If you DO recognize the picture from the study list, there are two options that indicate that you recognize this as an old background. If you remember the picture being presented with an F (forget) cue, click the F key. If you remember the picture being presented with an R (remember) cue, click the R key. If you recognize the picture as old but do not remember if it was presented with an F or R cue, please make your best guess.” The test phase in Experiment 3b was identical to that of 3a, except that participants were only asked to make “old” or “new” recognition judgments for each picture.

Results and discussion

Experiment 3a

The mean proportions of R old and F old responses and overall old responses for each type of test probe are given in Table 4. A one-way ANOVA comparing the overall proportion of old responses for old R-cued, old F-cued, and new pictures revealed a significant effect of test probe, F(2, 54) = 43.67, MSE = .009, ηp 2 = .618, p < .001. Paired t tests showed that the hit rate was greater than the false-alarm rate for both old R-cued pictures, t(27) = 8.84, p < .001, Cohen’s d = 1.718, and F-cued pictures, t(27) = 6.47, p <.001, Cohen’s d = 1.250. There was, though, no significant difference in the hit rates for R-cued and F-cued pictures, t(27) = 1.77, p = .089, Cohen’s d = 0.334, indicating that the directed forgetting instructions did not greatly affect memory for the background context pictures.
Table 4

Mean proportions of R old, F old, and total old responses for Experiment 3a and proportion of old responses for Experiment 3b for each picture test probe. Standard errors of the means are given in parentheses

 

Picture test type

Old R-cued

Old F-cued

New

Mean

Experiment 3a

R responses

.26 (.03)

.23 (.04)

.11 (.02)

.20 (.03)

F responses

.20 (.02)

.27 (.02)

.16 (.02)

.21 (.02)

 “Old” responses

.46 (.04)

.50 (.04)

.27 (.04)

.41 (.04)

Experiment 3b

 “Old” responses

.55 (.04)

.51 (.04)

.24 (.03)

.43 (.03)

To examine differences between R versus F responses, a 2 (response: R old vs. F old) × 3 (test probe: R-cue vs. F-cue vs. new) ANOVA was conducted. This analysis revealed there was no main effect of response, F(1, 27) < 1. The main effect of test probe was significant, F(2, 54) = 43.67, MSE = .005, ηp 2 = .618, p < .001, as hit rates were greater than false-alarm rates. The Response × Probe interaction, F(2, 54) = 4.81, MSE = .011, ηp 2 = .151, p = .012, was also significant. The proportion of R responses was greater for R-cued than F-cued pictures, but this difference was not reliable, t(27) = 1.16, p = .257, Cohen’s d = 0.220. The proportion of F responses was, though, significantly greater for F-cued than for R-cued pictures, t(27) = 3.25, p = .003, Cohen’s d = 0.618. These results indicate that participants’ ability to discriminate the nature of the cue that followed the pictures at study was very modest at best.

Experiment 3b

The mean proportions of old responses for each type of test probe are given in the bottom portion of Table 4. A one-way ANOVA comparing the proportion of old responses for R-cued, F-cued, and new pictures revealed a significant effect of test probe, F(2, 52) = 72.70, MSE = .011, ηp 2 = .737, p < .001. Paired t tests showed that the hit rate was greater than the false-alarm rate for both R-cued pictures, t(26) = 8.57, p < .001, Cohen’s d = 1.717, and F-cued pictures, t(26) = 12.95, p < .001, Cohen’s d = 2.544. There was, again, no reliable difference in the hit rates for R-cued and F-cued pictures, t(26) = 1.58, p = .127, Cohen’s d = 0.306.

Both Experiments 3a and 3b demonstrated that participants’ ability to discriminate old from new pictures was above chance, although the absolute level of performance was relatively modest. There was, though, no directed forgetting effect for pictures as discrimination performance did not differ for background pictures followed by R cues compared to F cues. This result indicates that the encoding of the information supporting memory for the background pictures likely occurred prior to the presentation of the R or F cue, and there was no additional processing of the pictures that were followed by an R cue.

General discussion

This study provides a further demonstration of the robustness of the item-based directed forgetting effect. R-cued target items were better recognized than were F-cued items tested in both old and new contexts. This directed forgetting effect, however, did not extend to the background picture contexts of the target words. In contrast to the target words, recognition of the background pictures did not differ significantly as to whether the target-picture displays were followed by an R or an F cue.1

This study was not designed to distinguish between the differential encoding and attentional inhibition accounts of item-based directed forgetting, and the observed results do not favor one account over the other. The results do, though, serve to show that the processes or operations that underlie either differential encoding or attentional inhibition are focused on the target information, as the availability and influence of context information at retrieval is similar for both R and F items, regardless of whether context was encoded incidentally (Experiment 1) or intentionally (Experiment 2). This finding also has implications for the question concerning possible effects of item-based directed forgetting at retrieval (e.g., Nowicka et al., 2009; Rummel et al., 2016; Ullsperger et al., 2000; van Hooff et al., 2009; van Hooff, & Ford, 2011) because they indicate that any such effects are separate from, and not greatly influenced by, the reinstatement of context at retrieval.

Experiments 1 and 2 demonstrated two different effects of context on recognition performance. Context-dependent recognition was observed in Experiment 1 when the encoding of context was incidental and in Experiment 2 when the encoding of the picture contexts for R-cued items was intentional. In both cases, the hit and false-alarm rates were greater for items tested in an old compared to a new context without a difference in overall discrimination. Context-dependent discrimination was also observed in Experiment 2. Instructions that encouraged participants to remember the context pictures for the to-be-remembered target items produced greater discrimination of target items when they were tested in their original study context compared to target items tested in a different old context. This recognition advantage was found to a similar extent for both R-cued and F-cued targets.

The results of Experiment 2 also showed that participants had a reasonable memory for the cue that followed the study presentations as R responses were significantly greater than were F responses for R-cued targets, and F responses were greater than R responses for F-cued targets. Reinstating the study context also conferred an advantage in remembering the R and F cues as correct responses were greater in the same old context compared to the rearranged and new contexts. Memory for the cue was not as reliable when old versus new pictures were discriminated in Experiment 3a. The proportion of R responses did not differ significantly between R-cued and F-cued pictures, although the proportion of F responses was significantly greater for F-cued than R-cued pictures. Overall, memory for the R and F cues showed a similar pattern of effects as was observed for recognition of the target items.

The novel contribution of the present study was the consistent finding in Experiments 1 and 2 that the effects of context and directed forgetting were statistically independent. Context benefited both R-cued and F-cued targets to a similar extent. It is, perhaps, not surprising that an effect of context dependent recognition would be observed for both R- and F-cued items in Experiments 1 and 2. As context dependent recognition is believed to be due to the familiarity of an old context increasing the familiarity of both old and new test items, it would be expected that the hit rates for both R- and F-cued targets would be influenced in a similar manner and to a similar extent by a familiar context.

It was not anticipated, however, that context dependent discrimination would also be observed for both R- and F-cued items. The higher hit rate for targets presented in their same study context compared to targets presented in an old but different study context is assumed to be a result of the encoding and retrieval of specific target-context associations. The encoding of such associations is more likely when participants are instructed to pay attention to the background context, as was the case in Experiment 2. The finding that the effects of context dependent discrimination did not differ for R- and F-cued targets, and the finding that recognition memory for the picture contexts did not differ for R- and F-cued targets in Experiments 3a and 3b, both indicate that there was no additional or differential processing of contextual information following the presentation of the R cue. The elaborative processing that followed an R cue was focussed on the target item and not the background context. Therefore, the encoding of the context must largely have occurred prior to the presentation of the cue.

The present results are consistent with, and provide further support for, Malmberg and Shiffrin’s (2005) one-shot hypothesis of context storage. According to this hypothesis, a fixed amount of context is stored in the first 1 to 2 s of the presentation of an item, and additional presentation time or encoding processes do not lead to additional increases in the amount of contextual information than what is stored in the first 1 to 2 s. In the present experiments, items and their picture contexts were presented for 2 s before the R or F cue was presented for 3 s. Thus, according to the one-shot hypothesis, all of the encoding of the context information occurred before the cue was presented and therefore the cue should have no effect on context information. As the effects of context were statistically independent of the effects of directed forgetting in both Experiments 1 and 2, the present results also indicate that the one-shot context storage hypothesis applies to both context that is encoded incidentally, and context that is encoded in a more deliberate and intentional manner. To account for the greater effects of context when encoding is intentional compared to when it is incidental, however, requires the additional assumption that the encoding of context in the first few seconds of stimulus presentation is modulated by the degree of attention that is directed to the context.

If context is encoded early, and before the directed forgetting cue is presented, why was memory for the R and F cues associated with the target items greater when the same old study context was reinstated in Experiment 2? Memory for the cue is likely associated with the target item, or can be derived from how the target item was processed. Reinstating the study context serves to benefit memory for the target item, which in turn supports memory for the cue. Thus, the effect of context on memory for the cue may be indirect and mediated through the target item. This suggestion is consistent with Starns and Hicks’ (2008) demonstration that different context dimensions are each bound to the target item, but are not bound to each other.

In summary, the present results show that incidental encoding of context information is sufficient to produce an effect of context-dependent recognition, and the intentional encoding of context can produce the additional benefit of context-dependent recognition. These effects of context were observed for both R-cued and F-cued targets in the item-based directed forgetting procedure. More importantly, these effects did not interact. The effects of context and item-based directed forgetting are independent effects because the encoding processes that support these effects occur at different times. The encoding of contextual information occurs early and before the presentation of the R or F cue, whereas the differential processing of the target item that gives rise to the directed forgetting effect occurs after the cue.

Footnotes

  1. 1.

    It should be noted that Experiments 1 and 2 were specifically designed to detect the effects of item-based directed forgetting and the effects of context reinstatement on recognition performance. As these experiments demonstrate, the magnitude of the effects of directed forgetting were greater than the effects of context reinstatement. In Experiments 3a and 3b, the effect of directed forgetting on memory for the context itself was assessed. It is possible that there is such an effect, but it is smaller in magnitude than the reinstatement effects of context seen in Experiments 1 and 2, and Experiments 3a and 3b did not have sufficient power to detect this effect. Based on the present results, it can only be concluded that the effects of directed forgetting on memory for the target items are much greater in magnitude than any possible effect on memory for the background context.

Notes

Acknowledgements

This study was conducted by N.B. for her honour’s thesis under the supervision of W.E.H. and K.L.H., and was supported by a Discovery Grant from the Natural Sciences Research Council of Canada awarded to W.E.H. Correspondence regarding this article can be addressed electronically to whockley@wlu.ca or khourihan@mun.ca.

References

  1. Bancroft, T. D., Hockley, W. E., & Farquhar, R. (2013). The longer we have to forget the more we remember: The ironic effect of post cue duration in the item-based directed forgetting. Journal of Experimental Psychology: Learning, Memory, and Cognition, 39, 691–699.PubMedGoogle Scholar
  2. Basden, B. H., & Basden, D. R. (1996). Directed forgetting: Further comparisons of the item and list methods. Memory, 4, 633–653.CrossRefPubMedGoogle Scholar
  3. Basden, B., & Basden, D. (1998). Directed forgetting: A contrast of methods and interpretations. In J. Golding & C. M. MacLeod (Eds.), Intentional forgetting: Interdisciplinary approaches (pp. 139–172). Mahwah, NJ: Erlbaum.Google Scholar
  4. Basden, B., Basden, D., & Gargano, G. (1993). Directed forgetting in implicit and explicit memory tasks: A comparison of methods. Journal of Experimental Psychology: Learning, Memory, and Cognition, 19, 603–616.Google Scholar
  5. Carr, H. A. (1925). Psychology: A study of mental activity. New York, NY: Longmans, Green.CrossRefGoogle Scholar
  6. Dalton, P. (1993). The role of stimulus familiarity in context-dependent recognition. Memory & Cognition, 21, 223–234.CrossRefGoogle Scholar
  7. Dulsky, S. G. (1935). The effect of a change in background on recall and relearning. Journal of Experimental Psychology, 18, 725–740.CrossRefGoogle Scholar
  8. Engle, R. W., Conway, A. R. A., Tuholski, S. W., & Shisler, R. J. (1995). A resource account of inhibition. Psychological Science, 6, 122–125.CrossRefGoogle Scholar
  9. Fawcett, J. M., & Taylor, T. L. (2008). Forgetting is effortful: Evidnece from reaction time probes in an intem-method directed forgetting task. Memory & Cognition, 36, 1168–1181.CrossRefGoogle Scholar
  10. Fawcett, J. M., & Taylor, T. L. (2010). Directed forgetting shares mechanisms with attentional withdrawal but not with stop-signal inhibition. Memory & Cognition, 38, 797–808.CrossRefGoogle Scholar
  11. Fawcett, J. M., & Taylor, T. L. (2012). The control of working memory resources in intentional forgetting: Evidence from incidental probe word recognition. Acta Psychologica, 139, 84–90.CrossRefPubMedGoogle Scholar
  12. Godden, D. R., & Baddeley, A. D. (1975). Context-dependent memory in two natural environments: On land and underwater. British Journal of Psychology, 66, 325–331.CrossRefGoogle Scholar
  13. Goernert, P., Widner, R., & Otani, H. (2007). Classification accuracy across multiple tests following item method directed forgetting. Quarterly Journal of Experimental Psychology, 60, 1178–1186.CrossRefGoogle Scholar
  14. Gruppuso, V., Lindsay, S., & Masson, M. E. J. (2007). I’d know that face anywhere! Psychonomic Bulletin & Review, 14, 1085–1089.CrossRefGoogle Scholar
  15. Hauswald, A., & Kissler, J. (2008). Directed forgetting of complex pictures in an item method paradigm. Memory, 16, 797–809.CrossRefPubMedGoogle Scholar
  16. Hockley, W. E. (2008). The effects of environmental context on recognition memory and claims of remembering. Journal of Experimental Psychology: Learning, Memory, and Cognition, 34, 1412–1429.PubMedGoogle Scholar
  17. Hockley, W. E., & Bancroft, T. D. (2015). Context dependent memory. In J. D. Wright (Ed.), International encyclopedia of the social and behavioural sciences (2nd ed., Vol. 5, pp. 787–795). Oxford, UK: Elsevier Limited.Google Scholar
  18. Hockley, W. E., Bancroft, T. D., & Bryant, E. (2012). Associative and familiarity-based effects of environmental context on memory. Canadian Journal of Experimental Psychology, 66, 81–89.CrossRefPubMedGoogle Scholar
  19. Hourihan, K. L., Ozubko, J. D., & McLeod, C. M. (2009). Directed forgetting of visual symbols: Evidence for nonverbal selective rehearsal. Memory & Cognition, 37, 1059–1068.CrossRefGoogle Scholar
  20. Hourihan, K. L., & Taylor, T. L. (2006). Cease remembering: Control processes in directed forgetting. Journal of Experimental Psychology: Human Performance and Attention, 32, 1354–1365.Google Scholar
  21. Johnson, H. M. (1994). Processes of successful intentional forgetting. Psychological Bulletin, 116, 274–292.CrossRefGoogle Scholar
  22. Macken, W. J. (2002). Environmental context and recognition: The role of recollection and familiarity. Journal of Experimental Psychology: Learning, Memory, and Cognition, 28, 153–161.PubMedGoogle Scholar
  23. MacLeod, C. (1998). Directed forgetting. In J. Golding & C. M. MacLeod (Eds.), Intentional forgetting: Interdisciplinary approaches (pp. 139–172). Mahwah, NJ: Erlbaum.Google Scholar
  24. Malmberg, K. J., & Shiffrin, R. M. (2005). The “one-shot” hypothesis for context storage. Journal of Experimental Psychology: Learning, Memory, and Cognition, 31, 322–336.PubMedGoogle Scholar
  25. Murnane, K., & Phelps, M. P. (1993). A global activation approach to the effect of changes in environmental context on recognition. Journal of Experimental Psychology: Learning, Memory, and Cognition, 19, 882–894.Google Scholar
  26. Murnane, K., & Phelps, M. P. (1994). When does a different environmental context make a difference in recognition? A global activation model. Memory & Cognition, 22, 584–590.CrossRefGoogle Scholar
  27. Murnane, K., & Phelps, M. P. (1995). Effects of changes in relative cue strength on context-dependent recognition. Journal of Experimental Psychology: Learning, Memory, and Cognition, 21, 158–172.Google Scholar
  28. Murnane, K., Phelps, M. P., & Malmberg, K. (1999). Context-dependent recognition memory: The ICE theory. Journal of Experimental Psychology: General, 4, 403–415.CrossRefGoogle Scholar
  29. Nowicka, A., Jednórog, K., Wypych, M., & Marchewka, A. (2009). Reversed old/new effect for intentionally forgotten words: An ERP study of directed forgetting. International Journal of Psychophysiology, 71, 97–102.CrossRefPubMedGoogle Scholar
  30. Quinlan, C. K., Taylor, T. L., & Fawcett, J. M. (2010). Directed forgetting: Comparing pictures and words. Canadian Journal of Experimental Psychology, 64, 41–46.CrossRefPubMedGoogle Scholar
  31. Reder, L. M., Victoria, L. W., Manelis, A., Oates, J. M., Dutcher, J. M., Bates, J. T., . . . Gyulai, F. (2013). Why it’s easier to remember seeing a face we already know than one we don’t: Preexisting memory representations facilitate memory formation. Psychological Science, 24, 363–372.Google Scholar
  32. Rummel, J., Marevic, I., & Kuhlman, B. G. (2016). Investigating storage and retrieval processes of directed forgetting: A model-based approach. Journal of Experimental Psychology: Learning, Memory, and Cognition, 42, 1526–1543.PubMedGoogle Scholar
  33. Smith, S. M. (1979). Remembering in and out of context. Journal of Experimental Psychology: Human Learning and Memory, 5, 460–471.Google Scholar
  34. Smith, S. M. (1988). Environmental context-dependent memory. In G. M. Davies & D. M. Thompson (Eds.), Memory in context: Context in memory (pp. 13–34). New York, NY: Wiley.Google Scholar
  35. Smith, S. M., & Vela, E. (2001). Environmental context-dependent memory: A review and meta-analysis. Psychonomic Bulletin & Review, 8, 203–220.CrossRefGoogle Scholar
  36. Spear, N. E. (1978). The processing of memories: Forgetting and retention. Hillsdale, NJ: Erlbaum.Google Scholar
  37. Starns, J. J., & Hicks, J. L. (2008). Context attributes in memory are bound to item information, but not to one another. Psychonomic Bulletin & Review, 15, 309–314.CrossRefGoogle Scholar
  38. Thompson, K. M., Fawcett, J. M., & Taylor, T. L. (2011). Tag, you’re it: Tagging as an alternative to yes/no recognition in item method directed forgetting. Acta Psychologica, 138, 171–175.CrossRefPubMedGoogle Scholar
  39. Ullsperger, M., Mecklinger, A., & Muller, U. (2000). An electrophysiological test of directed forgetting: The role of retrieval inhibition. Journal of Cognitive Neuroscience, 12, 924–940.CrossRefPubMedGoogle Scholar
  40. Vakil, E., Raz, T., & Levy, D. A. (2007). Multifactorial context effects on visual recognition memory. The Quarterly Journal of Experimental Psychology, 60, 916–923.CrossRefPubMedGoogle Scholar
  41. van Hooff, J. C., & Ford, R. M. (2011). Remember to forget: ERP evidence for inhibition in an item-method directed forgetting paradigm. Brain Research, 1392, 80–92.Google Scholar
  42. van Hooff, J. C., Whitaker, T. A. W., & Ford, R. (2009). Directed forgetting in direct and indirect tests of memory: Evaluating selective rehearsal versus inhibitory accounts with electrophysiological measures. Brain and Cognition, 71, 153–164.CrossRefPubMedGoogle Scholar
  43. Wilson, M. D. (1988). The MRC psycholinguistic database: Machine readable dictionary (Version 2). Behavioural Research Methods, Instruments and Computers, 20, 6–11.CrossRefGoogle Scholar
  44. Zacks, R. T., & Hasher, L. (1994). Directed ignoring: Inhibitory regulation of working memory. In D. Dagenback & T. H. Carr (Eds.), Inhibitory processes in attention, memory, and language (pp. 241–264). San Diego, CA: Academic Press.Google Scholar
  45. Zacks, R. T., Radvansky, G., & Hasher, L. (1996). Studies of directed forgetting in older adults. Journal of Experimental Psychology: Learning, Memory, and Cognition, 22, 143–156.PubMedGoogle Scholar

Copyright information

© Psychonomic Society, Inc. 2017

Authors and Affiliations

  • Nicole Burgess
    • 1
  • William E. Hockley
    • 1
    Email author
  • Kathleen L. Hourihan
    • 2
  1. 1.Department of PsychologyWilfrid Laurier UniversityWaterlooCanada
  2. 2.Memorial UniversitySt. John’sCanada

Personalised recommendations