The time course of episodic associative retrieval: Electrophysiological correlates of cued recall of unimodal and crossmodal pair-associate learning
- First Online:
- Cite this article as:
- Tibon, R. & Levy, D.A. Cogn Affect Behav Neurosci (2014) 14: 220. doi:10.3758/s13415-013-0199-x
- 336 Views
Little is known about the time course of processes supporting episodic cued recall. To examine these processes, we recorded event-related scalp electrical potentials during episodic cued recall following pair-associate learning of unimodal object-picture pairs and crossmodal object-picture and sound pairs. Successful cued recall of unimodal associates was characterized by markedly early scalp potential differences over frontal areas, while cued recall of both unimodal and crossmodal associates were reflected by subsequent differences recorded over frontal and parietal areas. Notably, unimodal cued recall success divergences over frontal areas were apparent in a time window generally assumed to reflect the operation of familiarity but not recollection processes, raising the possibility that retrieval success effects in that temporal window may reflect additional mnemonic processes beyond familiarity. Furthermore, parietal scalp potential recall success differences, which did not distinguish between crossmodal and unimodal tasks, seemingly support attentional or buffer accounts of posterior parietal mnemonic function but appear to constrain signal accumulation, expectation, or representational accounts.
KeywordsEpisodic memoryCued recallERPRecollectionFamiliarityParietal lobes
The retrieval of associative information from long-term episodic memory is a vital cognitive function. It enables us to represent the time course of incidents in which we have participated, reconstruct environments in which we have been present, and relive events that we have experienced. Associative retrieval is likely to be a complex procedure, requiring the contribution of multiple processes and brain substrates acting not only simultaneously but sequentially (Moscovitch & Winocur, 2002). Accordingly, in the investigation of associative retrieval, there is an important place not only for hemodynamic and neuropsychological investigations that may identify relevant brain substrates, but also for electrophysiological studies that can provide information about the time courses of constituent processes.
Cued recall studies offer an ecologically important perspective on associative retrieval not provided by recognition, since everyday remembering often involves reconstruction of events based on partial information. However, few physiological studies have investigated recall. In fMRI studies of stem completion of studied words, recall success was associated with activity in medial temporal lobes and the lateral parietal cortex (Okada, Vilberg & Rugg, 2012; Schott et al., 2005). Some electrophysiological cued recall studies employing word stem or word fragment completion have reported various event-related potential (ERP) dissociations related to retrieval success (Allan, Doyle & Rugg, 1996; Allan & Rugg, 1997; Allan, Wolf, Rosenthal & Rugg, 2001; Fay, Isingrini, Ragot & Pouthas, 2005; Johnson, Kreiter, Zhu & Russo, 1998). However, activations in those tasks may reflect priming or morphemic completion rather than episodic recall and, in any case, do not require associative recall.
Other cued recall studies have employed pair-associate learning paradigms. fMRI studies link cued recall success with the activation of various regions, including the hippocampus, additional MTL regions, the parahippocampal cortex, the middle temporal gyrus, the parietal cortex, the inferior prefrontal cortex, and posterior midline areas (de Zubicaray, McMahon, Eastburn, Pringle, Lorenz & Humphreys, 2007; Habib & Nyberg, 2008; Hayama, Vilberg & Rugg, 2012; Henson, Shallice, Josephs & Dolan, 2002; Meltzer & Constable, 2005). While this indicates that several brain regions either contribute to successful cued recall or are activated following recall success, hemodynamic imaging cannot distinguish between preretrieval, retrieval, and postretrieval processes. ERP studies, which offer superior temporal resolution, have addressed associative cued recall, but to our knowledge, none have provided direct observations of basic cued recall processes. Studies by Rösler and colleagues (Heil, Rösler & Hennighausen, 1996; Khader, Burke, Bien, Ranganath & Rösler, 2005; Khader, Heil & Rösler, 2005) tested recall after extensive associative overlearning, which is not reflective of ecological episodic memory, generally created in a single-trial learning experience. In other pair-associate cued recall studies (Donaldson & Rugg, 1998, 1999), participants made old/new judgments about cues before attempting to retrieve associated pair members, confounding cued recall and recognition processes. Recently, Siebert and colleagues (Seibert, Gimbel, Hagler & Brewer, 2011; Seibert, Hagler & Brewer, 2011) examined early parietal MEG activations elicited by recollecting the classification judgment made on the pair associate of a presented cue. However, since no comparisons between successful and unsuccessful trials were reported, those studies do not characterize recall success. Furthermore, none of the above studies investigated processes related to the recall of crossmodal associations.
We therefore examined the time course of associative cued recall minimally confounded by recognition processes, reflected by EEG signal recorded over frontal and parietal scalp areas. Cued recall is driven solely by recollection; unlike recognition, the recalled targets are unavailable for familiarity judgments. Therefore, ERPs elicited by cued recall enable appraisal of claims that early mid-frontal ERP old/new effects recorded during successful recognition reflect familiarity, while late parietal components reflect recollection (Mecklinger, 2000; Rugg & Curran, 2007). Since episodic cued recall cannot be driven by familiarity, early mid-frontal success effects should not be evident. In contrast, parietal components should be similar between recall and recognition.
The cued recall paradigm further allows the examination of several hypotheses that have been put forward regarding the nature of retrieval-related posterior parietal activations observed primarily in studies of recognition memory (recently reviewed by Kim, 2011; Levy, 2012). While most recent work has focused on hemodynamic studies of parietal mnemonic effects, it is generally asserted that parietal-maximal ERP retrieval success effects should be attributed to the same underlying processes as those indexed by fMRI (e.g., Rugg & Curran, 2007; Vilberg & Rugg, 2009; Wilding & Ranganath, 2011). One suggestion is that parietal activations at retrieval reflect the participation of the angular gyrus in multimodal representation (Shimamura, 2011). If that is the case, parietal ERPs should be stronger for successful crossmodal than for unimodal cued recall. Another suggestion is that parietal activations reflect a postretrieval buffer (Haramati, Soroker, Dudai & Levy, 2008; Rugg & Wilding, 2000). Such an account is most tenable if the ERP component associated with parietal processes follows an earlier component associated with a process reflecting recall success. According to the attentional process account (e.g., Cabeza, Ciaramelli & Moscovitch, 2012; Ciaramelli, Grady, Levine, Ween & Moscovitch, 2010), parietal ERP components should be found for cued recall and may reflect either preretrieval attentional focus or postretrieval attentional capture. In contrast, accounts proposing that parietal mnemonic activations reflect signal accumulation in the service of a recognition judgment (e.g., Donaldson, Wheeler & Petersen, 2010; Wagner, Shannon, Kahn & Buckner, 2005) or expectations regarding mnemonic status of a probe and their violations (Buchsbaum, Ye & D'Esposito, 2011; O'Connor, Han & Dobbins, 2010) would be challenged by the emergence of parietal ERP effects for cued recall, in which those processes have no substrate on which to act.
In order to best address these issues, we employed two types of single-trial pair-associate learning for which cued recall would be performed. In the unimodal task, stimulus pairs were color drawings of common objects (tools, animals, food, toys, vehicles, etc.), and in the crossmodal task, stimulus pairs consisted of an object picture and a brief nameable environmental sound (e.g., dog barking, glass breaking, harp arpeggio, etc.). In both cases, at study, participants were asked to create a mental image in which the presented objects interacted. At test, in both tasks, the cue was an object picture, and the recall target was the associated studied item (picture object name or sound object name, respectively). There was no explicit demand to make recognition judgments, as opposed to earlier recognition-then-cued-recall paradigms (Donaldson & Rugg, 1998, 1999). Furthermore, all cues were stipulated to be old and were expected by participants to be old. This paradigm therefore enabled us to the time course of associative cued recall minimally confounded by recognition processes.
Participants were 41 healthy right-handed (all scored positively in the Edinburgh Handedness Inventory; Oldfield, 1971) young adults (22 females; mean age 22.2 years, SD = 3.11 years, range 18–33), with normal or adjusted-to-normal vision. All were undergraduate students who volunteered in return for academic requirement credit or payment. Informed consent was obtained from all participants for a protocol approved by the Interdisciplinary Center's Institutional Review Board. Four participants were excluded from the analyses: 1 who did not complete the experiment, due to malaise, and 3 due to computer failure during the experiment, leaving us with 37 participants whose data were analyzed.
One hundred seventy-six environmental sounds were downloaded from various Internet sources. All stimuli were edited using Audacity audio editing software, at 44 kHz, 16-bit resolution, mono mode. They were all adjusted to the same amplitude level and edited to last 2 s. A behavioral pilot study was conducted to select the environmental sounds that were most recognizable. Five people who did not take part in the main study were asked to name candidate sounds and to rate the difficulty of naming on a scale ranging from 1 to 5, where 1 was very easy and 5 was very hard. Of these sounds, we selected the sounds that were correctly identified by a majority of participants with a difficulty level lower than 4. Altogether, 120 sounds matching these criteria were employed in the experiment. Twelve additional sounds were used for practice trials and examples.
Stimuli were 360 color drawings of common objects obtained from various Internet sources, including fruits and vegetables, tools, sporting goods, electrical and electronic devices, animals, food, transportation devices, body parts, furniture, and clothing, each approximately 6–8 cm in on-screen size.
To form the various experimental conditions, several stimulus lists were created. Two lists of 120 pictures served as cue lists for recollection. Two additional lists served as target lists, one containing 120 pictures and the other containing 120 sounds. Each entry in the cuing lists was pseudorandomly assigned to an entry in the target lists. List pairing was counterbalanced across participants. The two target lists were matched for stimulus domain (e.g., a picture of a violin in target list 1 was matched by a sound of an accordion in target list 2), while the domains of the to-be-associated entries from the cue and target lists were always different (e.g., a sound of a barking dog from the sound target list or a cat from the picture target list was never matched with a picture of a dog from the cue list). This pair construction process was intended to minimize the degree of preexisting semantic associative strength between pair members, such that the association to be generated by the participant would be unconfounded by preexisting semantic associations and its formation would constitute a discrete event, leading to a subsequent episodic memory.
The experiment consisted of two sessions, with an ~10-min break between them. In one session, participants performed the crossmodal pair-associate learning task, and in the other session, participants performed the unimodal task. Session order was counterbalanced across participants.
Participants were tested individually in a quiet room. Upon arrival at the lab, they signed an informed consent form and filled out the Edinburgh Handedness Inventory (Oldfield, 1971). Following EEG electrode cap preparation (described below), participants were seated at a distance of ~70 cm from a computer monitor. For the crossmodal session, in-ear headphones were applied. Participants were then told that they would be presented with pairs of stimuli (picture–picture pairs in the unimodal task and sound–picture pairs in the crossmodal task) and were instructed to remember those pairs. They were instructed to form an association between the stimuli, preferably by using imagery, in order to enhance their memory. They were further told that after 20 pairs (in the crossmodal task) or 40 pairs (in the unimodal task) had been presented, a test phase would ensue, in which cue pictures would be presented alone. They would then need to recall and use the keyboard to type the name of the object portrayed in the picture (in the unimodal task) or of the object making the sound (in the crossmodal task) that had accompanied that cue. Participants were asked to relax and to avoid eye movements and blinks as much as possible.
Pilot data indicated that performance in the unimodal task was superior to performance in the crossmodal task. We therefore employed different block lengths in order to match difficulty levels; behavioral results (see below) indicated that this manipulation did, indeed, match cued recall success measures across tasks. The participants viewed three blocks of 40 stimulus pairs each in the unimodal task and six blocks of 20 stimulus pairs each in the crossmodal task. During the encoding phase of each block, stimulus pairs were presented for 2 s, followed by a 500-ms blank screen. This was followed by a screen with the legend “Association?” to which participants were instructed to respond by hitting the "Enter" key once they had generated an association. This triggered a 1-s visual fixation cross, followed by the next stimulus pair. After all the pairs in the block (either 40 or 20) had been presented, the recall phase started. In this phase, the cue picture was presented alone, until participants responded by pressing, with their right hand, a green key (if they could recall its paired object) or a red key (if they could not). After keypress, a 500-ms blank screen appeared, followed by a screen with the legend “Pair Associate,” to which participants were instructed to respond by typing the correct answer and pressing the "Enter" key. This triggered a 1-s visual fixation cross, followed by the next cue stimulus. After all the cue pictures in the block had been presented, the next study block started. A practice block of 10 trials was provided before each session. During this practice session, the experimenter ascertained that the participant understood the nature of the association that was to be generated using the stimulus pairs. Self-paced rest breaks of several minutes duration were given between blocks.
Electrophysiological recording parameters and data processing
The EEG was recorded using the Active II system (BioSemi, The Netherlands) from 64 electrodes mounted in an elastic cap according to the extended 10–20 system. EOG (electro-occulogram) was recorded using four additional external electrodes, located above and below the right eye and on the outer canthi of both eyes. One electrode was placed on the tip of the nose. Two additional electrodes were placed over the left and right mastoid bones. The ground function during recording was provided by common-mode signal and direct right leg electrodes forming a feedback loop, placed over the parieto-occipital scalp. The online filter settings of the EEG amplifiers were 0.16–100 Hz. Both EEG and EOG were continuously sampled at 512 Hz and stored for offline analysis.
Using the Fieldtrip toolbox for MATLAB (Oostenveld, Fries, Maris & Schoffelen, 2011), stimulus-locked ERPs were segmented into epochs starting 200 ms before cue presentation and up to 1,000 ms afterward. EEG and EOG channels were then rereferenced to the average of the left and right mastoid channels, band-pass filtered with an offline cutoff of 0.1–30 Hz, and baseline-adjusted by subtracting the mean amplitude of the prestimulus period (200 ms) of each trial from all the data points in the segment. ICA was employed in order to remove eye movements and blink artifacts (Makeig et al., 1999). Additional trials containing electrode pop artifacts, resulting from a sudden change in the electrical potential between the electrode and the scalp, and muscle artifacts were rejected visually. Channels depicting drifts and other artifacts on individual trials were replaced with interpolated data from adjacent electrodes. Reaction time (RT) outliers (2 SDs above or below the participant’s average in each task) were removed from ERP analysis.
Mean performance indices (accuracy and reaction times [RTs]) for the cued recall tasks
Mean number (and range) of trials per participant
Not included in ERP analysis
Not included in ERP analysis
We examined task differences in recall accuracy with repeated measures ANOVAs, using recall success (success, FA, failure) and task (unimodal, crossmodal) as repeated factors. Greenhouse–Geisser-corrected degrees of freedom are reported as required. We found a significant difference in the number of trials between recall success possibilities, F(1.71, 42.16) = 116.3, p < .001, and a significant recall success × task interaction, F(2, 72) = 6.77, p = .002. Importantly, although follow-up pairwise comparisons revealed significant differences between the unimodal and crossmodal tasks in FA rates, t(36) = −4.13, p < .001, and in failure rates, t(27) = 2.28, p = .029, the tasks did not differ in success rates. Thus, although there was an imbalance in unsuccessful trial type across tasks (higher FA than failure rates in the crossmodal task and higher failure than FA rates in the unimodal task), FA trials were excluded from the ERP analysis, such that the main comparison was done between success and failure trials. The difference between the tasks in unsuccessful trial rates was one reason we chose to use mixed-effects models for statistical analysis of the ERP findings (see below). For RT analysis, outliers (2 SDs above or below the participant’s average in each condition) were removed. Additionally, to enable calculation of the ANOVA, two missing values (one subject who did not have any FA in the unimodal task and another who did not have any failure trials in the crossmodal task) were replaced with the group mean RTs in the corresponding conditions. The analysis revealed significant effects of recall success, F(1.6, 58.8) = 33.37, p < .001, and of task, F(1, 36) = 6.05, p = .019, and a significant recall success × task interaction, F(2, 72) = 3.52, p = .035. Follow-up pairwise comparisons revealed that unimodal RTs differed significantly from crossmodal RTs only for FA responses, t(36) = 2.93 (which were excluded from the ERP analyses), but not for success and failure responses.
Ultimately, even though some behavioral differences were found between tasks, both accuracy and RT analyses indicated that our use of different list lengths for unimodal and crossmodal tasks was generally successful in reducing the difficulty differences between those tasks found in pilot studies. Hence, any ERP differences found between the tasks should not be attributed solely to task difficulty differences.
In order to address the possibility that ERP recall success effects reflected not recall but, rather, cue familiarity, we conducted an ancillary behavioral experiment with a different set of comparable participants (n = 15). In this experiment, participants studied a list of picture pairs using the same encoding procedure as in the unimodal task. As in the main experiment, the task was divided into three encoding-retrieval blocks; each encoding block consisted of 80 pictures (40 pairs). At retrieval, 40 studied pictures, intermixed with 40 new foils were presented. Participants were asked to press one of the five buttons on the keyboard according to whether, and how well, they remembered that picture as having been presented in the study phase, by choosing between “definitely not,” “probably not,” “don’t know,” “probably yes,” or “definitely yes.” The data of 3 participants were removed as outliers (−2 SD in hits). For old items, “definitely yes” and “probably yes” responses were classified as hits, and for new items, “definitely not” and “probably not” responses were classified as correct rejections. The average hit rate for the 12 remaining participants was 94.6 %. Of those hit responses, 96.7 % were “definitely yes” responses, and the remaining 3.3 % were “probably yes” responses. The average correct rejection rate was 95.8 %. Of those correct rejection responses, 92 % were “definitely not” responses, and the remaining 8 % were “probably not” responses. These results suggest that the study procedure employed in the main experiment results in almost all cues being recognized with high confidence; that is, almost all would be regarded as familiar. As we will argue below, this is one of the reasons that it is unlikely that ERP cued recall effects result from differences in the familiarity signal elicited by cues leading to successful recall.
ERP analyses and results
For each participant, trials were averaged to compute four ERP waveforms: (1) unimodal–recall success, (2) unimodal–recall failure, (3) crossmodal–recall success, and (4) Crossmodal–recall failure. As was noted above, FA trials were omitted from the ERP analyses.
Mixed-effects models analysis
Outcomes of mixed-effects models analysis
Recall success × task
Recall success × anteriority
Laterality × anteriority
Recall success × task × anteriority
For the first time window (0–200 ms), decomposition of the recall success × task interaction revealed that successful recall trials elicited more positive deflections than did unsuccessful trials only in the unimodal task. In the second time window (200–350 ms), decomposition of the significant recall success × task × anteriority interaction revealed that unsuccessful trials exhibited more negative deflections in the unimodal task, but not in the crossmodal task. This unimodal recall success effect was most prominent in anterior locations. In the third time window (350–600 ms), recall success was characterized by positive deflections. Decomposition of the recall success × task × anteriority interaction in this epoch revealed that while, in the unimodal task, the recall success effect emerged in all topographic locations, in the crossmodal task, the effect was apparent only over central-posterior scalp electrodes. Finally, in the fourth time window (600–1,000 ms), decomposition of the two-way recall success × task interaction indicated that the recall success effect was found only in the crossmodal task.
In order to better characterize the recall success effect found in the third time window, we ran an additional analysis. Although the third time window in our analysis (350 – 600 ms) was delineated in order to best characterize effects observed in parietal electrodes, additional anterior effects emerged in the crossmodal task that appeared to peak earlier (~450 vs. ~500 ms) and offset earlier (~500 vs. 600 ms) than the posterior effects. To examine that effect, we conducted a mixed-effects model analysis that included the data from anterior channels in the 350- to 500-ms time window, with task, recall success, laterality, and relevant possible interactions as fixed factors. This analysis revealed a significant effect of recall success, F(1, 21111) = 39.33, p < .001, a significant effect of task, F(1, 21111) = 6.06, p = .014, a significant effect of laterality, F(2, 21111) = 10.99, p < .001, and a significant recall success × task interaction, F(1, 21111) = 10.53, p = .001, suggesting that although the recall success effect in anterior electrodes in the 350- to 500-ms window was larger in the unimodal task, there may be an anterior recall success effect for the crossmodal task as well.
Mixed-effects model analysis versus repeated measures ANOVA
As we explain above, mixed-effects model analysis appears to be the appropriate mode of inspecting data in which bin size is determined post hoc by individual performance. However, since that type of analysis is not yet widespread, we compared the results of that approach with those of a conventional repeated measures ANOVA, using the same factors as those used as fixed factors in our mixed-effects analyses (task, recall success, anteriority, and laterality). We identified the subset of participants who had at least 15 trials in each bin (n = 21). Eliminating participants with a low number of trials in certain bins not only increases the SNR ratio for each condition for each participant (which is necessary when the grand average is participants based, as in the standard ANOVA), but since the bins are mutually dependent (i.e., a participant who had 100 correct responses can have only 20 error responses, and vice versa), this also improves the balance between the bins. We initially conducted the analysis for the key 200- to 350-ms time window. This analysis concurred with the mixed-effects model results. It revealed a significant recall success × task × anteriority interaction, F(1.32, 26.43) = 4.7, p = .014. We further used pairwise comparisons that were subjected to Bonferroni correction (to control for type I error, significant results are those with p < ~.008). This revealed that in frontal sites, the effect was significant in the unimodal task (p = .001), but not in the crossmodal task (p = .213). Patterns similar to the ones that emerged using mixed-effects models were obtained for other locations and other components. By using the mixed-effects model analysis, we were able to obtain results in line with conventional repeated measures ANOVAs, without excluding the data of 16 participants who had small numbers of trials in some bins.
This first delineation of the electrophysiological correlates of episodic cued recall following crossmodal as well as unimodal pair-associate learning, minimally confounded by recognition processes, reveals a sequence of recall success effects. They are distinguished by their morphologies and polarities and onset at rather early latencies. Some of these effects may be analogous to ERP correlates of recognition and may raise questions regarding received interpretations of the functional significance of those components.
Recall-success-related ERP divergence beginning at the onset of cue presentation was observed in the unimodal task, but not in the crossmodal task. This divergence may reflect a kind of preparatory cue-processing orientation, in the following sense: In the unimodal task, study pairs comprised similar item types (visually presented objects), allowing unitized or within-domain associations to be formed at encoding. Such unitized associative representations could subsequently encourage a type of pattern completion strategy at retrieval, in which presentation of one member of a jointly encoded visual object pair is used to engender the activation of the other pair member. Trials on which this orientation to processing the retrieval cue is applied from its onset might achieve more eventual recall success. In contrast, in the crossmodal task, items were presented in different sensory modalities, yielding between-domain associations (Mayes, Montaldi & Migo, 2007; Tibon, Vakil, Goldstein & Levy, 2012), which are less amenable to such a strategy. The notion that such an early preparatory cue-processing orientation can be linked with recollective success is reminiscent of a recent report by Addante, Watrous, Yonelinas, Ekstrom and Ranganath (2011) that prestimulus oscillatory brain activity measured at frontal electrode sites may predict subsequent source memory retrieval success.
The following time window (200–350 ms) was characterized by a strong negative deflection for unsuccessful trials, again exclusive to the unimodal task, but notably larger than in the prior time window (thus, seemingly reflecting some different or additional retrieval process). This unimodal recall success divergence deflection remained robust until 600 ms, but during the earlier part of that epoch, it was not accompanied by posterior recall success effects or by anterior crossmodal recall success effects. Notably, the effect of onset latency precedes that of post-N1-P2 cued recall effects in previous reports. In several cued recall studies employing word stem completion, an anterior retrieval success effect onset at ~400 ms (Allan et al., 1996; Allan & Rugg, 1997; Allan et al., 2001; Fay et al., 2005). Nevertheless, the use of word stems or fragments as cues makes it is hard to determine whether those effects reflect recollection or, rather, generate-and-recognize strategies (Allan et al., 2001) or even priming effects. Two word-pair–cued-recall studies also report analogous recall success effects (Donaldson & Rugg, 1998, 1999) onsetting later than those we observed. However, in those experiments, an old/new recognition judgment on cue words was performed first; possibly, later onset of that effect is due to the dual-response procedure, in which some cues needed to be rejected as new before associative retrieval could commence. In contrast, in the present study, all cues were stipulated to be studied items, such that associative ecphory could commence immediately upon cue presentation and associative recall effects could emerge earlier. However, consideration of ERP studies of recognition suggests that the shorter frontal recall success effect latency here is a function of cue stimulus type—namely, object pictures versus visual words in those prior studies. Retrieval success ERP divergences for picture recognition at frontal sites have been reported to begin at ~250 ms (Ally & Budson, 2007; Herzmann, Jin, Cordes & Curran, 2012; Opitz, 2010), ~200 ms (Ecker, Zimmer, Groh-Bordin & Mecklinger, 2007; Jäger, Mecklinger & Kipp, 2006 [faces]), and even at ~180 ms (Iidaka, Matsumoto, Nogawa, Yamamoto & Sadato, 2006). That is notably earlier than the standard onset latency of the mid-frontal retrieval success effect (FN400) for words at ~300 ms (Mecklinger, 2000; Rugg & Curran, 2007). It should be noted, though, that the pattern is not universal: Earlier frontal effects for verbal stimuli are reported, for example, by Curran and Dien (2003; onset at 176 ms) and by Diana, Vilberg and Reder (2005; onset at 190 ms), while old/new divergences for pictures in Schloerscheidt and Rugg (1997) do not precede 300 ms.
Furthermore, while we have described the aforementioned unimodal frontal recall success effect as beginning in the 200- to 350-ms time window, the 350-ms end point was employed in analysis, since additional effects begin at that point both frontally for the crossmodal task and in posterior channels for both tasks (see below). As was mentioned earlier, the unimodal frontal recall success effect itself extends in a relatively stable fashion until 600 ms. Notably, in studies of picture recognition, a similar latency window is reported for the FN400 component. For example, in Iidaka et al. (2006), the greater negativity for correct rejections at F3, F4, and Fz is stable from 200 to 600 ms and then abruptly terminates, just as in the present study. In Opitz (2010), the difference onsets slightly later but is stable to 600 ms. In Yu and Rugg (2010), who reported differences between probes receiving confident old judgments and new probes, the divergence was stable over the 200- to 500-ms range, continuing but weaker until 800 ms, then strong again from 800 to 2,000 ms. In several other studies, divergences in frontal locations remained stable for extended periods, sometimes to the conclusion of the recording epoch (Ally & Budson, 2007 [400–1,200 ms]; Curran & Doyle, 2011 [Experiment 2; 200–1,000 ms]; Ecker et al., 2007 [200–1,200 ms]; Herzmann et al., 2012 [240–1,000 ms]; Nyhus & Curran, 2012 [300–1,000 ms]; Vilberg & Rugg, 2009, for remember responses [~250–2,000 ms]). Thus, the extension of the frontal retrieval success effect is a common feature for picture memoranda, and therefore, the recall success effect herein reported seems analogous in both onset and offset latencies to the mid-frontal old/new / FN400 component. Additionally, the FN400 effect is characterized by a more negative deflection at anterior electrodes for probes judged to be new (i.e., report of the absence of a mnemonic representation), just as in the present data, the parallel time window is characterized as a more negative deflection for trials in which the sought-after target is not retrieved from memory. Since hemodynamic neuroimaging studies indicate that there is substantial overlap between the brain substrates of successful cued recall and recognition (Habib & Nyberg, 2008; Okada et al., 2012), it seems most parsimonious to understand the present cued recall success deflections that are temporally parallel to morphologically alike recognition success deflections as reflecting similar underlying processes, rather than as a de novo mnemonic component.
It is commonly asserted that in recognition, the early mid-frontal effect (the FN400) reflects familiarity processes (reviewed by Friedman & Johnson, 2000; Mecklinger, 2000; Rugg & Curran, 2007; Wilding & Ranganath, 2011). Could familiarity processes account for the analogous cued recall success effects in the present study? Familiarity is considered to be a process that allows one to appreciate the fact that a presented item was previously encountered, even though contextual details of the encounter cannot be retrieved (Yonelinas, 2002). Inherent in this definition is the notion that some representation of the target item itself must be present in order for a familiarity signal to be elicited. In cued recall, a stored target representation must be activated by the cue and enter consciousness in order to serve as the required response. The target representation is not available to be judged as familiar or not, and therefore, target familiarity in its most common sense seems to play no role in this paradigm.
An alternative account is that this retrieval success effect reflects some form of associative familiarity, which is asserted to support associative recognition judgments, especially in the case of representations that were unitized at encoding (Mayes et al., 2007). In source memory paradigms, accurate recollection of source-specifying information regarding a recognition probe is generally asserted to be dependent on recollection. However, it has recently been demonstrated that when an item and its source information are unitized at encoding, familiarity can contribute to source memory retrieval (e.g., Bastin, Diana, Simon, Collette, Yonelinas & Salmon, 2013; Diana, Van den Boom, Yonelinas & Ranganath, 2011; Diana, Yonelinas & Ranganath, 2008; Mollison & Curran, 2012; Staresina & Davachi, 2010). Such findings raise the possibility that in the unimodal cued recall task, in which the stimuli were in the same sensory modality and, therefore, more readily unitized than in the crossmodal task (Mayes et al., 2007), recall success might similarly be based on associative familiarity processes. However, there is a substantial difference between source memory and pair-associate paradigms. In source memory paradigms, a limited number of source possibilities are employed (e.g., two-source options [Bastin et al., 2013; Diana et al., 2011; Diana et al., 2008]; four-source options [Staresina & Davachi, 2010]; six-source options [Mollison & Curran, 2012]), and therefore, given a specific item probe, all possible item–source combinations can be mentally generated and appraised, with one or more options possibly eliciting a familiarity signal. In contrast, in the present study, pair-associate learning was performed on stimuli with no preexisting semantic associations, such that the “retrieval space” for any cue was unbounded. Therefore, the generate-and-appraise approach cannot support recall success, and associative familiarity similarly does not seem to explain the recall success effect. It is possible that once cued recall is successfully accomplished, the retrieved answer may generate a feeling of associative familiarity and that the signal accompanying such a feeling might be stronger for unimodal than for crossmodal associations; in such an account, the recall success divergence reflects familiarity consequent upon recollective processes, rather than an alternative to them.
Yet another account of the FN400 sees it as reflecting conceptual priming of experienced stimuli (summarized by Paller, Lucas & Voss, 2012). However, since, in cued recall success, the target information is recollected and not re-presented, priming seems unable to account for the present early mid-frontal recall success effect, nor can it explain the presence of the effect only for cues in the unimodal, but not the crossmodal, task.
A final possibility is that the recall success differences in this time window are simply a function of differences in cue familiarity. Some considerations seem to argue against that possibility. All cues were stipulated to be old, encouraging a sense of familiarity, and no judgments regarding cues were required. While it is possible that the amount of familiarity the cues evoke might differ regardless of the nature of the task, the fact is that in our ancillary study, cues were likely to be recognized at a high level of confidence (with a 95 % hit rate in the control experiment reported above). Therefore, cue familiarity differences could, even theoretically, explain only the variance in EEG signal associated with a small percentage of trials on which the cue was not confidently recognized. More important, since the cues employed in the crossmodal and unimodal tasks were identical, cue familiarity effects should be similar in the two tasks. Our results indicate that this was not the case: In this time window, a recall success effect was observed only in the unimodal task, not in the crossmodal task. However, these considerations do not conclusively rule out cue familiarity as the basis of the recall success effect in this time window. Familiarity might differ spontaneously irrespective of whether judgments are made on the cues; even within recognized cues, levels of familiarity might differ; and task-driven differences might have lessened cue familiarity in the crossmodal condition.
Keeping these reservations in mind, the possibility exists that the unimodal cued recall success effect might reflect more than just cue familiarity differences. An alternative explanation is that effects beginning in this time window reflect retrieval orientation differences (Rugg & Wilding, 2000). This term is used to describe a wide range of effects on retrieval activity engendered by differences between categories of retrieval target, distinguished by perceptual form such as words versus pictures (Robb & Rugg, 2002), encoding task (Dzulkifli, Sharpe & Wilding, 2004), reward value (Halsband, Ferdinand, Bridger & Mecklinger, 2012), or specificity of study–test match (Ecker & Zimmer, 2009). To date, these effects have been reported only in studies of recognition, generally in the form of differences between responses to correctly rejected foils related to each category of targets. The report of retrieval orientation most similar to the conditions of the present study is that of Experiment 2b in Hornberger, Morcom and Rugg (2004). In that experiment, participants studied either auditory words or pictures, using the same encoding task, and later made recognition judgments using visual word probes. ERPs elicited by correct rejections of foil cues differed beginning at about 400 ms and continuing until 1,200 ms, depending on whether the targets were pictures or auditory words. The early aspect of that divergence appears analogous to the effect of the present study, since it describes a strong negative deflection immediately following the N1 and P2 waves. Also, analogous to the results reported herein, there was a more negative deflection during the picture target condition than during the auditory target condition. In contrast to this distinction between foil cues, there was no significant difference between hits of studied pictures or auditory words. Hornberger and colleagues (2004) suggested that the reason for the retrieval orientation difference in their study was that using visual words to retrieve studied pictures can rely only on conceptual overlap between the cue and target, thus requiring greater constraint of the cue representation and a stronger negative deflection, while using written words to retrieve auditory words may additionally utilize lexical and phonological overlap, less constraint of cue representation, and a weaker negative deflection. That approach does not appear to explain the data of the present study, in which there is no lexical, phonological, or semantic overlap between cue and target stimuli of either type. Furthermore, in the present study, successful recall trials were characterized by more positive deflections for the unimodal task than for the crossmodal task. Since this difference is in the opposite polarity than the task differences for recall failure, it cannot be the case that just the attempt to retrieve picture stimuli irrespective of success or failure (which would canonically constitute retrieval orientation based on target modality) is responsible for the ERP difference between tasks; rather, it is seemingly an actual recall success effect in the unimodal task. but not in the crossmodal task.
A slightly more general conceptualization of retrieval orientation—as the choice of strategy leading to retrieval optimization—might point to the mnemonic processes expressed in the cued recall success effect. For recognition, maximization of study–test overlap may be the most effective retrieval strategy. In contrast, for episodic cued recall, the optimal retrieval strategies seem likely to be those that activate hippocampal associative representations, such as cue elaboration and reentrant processing. Interestingly, local field potential recording in medial temporal lobes displays divergence of successful from unsuccessful item recognition beginning ~200 ms post-cue-presentation (Grunwald et al., 2003; Staresina, Fell, Lam, Axmacher & Henson, 2012), and significant divergence related to successful source-memory retrieval in the hippocampus is reported beginning at 250 ms (Staresina et al., 2012). This activity temporally overlaps the recall success effect over the frontal sites reported in our study and raises the possibility that this frontal component might be related to attempts at ecphory reflected by hippocampal activity. The greater negative-going wave observed for unsuccessful trials in the present study may reflect one type of frontal lobe “working-with-memory” operations (Moscovitch, 1992). In the unimodal task, in which unitized representations might be more readily formed at encoding, frontal mechanisms might engage in maintaining and elaborating the cue stimulus to probe medial temporal lobe representations via pattern completion attempts, continuing until a decision is made to cease retrieval efforts. It is interesting that both frontal recall success divergences and the subsequent parietal recall success divergences seem to taper off at 600 ms, yielding to a different electrophysiological signature for the following time window. That time point might mark the cessation of both pattern completion and recollective retrieval attempts.
Post hoc analysis revealed a frontal recall-success-related positivity observed in the 350- to 500-ms time window in the crossmodal task. Its short duration suggests that it is not simply a later onset of the same effect as in the unimodal task. We very preliminarily surmise that this may reflect the earliest manifestation of complex recollective reconstruction of the type that might have been most effective in retrieving the associated sound representation in the crossmodal task, in the absence of fast activation of the associated picture target by pattern completion that is possible in the unimodal task. Such recollective reconstruction might involve cue-based reactivation of elements of the encoding episode, including the association that was formed between the encoded pair members, which lead to the retrieval of the target pair member. This type of reconstruction is primarily dependent on hippocampal processes (e.g., Mayes et al., 2007), which may be reflected, indirectly or directly, by the observed frontal-maximal ERP differences. It remains to be seen whether this component will be confirmed by additional cued recall studies, which may clarify its possible functional significance.
The next recall success effect, observed in the 350- to 600-ms time window most prominently over the parietal scalp, naturally invites comparison with the recognition memory late positive component (LPC; also called the posterior “old/new effect”) assumed to reflect recollection (Friedman & Johnson, 2000; Mecklinger, 2000; Rugg & Curran, 2007; Wilding & Ranganath, 2011), since recollective processes are part and parcel of cued recall. Notably, in the present paradigm, this component onsets earlier (~350 ms) than in most recognition studies, even of pictures (e.g., ~500 ms in Mollison & Curran, 2012; Nyhus & Curran, 2012; ~450 in Ally & Budson, 2007; Curran & Doyle, 2011; Ecker et al., 2007; Herzmann et al., 2012; ~400 in Opitz, 2010.; Vilberg & Rugg, 2009; Yu & Rugg, 2010); in some cases, parietal recognition effects are as early (~350 ms in Iidaka et al., 2006; ~300 ms or earlier in Ally, Simons, McKeever, Peers & Budson, 2008). Seemingly, if task demands call directly for a recall operation, relevant processes may be initiated at a very early point in time, and retrieval success effects may emerge earlier in recall than in recognition tasks.
While cross-task and cross-study comparisons are difficult to quantify, the present posterior recall success effect seems very similar in morphology and scalp distribution to the recognition-related LPC. It may therefore be instructive to consider the present pattern of recall success effects recorded over the parietal scalp in light of the extensive recent discussion about the functional significance of parietal retrieval-related hemodynamic activations described above. These are most commonly studied in the context of recognition memory tasks and, as was noted in the introduction, have been linked to parietal recognition ERP effects (see also Wagner et al., 2005). The present findings of parietal ERP recall success effects that cannot readily be attributed to recognition processes may challenge some of those accounts of parietal mnemonic functions. The signal accumulation account attributes parietal activations to a signal indicating “the amount of evidence that retrieval cue corresponds to a studied item” (e.g., Okada et al., 2012). This approach relates to recognition decisions as signal detection style evaluation of test probes as being either old or new, analogous to perceptual decisions. In cued recall following pair-associate learning of unrelated stimuli, there are no probes to be assessed in a signal detection fashion, since the number of possible recall targets is unbounded. Therefore, even though a parietal-focal recall success effect cannot entirely rule out the possibility that the accumulator account might elucidate the role of the PPC in recognition memory, it does indicate that this account cannot be an exclusive explanation of parietal mnemonic activity. It is possible, though, that even in recall tasks, potential responses are evaluated in signal detection fashion, with the decision to produce a candidate recall response based on the pooling of some scalar signal of recall certainty accruing from activity in other brain regions. Similarly, the expectation account (Buchsbaum et al., 2011; O'Connor et al., 2010) focuses on violation or confirmation of expectations regarding the mnemonic strength of a recognition probe and also seems unable to account for recall success effects. Yet another account, the cortical binding of relational activation (Shimamura, 2011) account suggests that ventral parietal areas—specifically, the angular gyrus—serve as a multimodal perceptual convergence zone supporting consolidated representations that are activated during retrieval. In that view, the parietal recall success effect in the present study should be stronger in crossmodal than in unimodal tasks; this was not the case. It is furthermore instructive that the unimodal and crossmodal tasks in the present study exhibit similar component profiles (especially in peak amplitude latency and offset latency). MacKenzie and Donaldson (2009) noted that parietal activations are often posited to reflect material-independent retrieval because, in recognition studies, they have been observed across stimulus types—words, line drawings, object pictures, landscape/object compound stimuli, and sounds. We now confirm that when pictures and sounds are the retrieval targets, relevant ERP recall success effects over parietal scalp are comparable. This seems to indicate that parietal recollective processes are domain general and operate similarly across target modalities in recall, as well as in recognition.
As far as other proposals regarding the functional significance of parietal recall-related activations, the present findings are more equivocal. The attention-to-memory (Cabeza et al., 2012; Ciaramelli et al., 2010) account asserts that dorsal parietal activations reflect preretrieval top-down attentional processes required for challenging retrieval, while ventral parietal activations reflect bottom-up capture of attention by stronger (and less mnemonically effortful) retrieved representations. This account has not explicitly addressed the question of the assignment of parietal LPC in ERP studies of recognition to one or the other (or both) of those processes. The emergence of several significant recall success effects in earlier time windows, along with the short overall latency of the parietal effect, seems to be more in consonance with postretrieval capture of attention attributed to ventral parietal areas than with dorsal parietal preretrieval attentional allocation, which should precede other recall success differences, but this is not a conclusive argument. Yet another approach assigns parietal areas the function of postretrieval buffer, either in general (Rugg & Wilding, 2000) or specifically in support of action planning (Haramati et al., 2008). The pattern found in the present study is in consonance with that suggestion, since the parietal effect follows the earlier frontal components reflecting recollective success, such that a retrieved representation may be available for storage in an episodic buffer, but it does not provide conclusive evidence in favor of that approach.
Having offered the foregoing assessments of proposals about parietal mnemonic effects reported in hemodynamic imaging studies, we wish to add the caveat that such assessments are valid only on the basis of the assumption that ERP old/new effects recorded over the parietal scalp related to recollection have parietal generators. This view has been propounded by other researchers (e.g., Rugg & Curran, 2007; Wilding & Ranganath, 2011) and is supported by converging evidence (e.g., Vilberg & Rugg, 2009). However, while some recent source localization studies support that identification (e.g., Bergström, Henson, Taylor & Simons, 2013, who report a striking overlap of EEG/MEG generator source and fMRI activation in the medial parietal cortex for domain-general source memory success), other studies have reported additional sources of the posterior old/new EEG effect in the later time window under discussion (e.g., Herzmann et al., 2012), and yet other studies may indicate that parietal activations reported in hemodynamic studies are not captured by the posterior EEG old/new effects. The above observations must therefore be considered tentative.
The parietal recall success difference tapers off and disappears at 600 ms postcue, at which time an extended recall success difference emerges in frontal loci, with more pronounced negativity for successful than for unsuccessful trials. This component is most clearly observed in left anterior scalp electrodes in the crossmodal task and extends to the end of the recording period employed in the present study. This component may be compared with late frontal components reported in connection with recognition tasks also requiring retrieval of contextual information, such as source retrieval (e.g., Wilding & Rugg, 1996). Recent findings have led to an account that proposes the existence of two discernible postretrieval monitoring processes indexed by frontal ERP differences: a fronto-central effect linked with operations working over recollected content and a right-frontal old/new effect reflecting postretrieval monitoring but not tied specifically to recollection (Cruse & Wilding, 2011; Woodruff, Hayama & Rugg, 2006). It is possible that the late frontal component revealed in the present study is related to one or both of those processes. Retrieval monitoring might be understood to be more demanding in the crossmodal task, which requires richer episodic reconstruction in the absence of unitization at encoding. However, in the present study, the effect was strongest over left anterior sites, rather than having the right-hemisphere focus reported in many recognition studies (Wilding & Ranganath, 2011). Additionally, the present effect is of the opposite polarity: a more negative deflection for successful trials, while in recognition studies, source-correct or high-confidence trials yield a more positive late frontal deflection (e.g., Cruse & Wilding, 2011; Woodruff et al., 2006). Therefore, an alternative interpretation may be considered: that since successfully recalled information needed to be maintained until the full response was given, the stronger negativity for successful trials reflects a working memory process bridging the interval between recall and typed response. Such maintenance might be more demanding for identifications derived from environmental sounds, which may incorporate the generation of a verbal label that is less clear-cut than for object pictures.
In summary, using a cued recall paradigm, we have identified ERPs elicited in conjunction with successful episodic cued recall following pair-associate learning both within and across modality as early as, and perhaps earlier than, analogous components associated with retrieval success in recognition memory paradigms. In addition to adding to the characterization of the time course of cued recall, these findings invite reinterpretation of the functional significance of recognition-related ERP components.
This work was supported by the Israel Science Foundation grant 611/09. We thank Shir Ben-Zvi, Ayelet Peer, Inbar Shimson, Maia Duarte, and Odelia Nakar-Goldberg for assistance with stimuli preparation, data collection, and analysis and Profs. Axel Mecklinger and John T. Wixted for valuable comments.