Advertisement

Psychonomic Bulletin & Review

, Volume 25, Issue 1, pp 302–321 | Cite as

Distributed representations of action sequences in anterior cingulate cortex: A recurrent neural network approach

  • Danesh Shahnazian
  • Clay B. Holroyd
Theoretical Review

Abstract

Anterior cingulate cortex (ACC) has been the subject of intense debate over the past 2 decades, but its specific computational function remains controversial. Here we present a simple computational model of ACC that incorporates distributed representations across a network of interconnected processing units. Based on the proposal that ACC is concerned with the execution of extended, goal-directed action sequences, we trained a recurrent neural network to predict each successive step of several sequences associated with multiple tasks. In keeping with neurophysiological observations from nonhuman animals, the network yields distributed patterns of activity across ACC neurons that track the progression of each sequence, and in keeping with human neuroimaging data, the network produces discrepancy signals when any step of the sequence deviates from the predicted step. These simulations illustrate a novel approach for investigating ACC function.

Keywords

Anterior cingulate cortex Recurrent neural network Sequence learning Surprise 

A surge of research interest over the last 2 decades has resulted in widespread agreement that anterior cingulate cortex (ACC) is partly responsible for cognitive control and decision making (e.g., Euston, Gruber, & McNaughton, 2012; Ridderinkhof, van den Wildenberg, Segalowtiz, & Carter, 2004; Rolls, 2009; Rushworth & Behrens, 2008; Rushworth, Kolling, Sallet, & Mars, 2012; Shackman et al., 2011; Shenhav, Botvinick, & Cohen, 2013; Walton & Mars, 2007), but the development of a formal, comprehensive theory about its function has remained out of reach (e.g., Alexander & Brown, 2011; Botvinick, Braver, Barch, Carter, & Cohen, 2001; Brown & Braver, 2005; Holroyd & McClure, 2015; Khamassi, Lallee, Enel, Procyk, & Dominey, 2011; Silvetti, Seurinck, & Verguts, 2011; Verguts, 2017; Verguts, Vassena, & Silvetti, 2015; Yeung, Botvinick, & Cohen, 2004). Efforts in this direction have been hindered by a complex panoply of empirical findings. For example, a salient observation across neurophysiological studies is that individual cells in ACC tend to respond to multiple task events, which seems to implicate a role for ACC in everything (Ebitz & Hayden, 2016). Thus the exact computational function of ACC—and even whether it has one (Bush, 2009)—remains controversial (Holroyd & Yeung, 2011).

Here we present a simple computational model of ACC that illuminates many of these disparate findings. Our approach is motivated by three considerations. First, the model is based on the long-standing idea in cognitive psychology that cognitive processes are encoded across networks of interconnected processing units (Rumelhart & McClelland, 1986). Although this perspective has recently gained currency in behavioral neuroscience, which is increasingly emphasizing the collective activity of ensembles of neurons rather than the tuning properties of individual cells (e.g., Fusi, Miller & Rigotti, 2016; Rigotti et al., 2013; Yuste, 2015), to our knowledge it has yet to be applied to formal models of ACC. Second, the model takes into account ubiquitous findings from human neuroimaging and electrophysiological studies that ACC is sensitive to response conflict, errors, and surprising events (e.g., Botvinick et al., 2001). Third, the model is based on our previous argument, derived partly from lesion data in human and other animals, that ACC is concerned with the execution of extended, goal-directed action sequences (Holroyd & Yeung, 2012). Here we develop that idea by proposing that the ACC predicts each successive step in the sequence. On this account, the model yields distributed patterns of activity across ACC neurons that track the progression of action sequences (in keeping with the neurophysiological data), and produces a discrepancy signal when any step of a sequence deviates from the predicted step (in keeping with the neuroimaging data).

We begin by providing a very brief overview of neurophysiological data related to ACC. We then home in on a series of studies that suggest a special role for ACC in the execution of goal-directed action sequences. Next, we describe principles for modeling action sequences based on recurrent neural networks (RNNs), and apply these principles to simulate the activity of ACC neurons of rats performing a sequential task. Consistent with empirical findings, the individual units in the model respond to multiple task events, whereas distributed patterns of activity across the units follow the progression of each sequence. We also illustrate how these same principles account for the surprise, error and conflict signals produced in ACC when humans perform speeded response time (RT) tasks, as revealed by human event-related brain potential (ERP) and functional magnetic resonance imaging (fMRI) experiments. Finally, in supplementary materials, we illustrate how these signals could be applied to regulate action execution; these simulations suggest that ACC should be especially important for maintaining contextual information that disambiguates related action sequences. These observations are discussed in terms of our proposal that the ACC is responsible for motivating the execution of extended behaviors (Holroyd & Yeung, 2011, 2012).

Background

Neurophysiological and neuroimaging observations of ACC

A major complicating factor in understanding ACC is the wide range of neurophysiological findings that seem to implicate it in most task events (Ebitz & Hayden, 2016). For example, ACC neurons are seen to respond to stimulus events (Nishijo et al., 1997), motor activity (Backus, Ye, Russo, & Crutcher, 2001; Russo, Backus, Ye, & Crutcher, 2002; Shima et al., 1991), rewards (Amiez, Joseph, & Procyk, 2005; Kennerley, Behrens, & Wallis, 2011; Luk & Wallis, 2009; Sallet et al., 2007; Seo & Lee, 2008), errors (Shen et al., 2015; Totah, Kim, Homayoun, & Moghaddam, 2009; C. Wang, Ulbert, Schomer, Marinkovic, & Halgren, 2005), prediction errors and surprise signals (Bryden, Johnson, Tobia, Kashtelyan, & Roesch, 2011; Hayden, Heilbronner, Pearson, & Platt, 2011; Klavir, Genud-Gabai, & Paz, 2013; Matsumoto, Matsumoto, Abe, & Tanaka, 2007), pain and its anticipation (Koyama, Tanaka, & Mikami, 1998) and, more controversially, in conflict (Davis et al., 2005; Ebitz & Platt, 2015; Kaping, Vinck, Hutchison, Everling, & Womelsdorf, 2011). ACC neurons are also related to cognitive processes associated with working memory (Niki & Watanabe, 1976), long-term memory formation (Weible, Rowland, Monaghan, Wolfgang, & Kentros, 2012), effortful control (Davis, Hutchinson, Lozano, Tasker, & Dostrovsky, 2000; Hillman & Bilkey, 2010, 2012) and even grid cell representations (Jacobs et al., 2013). Complicating matters further, many of these neurons reveal interdependencies across events. For example, ACC cells are said to provide a “gateway” through which decision making systems affect behavior (Cai & Padoa-Schioppa, 2012) by linking or “multiplexing” information about rewards and actions (Hayden & Platt, 2010; Shima & Tanji, 1998; Tanji, Shima, & Matsuzaka, 2002) and predictive stimuli (Takenouchi et al., 1999). The firing patterns of many ACC neurons are multi-determined (Shidara, Mizuhiki, & Richmond, 2005), reflecting multiple aspects of value-based decision making (Hoshi, Sawamura, & Tanji, 2005; Kennerley, Dahmubed, Lara, & Wallis, 2009; Kennerley & Wallis, 2009; Khamassi, Quilodran, Enel, Dominey, & Procyk, 2015) including high-level aspects of task performance such as task switching and behavior shifts (Johnston, Levin, Koval, & Everling, 2007; Kuwabara, Mansouri, Buckley, & Tanaka, 2014; Quilodran, Rothé, & Procyk, 2008). In humans, individual ACC neurons are recruited across multiple different tasks, especially when these tasks demand effort or attention (Davis et al., 2000; Wang et al., 2005).

Neuroimaging and scalp-recorded electrophysiological recordings in humans have also associated ACC with a wide range of phenomena (Holroyd & Yeung, 2011, 2012). Yet, a core set of highly replicable findings have implicated ACC function specifically in the processing of response conflict (Botvinick et al., 2001; Botvinick, Cohen, & Carter, 2004; Yeung, 2013; Yeung et al., 2004), errors (Falkenstein, Hohnsbein, Hoormann, & Blanke, 1990; Gehring et al., 1993; Wessel, Danielmeier, Morton, & Ullsperger, 2012) and otherwise surprising or unexpected events (Alexander & Brown, 2011; Braver, Barch, Gray, Molfese, & Snyder, 2001; Ferdinand & Ovitz, 2014; Forster & Brown, 2011; HajiHosseini & Holroyd, 2013; Holroyd, 2004; Holroyd, Pakzad-Vaezi, & Krigolson, 2008; Jessup, Busemeyer, & Brown, 2010; Metereau & Dreher, 2013; Nee, Kastner, & Brown, 2011; Oliveira, McDonald, & Goodman, 2007; O’Reilly et al., 2013; Silvetti et al., 2011; Wessel et al., 2012). For example, the error-related negativity (ERN) is an ERP component elicited by error commission in speeded response-time tasks (Falkenstein et al., 1990; Gehring et al., 1993; for review see Gehring, Liu, Orr, & Carp, 2012), and the N2 is an ERP component elicited by unexpected, task-relevant stimuli (Donkers, Nieuwenhuis, & van Boxtel, 2005; Ferdinand, Mecklinger, Kray, & Gehring, 2012; Gehring, Gratton, Coles, & Donchin, 1992; Holroyd, 2004; Kopp & Wolff, 2000; Oliveira et al., 2007; Warren & Holroyd, 2012), especially when the stimuli mismatch with a perceptual template of ongoing events (Jia et al., 2007; Sams, Alho, & Näätänen, 1983; Y. Wang, Cui, Wang, Tian, & Zhang, 2004; for review, see Folstein & Van Petten, 2008). The neural sources of the N2 and the ERN colocalize in ACC with an enhanced fMRI blood-oxygen-level dependent (BOLD) response to the same events (Mathalon, Whitfield, & Ford, 2003; Wessel et al., 2012), and it has been proposed that the two ERP components reflect different manifestations of a single underlying cognitive process by ACC (Cavanagh, Zambrano-Vazquez, & Allen, 2012; Wessel et al., 2012; Yeung et al., 2004).

These observations suggest that, when inspected at the cellular level, ACC function is associated with a wide range of task-related events, but when inspected at a more global level, ACC activation is strongly modulated by events related to surprise, conflict and errors.

ACC and action sequences

Although the response profiles of ACC neurons are heterogeneous, a telling set of observations have implicated ACC specifically in the production of goal-directed action sequences. Early on, Procyk, Tanaka, and Joseph (2000) and Procyk and Joseph (2001) observed that motor neurons in ACC are sensitive to the serial order of actions executed in a sequence irrespective of the actual movements performed. Shidara and Richmond (2002) also reported that ACC neurons code for the degree of reward expectancy through multi-stage tasks, and Mulder, Nordquist, Örgüt, and Pennartz (2003) observed that ACC neurons implement a “response set,” which they defined as “a cognitive-motor predisposition to organize and execute an action sequence directed towards a particular goal.” This research presaged a series of studies indicating that ACC neurons are sensitive to the sequential order of task progression across multiple task stages (Cowen & McNaughton, 2007; Fujisawa, Amarasingham, Harrison, & Buzsáki, 2008; Hayden, Pearson, & Platt, 2011; Hoshi et al., 2005; Shidara et al., 2005), as represented by distinct patterns of activity of ensembles of ACC neurons (Blanchard, Strait & Hayden, 2015; Cowen, Davis, & Nitz, 2012). These sequential activations have been said to be mediated by a working memory process intrinsic to ACC (Baeg et al., 2003), or alternatively by inputs carrying sensory and motor efferent information to ACC (Euston & McNaughton, 2006). The sequential activity is also coordinated with respect to the phase of theta oscillations of local field potentials in ACC (Remondes & Wilson, 2013), which are said to synchronize the computations of widespread brain networks (Holroyd, 2016; Verguts, 2017).

Recently, Jeremy Seamans and colleagues have systematically investigated the activity of ACC neurons during the execution of task sequences (Balaguer-Ballester, Lapish, Seamans, & Durstewitz, 2011; Caracheo, Emberly, Hadizadeh, Hyman, & Seamans, 2013; Durstewitz, Vittoz, Floresco, & Seamans, 2010; Hyman, Ma, Balaguer-Ballester, Durstewitz, & Seamans, 2012; Hyman, Whitman, Emberly, Woodward, & Seamans, 2013; Lapish, Durstewitz, Chandler, & Seamans, 2008; Ma, Hyman, Lindsay, Phillips, & Seamans, 2014; Ma, Hyman, Phillips, & Seamans, 2014). Their work focuses on distributed patterns of activity that characterize entire ensembles of neurons, rather than on the responses of the individual neurons that comprise the ensembles. Application of dimension reduction techniques reveal that network-wide ACC activity is characterized by complex trajectories through an abstract state space (Balaguer-Ballester et al., 2011), the disruption to which predicts behavioral errors (Lapish et al., 2008; see also Hyman et al., 2013; Stokes et al., 2013). On this view, ACC network activity tracks task progression through a task-dependent frame of reference, or “task space” (Lapish et al., 2008), toward the animal’s goal (Ma, Ma, Hyman, et al., 2014). The network activity is especially sensitive to transitions through subcomponents of cognitive tasks (Balaguer-Ballester et al., 2011) and is accompanied by abrupt transitions in the state space when the animal learns new rules (Durstewitz et al., 2010) or is exposed to important environmental changes (Caracheo et al., 2013). ACC network activity also discriminates between task sequences more accurately than comparable activity in the dorsal striatum (Ma, Hyman, Lindsay, et al., 2014; Ma, Hyman, Phillips, et al., 2014), and can be distinguished from the function of the hippocampus, which famously encodes sequences related to spatial navigation and other temporally organized events (Hyman et al., 2012).

In humans, fMRI evidence indicates that ACC is responsible for learning and evaluating the execution of hierarchically organized sequences of cognitive tasks (Koechlin, Danek, Burnod, & Grafman, 2002). Further, unexpected changes in sequence production tasks elicit an increased ACC BOLD response (Berns, Cohen, & Mintun, 1997; Ursu, Clark, Aizenstein, Stenger, & Carter, 2009) and larger N2 amplitudes (Eimer, Goschke, Schlaghecken, & Stürmer, 1996; Ferdinand, Mecklinger, & Kray, 2008; Fu, Bin, Dienes, Fu, & Gao, 2013; Jongsma et al., 2013; Lang & Kotchoubey, 2000; Miyawaki, Sato, Yasuda, Kumano, & Kuboki, 2005; Rüsseler, Hennighausen, Münte, & Rösler, 2003; Rüsseler, Kuhlicke, & Münte, 2003; Rüsseler & Rösler, 2000; Schlaghecken, Stürmer, & Eimer, 2000). These observations are in line with our suggestion, based largely on lesion evidence in humans and other animals, that ACC supports the execution of extended, goal directed behaviors involving multiple actions (Holroyd & McClure, 2015; Holroyd & Yeung, 2012), and that unexpected deviations from the intended sequence elicit surprise signals from ACC.

The current approach: recurrent neural networks

Accordingly, we aimed to illustrate these principles in a computational model that simulates ACC neuron activity as animals execute goal-directed task sequences and that produces surprise signals to unexpected events during those sequences. Crucially, the widely varying response profiles of individual ACC neurons suggests that this level of analysis may not be optimal for inferring ACC function. Instead, our approach leverages a long-standing idea in cognitive psychology (Rumelhart & McClelland, 1986) and computational neuroscience (Churchland & Sejnowski, 1992) that holds that neurocognitive functions are encoded across distributed populations of units rather than by individual cells. Although fundamental to computational neuroscience, this principle has only recently become more widely recognized in behavioral neuroscience (e.g., Fusi et al., 2016; Yuste, 2015)—a change in perspective driven partly by methodological advances that have allowed for the simultaneous collection and statistical analysis of data from multiple neural units (Cunningham & Yu, 2014). Thus, a basic principle of neural network theory suggests that we should simulate and analyze the ensemble activity of ACC neurons rather than the single-cell activity per se.

Here, we were inspired by the observation that recurrent connections in neural networks provide a natural means for supporting the execution of extended, goal-directed behaviors (Durstewitz, Seamans, & Sejnowski, 2000). In particular, “connectionist” style recurrent neural networks have been used extensively to study the cognitive psychology of sequence processing (e.g., Cleeremans & McClelland, 1991; Elman, 1990). RNNs provide a computational platform for exploring how distributed representations of hierarchically organized sequences unfold dynamically over time (Elman, 1990). This approach has proven especially fruitful as a proving ground for understanding the neural and cognitive representations that underpin complex action sequences with hierarchical structure. For example, previous work has illustrated how patterns of activity observed across network units can provide insight into representations shared within and across action sequences (Botvinick & Plaut, 2002, 2004). We therefore adopted RNNs for this purpose.

Connectionist networks consist of sets of interconnected, abstract units that process information passed between them (Rumelhart & McClelland, 1986). Some connectionist networks exploit a simple but powerful architecture consisting of three layers of nonlinear processing units, including an input layer, output layer, and an intermediating so-called hidden layer, connected with feed-forward excitatory projections (or weights), that can be augmented with additional layers. With the appropriate connection weights, which are determined through an iterative training process, these networks are capable of approximating any mathematical function to an arbitrary degree of accuracy (Hornik, 1991). It should be emphasized that although the connectionist framework in psychology is motivated by the massively parallel and interconnected structure of the human nervous system, the units in connectionist models are not normally understood to represent individual neurons. Rather, the networks describe in the abstract how distributed processing systems can give rise to observable behavior (Rumelhart & McClelland, 1986).

By definition, any network that contains feedback connections from downstream units to upstream units—that is, that contains connective loops—is recurrent. An important consequence of recurrence is the introduction of activity-dependent memory: Because the output of the recurrent units is subsequently passed back as input, the network retains a memory of previous events. This architecture introduces temporal structure into network processing by integrating current input with prior output. Here we adopt a particular network architecture in which the output of the hidden layer is passed, via one-to-one connections, as an exact copy to a context layer. In turn, the context layer maintains the information in a memory buffer for one discrete time step, after which it is fed back to the hidden layer as input (see Fig. 1; Elman, 1990).
Fig. 1

Elman network. Boxes indicate collections (“layers”) of processing units (not shown); arrows indicate feed-forward excitatory connections from each layer to the next. Elman networks are recurrent neural networks consisting of an input layer, a hidden layer, an output layer, and a context layer. The input units connect fully to the hidden units, the hidden units connect fully the output units, and the context units connect fully to the hidden units; connections between the hidden layer and context layer units are one to one, such that on each time step the context layer receives an identical copy of the hidden layer activity. The networks are trained to produce each element of a sequence given the previous element as input. The number of units in each layer is task dependent

Given an initial element of a sequence as input, these so-called Elman networks can be trained to reproduce the subsequent elements of that sequence. Of interest in these simulations is how the networks disambiguate elements in a sequence by maintaining a memory of the context in which each element occurs. For example, a network may be trained to produce the sequences A->B->C and D->B->E. Recurrence enables the network to produce correctly C or E following the element that is common to both sequences (B), by retaining information about the context in which that element occurred (following an A or D). This contextual encoding can be supported across even longer and more complex sequences, such as A->B->B->B->B->C and D->B->B->B->B->E (Servan-Schreiber, Cleeremans, & McClelland, 1991).

We used an Elman network to represent the function of ACC (Elman, 1990; see Fig. 2). On this account, ACC takes as input information related to external states of the environment and actions produced by the agent, and uses that information to predict the immediately forthcoming environmental state or action. The network then detects discrepancies between what it predicts and what actually occurs, that is, it detects any unexpected deviations from the goal-directed action sequence under execution. Further, the ensemble activity of units in the hidden and context layers encode, in an abstract state-space, the progression of the action sequence as it unfolds. Note that this account holds ACC responsible only for predicting upcoming events and actions, not for producing them. In this way the model instantiates a type of forward model applied to high-level action selection, which in other domains have been used to predict the sensory consequences of low-level motor actions (Shadmehr & Wise, 2005; Wolpert & Ghahramani, 2000).
Fig. 2

Schematic of the recurrent neural network model of anterior cingulate cortex (ACC). Each box corresponds to a layer of an Elman network as shown in Fig. 1; circles within boxes represent individual units. ACC takes as input information related to external states of the environment and actions produced by the actor (e.g., a red stimulus, that a lever was pressed, and so on), and uses that information to predict the immediately forthcoming environmental state or action. Note that the input and output layers can represent the environment in different ways, and so can be composed of different units. Differences between the model predictions (the output of the network, delayed by one sequence step; not shown) and what actually occurs (the input to the network) produce discrepancy signals (circle labeled “x”), as revealed in functional magnetic resonance imaging and event-related brain potential experiments. The role of ACC in sequence execution tasks can be inferred from distributed representations encoded by units in the hidden and context layers, as observed in the ensemble activity of ACC neurons in nonhuman animal studies

All of the simulations herein conform to the network architecture and set of computational principles illustrated in Fig. 1. Each model consists of four layers of abstract processing units including an input layer, context layer, hidden layer, and output layer. Each unit transforms, in discrete time steps, the input it receives into output according to the logistic function
$$ f(x) = 1/\left\{1+{e}^{-\left( Gx+ B\right)}\right\} $$
(1)
where x is the net input to the unit, G is the gain parameter, and B is the bias parameter. The input layer receives information about external events including stimuli in the environment and actions produced by the agent (or alternatively, “efference copies” of issued motor commands; Angel, 1976). This information is represented “locally” such that individual elements of a task correspond to designated units in the input and output layers. Thus, a sequence is simulated by activating, on each time step, the input unit corresponding to the event at that time (by setting the activation value of that unit to 1). The hidden layer then computes the linear combination of the input unit and context unit activations, weighted by the size of the connecting weights, and transforms the result x through the logistic function (Equation 1). In turn, the output layer computes the linear combination of the hidden layer outputs with the connecting weights, the results of which are also nonlinearly transformed through the logistic function. Units with high activity are assumed to reflect the network's outputs, thus output layer activation values less than a threshold of .9 are set equal to zero. Crucially, on each time step the output of each hidden unit is copied to a corresponding unit in the context layer, which maintains that information into the following time step. This memory buffer enables the hidden layer to integrate new information received from the environment with information about past events maintained in the context layer.
The job of the RNN is to produce, on each time step, an output corresponding to the input to be received on the subsequent time step. To develop the internal representations that allow for successful task performance, the network weights are adjusted through a gradual, iterative training process. Training is achieved by computing an error term error that indicates the discrepancy between the actual output of the network and the desired output, that is, between the predicted event and the event that actually happened,
$$ \mathrm{error} = \mathrm{actual}\ \hbox{-}\ \mathrm{predicted} $$
(2)
where error, actual and predicted are all vectors of length equal to the number of units in the output layer. The weights are then adjusted in a way that minimizes the Euclidean length of the error vector across iterations according to the back-propagation through time learning algorithm (Williams & Zipser, 1995). On this basis the RNN is able to learn to predict which action will be produced given previous environmental states and actions, even for ambiguous states that afford multiple potential actions. Note that a single RNN can implement multiple sequences, the limit to which would depend on the number and length of the sequences, their degree of shared structure, and the representational capacity of the network.

In what follows we first examine how ACC tracks task progression. The RNN implementation of this function predicts that ACC should exhibit formal network properties that should be evident at the neural level, which we explore by comparing our results with neurophysiological data recorded from rats. Second, we compare the discrepancy signals produced by the model with ERP and fMRI indicators of ACC activation in humans. Last, in online supplementary materials, we explore how this signal could be utilized to regulate behavior. All of the following simulations conform to the RNN architecture described above. For each RNN, the number of units in each layer is specific to each problem. Each task was simulated multiple times with weights that were initialized with small random values drawn from a uniform distribution between −0.01 and 0.01. Input units were activated by setting their activation values to 1, and only one input unit was activated at a given time. The learning rate parameter was set to 0.5 across simulations, except where indicated (Williams & Zipser, 1995).

Simulations of multivariate ACC activity

We begin by simulating the role of ACC in sequence execution. For this purpose, we trained an RNN to predict each step of a sequence production task recently conduced by Ma and colleagues (Ma, Hyman, Lindsay, et al., 2014; Ma, Hyman, Phillips, et al., 2014), who trained rats to execute a series of 3 different lever presses according to 3 different sequences (see Fig. 3). The input layer of the network contained seven input units, including three “sequence” units that indicated which of the three sequences to execute, three “orientation” units that indicated which of the three levers the animal faced (as specified by textile cues, but which are color-coded as red, yellow, and blue in Fig. 3 for heuristic purposes), and a “press” unit that indicated that the animal recently pressed a lever. Eight output units coded for all possible events that could occur during the sequence, including six units representing the three stimulus locations (left lever, middle lever, right lever) and three stimulus types (red lever, green lever, blue lever), a unit representing a lever press, and a unit representing that the sequence had terminated. The hidden and context layers contained 50 units each. In order to press a lever, the rat was assumed first to orient to the lever and then to press it; correspondingly, for each press the network was trained to predict, in a two-step sequence, first the lever location and lever type simultaneously, and then a lever press.
Fig. 3

Rat sequence production task. Rats were trained to press three different levers according to three different sequences. Each lever was characterized by a unique sensory cue, color-coded here for heuristic purposes as red, yellow and blue. Sequence A consisted of the sequence right, middle, and left lever presses, followed by reward receipt. Sequence B consisted of the sequence middle, left, and right lever presses, followed by reward receipt. Sequence C consisted of the sequence left, right, and middle lever presses, followed by reward receipt. Each sequence was conducted for 10 trials. Note that the order of sensory cues was maintained across the sequences whereas their spatial location changed. Note. Republished with permission of The Journal of Neuroscience: The Official Journal of the Society for Neuroscience, from Tracking Progress Toward a Goal in Corticostriatal Ensembles, Ma, Hyman, Phillips, et al., 34, 2014; permission conveyed through Copyright Clearance Center, Inc. (Color figure online)

Trial progression was simulated across a sequence of seven discrete time steps in which a different input unit was activated at each step (by setting the activation value for that unit to 1): first, the sequence unit representing the sequence to be executed, followed by three iterations of orient and press corresponding to the given sequence. The network was trained to predict what event would occur following each action. For example, if the animal oriented to the red lever, then the network was trained to predict a lever press on the next time step. And if the animal pressed a lever, then the network predicted the location and color of the lever to which the rat would orient on the subsequent step in the sequence. Two hundred RNNs initialized with different weights were trained on the three sequences for 6,000 trials each. All of the network achieved 100% accuracy.

Although the RNNs were not intended to represent how these processes are instantiated at a biological level, we assumed that the abstract neural networks and actual biological networks obey similar computational principles and therefore display comparable patterns of activity. We thus began our analysis by inspecting the properties of individual units in the hidden layers of the RNNs (Figure 2).1 We separately classified hidden units that were exclusively activated by input from the sequence units, orientation units, or response units (see Fig. 4, left panel; note that because the first orientation action in each sequence was taken in response to the sequence cue input, the unit activity that preceded the first orientation action was concurrent with activity elicited by the sequence cue input). Individual units in the hidden layer responded exclusively to the contextual input indicating the sequence to be performed on that trial (16% +/− 4%), to orienting to any of the three levers (21% +/− 4%), or to pressing any of the levers (41% +/− 3%). Overall, nearly 90% of units in the simulations were task responsive; conversely, about 10% of the units were nonresponsive. In addition, because Ma, Hyman, Phillips, et al. (2014) and other groups (e.g., Blanchard et al., 2015; Shidara & Richmond, 2002) have observed ACC neurons that increase in activity throughout the execution of action sequences, we inspected the results for “ramping” units that exhibited comparable behavior. Overall, 14% +/− 5% of the units exhibited ramping behavior (see Fig. 4, left panel).
Fig. 4

Overview of hidden unit activity in the sequence production task. Left panel: Proportion of all hidden layer units classified as “sequence” (S), “orientation” (O), “press” (P) and “ramping” (R) units. Error bars indicate standard error of the mean across simulations. Right panel: Sequence trajectories of the first two principal components of hidden layer unit activity (PC1 and PC2). Each point in the space is labeled according to the input cue for that element of the sequence. S= sequence cue; O1, O2, O3 = the first, second, and third orientation responses, respectively; P1, P2, and P3 = the first, second and third lever presses, respectively; S1, S2, and S3 = Sequence 1, Sequence 2, and Sequence 3, respectively. Note that the trajectories for the three sequences are indistinct because they nearly overlap. (Color figure online)

Thus, separate units in the model encoded each of several major elements of task execution. Many neurons were also sensitive to multiple task events. For example, 10% +/− 3% of all hidden units across all networks were selective to both press actions and orientation actions, 2% +/− 2% of all units were sensitive to both ramping and press actions, and 7.5% +/− 2.5% of all units were sensitive to both ramping and orientation actions. Conversely, around 70%+/− 4% of all the press-sensitive units were uniquely sensitive to presses, and 43% +/− 3.2% of the orientation-sensitive units were uniquely sensitive to orientation.

We expected that this single-unit activity reflected idiosyncratic aspects of the network’s overall function that is encoded across the collective activity of the entire network. To investigate the nature of these representations, we further analyzed the dynamics of the network during task execution using a dimension reduction approach motivated by previous RNN simulations of hierarchically organized action sequences in humans (Botvinick & Plaut, 2002, 2004). In particular, we applied principal components analysis (PCA) to characterize the evolving patterns of activation observed across the network hidden units (see Fig. 2), and computed the distance between the different network representations in the network state space.

PCA was applied to the hidden unit activation levels during task execution across all the 200 networks (50 units per network = 50 variables; three sequences with seven steps per sequence across 200 networks = 4,200 observations). Figure 4, right panel, illustrates the factor scores for the first two principal components, which together account for 70% of the variance for each event in each sequence. Each line (S1, S2, S3) represents the configuration of the network through a sequence of seven events, beginning with the initial sequence cue (S) followed by three cycles of orient (O1, O2, O3) and press (P1, P2, P3). Visual inspection suggests that the network travels through state space in an orderly fashion, with PC1 encoding sequence progression. These two factors encode different aspects of task performance that correspond well with the empirical data (see Fig. 5). Principal Component 1 appears to represent the sequential position of each task event, with later steps in the sequence represented with higher component scores (see Fig. 5a). In the original study by Ma, Hyman, Lindsay, et al. (2014), a similar increase in activity was seen in the firing rates of 107 of 637 ACC neurons (see Fig. 5b). By contrast, Principal Component 2 distinguishes between different events within the same sequence, namely by separating lever presses from orientation responses (see Fig. 5c), as also observed for a smaller collection of 37 ACC neurons (see Fig. 5d). The figures also suggest that the network is relatively insensitive to the specific levers pressed, coding instead for the serial position of each action within each sequence. Consistent with this observation, the Euclidean distance between representations in the 50-dimension hidden unit state space was larger for a given lever press across the three different sequences (“serial”) as compared to a control condition (“control”; see Fig. 5e). A comparable analysis of ACC network activity for rats performing the task yielded similar results, indicating larger differences between lever presses of a given lever across the three different sequences (“serial”) compared to a random control condition (“control”; see Fig. 5f).2
Fig. 5

Simulated and empirical anterior cingulate cortex (ACC) ensemble activity in the sequence production task. Simulated data are presented in the left column and empirical data are presented in the right column. a Factor scores for Principal Component 1 (PC1) of hidden unit activation states as a function of serial position. b Firing rate (FR) of ACC neurons sensitive to sequence progression as a function of serial position. c Factor scores for PC2 of hidden unit activation states as a function of serial position. d FR of ACC neurons sensitive to lever presses as a function of serial position. e Euclidean distance between hidden unit network representations associated with presses on the same lever occurring at different serial positions across sequences (black, “serial”) and a control comparison (white, “control”). f The Mahalanobis distance between population vectors in the multiple single unit activity space for presses on the same lever occurring at different serial positions across sequences (black, “serial”) and a control comparison (white, “control”). ***p < .00001. S, O1, P1, O2, P2, O3, P3, S1, S2, S3 = Same as in Fig. 4; 1st, 2nd, 3rd = the first, second, and third lever presses on each sequence, respectively; Approach = the reward-approach period. Error bars indicate standard error of the mean (SEM). Units in panels a, c, e, and f are arbitrary. Note that the trajectories for the three sequences in panels a and c are indistinct because they nearly overlap. Panels b & d: Republished with permission of The Journal of Neuroscience: The Official Journal of the Society for Neuroscience, from Tracking Progress Toward a Goal in Corticostriatal Ensembles, Ma, Hyman, Phillips, et al., 34, 2014; permission conveyed through Copyright Clearance Center, Inc. Panel f: Adapted by permission from Macmillan Publishers Ltd: Nature Neuroscience, Ma, Hyman, Lindsay, et al., (2014), copyright 2014

The model thus yielded individual units that encoded specific features in the trial that together as a group represented higher order aspects of the task sequence. These results suggest that the responses of individual ACC neurons to various task events reflect a deeper role for ACC in tracking the progression of the organism through a series of goal-directed actions, as revealed by the collective activity of entire ensemble of units. Importantly, the characteristics were not explicitly coded into the model architecture but rather emerged as a natural consequence of recurrent neural network function.

Simulations of univariate ACC activation

Above, we used an RNN to illustrate the role of ACC in predicting each event in the execution of goal-directed action sequences, which revealed distributed patterns of activity across ensembles of ACC neurons that evolve dynamically throughout the sequence. Here, we show that these same principles naturally allow for the detection of unexpected events that fall outside the domain of the sequence under execution. RNNs are inherently predictive mechanisms: On the condition that the output of the network is compared to the actual state of the environment, the networks are trained based on the current state of the environment to predict the immediately subsequent state. This training is enforced with an error signal that quantifies the discrepancy between the internal model predictions and the actual outcomes. Here we compute in the testing phase the Euclidean length D of the same signal used to train the network during the training phase (Equation 2), namely
$$ D = \left|\left|\ error\ \right|\right| $$
(3)

As we will show, this discrepancy measure aligns with univariate signals recorded in ACC that reflect the collective activity of multiple neurons.

Electrophysiological signals in humans

Toward this end, we first simulated subject performance in a study by Wessel and colleagues (2012) that combined elements of response compatibility tasks and oddball tasks (see Fig. 6). On each trial subjects responded to an imperative stimulus that indicated which of two buttons to press. The imperative stimulus was flanked on either side by distractors that mapped either to the same response or to the other response. Button presses were followed immediately by the appearance of a “standard” stimulus on most trials, a “novel” stimulus on a small, random subset of correct trials, and a “target” stimulus on only three trials, which the subjects were instructed to count silently.
Fig. 6

Speeded response time task. On each trial subjects were required to fixate on an initial stimulus and then to respond to an imperative stimulus consisting of a central target stimulus that was mapped to either a left or right button press, flanked by two distractor stimuli on each side. A second stimulus appeared shortly after the participant’s response. On most trials the stimulus was an upward-pointing triangle (standard). Three upside-down triangles appeared at equal intervals over the duration of the experiment (target). And a picture of an object or an animal occasionally appeared following correct trials (novel). Participants were instructed to press a third button upon appearance of the target stimuli. Republished with permission of The Journal of Neuroscience: The Official Journal of the Society for Neuroscience, from Surprise and Error: Common Neuronal Architecture for the Processing of Errors and Novelty, Wessel et al., 32, 2012; permission conveyed through Copyright Clearance Center, Inc.

We simulated the task with an RNN composed of nine input units, corresponding to the eight possible imperative stimuli in the task (four stimuli in which the responses were compatible with the flanker stimuli, and four stimuli in which the responses were incompatible with the flanker stimuli), and one unit indicating that the response had been completed; four output units, corresponding to the left and right button presses and the novel and standard stimuli; and 10 units in the hidden and context layers, each. Each trial was represented in two time steps: On the first time step the input corresponding to the stimulus for that trial was activated, and on the second time step the input unit corresponding to the selected response was activated. Networks were trained to predict on each trial the response that was executed to the imperative stimulus and the stimulus that would occur following response completion. Five hundred networks were trained 3,000 times each on data derived from the subject accuracy levels reported in Wessel et al. (2012). For example, if accuracy for a particular stimulus was 90%, then the model was trained to predict the correct response on 90% of these trials and the incorrect response on the remaining 10% of the trials, randomly interleaved.

After training, we tested the response of the network to trials with correct responses, error responses, standard stimuli, and novel stimuli. The discrepancy signal at each time step was quantified as the Euclidean distance in a four-dimension state space between the output unit activation levels (indicating what the network predicted) and the output unit target values (indicating what actually occurred; see Equation 3). Response errors were evaluated at Step 1 and the novel stimuli at Step 2. The network produced relatively larger discrepancy signals to errors and to novel stimuli, which occurred less often than correct responses and standard stimuli, respectively, and were therefore relatively unexpected (see Fig. 7, left panel).
Fig. 7

Simulated and empirical human event-related brain potential (ERP) data. Left panel: Simulated ERP amplitudes. Discrepancy signals are shown for correct and error responses on Step 1 of the sequence for standard trials, and on Step 2 of the sequence of standard and novel trials. Conditions are labeled according to their associated ERP components: correct-related negativity (CRN), error-related negativity (ERN), and N2. Units are arbitrary. Error bars show standard error of the mean. Right panel: ERPs time-locked to response onset, recorded over central areas of the scalp (channel Cz). Note that the novel and standard stimuli appear 10 ms after the response. Separate ERPs are presented for error trials (solid black), novel correct trials (solid gray), and standard correct trials (dashed grey). Negative is plotted up by convention. The ERN peaks on error trials about 50 ms following the response. A smaller negative deflection occurring at the same time on correct (standard) trials is termed the CRN (Gehring et al., 2012). The N2 peaks about 300 ms following the onset of the novel stimuli. Republished with permission of The Journal of Neuroscience: The Official Journal of the Society for Neuroscience, from Surprise and Error: Common Neuronal Architecture for the Processing of Errors and Novelty, Wessel et al., 32, 2012; permission conveyed through Copyright Clearance Center, Inc.

Notably, this simple measure conforms to the behavior of two ERP components related to surprise and error processing (see Fig. 7, right panel). As commonly observed, Wessel and colleagues (2012) found a large ERN following error commission to the imperative stimuli and a large N2 elicited by the infrequently occurring novel stimuli (see Fig. 7, right panel). Our simulations agree with these findings but suggest a more nuanced explanation for their functional significance: The N2 and ERN indicate that the task sequence is not unfolding as predicted.

FMRI BOLD signals in humans

Importantly, although model details such as the number of units in each layer necessarily vary from simulation to simulation, the same model architecture and computational principles were used to simulate both the behavior of rats in a sequence production task and of humans in a speeded reaction time task. Taken together, the results suggest that ACC tracks the execution of goal-directed action sequences as a distributed pattern of activity across multiple cells and produces discrepancy signals when ongoing events deviate from the given plan.

These ideas share much in common with an influential model of ACC function called the predicted response-outcome (PRO) model which, rather than holding ACC responsible for detecting unexpected events, holds that it detects the omission of expected events (Alexander & Brown, 2011). To compare the two models directly, we simulated the fMRI BOLD response to events in the stop-change task (Brown & Braver, 2005) that have previously been simulated using the PRO model (Alexander & Brown, 2011). These data therefore provide a useful benchmark for comparison.

On each trial of the stop-change task, subjects are required to press a button as quickly as possible to an imperative stimulus, while occasionally inhibiting that response in favor of an alternative choice when presented with a second “change” stimulus; crucially, the timing between the imperative stimulus and the change stimulus is controlled such that some trials are harder than others. Participants are first presented with a cue indicating the trial difficulty (low error vs. high error), followed by an arrow indicating which of two buttons to press (see Fig. 8). On the change trials, the arrow is followed by a second arrow that indicates to participants to cancel the initial response and press the other button; the remaining trials are called “go trials.” Go and change trials occur with 67% and 33% probability, respectively. Trial difficulty is determined for each subject individually according to a staircase procedure that adjusts the time between the imperative cue and the change signal (the “change signal delay”; CSD), enforcing a 50% error rate on difficult change trials and a 4% error rate on easy change trials.
Fig. 8

The stop-change task. See text for a description. From “Learned Predictions of Error Likelihood in the Anterior Cingulate Cortex, “ by J. W. Brown and T. S. Braver, 2005 Science, 307, 1118–1121. Reprinted with permission from AAAS

Figure 9a presents the results of the PRO model simulation to key task events, which parallel the empirical BOLD signal findings (not shown). As can be seen, the model yields four essential findings. First, larger signals are produced to correct trials requiring a change in response compared to correct trials requiring no such change, yielding a contrast related to response conflict. Second, larger signals are elicited by errors than by correct responses, yielding an error signal. Third, errors occurring on less difficult trials (“low error likelihood”; LEL) produce larger signals than errors occurring on harder trials (“high error likelihood”; HEL). Finally, larger signals are elicited by correct go-responses on HEL trials compared to correct go-responses on LEL trials, called the “error-likelihood” effect. As described elsewhere, these signals all result in the PRO model from the omission of expected events (Alexander & Brown, 2011).
Fig. 9

Simulated human blood-oxygen-level dependent (BOLD) response to events in the stop-change task. a Simulated BOLD response by the predicted response-outcome model of anterior cingulate cortex (Alexander & Brown, 2011). The conflict effect is derived by comparing correct change trials with correct go trials. The error effect is derived by comparing error trials with correct trials. The error unexpectedness effect is derived by comparing errors on high error-likelihood (HEL) trials with errors on low error-likelihood (LEL) trials. The error likelihood effect is derived by comparing HEL trials with LEL trials. Adapted from “Medial Prefrontal Cortex as an Action-Outcome Predictor,” by W. H. Alexander and J. W. Brown, 2011, Nature Neuroscience, 14, 1338–1344. Copyright 2011 by Macmillan Publishers Ltd. Adapted with permission. b Simulated BOLD response by the recurrent neural network model for the same four comparisons shown in the left panel. Cont = contrast; Err = error; Cor = correct. High and Low indicate conditions with high or low conflict, error expectancy, and error likelihood, respectively, for these three comparisons. Error bars indicate standard error of the mean (SEM)

We simulated task performance with an RNN composed of five input units, corresponding to the two difficulty cues (hard, easy), the two arrow directions (pointing left, pointing right), and the change signal; five output units, corresponding to the two arrow directions, the change stimulus, and the two responses (left button press, right button press); and five units for each of the hidden and context layers. Each trial was simulated as a sequence of three time steps that indicated the difficulty cue (Step 1), the arrow direction (Step 2), and the appearance of the change stimulus (Step 3). Based on this input the network was trained to predict the direction of the subsequent arrow stimulus (Step 1), whether or not the change signal would appear (Step 2), and the forthcoming button press (Step 3). Two hundred networks starting with different initial weights were trained for 8,000 trials each, with target values derived from the empirical error rates for each condition, namely, 4% errors on easy change trials and 50% error on hard change trials, pseudorandomly interspersed. As with the actual experiment, 33% of all of the trials were change trials (Brown & Braver, 2005). In keeping with evidence that the BOLD response in this task is sensitive to multiple events within each trial (Nieuwenhuis, Schweizer, Mars, Botvinick, & Hajcak, 2007), the discrepancy signal on each trial was taken as the average discrepancy across the three-step sequence.

Across three of the four contrasts, the discrepancy signals produced by the RNN qualitatively replicated the output of the PRO model (and the associated empirical observations of ACC BOLD response; see Fig. 9b). This correspondence stems from the fact that both models operate on similar principles. The activity level of each output unit of the RNN reflects the probability that a particular event will occur at that time step, as determined by the frequency of occurrence of that event at that time during the course of training. Thus, whereas the PRO model produces a stronger response on error trials to the unexpected omission of the correct response, the RNN model produces a larger discrepancy signal to the unexpected commission of the error itself. For the same reason, errors that are unexpected produce a larger discrepancy signal than errors that are expected. And because correct responses occur less frequently on change trials than on go trials, they elicit larger discrepancy signals on the former compared to the latter, producing a conflict effect.

Unlike the PRO model, however, the RNN does not consistently predict a larger discrepancy signal to correct responses on go trials with high error likelihood compared to correct responses on go trials with low error likelihood. This result obtains because the go signal is equally predictive of a correct outcome on both HEL and LEL trials, yielding discrepancy signals of about equal magnitude. However, we note that the empirical result itself is controversial, which we address in the discussion below.

Taken together, our simulations suggest that ACC tracks the execution of extended behaviors and produces large discrepancy signals to unexpected events that deviate from the task domain. In online supplementary materials, we illustrate how these discrepancy signals could be used to regulate task performance, as discussed below.

Discussion

ACC research has been complicated by a plethora of challenging empirical findings such as the common observation that ACC neurons tend to respond to multiple task-related events (Ebitz & Hayden, 2016; Holroyd & Yeung, 2011). Here, we elucidate these findings with a computational approach that is motivated by the long-standing principle in cognitive psychology (Rumelhart & McClelland, 1986) and computational neuroscience (Churchland & Sejnowski, 1992) that neural functions are encoded as distributed representations across ensembles of neurons (e.g., Fusi et al., 2016; Yuste, 2015). Based in part on our argument that ACC is concerned with the execution of extended, goal-directed action sequences, which we have articulated elsewhere (Holroyd & Yeung, 2012; see also Holroyd & McClure, 2015), we trained a model to predict the behaviors of human and nonhuman animals in sequential tasks. The simulations illustrate how the collective activity of ACC neurons, encoded as a distributed representation across ensembles of units, can track the progression of an agent throughout the execution of goal-directed action sequences, in line with observations of ensemble ACC activity in rats and other animals (e.g., Baeg et al., 2003; Balaguer-Ballester et al., 2011; Blanchard et al., 2015; Cowen et al., 2012; Cowen & McNaughton, 2007; Durstewitz et al., 2010; Euston & McNaughton, 2006; Fujisawa et al., 2008; Hayden, Pearson, et al., 2011; Hyman et al., 2012, 2013; Lapish et al., 2008; Ma, Hyman, Lindsay, et al., 2014; Ma, Hyman, Phillips, et al., 2014; Remondes & Wilson, 2013). Further, because the model was trained to predict task events as they occurred, the simulations were sensitive to unexpected deviations from each sequence, consistent with ubiquitous evidence from fMRI and ERP studies that ACC responds to surprising or conflict-eliciting events (Alexander & Brown, 2011; Botvinick et al., 2001, 2004; Braver et al., 2001; Cavanagh et al., 2012; Donkers et al., 2005; Ferdinand et al., 2012; Ferdinand & Opitz, 2014; Folstein & Van Petten, 2008; Gehring et al., 1992; HajiHosseini & Holroyd, 2013; Holroyd, 2004; Holroyd et al., 2008; Jessup et al., 2010; Jia et al., 2007; Kopp & Wolff, 2000; Mathalon et al., 2003; Nee et al., 2011; Oliveira et al., 2007; O’Reilly et al., 2013; Sams et al., 1983; Silvetti et al., 2011; Y. Wang et al., 2004; Warren & Holroyd, 2012; Wessel et al., 2012; Yeung, 2013; Yeung et al., 2004).

Although a handful of computational models have simulated ACC function previously (e.g., Alexander & Brown, 2011; Botvinick et al., 2001; Brown & Braver, 2005; Holroyd & Coles, 2002, 2008; Holroyd & McClure, 2015; Holroyd, Yeung, Coles, & Cohen, 2005; Khamassi, et al., 2011; Silvetti et al., 2011; Verguts, 2017; Verguts et al., 2015; Yeung et al., 2004), to our knowledge none of these models have examined the essential role of ACC in encoding task execution as distributed representations that evolve dynamically with time (Balaguer-Ballester et al., 2011; Caracheo et al., 2013; Durstewitz et al., 2010; Lapish et al., 2008). Our simulations are thus the first to provide a formal account of this crucial aspect of ACC function. In so doing, the simulations also yield—without any additional assumptions—the surprise, conflict, and error signals that are the mainstay of many previous ACC models (e.g., Alexander & Brown, 2011; Botvinick et al., 2001; Silvetti et al., 2011) but explains these events as resulting from unexpected deviations in the execution of goal-directed action sequences. Consonant with recent calls for the development of theoretical frameworks that bridge studies between human and nonhuman animals (Badre, Frank, & Moore, 2015), these simulations provide a common framework for relating single-cell findings associated with task execution in nonhuman animals with ubiquitous electrophysiological and neuroimaging findings in humans.

Relationship to the PRO model

We simulated the fMRI BOLD response in the stop-change task (Brown & Braver, 2005) as a benchmark for comparison with one of the most comprehensive and successful models of ACC function, the PRO model (Alexander & Brown, 2011; see also Silvetti et al., 2011, for comparable findings). Like the PRO model, the simulations successfully accounted for what are arguably the most salient aspects of ACC activity—conflict, error, and surprise signals (see Fig. 9b). Yet in contrast to the PRO model, our simulations failed to capture the error likelihood effect previously observed in the original stop-change study (Brown & Braver, 2005). That effect is specifically associated with the contrast between correct-go, HEL trials and correct-go, LEL trials in the stop-change task (Brown & Braver, 2005; see Fig. 9a). This divergence between the PRO and RNN models appears to reflect a fundamental difference between their computational properties. The PRO model accounts for the error likelihood effect with separate units for the correct and incorrect response that are simultaneously activated on HEL trials; because subsequent production of the correct response is also associated with the omission of the expected incorrect response, a larger surprise signal is produced on HEL trials than on LEL trials, the latter of which predict mostly correct responses. By contrast, the RNN model predicts what should occur following the appearance of the initial go stimulus, which is either the change stimulus on change trials or the response on go trials. Because these two events occur about equally often for both the HEL and LEL conditions—for both conditions, the change stimulus occurs on 33% of the trials, and the correct response follows on nearly all of the go trials—the RNN produces about equally large discrepancy signals to go-correct trials across the HEL and LEL conditions.

With respect to predicting univariate ACC signals, in our view the PRO and RNN models are more alike than different. Nevertheless, the error likelihood effect may constitute diagnostic evidence between the two models. In this regard it is worth noting that the effect has received less empirical support than the other observations of ACC function related to conflict, errors, and surprise. Although the effect is reproducible (Alexander & Brown, 2010), it appears to be relatively weak (Brown, 2009), and varies substantially across individuals in terms of their sensitivity to risk (Alexander, Fukunaga, Finn, & Brown, 2015; Brown & Braver, 2007; cf. Brown & Braver, 2008). Further, one laboratory failed to replicate the result in two fMRI experiments (Nieuwenhuis, Schweizer, Mars, Botvinick, & Hajcak, 2007) and also found ERP evidence that is inconsistent with the theory (Yeung & Nieuwenhuis, 2009; see also Hammer, Rautzenberg, Heldmann, & Münte, 2011). These considerations suggest that the error likelihood effect warrants further investigation.

By contrast, only the RNN model has yet been shown to account for the type of multivariate activity illustrated in Figs. 4 and 5. A useful exercise would entail training the PRO, RNN, and other models of ACC function on the same task and then applying representational similarity analysis to compare the network properties of each of the models against empirical data (Kriegeskorte & Kievit, 2013; Kriegeskorte, Mur, & Bandettini, 2008).

Discrepancy signal function

Although these simulations illustrate the role of ACC in tracking the execution of goal-directed action sequences and in detecting discrepancies between predicted and actual events in the sequence, they do not apply the discrepancy signals for any functional purpose. A perplexing characteristic of ACC is that despite clear evidence that such discrepancy signals are correlated with adaptive adjustments to behavior, suggesting that ACC is involved in behavioral regulation (Cavanagh & Shackman, 2015), lesions to ACC only minimally impair these adjustments, indicating that the trial-to-trial changes in performance are actually carried out by other brain areas (Holroyd & Yeung, 2012). For this reason, we have argued elsewhere that ACC is responsible for motivating control over the execution of extended action sequences rather than in adjusting behavior from one moment to the next (Holroyd & Yeung, 2012), a property that we have implemented in a previous model of ACC (Holroyd & McClure, 2015). In keeping with this perspective, additional simulations (see online Supplementary Materials) suggest that the discrepancy signals might regulate a control signal that is used by an action production system to produce actions that comport to the task objectives, for example, by providing contextual information that prevents against the production of capture errors and other action slips during the execution of goal-directed action sequences. Alternatively, the discrepancy signals might promote exploration of alternative task strategies (Donoso, Collins, & Koechlin, 2014; Kuwabara et al., 2014; Schuck et al., 2015; Tervo et al., 2014), in parallel to a proposed switching mechanism regulated by tonic dopamine levels in ACC (Holroyd & McClure, 2015) or could provide an update or training signal to the predictive model mediated by ACC (e.g., O’Reilly et al., 2013). We believe that the evidence to date is in insufficient to decide between these possibilities, which in any case are not mutually exclusive.

Future directions

A promising avenue for investigation would entail instantiating the model in a more biologically realistic network that incorporates finer temporal dynamics into the recurrent activity (Sussillo, 2014), in line with previous efforts to simulate the role of prefrontal cortex in response generation and decision making (Erlich & Brody, 2013; Mante, Sussillo, Shenoy, & Newsome, 2013; Moody, Wise, di Pellegrino, & Zipser, 1998; Nakahara & Doya, 1998; X.-J. Wang, 2008). These networks typically exhibit complex temporal dynamics that are amenable to investigation using an arsenal of mathematical tools from nonlinear dynamical systems analysis (e.g., Durstewitz & Deco, 2008; Sussillo, 2014; Sussillo & Barak, 2013; Wolf, Engelken, Puelma-Touzel, Weidinger, & Neef, 2014). Although notoriously difficult to train (Sussillo, 2014), recent advances (e.g., Martens, 2010; Martens & Sutskever, 2011; Song, Yang, & Wang, 2016) have been encouraging (Abbott, DePasquale, & Memmesheimer, 2016; Ardid & Wang, 2013; Song et al., 2016; Sussillo & Abbott, 2009). For example, networks of sparsely and recurrently connected spiking neurons have been trained using a biologically realistic reward signal to compute rule-specific decisions based on information maintained in working memory (Hoerzer, Legenstein, & Maas, 2014), though it remains an open question whether or not such principles can be utilized to simulate the neural mechanisms of hierarchically organized action sequences (for efforts in this directions, see Namikawa, Nishimoto, & Tani, 2011; Nishimoto & Tani, 2004; Rao & Sejnowski, 2000; Starzyk & He, 2007; Yamashita & Tani, 2008).

In other work, we have simulated the role of ACC in learning the value of and selecting tasks based on principles of hierarchical reinforcement learning (HRL; Holroyd & McClure, 2015). An obvious next step would therefore be to integrate the RNN model of ACC into the HRL framework. A hybrid model of ACC would see the action policy for each option implemented in a separate RNN and would select and execute the RNNs based on their learned reward values. This approach could help resolve a debate about whether the execution of hierarchically organized action sequences are better represented with connectionist-style RNNs (Botvinick, 2005; Botvinick & Plaut, 2002, 2004), or with rule-following symbolic processes that explicitly organize action sequences according to goals and sub-goals (Cooper & Shallice, 2000, 2006). An integrated model would allow for options to be flexibly combined in novel configurations according to principles of HRL (Hengst, 2012) while retaining the strengths of the connectionist implementation, such as the ability to generalize learned structure across contextual domains (Botvinick & Plaut, 2002, 2004). These two approaches are not actually incompatible (Cooper & Shallice, 2006), and recent efforts have seen the development of hybrid models that explicitly encode goal and subgoal states in RNNs (Cooper, Ruh, & Mareschal, 2014). Related work has investigated how an RNN can implement an “actor-critic” architecture that executes action sequences according to principles of reinforcement learning (Ruh, Cooper, & Mareschal, 2005; see also Cooper & Glasspool, 2001).

Conclusion

Decades of research on ACC have revealed a colorful but bewildering landscape of empirical findings. Our simulations show the proverbial forest for the trees, where the trees consist of observations of ACC activity to individual task events and the forest is the dynamically evolving relationship between these observations. This proposal is cast in a formal theoretical framework that accounts for existing ensemble activity in ACC of nonhuman animals as well as for conflict, surprise, and related signals commonly observed in ACC in human functional neuroimaging and electrophysiological experiments. In the context of previous work on HRL (Holroyd & McClure, 2015; Holroyd & Yeung, 2012), these efforts point toward a unified account of the role of ACC in action selection and performance monitoring.

Footnotes

  1. 1.

    Unit sensitivity was determined as follows. Press units were defined as units with high average activity to the nine press actions, as determined by the following permutation analysis. The distribution of average unit activities to all combinations of nine out of the 21 possible events (seven events across each of three sequences) was estimated by selecting 10,000 different permutations of nine randomly chosen events, and then averaging the activities of each unit across each set of nine events. The threshold (.77) was defined as the mean (0.53) plus the standard deviation (0.24) of the distribution. Orientation units were identified as units with high average activity to the six orientation actions, following an analogous permutation approach (across multiple combinations of 6 non-specific events), which yielded a threshold of .80 (mean = 0.53, standard deviation = 0.27). Sequence units were defined as units with higher average activity to the first event in each sequence (i.e., to the three sequence cues) relative to the average activity to all the remaining events in the sequence. (Note that this definition of sequence unit refers to the hidden layer, which should not be confused with the sequence units in the input layer that indicate which of the three sequences to execute.) For the permutation analysis, each units’ activity averaged across 18 randomly selected events was subtracted from the average activity of three randomly selected events, which yielded a threshold of 0.71 (M = −0.001, SD = 0.71). Ramping units were defined as units with average activity to each of three pairs of orient (O1, O2, O3) and press (P1, P2, P3) actions—that is, (O1, P1), (O2, P2), (O3, P3)—that systematically increased by more than .1 across each successive pair of pairs—that is, activity(O2, P2) > .1 + activity(O1, P1) and activity(O3, P3) > .1 + activity(O2, P2). Simulation results were broadly consistent across a range of hidden layer sizes (20, 50, and 100 units).

  2. 2.

    Figure 5e indicates the Euclidean distance between representations and Fig. 5f indicates the Mahalanobis distance. The Mahalanobis distance corresponds to Euclidean distance when each axis is rescaled to have unit variance. Further, the control condition in Fig. 5f is derived from a within-subjects random shuffling of trials across conditions. Because each network (“subject”) in the simulation performed each sequence only once, the same within-subjects comparison for the simulation was not possible. Therefore, the control condition in Fig. 5e reflects the Euclidean distance between hidden unit network representations associated with different lever presses occurring at the same serial position across sequences.

Notes

Acknowledgements

The order of authorship is arbitrary; both authors made equal contributions to the research and preparation of this article. This research was supported in part by funding from the Canada Research Chairs program and a Natural Sciences and Engineering Research Council of Canada Discovery Grant (312409–05) awarded to C.B.H.

Supplementary material

13423_2017_1280_Fig10_ESM.gif (94 kb)
ESM 1

(GIF 93 kb)

13423_2017_1280_MOESM1_ESM.tif (664 kb)
High resolution image (TIF 663 kb)
13423_2017_1280_Fig11_ESM.gif (44 kb)
ESM 2

(GIF 43 kb)

13423_2017_1280_MOESM2_ESM.tif (225 kb)
High resolution image (TIF 225 kb)

References

  1. Abbott, L. F., DePasquale, B., & Memmesheimer, R.-M. (2016). Building functional networks of spiking model neurons. Nature Neuroscience, 19, 350–355. doi: 10.1038/nn.4241
  2. Alexander, W. H., & Brown, J. W. (2010). Competition between learned reward and error outcome predictions in anterior cingulate cortex. NeuroImage, 49, 3210–3218. doi: 10.1111/j.1756-8765.2010.01085.x PubMedCrossRefGoogle Scholar
  3. Alexander, W. H., & Brown, J. W. (2011). Medial prefrontal cortex as an action-outcome predictor. Nature Neuroscience, 14, 1338–1344. doi: 10.1038/nn.2921 PubMedPubMedCentralCrossRefGoogle Scholar
  4. Alexander, W. H., Fukunaga, R., Finn, P., & Brown, J. W. (2015). Reward salience and risk aversion underlie differential ACC activity in substance dependence. NeuroImage: Clinical, 8, 59–71. doi: 10.1016/j.nicl.2015.02.025 CrossRefGoogle Scholar
  5. Amiez, C., Joseph, J.-P., & Procyk, E. (2005). Anterior cingulate error-related activity is modulated by predicted reward. European Journal of Neuroscience, 21, 3447–3452. doi: 10.1111/j.1460-9568.2005.04170.x PubMedPubMedCentralCrossRefGoogle Scholar
  6. Angel, R. W. (1976). Efference copy in the control of movement. Neurology, 26, 1164–1168. doi: 10.1212/wnl.26.12.1164 PubMedCrossRefGoogle Scholar
  7. Ardid, S., & Wang, X.-J. (2013). A tweaking principle for executive control: Neuronal circuit mechanism for rule-based task switching and conflict resolution. Journal of Neuroscience, 33, 19504–19517. doi: 10.1523/jneurosci.1356-13.2013 PubMedCrossRefGoogle Scholar
  8. Backus, D. A., Ye, S., Russo, G. S., & Crutcher, M. D. (2001). Neural activity correlated with the preparation and execution of visually guided arm movements in the cingulate motor area of the monkey. Experimental Brain Research, 140, 182–189. doi: 10.1007/s002210100807 PubMedCrossRefGoogle Scholar
  9. Badre, D., Frank, M. J., & Moore, C. I. (2015). Interactionist neuroscience. Neuron, 88, 855–860. doi: 10.1016/j.neuron.2015.10.021 PubMedCrossRefGoogle Scholar
  10. Baeg, E. H., Kim, Y. B., Huh, K., Mook-Jung, I., Kim, H. T., & Jung, M. W. (2003). Dynamics of population code for working memory in the prefrontal cortex. Neuron, 40, 177–188. doi: 10.1016/s0896-6273(03)00597-x PubMedCrossRefGoogle Scholar
  11. Balaguer-Ballester, E., Lapish, C. C., Seamans, J. K., & Durstewitz, D. (2011). Attracting dynamics of frontal cortex ensembles during memory-guided decision-making. PLoS Computational Biology, 7, e1002057. doi: 10.1371/journal.pcbi.1002057 PubMedPubMedCentralCrossRefGoogle Scholar
  12. Berns, G. S., Cohen, J. D., & Mintun, M. A. (1997). Brain regions responsive to novelty in the absence of awareness. Science, 276, 1272–1275. doi: 10.1126/science.276.5316.1272 PubMedCrossRefGoogle Scholar
  13. Blanchard, T. C., Strait, C. E., & Hayden, B. Y. (2015). Ramping ensemble activity in dorsal anterior cingulate neurons during persistent commitment to a decision. Journal of Neurophysiology, 114, 2439–2449.PubMedPubMedCentralCrossRefGoogle Scholar
  14. Botvinick, M. M. (2005). Modeling routine sequential action with recurrent neural nets. In J. J. Bryson, T. J. Prescott, & A. K. Seth (Eds.), Modeling natural action selection (pp. 180–187). Cambridge: Cambridge University Press.Google Scholar
  15. Botvinick, M. M., Braver, T. S., Barch, D. M., Carter, C. S., & Cohen, J. D. (2001). Conflict monitoring and cognitive control. Psychological Review, 108, 624–652. doi: 10.1037/0033-295x.108.3.624 PubMedCrossRefGoogle Scholar
  16. Botvinick, M. M., Cohen, J. D., & Carter, C. S. (2004). Conflict monitoring and anterior cingulate cortex: An update. Trends in Cognitive Sciences, 8, 539–546. doi: 10.1016/j.tics.2004.10.003 PubMedCrossRefGoogle Scholar
  17. Botvinick, M., & Plaut, D. C. (2002). Representing task context: Proposals based on a connectionist model of action. Psychological Research, 66, 298–311. doi: 10.1007/s00426-002-0103-8 PubMedCrossRefGoogle Scholar
  18. Botvinick, M., & Plaut, D. C. (2004). Doing without schema hierarchies: A recurrent connectionist approach to normal and impaired routine sequential action. Psychological Review, 111, 395–429. doi: 10.1037/0033-295x.111.2.395 PubMedCrossRefGoogle Scholar
  19. Braver, T. S., Barch, D. M., Gray, J. R., Molfese, D. I., & Snyder, A. (2001). Anterior cingulate cortex and response conflict: Effects of frequency, inhibition and errors. Cerebral Cortex, 11, 825–836. doi: 10.1093/cercor/11.9.825 PubMedCrossRefGoogle Scholar
  20. Brown, J. W. (2009). Conflict effects without conflict in anterior cingulate cortex: Multiple response effects and context specific representations. NeuroImage, 47, 334–341. doi: 10.1016/j.neuroimage.2009.04.034 PubMedPubMedCentralCrossRefGoogle Scholar
  21. Brown, J. W., & Braver, T. S. (2005). Learned predictions of error likelihood in the anterior cingulate cortex. Science, 307, 1118–1121. doi: 10.1126/science.1105783 PubMedCrossRefGoogle Scholar
  22. Brown, J. W., & Braver, T. S. (2007). Risk prediction and aversion by anterior cingulate cortex. Cognitive, Affective, & Behavioral Neuroscience, 7, 266–277. doi: 10.3758/cabn.7.4.266 CrossRefGoogle Scholar
  23. Brown, J. W., & Braver, T. S. (2008). A computational model of risk, conflict, and individual difference effects in the anterior cingulate cortex. Brain Research, 1202, 99–108. doi: 10.1016/j.brainres.2007.06.080 PubMedCrossRefGoogle Scholar
  24. Bryden, D. W., Johnson, E. E., Tobia, S. C., Kashtelyan, V., & Roesch, M. R. (2011). Attention for learning signals in anterior cingulate cortex. Journal of Neuroscience, 31, 18266–18274. doi: 10.1523/jneurosci.4715-11.2011 PubMedPubMedCentralCrossRefGoogle Scholar
  25. Bush, G. (2009). Dorsal anterior midcingulate cortex: Roles in normal cognition and disruption in attention-deficit/hyperactivity disorder. In B. A. Vogt (Ed.), Cingulate neurobiology and disease (pp. 245–274). Oxford: Oxford University Press.Google Scholar
  26. Cai, X., & Padoa-Schioppa, C. (2012). Neuronal encoding of subjective value in dorsal and ventral anterior cingulate cortex. Journal of Neuroscience, 32, 3791–3808. doi: 10.1523/jneurosci.3864-11.2012 PubMedPubMedCentralCrossRefGoogle Scholar
  27. Caracheo, B. F., Emberly, E., Hadizadeh, S., Hyman, J. M., & Seamans, J. K. (2013). Abrupt changes in the patterns and complexity of anterior cingulate cortex activity when food is introduced into an environment. Frontiers in Neuroscience, 7, 74. doi: 10.3389/fnins.2013.00074 PubMedPubMedCentralCrossRefGoogle Scholar
  28. Cavanagh, J. F., & Shackman, A. J. (2015). Frontal midline theta reflects anxiety and cognitive control: Meta-analytic evidence. Journal of Physiology, Paris, 109, 3–15. doi: 10.1016/j.jphysparis.2014.04.003 PubMedCrossRefGoogle Scholar
  29. Cavanagh, J. F., Zambrano-Vazquez, L., & Allen, J. J. B. (2012). Theta lingua franca: A common mid-frontal substrate for action monitoring processes. Psychophysiology, 49, 220–238. doi: 10.1111/j.1469-8986.2011.01293.x PubMedCrossRefGoogle Scholar
  30. Churchland, P. S., & Sejnowski, T. J. (1992). The computational brain. Cambridge: MIT Press.Google Scholar
  31. Cleeremans, A., & McClelland, J. L. (1991). Learning the structure of event sequences. Journal of Experimental Psychology: General, 120, 235–253. doi: 10.1037/0096-3445.120.3.235 CrossRefGoogle Scholar
  32. Cooper, R., & Glasspool, D. (2001). Learning action affordances and action schemas. In R. M. French & J. P. Sougné (Eds.), Connectionist models of learning, development and evolution (pp. 133–42). London: Springer. doi: 10.1007/978-1-4471-0281-6_14 CrossRefGoogle Scholar
  33. Cooper, R. P., Ruh, N., & Mareschal, D. (2014). The goal circuit model: A hierarchical multi-route model of the acquisition and control of routine sequential action in humans. Cognitive Science, 38, 244–274. doi: 10.1111/cogs.12067 PubMedCrossRefGoogle Scholar
  34. Cooper, R., & Shallice, T. (2000). Contention scheduling and the control of routine activities. Cognitive Neuropsychology, 17, 297–338. doi: 10.1080/026432900380427 PubMedCrossRefGoogle Scholar
  35. Cooper, R. P., & Shallice, T. (2006). Hierarchical schemas and goals in the control of sequential behavior. Psychological Review, 113, 887–916. doi: 10.1037/0033-295x.113.4.887 PubMedCrossRefGoogle Scholar
  36. Cowen, S. L., Davis, G. A., & Nitz, D. A. (2012). Anterior cingulate neurons in the rat map anticipated effort and reward to their associated action sequences. Journal of Neurophysiology, 107, 2393–2407. doi: 10.1152/jn.01012.2011 PubMedCrossRefGoogle Scholar
  37. Cowen, S. L., & McNaughton, B. L. (2007). Selective delay activity in the medial prefrontal cortex of the rat: Contribution of sensorimotor information and contingency. Journal of Neurophysiology, 98, 303–316. doi: 10.1152/jn.00150.2007 PubMedCrossRefGoogle Scholar
  38. Cunningham, J. P., & Yu, B. M. (2014). Dimensionality reduction for large-scale neural recordings. Nature Neuroscience, 17, 1500–1509. doi: 10.1038/nn.3776 PubMedPubMedCentralCrossRefGoogle Scholar
  39. Davis, K. D., Hutchinson, W. D., Lozano, A. M., Tasker, R. R., & Dostrovsky, J. O. (2000). Human anterior cingulate cortex neurons modulated by attention-demanding tasks. Journal of Neurophysiology, 83, 3575–3577.PubMedCrossRefGoogle Scholar
  40. Davis, K. D., Taylor, K. S., Hutchison, W. D., Dostrovsky, J. O., McAndrews, M. P., Richter, E. O., & Lozano, A. M. (2005). Human anterior cingulate cortex neurons encode cognitive and emotional demands. Journal of Neuroscience, 25, 8402–8406. doi: 10.1523/jneurosci.2315-05.2005 PubMedCrossRefGoogle Scholar
  41. Donkers, F. C. L., Nieuwenhuis, S., & van Boxtel, G. J. M. (2005). Mediofrontal negativities in the absence of responding. Cognitive Brain Research, 25, 777–787. doi: 10.1016/j.cogbrainres.2005.09.007 PubMedCrossRefGoogle Scholar
  42. Donoso, M., Collins, A. G. E., & Koechlin, E. (2014). Foundations of human reasoning in the prefrontal cortex. Science, 344, 1481–1486. doi: 10.1126/science.1252254 PubMedCrossRefGoogle Scholar
  43. Durstewitz, D., & Deco, G. (2008). Computational significance of transient dynamics in cortical networks. European Journal of Neuroscience, 27, 217–227. doi: 10.1111/j.1460-9568.2007.05976.x PubMedCrossRefGoogle Scholar
  44. Durstewitz, D., Seamans, J. K., & Sejnowski, T. J. (2000). Neurocomputational models of working memory. Nature Neuroscience, 3, 1184–1191. doi: 10.1038/81460 PubMedCrossRefGoogle Scholar
  45. Durstewitz, D., Vittoz, N. M., Floresco, S. B., & Seamans, J. K. (2010). Abrupt transitions between prefrontal neural ensemble states accompany behavioral transitions during rule learning. Neuron, 66, 438–448. doi: 10.1016/j.neuron.2010.03.029 PubMedCrossRefGoogle Scholar
  46. Ebitz, R. B., & Hayden, B. Y. (2016). Dorsal anterior cingulate: A Rorschach test for cognitive neuroscience. Nature Neuroscience, 19, 1278–1279.PubMedCrossRefGoogle Scholar
  47. Ebitz, R. B., & Platt, M. L. (2015). Neuronal activity in primate dorsal anterior cingulate cortex signals task conflict and predicts adjustments in pupil-linked arousal. Neuron, 85, 628–640. doi: 10.1016/j.neuron.2014.12.053 PubMedPubMedCentralCrossRefGoogle Scholar
  48. Eimer, M., Goschke, T., Schlaghecken, F., & Stürmer, B. (1996). Explicit and implicit learning of event sequences: Evidence from event-related brain potentials. Journal of Experimental Psychology: Learning, Memory, and Cognition, 22, 970–987. doi: 10.1037/0278-7393.22.4.970 PubMedGoogle Scholar
  49. Elman, J. L. (1990). Finding structure in time. Cognitive Science, 14, 179–211. doi: 10.1207/s15516709cog1402_1 CrossRefGoogle Scholar
  50. Erlich, J. C., & Brody, C. D. (2013). What to do and how. Nature, 503, 45–47. doi: 10.1038/503045a PubMedCrossRefGoogle Scholar
  51. Euston, D. R., Gruber, A. J., & McNaughton, B. L. (2012). The role of medial prefrontal cortex in memory and decision making. Neuron, 76, 1057–1070. doi: 10.1016/j.neuron.2012.12.002 PubMedPubMedCentralCrossRefGoogle Scholar
  52. Euston, D. R., & McNaughton, B. L. (2006). Apparent encoding of sequential context in rat medial prefrontal cortex is accounted for by behavioral variability. Journal of Neuroscience, 26, 13143–13155. doi: 10.1523/jneurosci.3803-06.2006 PubMedCrossRefGoogle Scholar
  53. Falkenstein, M., Hohnsbein, J., Hoormann, J., & Blanke, L. (1990). Effects of errors in choice reaction tasks on the ERP under focused and divided attention. In C. H. M. Brunia, A. W. K. Gaillard, & A. Kok (Eds.), Psychophysiological brain research (pp. 192–195). Tilberg: Tilburg University Press.Google Scholar
  54. Ferdinand, N. K., Mecklinger, A., Kray, J., & Gehring, W. J. (2012). The processing of unexpected positive response outcomes in the mediofrontal cortex. Journal of Neuroscience, 32, 12087–12092. doi: 10.1523/jneurosci.1410-12.2012 PubMedCrossRefGoogle Scholar
  55. Ferdinand, N. K., Mecklinger, A., & Kray, J. (2008). Error and deviance processing in implicit and explicit sequence learning. Journal of Cognitive Neuroscience, 20, 629–642. doi: 10.1162/jocn.2008.20046 PubMedCrossRefGoogle Scholar
  56. Ferdinand, N. K., & Opitz, B. (2014). Different aspects of performance feedback engage different brain areas: Disentangling valence and expectancy in feedback processing. Scientific Reports, 4, 5986. doi: 10.1038/srep05986 PubMedPubMedCentralCrossRefGoogle Scholar
  57. Folstein, J. R., & Van Petten, C. (2008). Influence of cognitive control and mismatch on the N2 component of the ERP: A review. Psychophysiology, 45, 152–170. doi: 10.1111/j.1469-8986.2007.00602.x PubMedCrossRefGoogle Scholar
  58. Forster, S. E., & Brown, J. W. (2011). Medial prefrontal cortex predicts and evaluates the timing of action outcomes. NeuroImage, 55, 253–265. doi: 10.1016/j.neuroimage.2010.11.035 PubMedCrossRefGoogle Scholar
  59. Fu, Q., Bin, G., Dienes, Z., Fu, X., & Gao, X. (2013). Learning without consciously knowing: Evidence from event-related potentials in sequence learning. Consciousness and Cognition, 22, 22–34. doi: 10.1016/j.concog.2012.10.008 PubMedCrossRefGoogle Scholar
  60. Fujisawa, S., Amarasingham, A., Harrison, M. T., & Buzsáki, G. (2008). Behavior-dependent short-term assembly dynamics in the medial prefrontal cortex. Nature Neuroscience, 11, 823–833. doi: 10.1038/nn.2134 PubMedPubMedCentralCrossRefGoogle Scholar
  61. Fusi, S., Miller, E. K., & Rigotti, M. (2016). Why neurons mix: High dimensionality for higher cognition. Current Opinion in Neurobiology, 37, 66–74. doi: 10.1016/j.conb.2016.01.010 PubMedCrossRefGoogle Scholar
  62. Gehring, W. J., Goss, B., Coles, M. G. H., Meyer, D. E., & Donchin, E. (1993). A neural system for error detection and compensation. Psychological Science, 4, 385–390. doi: 10.1111/j.1467-9280.1993.tb00586.x CrossRefGoogle Scholar
  63. Gehring, W. J., Gratton, G., Coles, M. G. H., & Donchin, E. (1992). Probability effects on stimulus evaluation and response processes. Journal of Experimental Psychology: Human Perception and Performance, 18, 198–216. doi: 10.1037/0096-1523.18.1.198 PubMedGoogle Scholar
  64. Gehring, W. J., Liu, Y., Orr, J. M., & Carp, J. (2012). The error-related negativity (ERN/Ne). In S. J. Luck & E. Kappenm (Eds.), The Oxford handbook of event-related potential components (pp. 231–291). Oxford: Oxford University Press. doi: 10.1093/oxfordhb/9780195374148.013.0120 Google Scholar
  65. HajiHosseini, A., & Holroyd, C. B. (2013). Frontal midline theta and N200 amplitude reflect complementary information about expectancy and outcome evaluation. Psychophysiology, 50, 550–562. doi: 10.1111/psyp.12040 PubMedCrossRefGoogle Scholar
  66. Hammer, A., Rautzenberg, A., Heldmann, M., & Münte, T. F. (2011). Challenging the error-likelihood model with a recognition paradigm: An electrophysiological study. Biological Psychology, 88, 180–187. doi: 10.1016/j.biopsycho.2011.07.010 PubMedCrossRefGoogle Scholar
  67. Hayden, B. Y., Heilbronner, S. R., Pearson, J. M., & Platt, M. L. (2011). Surprise signals in anterior cingulate cortex: Neuronal encoding of unsigned reward prediction errors driving adjustment in behavior. Journal of Neuroscience, 31, 4178–4187. doi: 10.1523/jneurosci.4652-10.2011 PubMedPubMedCentralCrossRefGoogle Scholar
  68. Hayden, B. Y., Pearson, J. M., & Platt, M. L. (2011). Neuronal basis of sequential foraging decisions in a patchy environment. Nature Neuroscience, 14, 933–939. doi: 10.1038/nn.2856 PubMedPubMedCentralCrossRefGoogle Scholar
  69. Hayden, B. Y., & Platt, M. L. (2010). Neurons in anterior cingulate cortex multiplex information about reward and action. Journal of Neuroscience, 30, 3339–3346. doi: 10.1523/jneurosci.4874-09.2010 PubMedPubMedCentralCrossRefGoogle Scholar
  70. Hengst, B. (2012). Hierarchical approaches. In M. Wiering & M. Van Otterlo (Eds.), Reinforcement learning (pp. 293–323). Berlin: Springer. doi: 10.1007/978-3-642-27645-3 CrossRefGoogle Scholar
  71. Hillman, K. L., & Bilkey, D. K. (2010). Neurons in the rat anterior cingulate cortex dynamically encode cost-benefit in a spatial decision-making task. Journal of Neuroscience, 30, 7705–7713. doi: 10.1523/jneurosci.1273-10.2010 PubMedCrossRefGoogle Scholar
  72. Hillman, K. L., & Bilkey, D. K. (2012). Neural encoding of competitive effort in the anterior cingulate cortex. Nature Neuroscience, 15, 1290–1297. doi: 10.1038/nn.3187 PubMedCrossRefGoogle Scholar
  73. Hoerzer, G. M., Legenstein, R., & Maass, W. (2014). Emergence of complex computational structures from chaotic neural networks through reward-modulated Hebbian learning. Cerebral Cortex, 24, 677–690. doi: 10.1093/cercor/bhs348 PubMedCrossRefGoogle Scholar
  74. Holroyd, C. B. (2004). A note on the oddball N200 and the feedback ERN. In M. Ullsperger & M. Falkenstein (Eds.), Errors, conflicts, and the brain: Current opinions on performance monitoring (pp. 211–218). Leipzig: MPI of Cognitive Neuroscience.Google Scholar
  75. Holroyd, C. B. (2016). The waste disposal problem of effortful control. In T. Braver (Ed.), Motivation and cognitive control (pp. 235–260). New York: Psychology Press.Google Scholar
  76. Holroyd, C. B., & Coles, M. G. H. (2002). The neural basis of human error processing: Reinforcement learning, dopamine, and the error-related negativity. Psychological Review, 109, 679–709. doi: 10.1037//0033-295x.109.4.679 PubMedCrossRefGoogle Scholar
  77. Holroyd, C. B., & Coles, M. G. H. (2008). Dorsal anterior cingulate cortex integrates reinforcement history to guide voluntary behavior. Cortex, 44, 548–559. doi: 10.1016/j.cortex.2007.08.013 PubMedCrossRefGoogle Scholar
  78. Holroyd, C. B., & McClure, S. M. (2015). Hierarchical control over effortful behavior by rodent medial frontal cortex: A computational model. Psychological Review, 122, 54–83. doi: 10.1037/a0038339 PubMedCrossRefGoogle Scholar
  79. Holroyd, C. B., Pakzad-Vaezi, K. L., & Krigolson, O. E. (2008). The feedback correct-related positivity: Sensitivity of the event-related brain potential to unexpected positive feedback. Psychophysiology, 45, 688–697. doi: 10.1111/j.1469-8986.2008.00668.x PubMedCrossRefGoogle Scholar
  80. Holroyd, C. B., & Yeung, N. (2011). An integrative theory of anterior cingulate cortex function: Option selection in hierarchical reinforcement learning. In R. B. Mars, J. Sallet, M. F. S. Rushworth, & N. Yeung (Eds.), Neural basis of motivational and cognitive control (pp. 333–349). Cambridge: MIT Press. doi: 10.7551/mitpress/9780262016438.003.0018 Google Scholar
  81. Holroyd, C. B., & Yeung, N. (2012). Motivation of extended behaviors by anterior cingulate cortex. Trends in Cognitive Sciences, 16, 122–128. doi: 10.1016/j.tics.2011.12.008 PubMedCrossRefGoogle Scholar
  82. Holroyd, C. B., Yeung, N., Coles, M. G. H., & Cohen, J. D. (2005). A mechanism for error detection in speeded response time tasks. Journal of Experimental Psychology: General, 134, 163–191. doi: 10.1037/0096-3445.134.2.163 CrossRefGoogle Scholar
  83. Hornik, K. (1991). Approximation capabilities of multilayer feedforward networks. Neural Networks, 4, 251–257. doi: 10.1016/0893-6080(91)90009-t CrossRefGoogle Scholar
  84. Hoshi, E., Sawamura, H., & Tanji, J. (2005). Neurons in the rostral cingulate motor area monitor multiple phases of visuomotor behavior with modest parametric selectivity. Journal of Neurophysiology, 94, 640–656. doi: 10.1152/jn.01201.2004 PubMedCrossRefGoogle Scholar
  85. Hyman, J. M., Ma, L., Balaguer-Ballester, E., Durstewitz, D., & Seamans, J. K. (2012). Contextual encoding by ensembles of medial prefrontal cortex neurons. Proceedings of the National Academy of Sciences, 109, 5086–5091. doi: 10.1073/pnas.1114415109 CrossRefGoogle Scholar
  86. Hyman, J. M., Whitman, J., Emberly, E., Woodward, T. S., & Seamans, J. K. (2013). Action and outcome activity state patterns in the anterior cingulate cortex. Cerebral Cortex, 23, 1257–1268. doi: 10.1093/cercor/bhs104 PubMedCrossRefGoogle Scholar
  87. Jacobs, J., Weidemann, C. T., Miller, J. F., Solway, A., Burke, J. F., Wei, X.-X., & Kahana, M. J. (2013). Direct recordings of grid-like neuronal activity in human spatial navigation. Nature Neuroscience, 16, 1188–1190. doi: 10.1038/nn.3466 PubMedPubMedCentralCrossRefGoogle Scholar
  88. Jessup, R. K., Busemeyer, J. R., & Brown, J. W. (2010). Error effects in anterior cingulate cortex reverse when error likelihood is high. Journal of Neuroscience, 30, 3467–3472. doi: 10.1523/jneurosci.4130-09.2010 PubMedPubMedCentralCrossRefGoogle Scholar
  89. Jia, S., Li, H., Luo, Y., Chen, A., Wang, B., & Zhou, X. (2007). Detecting perceptual conflict by the feedback-related negativity in brain potentials. NeuroReport, 18, 1385–1388. doi: 10.1097/wnr.0b013e3282c48a90 PubMedCrossRefGoogle Scholar
  90. Johnston, K., Levin, H. M., Koval, M. J., & Everling, S. (2007). Top-down control-signal dynamics in anterior cingulate and prefrontal cortex neurons following task switching. Neuron, 53, 453–462. doi: 10.1016/j.neuron.2006.12.023 PubMedCrossRefGoogle Scholar
  91. Jongsma, M. L. A., van Rijn, C. M., Gerrits, N. J. H. M., Eichele, T., Steenbergen, B., Maes, J. H. R., & Quiroga, R. Q. (2013). The learning-oddball paradigm: Data of 24 separate individuals illustrate its potential usefulness as a new clinical tool. Clinical Neurophysiology, 124, 514–521. doi: 10.1016/j.clinph.2012.09.009 PubMedCrossRefGoogle Scholar
  92. Kaping, D., Vinck, M., Hutchison, R. M., Everling, S., & Womelsdorf, T. (2011). Specific contributions of ventromedial, anterior cingulate, and lateral prefrontal cortex for attentional selection and stimulus valuation. PLoS Biology, 9, e1001224. doi: 10.1371/journal.pbio.1001224 PubMedPubMedCentralCrossRefGoogle Scholar
  93. Kennerley, S. W., Behrens, T. E. J., & Wallis, J. D. (2011). Double dissociation of value computations in orbitofrontal and anterior cingulate neurons. Nature Neuroscience, 14, 1581–1589. doi: 10.1038/nn.2961 PubMedPubMedCentralCrossRefGoogle Scholar
  94. Kennerley, S. W., Dahmubed, A. F., Lara, A. H., & Wallis, J. D. (2009). Neurons in the frontal lobe encode the value of multiple decision variables. Journal of Cognitive Neuroscience, 21, 1162–1178. doi: 10.1162/jocn.2009.21100 PubMedPubMedCentralCrossRefGoogle Scholar
  95. Kennerley, S. W., & Wallis, J. D. (2009). Evaluating choices by single neurons in the frontal lobe: Outcome value encoded across multiple decision variables. European Journal of Neuroscience, 29, 2061–2073. doi: 10.1111/j.1460-9568.2009.06743.x PubMedPubMedCentralCrossRefGoogle Scholar
  96. Khamassi, M., Lallee, S., Enel, P., Procyk, E., & Dominey, P. F. (2011). Robot cognitive control with a neurophysiologically inspired reinforcement learning model. Frontiers in Neurorobotics, 6(1), 1–14.Google Scholar
  97. Khamassi, M., Quilodran, R., Enel, P., Dominey, P. F., & Procyk, E. (2015). Behavioral regulation and the modulation of information coding in the lateral prefrontal and cingulate cortex. Cerebral Cortex, 25, 3197–3218. doi: 10.1093/cercor/bhu114 PubMedCrossRefGoogle Scholar
  98. Klavir, O., Genud-Gabai, R., & Paz, R. (2013). Functional connectivity between amygdala and cingulate cortex for adaptive aversive learning. Neuron, 80, 1290–1300. doi: 10.1016/j.neuron.2013.09.035 PubMedCrossRefGoogle Scholar
  99. Koechlin, E., Danek, A., Burnod, Y., & Grafman, J. (2002). Medial prefrontal and subcortical mechanisms underlying the acquisition of motor and cognitive action sequences in humans. Neuron, 35, 371–381. doi: 10.1016/s0896-6273(02)00742-0 PubMedCrossRefGoogle Scholar
  100. Kopp, B., & Wolff, M. (2000). Brain mechanisms of selective learning: Event-related potentials provide evidence for error-driven learning in humans. Biological Psychology, 51, 223–246. doi: 10.1016/s0301-0511(99)00039-3 PubMedCrossRefGoogle Scholar
  101. Koyama, T., Tanaka, Y. Z., & Mikami, A. (1998). Nociceptive neurons in the macaque anterior cingulate activate during anticipation of pain. NeuroReport, 9, 2663–2667. doi: 10.1097/00001756-199808030-00044 PubMedCrossRefGoogle Scholar
  102. Kriegeskorte, N., & Kievit, R. A. (2013). Representational geometry: Integrating cognition, computation and the brain. Trends in Cognitive Sciences, 17, 401–412. doi: 10.1016/j.tics.2013.06.007 PubMedPubMedCentralCrossRefGoogle Scholar
  103. Kriegeskorte, N., Mur, M., & Bandettini, P. (2008). Representational similarity analysis—Connecting the branches of systems neuroscience. Frontiers in Systems Neuroscience, 2, Article 4, 1–28. doi: 10.3389/neuro.06.004.2008 Google Scholar
  104. Kuwabara, M., Mansouri, F. A., Buckley, M. J., & Tanaka, K. (2014). Cognitive control functions of anterior cingulate cortex in macaque monkeys performing a Wisconsin card sorting test analog. Journal of Neuroscience, 34, 7531–7547. doi: 10.1523/jneurosci.3405-13.2014 PubMedPubMedCentralCrossRefGoogle Scholar
  105. Lang, S., & Kotchoubey, B. (2000). Learning effects on event-related brain potentials. NeuroReport, 11, 3327–3331. doi: 10.1097/00001756-200010200-00013 PubMedCrossRefGoogle Scholar
  106. Lapish, C. C., Durstewitz, D., Chandler, L. J., & Seamans, J. K. (2008). Successful choice behavior is associated with distinct and coherent network states in anterior cingulate cortex. Proceedings of the National Academy of Sciences, 105, 11963–11968. doi: 10.1073/pnas.0804045105 CrossRefGoogle Scholar
  107. Luk, C.-H., & Wallis, J. D. (2009). Dynamic encoding of responses and outcomes by neurons in medial prefrontal cortex. Journal of Neuroscience, 29, 7526–7539. doi: 10.1523/jneurosci.0386-09.2009 PubMedPubMedCentralCrossRefGoogle Scholar
  108. Ma, L., Hyman, J. M., Lindsay, A. J., Phillips, A. G., & Seamans, J. K. (2014). Differences in the emergent coding properties of cortical and striatal ensembles. Nature Neuroscience, 17, 1100–1106. doi: 10.1038/nn.3753 PubMedPubMedCentralCrossRefGoogle Scholar
  109. Ma, L., Hyman, J. M., Phillips, A. G., & Seamans, J. K. (2014). Tracking progress toward a goal in corticostriatal ensembles. Journal of Neuroscience, 34, 2244–2253. doi: 10.1523/jneurosci.3834-13.2014 PubMedCrossRefGoogle Scholar
  110. Mante, V., Sussillo, D., Shenoy, K. V., & Newsome, W. T. (2013). Context-dependent computation by recurrent dynamics in prefrontal cortex. Nature, 503, 78–84. doi: 10.1038/nature12742 PubMedPubMedCentralCrossRefGoogle Scholar
  111. Martens, J. (2010). Deep learning via Hessian-free optimization. Proceedings of the 27th International Conference on Machine Learning (ICML-10), Haifa, Israel.Google Scholar
  112. Martens, J., & Sutskever, I. (2011). Learning recurrent neural networks with hessian-free optimization. Proceedings of the 28th International Conference on Machine Learning (ICML-11), Bellevue, WA.Google Scholar
  113. Mathalon, D. H., Whitfield, S. L., & Ford, J. M. (2003). Anatomy of an error: ERP and fMRI. Biological Psychology, 64, 119–141. doi: 10.1016/s0301-0511(03)00105-4 PubMedCrossRefGoogle Scholar
  114. Matsumoto, M., Matsumoto, K., Abe, H., & Tanaka, K. (2007). Medial prefrontal cell activity signaling prediction errors of action values. Nature Neuroscience, 10, 647–656. doi: 10.1038/nn1890 PubMedCrossRefGoogle Scholar
  115. Metereau, E., & Dreher, J.-C. (2013). Cerebral correlates of salient prediction error for different rewards and punishments. Cerebral Cortex, 23, 477–487. doi: 10.1093/cercor/bhs037 PubMedCrossRefGoogle Scholar
  116. Miyawaki, K., Sato, A., Yasuda, A., Kumano, H., & Kuboki, T. (2005). Explicit knowledge and intention to learn in sequence learning: An event-related potential study. NeuroReport, 16, 705–708. doi: 10.1097/00001756-200505120-00010 PubMedCrossRefGoogle Scholar
  117. Moody, S. L., Wise, S. P., di Pellegrino, G., & Zipser, D. (1998). A model that accounts for activity in primate frontal cortex during a delayed matching-to-sample task. Journal of Neuroscience, 18, 399–410.PubMedGoogle Scholar
  118. Mulder, A. B., Nordquist, R. E., Örgüt, O., & Pennartz, C. M. A. (2003). Learning-related changes in response patterns of prefrontal neurons during instrumental conditioning. Behavioural Brain Research, 146, 77–88. doi: 10.1016/j.bbr.2003.09.016 PubMedCrossRefGoogle Scholar
  119. Nakahara, H., & Doya, K. (1998). Near-saddle-node bifurcation behavior as dynamics in working memory for goal-directed behavior. Neural Computation, 10, 113–132. doi: 10.1162/089976698300017917 PubMedCrossRefGoogle Scholar
  120. Namikawa, J., Nishimoto, R., & Tani, J. (2011). A neurodynamic account of spontaneous behaviour. PLoS Computational Biology, 7, e1002221. doi: 10.1371/journal.pcbi.1002221 PubMedPubMedCentralCrossRefGoogle Scholar
  121. Nee, D. E., Kastner, S., & Brown, J. W. (2011). Functional heterogeneity of conflict, error, task-switching, and unexpectedness effects within medial prefrontal cortex. NeuroImage, 54, 528–540. doi: 10.1016/j.neuroimage.2010.08.027 PubMedCrossRefGoogle Scholar
  122. Nieuwenhuis, S., Schweizer, T. S., Mars, R. B., Botvinick, M. M., & Hajcak, G. (2007). Error-likelihood prediction in the medial frontal cortex: A critical evaluation. Cerebral Cortex, 17, 1570–1581. doi: 10.1093/cercor/bhl068 PubMedCrossRefGoogle Scholar
  123. Niki, H., & Watanabe, M. (1976). Cingulate unit activity and delayed response. Brain Research, 110, 381–386. doi: 10.1016/0006-8993(76)90412-1 PubMedCrossRefGoogle Scholar
  124. Nishijo, H., Yamamoto, Y., Ono, T., Uwano, T., Yamashita, J., & Yamashima, T. (1997). Single neuron responses in the monkey anterior cingulate cortex during visual discrimination. Neuroscience Letters, 227, 79–82. doi: 10.1016/s0304-3940(97)00310-8 PubMedCrossRefGoogle Scholar
  125. Nishimoto, R., & Tani, J. (2004). Learning to generate combinatorial action sequences utilizing the initial sensitivity of deterministic dynamical systems. Neural Networks, 17, 925–933. doi: 10.1016/j.neunet.2004.02.003 PubMedCrossRefGoogle Scholar
  126. O’Reilly, J. X., Schüffelgen, U., Cuell, S. F., Behrens, T. E. J., Mars, R. B., & Rushworth, M. F. S. (2013). Dissociable effects of surprise and model update in parietal and anterior cingulate cortex. Proceedings of the National Academy of Sciences, 110, E3660–E3669. doi: 10.1073/pnas.1305373110 CrossRefGoogle Scholar
  127. Oliveira, F. T. P., McDonald, J. J., & Goodman, D. (2007). Performance monitoring in the anterior cingulate is not all error related: Expectancy deviation and the representation of action-outcome associations. Journal of Cognitive Neuroscience, 19, 1994–2004. doi: 10.1162/jocn.2007.19.12.1994 PubMedCrossRefGoogle Scholar
  128. Procyk, E., & Joseph, J. P. (2001). Characterization of serial order encoding in the monkey anterior cingulate sulcus. European Journal of Neuroscience, 14, 1041–1046. doi: 10.1046/j.0953-816x.2001.01738.x PubMedPubMedCentralCrossRefGoogle Scholar
  129. Procyk, E., Tanaka, Y. L., & Joseph, J. P. (2000). Anterior cingulate activity during routine and non-routine sequential behaviors in macaques. Nature Neuroscience, 3, 502–508. doi: 10.1038/74880 PubMedCrossRefGoogle Scholar
  130. Quilodran, R., Rothé, M., & Procyk, E. (2008). Behavioral shifts and action valuation in the anterior cingulate cortex. Neuron, 57, 314–325. doi: 10.1016/j.neuron.2007.11.031 PubMedCrossRefGoogle Scholar
  131. Rao, R. P. N., & Sejnowski, T. J. (2000). Predictive learning of temporal sequences in recurrent neocortical circuits. In S. A. Solla, T. K. Leen, & K.-R. Müller (Eds.), Advances in neural information processing systems 12 (pp. 164–70). Cambridge: MIT Press.Google Scholar
  132. Remondes, M., & Wilson, M. A. (2013). Cingulate-hippocampus coherence and trajectory coding in a sequential choice task. Neuron, 80, 1277–1289. doi: 10.1016/j.neuron.2013.08.037 PubMedCrossRefGoogle Scholar
  133. Ridderinkhof, K. R., van den Wildenberg, W. P. M., Segalowtiz, S. J., & Carter, C. S. (2004). Neurocognitive mechanisms of cognitive control: The role of prefrontal cortex in action selection, response inhibition, performance monitoring, and reward-based learning. Brain and Cognition, 56, 129–140. doi: 10.1016/j.bandc.2004.09.016 PubMedCrossRefGoogle Scholar
  134. Rigotti, M., Barak, O., Warden, M. R., Wang, X.-J., Daw, N. D., Miller, E. K., & Fusi, S. (2013). The importance of mixed selectivity in complex cognitive tasks. Nature, 497, 585–590. doi: 10.1038/nature12160 PubMedPubMedCentralCrossRefGoogle Scholar
  135. Rolls, E. G. (2009). The anterior and midcingulate cortices and reward. In B. A. Vogt (Ed.), Cingulate neurobiology and disease (pp. 191–218). Oxford: Oxford University Press.Google Scholar
  136. Ruh, N., Cooper, R. P., & Mareschal, D. (2005). Routine action: Combining familiarity and goal orientedness. In J. J. Bryson, T. J. Prescott, & A. K. Seth (Eds.), Modeling natural action selection: Proceedings of an international workshop (pp. 174–179). Edinburgh: AISB Press.Google Scholar
  137. Rumelhart, D. E., & McClelland, J. L. (1986). Parallel distributed processing: Explorations in the microstructure of cognition (vol. 1). Cambridge: MIT Press.Google Scholar
  138. Rushworth, M. F. S., & Behrens, T. E. J. (2008). Choice, uncertainty and value in prefrontal and cingulate cortex. Nature Neuroscience, 11, 389–397. doi: 10.1038/nn2066 PubMedCrossRefGoogle Scholar
  139. Rushworth, M. F. S., Kolling, N., Sallet, J., & Mars, R. B. (2012). Valuation and decision-making in frontal cortex: One or many serial or parallel systems? Current Opinion in Neurobiology, 22, 946–955. doi: 10.1016/j.conb.2012.04.011 PubMedCrossRefGoogle Scholar
  140. Rüsseler, J., Hennighausen, E., Münte, T. F., & Rösler, F. (2003). Differences in incidental and intentional learning of sensorimotor sequences as revealed by event-related brain potentials. Cognitive Brain Research, 15, 116–126. doi: 10.1016/s0926-6410(02)00145-3 PubMedCrossRefGoogle Scholar
  141. Rüsseler, J., Kuhlicke, D., & Münte, T. F. (2003). Human error monitoring during implicit and explicit learning of a sensorimotor sequence. Neuroscience Research, 47, 233–240. doi: 10.1016/s0168-0102(03)00212-8 PubMedCrossRefGoogle Scholar
  142. Rüsseler, J., & Rösler, F. (2000). Implicit and explicit learning of event sequences: Evidence for distinct coding of perceptual and motor representations. Acta Psychologica, 104, 45–67. doi: 10.1016/s0001-6918(99)00053-0 PubMedCrossRefGoogle Scholar
  143. Russo, G. S., Backus, D. A., Ye, S., & Crutcher, M. D. (2002). Neural activity in monkey dorsal and ventral cingulate motor areas: Comparison with the supplementary motor area. Journal of Neurophysiology, 88, 2612–2629. doi: 10.1152/jn.00306.2002 PubMedCrossRefGoogle Scholar
  144. Sallet, J., Quilodran, R., Rothe, M., Vezoli, J., Joseph, J.-P., & Procyk, E. (2007). Expectations, gains, and losses in the anterior cingulate cortex. Cognitive, Affective, & Behavioral Neuroscience, 7, 327–336. doi: 10.3758/cabn.7.4.327 CrossRefGoogle Scholar
  145. Sams, M., Alho, K., & Näätänen, R. (1983). Sequential effects on the ERP in discriminating two stimuli. Biological Psychology, 17, 41–58. doi: 10.1016/0301-0511(83)90065-0 PubMedCrossRefGoogle Scholar
  146. Schlaghecken, F., Stürmer, B., & Eimer, M. (2000). Chunking processes in the learning of event sequences: Electrophysiological indicators. Memory & Cognition, 28, 821–831. doi: 10.3758/bf03198417 CrossRefGoogle Scholar
  147. Schuck, N. W., Gaschler, R., Wenke, D., Heinzle, J., Frensch, P. A., Haynes, J.-D., & Reverberi, C. (2015). Medial prefrontal cortex predicts internally driven strategy shifts. Neuron, 86, 331–340. doi: 10.1016/j.neuron.2015.03.015 PubMedPubMedCentralCrossRefGoogle Scholar
  148. Seo, H., & Lee, D. (2008). Cortical mechanisms for reinforcement learning in competitive games. Philosophical Transactions of the Royal Society B, 363, 3845–3857. doi: 10.1098/rstb.2008.0158 CrossRefGoogle Scholar
  149. Servan-Schreiber, D., Cleeremans, A., & McClelland, J. L. (1991). Graded state machines: The representation of temporal contingencies in simple recurrent networks. Machine Learning, 7, 57–89. doi: 10.1007/978-1-4615-4008-3_4 Google Scholar
  150. Shackman, A. J., Salomons, T. V., Slagter, H. A., Fox, A. S., Winter, J. J., & Davidson, R. J. (2011). The integration of negative affect, pain and cognitive control in the cingulate cortex. Nature Reviews Neuroscience, 12, 154–167. doi: 10.1038/nrn2994 PubMedPubMedCentralCrossRefGoogle Scholar
  151. Shadmehr, R., & Wise, S. P. (2005). The computational neurobiology of reaching and pointing. Cambridge: MIT Press.Google Scholar
  152. Shen, C., Ardid, S., Kaping, D., Westendorff, S., Everling, S., & Womelsdorf, T. (2015). Anterior cingulate cortex cells identify process-specific errors of attentional control prior to transient prefrontal-cingulate inhibition. Cerebral Cortex, 25, 2213–2228. doi: 10.1093/cercor/bhu028 PubMedCrossRefGoogle Scholar
  153. Shenhav, A., Botvinick, M. M., & Cohen, J. D. (2013). The expected value of control: An integrative theory of anterior cingulate cortex function. Neuron, 79, 217–240. doi: 10.1016/j.neuron.2013.07.007 PubMedPubMedCentralCrossRefGoogle Scholar
  154. Shidara, M., Mizuhiki, T., & Richmond, B. J. (2005). Neuronal firing in anterior cingulate neurons changes modes across trials in single states of multitrial reward schedules. Experimental Brain Research, 163, 242–245. doi: 10.1007/s00221-005-2232-y PubMedCrossRefGoogle Scholar
  155. Shidara, M., & Richmond, B. J. (2002). Anterior cingulate: Single neuronal signals related to degree of reward expectancy. Science, 296, 1709–1711. doi: 10.1126/science.1069504 PubMedCrossRefGoogle Scholar
  156. Shima, K., Aya, K., Mushiake, H., Inase, M., Aizawa, H., & Tanji, J. (1991). Two movement-related foci in the primate cingulate cortex observed in signal-triggered and self-paced forelimb movements. Journal of Neurophysiology, 65, 188–202.PubMedCrossRefGoogle Scholar
  157. Shima, K., & Tanji, J. (1998). Role for cingulate motor area cells in voluntary movement selection based on reward. Science, 282, 1335–1338. doi: 10.1126/science.282.5392.1335 PubMedCrossRefGoogle Scholar
  158. Silvetti, M., Seurinck, R., & Verguts, T. (2011). Value and prediction error in medial frontal cortex: Integrating the single-unit and systems levels of analysis. Frontiers in Human Neuroscience, 5, 75. doi: 10.3389/fnhum.2011.00075 PubMedPubMedCentralCrossRefGoogle Scholar
  159. Song, H. F., Yang, G. R., & Wang, X.-J. (2016). Training excitatory-inhibitory recurrent neural networks for cognitive tasks: A simple and flexible framework. PLOS Computational Biology, 1–30. doi: 10.1371/journal.pcbi.1004792
  160. Starzyk, J. A., & He, H. (2007). Anticipation-based temporal sequences learning in hierarchical structure. IEEE Transactions on Neural Networks, 18, 344–358. doi: 10.1109/tnn.2006.884681 PubMedCrossRefGoogle Scholar
  161. Stokes, M. G., Kusunoki, M., Sigala, N., Nili, H., Gaffan, D., & Duncan, J. (2013). Dynamic coding for cognitive control in prefrontal cortex. Neuron, 78, 364–375. doi: 10.1016/j.neuron.2013.01.039 PubMedPubMedCentralCrossRefGoogle Scholar
  162. Sussillo, D. (2014). Neural circuits as computational dynamical systems. Current Opinion in Neurobiology, 25, 156–163. doi: 10.1016/j.conb.2014.01.008 PubMedCrossRefGoogle Scholar
  163. Sussillo, D., & Abbott, L. F. (2009). Generating coherent patterns of activity from chaotic neural networks. Neuron, 63, 544–557. doi: 10.1016/j.neuron.2009.07.018 PubMedPubMedCentralCrossRefGoogle Scholar
  164. Sussillo, D., & Barak, O. (2013). Opening the black box: Low-dimensional dynamics in high-dimensional recurrent neural networks. Neural Computation, 25, 626–649. doi: 10.1162/neco_a_00409 PubMedCrossRefGoogle Scholar
  165. Takenouchi, K., Nishijo, H., Uwano, T., Tamura, R., Takigawa, M., & Ono, T. (1999). Emotional and behavioral correlates of the anterior cingulate cortex during associative learning in rats. Neuroscience, 93, 1271–1287. doi: 10.1016/s0306-4522(99)00216-x PubMedCrossRefGoogle Scholar
  166. Tanji, J., Shima, K., & Matsuzaka, Y. (2002). Reward-based planning of motor selection in the rostral cingulate motor area. In S. C. Gandevia, U. Proske, & D. G. Stuart (Eds.), Sensorimotor control of movement and posture (pp. 417–423). New York: Springer. doi: 10.1007/978-1-4615-0713-0_47 CrossRefGoogle Scholar
  167. Tervo, D. G. R., Proskurin, M., Manakov, M., Kabra, M., Vollmer, A., Branson, K., & Karpova, A. Y. (2014). Behavioral variability through stochastic choice and its gating by anterior cingulate cortex. Cell, 159, 21–32. doi: 10.1016/j.cell.2014.08.037 PubMedCrossRefGoogle Scholar
  168. Totah, N. K. B., Kim, Y. B., Homayoun, H., & Moghaddam, B. (2009). Anterior cingulate neurons represent errors and preparatory attention within the same behavioral sequence. Journal of Neuroscience, 29, 6418–6426. doi: 10.1523/jneurosci.1142-09.2009 PubMedPubMedCentralCrossRefGoogle Scholar
  169. Ursu, S., Clark, K. A., Aizenstein, H. J., Stenger, V. A., & Carter, C. S. (2009). Conflict-related activity in the caudal anterior cingulate cortex in the absence of awareness. Biological Psychology, 80, 279–286. doi: 10.1016/j.biopsycho.2008.10.008 PubMedCrossRefGoogle Scholar
  170. Verguts, T. (2017). Binding by random bursts: A computational model of cognitive control. Journal of Cognitive Neuroscience. doi: 10.1162/jocn_a_01117 PubMedGoogle Scholar
  171. Verguts, T., Vassena, E., & Silvetti, M. (2015). Adaptive effort investment in cognitive and physical tasks: A neurocomputational model. Frontiers in Behavioral Neuroscience, 9, 57. doi: 10.3389/fnbeh.2015.00057 PubMedPubMedCentralCrossRefGoogle Scholar
  172. Walton, M. E., & Mars, R. B. (2007). Probing human and monkey anterior cingulate cortex in variable environments. Cognitive, Affective, & Behavioral Neuroscience, 7, 413–422. doi: 10.3758/cabn.7.4.413 CrossRefGoogle Scholar
  173. Wang, X.-J. (2008). Decision making in recurrent neuronal circuits. Neuron, 60, 215–234. doi: 10.1016/j.neuron.2008.09.034 PubMedPubMedCentralCrossRefGoogle Scholar
  174. Wang, Y., Cui, L., Wang, H., Tian, S., & Zhang, X. (2004). The sequential processing of visual feature conjunction mismatches in the human brain. Psychophysiology, 41, 21–29. doi: 10.1111/j.1469-8986.2003.00134.x PubMedCrossRefGoogle Scholar
  175. Wang, C., Ulbert, I., Schomer, D. L., Marinkovic, K., & Halgren, E. (2005). Responses of human anterior cingulate cortex microdomains to error detection, conflict monitoring, stimulus–response mapping, familiarity, and orienting. Journal of Neuroscience, 25, 604–613. doi: 10.1523/jneurosci.4151-04.2005 PubMedCrossRefGoogle Scholar
  176. Warren, C. M., & Holroyd, C. B. (2012). The impact of deliberative strategy dissociates ERP components related to speeded responding vs. reinforcement learning. Frontiers in Decision Neuroscience, 6, 43. doi: 10.3389/fnins.2012.00043 Google Scholar
  177. Weible, A. P., Rowland, D. C., Monaghan, C. K., Wolfgang, N. T., & Kentros, C. G. (2012). Neural correlates of long-term object memory in the mouse anterior cingulate cortex. Journal of Neuroscience, 32, 5598–5608. doi: 10.1523/jneurosci.5265-11.2012 PubMedCrossRefGoogle Scholar
  178. Wessel, J. R., Danielmeier, C., Morton, J. B., & Ullsperger, M. (2012). Surprise and error: Common neuronal architecture for the processing of errors and novelty. Journal of Neuroscience, 32, 7528–7537. doi: 10.1523/jneurosci.6352-11.2012 PubMedCrossRefGoogle Scholar
  179. Williams, R. J., & Zipser, D. (1995). Gradient-based learning algorithms for recurrent networks and their computational complexity. In Y. Chauvin & D. E. Rumelhart (Eds.), Back-propagation: Theory, architectures and applications (pp. 433–486). Hillsdale: Erlbaum.Google Scholar
  180. Wolf, F., Engelken, R., Puelma-Touzel, M., Weidinger, J. D. F., & Neef, A. (2014). Dynamical models of cortical circuits. Current Opinion in Neurobiology, 25, 228–236. doi: 10.1016/j.conb.2014.01.017 PubMedCrossRefGoogle Scholar
  181. Wolpert, D. M., & Ghahramani, A. (2000). Computational principles of movement neuroscience. Nature Neuroscience Supplement, 3, 1212–1217.CrossRefGoogle Scholar
  182. Yamashita, Y., & Tani, J. (2008). Emergence of functional hierarchy in a multiple timescale neural network model: A humanoid robot experiment. PLoS Computational Biology, 4, e1000220. doi: 10.1371/journal.pcbi.1000220 PubMedPubMedCentralCrossRefGoogle Scholar
  183. Yeung, N. (2013). Conflict monitoring and cognitive control. In K. Ochsner & S. Kosslyn (Eds.), Oxford handbook of cognitive neuroscience (pp. 275–299). Oxford: Oxford University Press. doi: 10.1093/oxfordhb/9780199988709.013.0018 Google Scholar
  184. Yeung, N., Botvinick, M. M., & Cohen, J. D. (2004). The neural basis of error detection: Conflict monitoring and the error-related negativity. Psychological Review, 111, 931–959. doi: 10.1037/0033-295x.111.4.931 PubMedCrossRefGoogle Scholar
  185. Yeung, N., & Nieuwenhuis, S. (2009). Dissociating response conflict and error likelihood in anterior cingulate cortex. Journal of Neuroscience, 29, 14506–14510. doi: 10.1523/jneurosci.3615-09.2009 PubMedPubMedCentralCrossRefGoogle Scholar
  186. Yuste, R. (2015). From the neuron doctrine to neural networks. Nature Reviews Neuroscience, 16, 487–497. doi: 10.1038/nrn3962 PubMedCrossRefGoogle Scholar

Copyright information

© Psychonomic Society, Inc. 2017

Authors and Affiliations

  1. 1.Department of PsychologyUniversity of VictoriaVictoriaCanada

Personalised recommendations