Journal of Computational Neuroscience

, Volume 43, Issue 3, pp 273–294 | Cite as

Short-term depression and transient memory in sensory cortex

  • Grant Gillary
  • Rüdiger von der Heydt
  • Ernst Niebur
Article

Abstract

Persistent neuronal activity is usually studied in the context of short-term memory localized in central cortical areas. Recent studies show that early sensory areas also can have persistent representations of stimuli which emerge quickly (over tens of milliseconds) and decay slowly (over seconds). Traditional positive feedback models cannot explain sensory persistence for at least two reasons: (i) They show attractor dynamics, with transient perturbations resulting in a quasi-permanent change of system state, whereas sensory systems return to the original state after a transient. (ii) As we show, those positive feedback models which decay to baseline lose their persistence when their recurrent connections are subject to short-term depression, a common property of excitatory connections in early sensory areas. Dual time constant network behavior has also been implemented by nonlinear afferents producing a large transient input followed by much smaller steady state input. We show that such networks require unphysiologically large onset transients to produce the rise and decay observed in sensory areas. Our study explores how memory and persistence can be implemented in another model class, derivative feedback networks. We show that these networks can operate with two vastly different time courses, changing their state quickly when new information is coming in but retaining it for a long time, and that these capabilities are robust to short-term depression. Specifically, derivative feedback networks with short-term depression that acts differentially on positive and negative feedback projections are capable of dynamically changing their time constant, thus allowing fast onset and slow decay of responses without requiring unrealistically large input transients.

Keywords

Positive feedback network Derivative feedback network Balanced network Sensory memory Persistent neuronal activity Short-term depression 

1 Introduction

Persistence of activity after the removal of a stimulus has been observed in many areas of cortex. In the prefrontal and parietal lobes, persistent activity can last for tens of seconds during working memory tasks, for a review see (Curtis and D’Esposito 2003). Shorter decaying memory traces have recently been found in early sensory areas including visual, auditory and somatosensory cortex, and the olfactory bulb (Super et al. 2001; Shuler and Bear 2006; O’Herron and von der Heydt 2009; Nikolić et al. 2009; O’Herron and von der Heydt 2011; Petreanu et al. 2012; Patterson et al. 2013). These traces rise over tens of milliseconds and decay over seconds. For instance, Shuler and Bear (Shuler and Bear 2006) observed decaying activity in primary visual cortex which was tied to the learned time of reward. O’Herron and von der Heydt (2009) showed that activity representing border ownership relationships in cortical area V2 lasts for a time period of about one second after the end of stimulus presentation.

Physiologically plausible computational models of cortical activity in networks driven by recurrent connectivity have previously shown the ability to produce widely varying time constants (Hardy and Buonomano 2016). These models fall into two categories: those which maintain activity through positive feedback, e.g. Gavornik et al. (2009) and Gavornik and Shouval (2011), or through derivative feedback (Lim and Goldman 2013). Both types of networks depend on their recurrent connections to maintain activity after stimulus offset. Previous studies of the decay times of both network types have not taken into account history dependent changes of synaptic strength (Gavornik et al. 2009; Lim and Goldman 2013; 2014), even though the strengths of individual synapses in cortex change constantly due to both short-term depression (STD) and short-term facilitation (STF). Specifically, it is known that the strengths of synapses in early sensory cortex are dominated by STD (Beck et al. 2005; Tsodyks and Markram 1997; Petersen 2002). We therefore examine how STD impacts the transient activity of both positive feedback and derivative feedback networks. We show that the decay times of positive feedback networks are highly sensitive to STD on their recurrent connections, whereas the decay times of derivative feedback networks have decay times which are robust within the physiological range of STD parameters.

A defining characteristic of any memory mechanism is that memories are acquired much faster than they decay (other persistent activity patterns exist where this is not the case, e.g. sensory after-effects, but those are usually not considered memories). Such behavior cannot be explained in linear systems whose dynamics are symmetric with respect to rise and decay. Most current memory models create the asymmetry by positive feedback which makes activity patterns fall into particular attractor states (Wang 1999; Compte et al. 2000; Major and Tank 2004; Barak and Tsodyks 2007). Such models can explain working memory behavior but not sensory memory since, in the latter, when the stimulus is removed activity decays to the pre-stimulus level. Furthermore, our results show that even small contributions of STD in the recurrent connections, a feature commonly observed in sensory cortex, eliminates persistent activity of positive feedback networks.

A class of models whose activity does return to pre-stimulus levels are negative-derivative networks (Lim and Goldman 2013), which for simplicity we will call derivative feedback networks in the following. This is a special case of balanced networks that have been proposed as implementing a computing paradigm of much higher efficiency and representational power than conventional rate coding (Denève and Machens 2016). Recurrent activity in these networks approximates a negative derivative with respect to time which is obtained from the combination of fast inhibitory feedback followed immediately by slower excitatory feedback (Lim and Goldman 2013). This negative derivative feedback is mathematically equivalent to a friction/damping term in a mechanical system which slows down the decay of activity after a transient excursion. The defining characteristic of derivative feedback networks is that the response time of the network is strongly dependent on the difference in time course between the excitatory-to-excitatory (EE) and excitatory-to-inhibitory (IE) projections.

We have shown previously (Gillary and Niebur 2016) that small changes in the relative timing can produce disproportionately larger (by two orders of magnitude) changes in the response time of the network. We will show below that, different from positive feedback models, derivative feedback models are robust to decreases in synaptic strength which are due to STD on their recurrent connections. However, previously described derivative feedback networks cannot explain the observed persistence of responses in sensory cortex. We will show that to produce the relative difference in onset vs. offset timing that is observed in cortex, derivative feedback networks require very large onset transients. One way to produce such large transients is through STD on the afferents, the feedforward connections to the population of interest. We show that for derivative feedback networks, STD on the afferent must be stronger than what is generally observed in cortex to produce a sufficiently strong onset transient. Therefore, neither positive feedback networks nor the derivative feedback networks described so far can explain the vast differences between onset and offset time courses.

Understanding the observed neuronal behavior requires a slight generalization of the previously discussed derivative networks which, at the same time, makes the model more biophysically realistic. We have, so far, adopted the commonly made simplification that STD can be described by a single dynamic variable that describes the activity-related depletion of synaptic resources, e.g. by using up synaptic vesicles at a higher rate than they can be renewed. We call such networks “homogeneous” derivative networks. While this is a reasonable point of departure, it is unlikely that all synapses will have completely identical STD characteristics. It is more likely that different synapses, neurons and projections express some range of activity-related synaptic depression (on longer time scales which are better understood, it is known that synaptic strength even can switch signs, from long term depression to long term potentiation and vice-versa, when calcium concentration or intracellular voltage change (Tsumoto and Yasuda 1996; Hansel et al 1997; Cormier et al 2001). While the assumption of completely identical STD characteristics in all synapses is unlikely to be true, few details about the range and distribution of STD parameters are known. We therefore explore the assumption that the distribution of STD takes on the simplest non-homogeneous form. That is, we assume that one synaptic population has stronger STD than the other, with no further differentiation within these populations. We call derivative networks with this property “heterogeneous” networks.

After we have relaxed the unphysiological restriction to homogeneous networks, we are now in a position to explain the observed neuronal behavior by making one additional assumption. Glutamatergic pathways in cortex typically have synapses which are composed of a mixture of α-amino-3-hydroxy-5-methyl-4-isoxazolepropionic acid (AMPA) and N-methyl-D-aspartate (NMDA) receptors, a property that we make use of in our model to allow for a variety of time courses in the recurrent connections. Different time courses are due to different relative contributions from synapses with different ratios of fast (AMPA) and slow (NMDA) receptors. We approximate the possible range of relative contributions of AMPA and NMDA by two synapse types, one which is AMPA-dominated with fast kinetics and another which is NMDA-dominated, with slow kinetics. These appear in the two recurrent pathways of our network, excitatory-to-excitatory and excitatory-to-inhibitory. The assumption that we will make is that the two types of STD (stronger and weaker) are assigned specifically to the fast and slow pathways, as described in more detail below.

While it is unlikely that the effect of STD is identical in different pathways, we want to make clear that, at this time, the specific assignment of STD to the fast and slow pathways as it is described below Eqs. (16-18) is a prediction of our model. There are many ways in which STD could differ between pathways. While we are not aware of any experimental evidence arguing against our assumption, we also do not know of direct evidence that would turn our assumption into a known fact. We make this prediction because it provides a simple and, we feel, plausible mechanism to explain the existence of sensory memory in cortical networks, with fast rise and slow decay of neuronal activity. We also note that slightly more general mechanisms than those we adopted can explain the behavior. We have chosen the simplest and most parsimonious model requiring the smallest number of free parameters but so-far unknown biophysical or physiological constraints may require a generalization of the model while still allowing it to explain the occurrence of sensory memory.

If STD decreases the strength of, say, the faster component of a projection more than that of its slower component, the overall time constant of the projection will change (become longer, in this case) over the course of a stimulus presentation. As we have shown in previous work (Gillary and Niebur 2016), changes in the relative time courses of positive and negative feedback connections in this balanced network lead to proportionally larger changes in the response time course of the network as a whole. If the EE projection slows relative to the IE projection, the speed of response of the network can also change significantly over a single stimulus presentation. With appropriate starting parameters these networks are capable of rising quickly at stimulus onset and then decaying slowly after stimulus offset. Simulation results (below) demonstrate that our conclusions hold for not only for rate models but also for networks of spiking neurons.

Finally, since STD causes the strength of individual connections to change over the course of a stimulus presentation and is also dependent upon the firing rate of the afferent projection, one might expect the decay times of such networks to change with different stimulus presentation lengths and for different steady state firing rates. Contrary to this expectation, previous experimental work has shown that the decay time of networks with persistent activity does not depend strongly on either parameter (O’Herron and von der Heydt 2009). Indeed, we show that the decay time of a network with heterogeneous feedback connections is well maintained over a range of stimulus presentation lengths as well as for different baseline activity levels. This is due to the reduced synaptic strengths being balanced by a slowing of the EE projection relative to the IE projection. In contrast, derivative feedback networks with homogeneous STD have network decay times which are much more strongly dependent on both the stimulus length and steady state firing rate than the network with heterogeneous STD.

2 Methods

2.1 Positive feedback model

If Fig. 1a we define a positive feedback model with homogenerous STD on the recurrent connections where homogeneous STD means that only a single set of STD parameters are used. This network is described by a fourth-order nonlinear ordinary differential equation:
$$\begin{array}{@{}rcl@{}} \tau_{e}\frac{dR_{e}}{dt}&=&-R_{e}+w(S^{ampa}+S^{nmda})+I(t) \end{array} $$
(1)
$$\begin{array}{@{}rcl@{}} \tau^{ampa}\frac{dS^{ampa}}{dt}&=&-S^{ampa}+(1-q)xR_{e} \end{array} $$
(2)
$$\begin{array}{@{}rcl@{}} \tau^{nmda}\frac{dS^{nmda}}{dt}&=&-S^{nmda}+qxR_{e} \end{array} $$
(3)
$$\begin{array}{@{}rcl@{}} \frac{dx}{dt}&=&\frac{1-x}{\tau_{r}}-uxR_{e} \end{array} $$
(4)
where Re is the firing rate of an excitatory population with intrinsic time constant τe = 20 ms and w represents the strength of the recurrent connections whose activity is filtered through two synaptic activation variables Sampa and Snmda. These variables represent AMPA and NMDA type receptors with time constants τampa = 5 ms and τnmda = 100 ms. These time constants are used for all rate based simulations in this study for both the positive feedback network and the derivative feedback networks defined below. The recurrent activity from the excitatory population is split between the two receptor types by the variable q which represents the fraction of synaptic strength carried by the NMDA receptors, q ∈ [0, 1]. We set q = 0.5 for all simulations of the positive feedback network. The STD on the recurrent synapses is represented by the variable x where x ∈ [0, 1] and x(0) = 1. For simulations without STD we set x(t) = 1. For synapses with STD, the recovery rate of the synapse is represented by τr, chosen as τr = 500 ms in all simulations with recurrent STD. The usage rate of the synapse, u, is the fraction of synaptic resources expended by each action potential.
Fig. 1

Dependence of decay time in the positive feedback network on STD parameters. a Network schematic showing the positive feedback network. The strength of the recurrent projection is x(t)w where x(t) is the time-dependent STD variable and w is the resting synaptic strength. There is only a single set of STD parameters for this projection. b Response of the positive feedback network to a step input at t = 0 s and ending at t = 2 s for four different values of usage rate, u. c Decay time of a linear system (i.e. with no STD) with instantaneous recurrent strength wlin = x(t)w at each time point in the simulation in panel b. Note that the network with no STD is already linear and so the decay time is constant. d Decay time of the positive feedback network without STD. e Decay time of the positive feedback network with STD for three different usage rates. Note the different scale when compared to panel d. f Maximum decay time of the positive feedback network with STD as a function of usage rate, u. The maximum decay time occurs when w = 1. g Response of positive feedback network composed of 9,600 LIF neurons to a step input at t = 0 s and ending at t = 2 s for four different values of usage rate, u. Each run uses the same recurrent strength, w = 0.145, which was chosen to produce a decay time of 2 s for the LIF network without STD. All rate based networks use w = 0.9936 chosen to yield a decay time of 25 s for the network without STD. All networks use τr = 500 ms when STD is present and have I(t) chosen to produce a steady state firing rate of 20 Hz

The input Iff(t) to the network is a “box car” function whose value jumps from zero to Iff at time t = 0, and back to zero after a time specified for each condition below. Half its strength is filtered through AMPA and the other half through NMDA type receptors such that the input current I(t) in Eq. (1) is computed as,
$$\begin{array}{@{}rcl@{}} I(t) &=& \frac{1}{2}S^{ampa}_{ff}+ \frac{1}{2}S^{nmda}_{ff} \end{array} $$
(5)
$$\begin{array}{@{}rcl@{}} \tau^{ampa}\frac{dS^{ampa}_{ff}}{dt}&=&-S^{ampa}_{ff}+I_{ff}(t) \end{array} $$
(6)
$$\begin{array}{@{}rcl@{}} \tau^{nmda}\frac{dS^{nmda}_{ff}}{dt}&=&-S^{nmda}_{ff}+I_{ff}(t) \end{array} $$
(7)

We use the terms “rise time” and “decay time” to describe the speed of network response to the onset and offset of a stimulus respectively. We define the rise time as the time for Re to go from 10% to 90% of its steady state value \(R_{e}^{ss}\) (computed below) when the stimulus is present, and vice-versa for the decay time. We refer to rise time and decay time generally as response times of the network.

The steady state firing rate \(R_{e}^{ss}\) of the system defined in Eqs. (1)-(4) is computed analytically, as the solution to a second order polynomial:
$$ R_{e}^{ss}=-\frac{1}{2}\frac{1-w}{u\tau_{r}}\pm\frac{1}{2}\sqrt{\left( \frac{1-w}{u\tau_{r}}\right)^{2}+4\frac{I}{u\tau_{r}}} $$
(8)
When I = 0 the steady states of the network are \(R_{e}^{ss}=(0,(w-1)/(u\tau _{r}))\). If w ≤ 1 this network will always decay back to its baseline value after stimulus offset (note that the negative steady state can not be reached by the system since Re ≥ 0). Therefore, we limit w to the range [0, 1] for all our simulations of the positive feedback network.

2.2 Derivative feedback model

The dynamics of the rate-based derivative feedback network, shown in Fig. 2a, can be described by the following set of ordinary differential equations,
$$\begin{array}{@{}rcl@{}} \tau_{e}\frac{dR_{e}}{dt}&=&-R_{e}+w(S^{ampa}_{ee}+S^{nmda}_{ee}) \\ &-&kwS^{gaba}_{ei}+I(t) \end{array} $$
(9)
$$\begin{array}{@{}rcl@{}} \tau_{i}\frac{dR_{i}}{dt}=&-&R_{i}+w(S^{ampa}_{ie}+S^{nmda}_{ie}) \\&-&kwS^{gaba}_{ii} \end{array} $$
(10)
$$\begin{array}{@{}rcl@{}} \tau^{ampa}\frac{dS^{ampa}_{mn}}{dt}=&-&S^{ampa}_{mn}+\frac{1}{N}\sum\limits_{j}(1-q^{j}_{mn})x^{j}_{mn}R_{e} \end{array} $$
(11)
$$\begin{array}{@{}rcl@{}} &&\kern-.5pc\tau^{nmda}\frac{dS^{nmda}_{mn}}{dt}=-S^{nmda}_{mn}+\frac{1}{N}\sum\limits_{j}q^{j}_{mn}x^{j}_{mn}R_{e} \end{array} $$
(12)
$$\begin{array}{@{}rcl@{}} &&\kern-.5pc\tau^{gaba}\frac{dS^{gaba}_{mn}}{dt}=-S^{gaba}_{mn}+R_{i} \end{array} $$
(13)
$$\begin{array}{@{}rcl@{}} &&\kern-.5pc\frac{dx^{j}_{mn}}{dt}=\frac{1-x^{j}_{mn}}{(1+p^{j}_{mn})\tau_{r}}-(1+p^{j}_{mn})ux^{j}_{mn}R_{e} \end{array} $$
(14)
The variables Re and Ri in Eqs. (9) and (10) represent the firing rates of the excitatory and inhibitory populations, respectively, with intrinsic time constants τe = 20 ms and τi = 10 ms. As for the positive feedback model, the parameter w in Eqs. (9), (10) represents the maximal strength for the excitatory projections. The inhibitory connection is w multiplied by a factor k which must fulfill k ≥ 1 to ensure stability. All rate based derivative feedback models in this work use w = 100 and k = 1.1. This parameterization of the derivative feedback network is similar to that used by Murphy and Miller (2009).
Fig. 2

Dependence of decay time in the derivative feedback network on the strength of STD on the excitatory projections. a Network schematic showing the derivative feedback network. The strength of the recurrent excitatory projections are x(t)w where x(t) is STD and w is the resting synaptic strength. Colored synapses indicate combinations of AMPA and NMDA receptors yielding either a longer (red) or shorter (green) average synaptic time constant respectively. For brevity, we refer to synapses with longer and shorter average synaptic time constants as slow and fast synapses respectively. All excitatory synapses have the same STD parameters. Inhibitory projections have no STD and a total synaptic strength of kw. b Response of derivative feedback network with homogeneous STD on its excitatory projections to a step input at t = 0 s and ending at t = 2 s for four different values of usage rate, u. c Decay time of a linear system (i.e.. with no STD) where the strength of excitatory projections are wlin = x(t)w at each time point in the simulation in panel b. Note that the network with no STD is already linear and so the decay time is constant. d Decay time of the derivative feedback network without STD for a range of values of w. Note the different scale when compared to panel e. e Decay time of the derivative feedback network with STD as a function of w for three different usage rates. f Response of derivative feedback network composed of 9,600 excitatory LIF neurons and 2,400 inhibitory LIF neurons to a step input at t = 0 s and ending at t = 2 s for four different values of usage rate, u. Each run uses the same parameters w = 10, k = 2, Δτ = 38 ms and τr = 500 ms (for simulations with STD). All rate based networks use w = 100, k = 1.1, Δτ = 10.925 ms unless otherwise noted. For simulations with STD τr = 500 ms

Smnl in Eqs. (11)-(13) is the synaptic activation level of the projection from population n to population m where l can be either ampa or nmda for AMPA or NMDA receptors on an excitatory projection, or gaba for GABA (γ −Amino-Butyric Acid) receptors on inhibitory projections where τampa = 5 ms, τnmda = 100 ms and τgaba = 10 ms.

Projections from the inhibitory populations, Eq. (13), have only a single type of receptor, l = gaba, and do not have STD. Conversely, excitatory projections have multiple receptor types, l ∈{ampa, nmda}, which may occur in different ratios across synapses. They also have STD whose parameters may have different values across synapses allowing multiple possible combinations. We define a particular combination of AMPA/NMDA ratio, \(q_{mn}^{j}\), and set of STD, \(x_{mn}^{j}\), parameters by the index j and call it a synapse type.N is the number of synapse types for each excitatory projection, defined below. Note that Eq. (11) represents two equations, one for the (AMPA-)receptors on the projection from the excitatory population to itself (m = n = e), and one from the excitatory to the inhibitory projection (m = i, n = e). The analogous comment applies to Eqs. (12)-(14).

For projections from the excitatory population, \(q_{mn}^{j}\) defines the fraction of total synaptic strength for synapse type j from population n to population m that is represented by NMDA receptors. \(x_{mn}^{j}\) is the STD variable for synapse type j on the projection from populations n to m with baseline recovery time constant τr, and usage rate u, as computed in Eq. (14). We set τr = 500 ms for all simulations. We add an additional variable \(p_{mn}^{j}\) which defines a percentage shift from the baseline STD parameters.

We define two different models of STD, “homogeneous” and “heterogeneous”. For “homogeneous STD”, STD is the same for all excitatory projections, \(p_{mn}^{j}=0\) in Eq. (14). For “heterogeneous STD,” \(p_{mn}^{j}\) has multiple multiple non-zero values on each excitatory projection, see below for details. Note that in the heterogeneous STD model, STD is modified τr and through u, and that in both cases the dependence on \(p_{mn}^{j}\) is the same (both have a \((1+p^{j}_{mn})\) multiplicator). We make this choice for the sake of simplicity and parsimony but, if needed, it is possible to modify STD dynamics by changing only one of these parameters, or selecting different functions of \(p_{mn}^{j}\) for one or both of these parameters.

For networks without STD or with homogeneous STD (Figs. 2b-d, 3 and 4), there is only a single type of synapse for each excitatory projection so we have N = 1 and we can ignore the summation over j in Eqs. (11)-(12). The value of qie is always 0.5. The value of qee is used to select the response time of the overall network. As qee increases, the relative contribution of the slow-decay NMDA currents to EE currents increases and therefore the time constant on the EE projection increases relative to that of the IE projection. The value Δτ characterizes this change in relative timing between the EE and IE projections. It is defined as the difference between the average synaptic time constant of the EE projection (first term in the following equation) and that of the IE projection (second term),
$$\begin{array}{@{}rcl@{}} {\Delta}\tau&=&\left( (1-q_{ee})\tau^{ampa}+q_{ee}\tau^{nmda}\right) \\ &-&\left( (1-q_{ie})\tau^{ampa}+q_{ie}\tau^{nmda}\right) \end{array} $$
(15)
For networks with homogeneous STD, a single set of STD parameters is used on both EE and IE projections defined by τr and u where, as a reminder, \(p_{mn}^{j}=0\).
Fig. 3

Dependence on Δτ of rise time and decay time of the derivative feedback network with STD. a Activity of the derivative feedback network with STD for different values of Δτ with a step input starting at t = 0 s and ending at t = 2 s. b Rise time as a function of Δτ and u. c Decay time as a function of Δτ and u. d Ratio of rise time to decay time as a function of Δτ and u. All simulations in this figure use w = 100, k = 1.1, τr = 500 ms and u = 0.1 unless otherwise noted

Fig. 4

STD on the input to the derivative feedback network yields a large initial transient. a Schematic of input with STD to the excitatory population. Input depends on the starting feedforward weight, wff, the feedforward STD, xff and the firing rate of the feedforward connections, Rff. The step input is a step in Rff as depicted in the bottom half of the panel. (b) Plot of xff for three different usage rates, uff. c Input seen by the excitatory population after being filtered through half AMPA and half NMDA receptors (Eqs. 5-7) for the feedforward system in panel a and the STD values in panel b. d Simulation showing decreasing rise time for derivative feedback networks with stronger STD on their feedforward connections (i.e. when the inputs to the network are those shown in panel c). In this network Δτ = 1.6 ms yields a decay time of 1 second. Other parameters used are w = 100, k = 1.1, u = 0.1, τr = 500 ms and \(\tau _{r}^{ff}\) = 500 ms

For heterogeneous STD (Figs. 610), \(p_{mn}^{j}\) ≠ 0, we have two types of synapses on each excitatory projection (N = 2) and j takes the value slow or fast. To avoid confusion, we note that, following generally applied practice, we have previously referred to NMDA and AMPA receptors as slow and fast, respectively. In the following we formally define entire synapses (that have both AMPA and NMDA channels) as slow or fast, with the former having a higher proportion of NMDA channels than the latter. Specifically, the EE and the IE pathways each have a slow and a fast synaptic component. For both, we assume a 3:1 ratio of NMDA to AMPA for slow synapses, and the converse for fast synapses:
$$\begin{array}{@{}rcl@{}} &&q_{ee}^{slow}=q_{ie}^{slow}=0.75\\ &&q_{ee}^{fast}=q_{ie}^{fast}=0.25 \end{array} $$
(16)
We make a specific assumption (which is at this time a genuine prediction of our model) that STD differs between the excitatory pathways as follows. The STD recovery function, Eq. (14) is characterized by a single parameter p where
$$\begin{array}{@{}rcl@{}} p_{ee}^{fast}&&=p_{ie}^{slow}=p\\ p_{ee}^{slow}&=&p_{ie}^{fast}=-p \end{array} $$
(17)
Recall from Eq. (14) that increasing the value of \(p_{mn}^{j}\) increases STD. For the parameterization in Eq. (17), higher network activity suppresses the fast component of the EE projection relative to its slow component, and does the opposite for the IE projection. This causes the average synaptic time constant of the EE projection to increase relative to the IE projection when the network is active. The use of anti-symmetric values of p between the EE and IE connections reduces the number of free parameters while ensuring that the balance of positive and negative feedback, needed for the stability of the network (Gillary and Niebur 2016), is maintained throughout stimulus presentation. We will study generalizations of this assumption below.
As before, the change in relative timing is determined by the difference in average synaptic time constants between the EE and IE projections, Δτ. For the heterogeneous network this is obtained from a straightforward generalization of Eq. (15),
$$\begin{array}{@{}rcl@{}} {\Delta}\tau&=&\frac{1}{{\sum}_{j}x_{ee}^{j}}\sum\limits_{j}x_{ee}^{j}\left[(1-q_{ee}^{j})\tau^{ampa}+q_{ee}^{j}\tau^{nmda} \right] \\ &-&\frac{1}{{\sum}_{j}x_{ie}^{j}}\sum\limits_{j}x_{ie}^{j}\left[(1-q_{ie}^{j})\tau^{ampa}+q_{ie}^{j}\tau^{nmda}\right] \end{array} $$
(18)
The difference, Δτ, is now a function of the time-dependent variables \(x_{mn}^{j}\), rather than a constant as in the linear and homogeneous networks. We use the starting value of Δτ as a parameter of the network. Therefore, when we state the value of Δτ for the heterogeneous networks we are referring to the value of Δτ at t = 0 where \(x_{mn}^{j}(0)=1\) for all values of m, n and j, see Figs. 6-10.
Time varying input to the excitatory population is denoted by I(t). For simulations without feedforward STD, the input is a boxcar input filtered through half AMPA and half NMDA receptors as in the positive feedback model, Eqs. (5)-(7). For feedforward input with STD the input is again as in Eq. (5) but Eqs. (6)-(7) are replaced by
$$\begin{array}{@{}rcl@{}} \tau^{ampa}\frac{dS^{ampa}_{ff}}{dt}&=&-S^{ampa}_{ff}+w_{ff}x_{ff}R_{ff}(t) \end{array} $$
(19)
$$\begin{array}{@{}rcl@{}} \tau^{nmda}\frac{dS^{nmda}_{ff}}{dt}&=&-S^{nmda}_{ff}+w_{ff}x_{ff}R_{ff}(t) \end{array} $$
(20)
Here, feedforward STD xff is defined as,
$$ \frac{dx_{ff}}{dt}=\frac{1-x_{ff}}{\tau_{r}^{ff}}-u_{ff}x_{ff}R_{ff} $$
(21)
with uff and \(\tau _{r}^{ff}\) referring to the usage rate and recovery time constant of the feedforward connections, respectively. As before, the feed forward input firing rate Rff is boxcar-shaped. In order to maintain the same steady state input across feedforward STD parameters, we set the feedforward synaptic strength, wff, to yield a constant steady state input for each simulation, see Figs. 46 and 10.
The response time of the network is computed in the same way as for the positive feedback network. We again use the analytically derived steady state, \(R_{e}^{ss}\) for this computation. For the homogeneous network \(R_{e}^{ss}\) is the solution to a second order polynomial,
$$ a_{2}s^{2}+a_{1}s+a_{0}=0 $$
(22)
with coefficients:
$$\begin{array}{@{}rcl@{}} &&\kern-.5pc a_{0} = \frac{-I}{\tau_{r}u} \end{array} $$
(23)
$$\begin{array}{@{}rcl@{}} &&\kern-.5pc a_{1} = -I+\frac{1+w(k-1)}{\tau_{r}u(1+kw)} \end{array} $$
(24)
$$\begin{array}{@{}rcl@{}} &&\kern-.5pc a_{2} = 1. \end{array} $$
(25)
\(R_{e}^{ss}\) for the heterogeneous network is the solution to a third order polynomial,
$$ a_{3}s^{3}+a_{2}s^{2}+a_{1}s+a_{0}=0 $$
(26)
with coefficients:
$$\begin{array}{@{}rcl@{}} a_{0} &=& \frac{-I}{\tau_{r}^{fast}u^{fast} \tau_{r}^{slow}u^{slow}} \end{array} $$
(27)
$$\begin{array}{@{}rcl@{}} a_{1} &=& -I\left( \frac{1}{\tau_{r}^{fast}u^{fast}}+\frac{1}{\tau_{r}^{slow}u^{slow}}\right)\\ &&\qquad\qquad\qquad\qquad+\frac{1-\frac{w}{1+kw}}{\tau_{r}^{fast}u^{fast} \tau_{r}^{slow}u^{slow}} \end{array} $$
(28)
$$\begin{array}{@{}rcl@{}} a_{2} &=& -I+\left( \!1\!-\frac{1}{2}\frac{w}{1+kw}\right)\left( \frac{1}{\tau_{r}^{fast}u^{fast}}\! +\frac{1}{\tau_{r}^{slow}u^{slow}}\right) \end{array} $$
(29)
$$\begin{array}{@{}rcl@{}} a_{3} &=& 1. \end{array} $$
(30)
The largest root of each polynomial provides \(R_{e}^{ss}\) and defines the network response time.

2.3 Leaky integrate and fire networks

All our LIF networks have Ne = 9,600 excitatory neurons and the derivative feedback networks have, in addition, Ni = 2,400 inhibitory neurons. Neurons are randomly connected, with the probability of making a connection from any neuron to any other being ρ = 0.2. Each neuron is represented by the standard LIF equation, and synapses have exponentially decaying activation,
$$\begin{array}{@{}rcl@{}} \tau_{\theta}\frac{dV_{m}}{dt}&=&-(V_{m}-E_{l})+\sum\limits_{n,l}S^{l}_{mn} +I(t) \end{array} $$
(31)
$$\begin{array}{@{}rcl@{}} \tau^{l}\frac{dS^{l}_{mn}}{dt}&=&-S^{l}_{mn}+\sum\limits_{n,k}x^{j}_{mn}J^{l}_{mn}\delta(t-t_{mn}^{\alpha}) \end{array} $$
(32)
$$\begin{array}{@{}rcl@{}} \frac{dx^{j}_{mn}}{dt}&=&\frac{1-x^{j}_{mn}}{(1+p^{j}_{mn})\tau_{r}}-\sum\limits_{k}(1+p^{j}_{mn})ux^{j}_{mn}\delta(t-t_{mn}^{\alpha}) \end{array} $$
(33)
where τ𝜃, the membrane time constant, is τe for excitatory and τi for inhibitory neurons. We set both membrane time constants to 20 ms, again following Murphy and Miller (2009). El = −60 mV is the reversal potential. Vm is the membrane voltage for neuron m with firing threshold − 40 mV, a reset potential after firing of − 52 mV and a refractory period of 2 ms. \(S^{l}_{mn}\) is the synaptic activation for receptors of type l from neuron n to neuron m, \(t^{\alpha }_{mn}\) is the time of the α −th incoming action potential at this synapse and τl is its decay time of that receptor type, where τampa = τgaba = 10 ms and τnmda = 200 ms for all m and n. The strength of this synapse is \(J^{l}_{mn}\), with
$$\begin{array}{@{}rcl@{}} J^{ampa}_{mn}&=&(1-q^{j}_{mn})w/(N_{e}\rho) \end{array} $$
(34)
$$\begin{array}{@{}rcl@{}} J^{nmda}_{mn}&=&q^{j}_{mn}w/(N_{e}\rho) \end{array} $$
(35)
for excitatory synapses and
$$ J^{gaba}_{mn}=kw/(N_{i}\rho) $$
(36)
for inhibitory synapses.

We use j to define different types of synapses. For excitatory projections, a synapse is defined by its ratio of NMDA to AMPA receptors, \(q^{j}_{mn}\), and its STD parameter \(p_{mn}^{j}\). The index l separates the types of receptors and their activations in Eqs. (31)-(32) such that each synapse indexed by m and n has a type j which sets the relationship between AMPA and NMDA receptors indexed by l, see Eqs. (34)-(35). For all LIF networks the excitatory projections are evenly split between synapses which are 25% NMDA (\(q^{fast}_{mn}=0.25\)) and those which are 75% NMDA (\(q^{slow}_{mn}=0.75\)) so that the total amount of current through NMDA receptors is the same as through AMPA receptors. We define Δτ as in the rate based model using Eqs. (15) and (18). For inhibitory synapses there is only a single receptor type and no STD so Eq. (36) fully defines the inhibitory synapses.

The overall connectivity pattern of each network is the same as in the rate based simulations (see Fig. 1a for the positive feedback network and Fig. 1b for the derivative feedback network). For the positive feedback network with LIF neurons, the strength of recurrent connections is defined by a single overall parameter w and there is no inhibitory population. For the derivative feedback networks, excitatory projections have maximal strength w and inhibitory projection have strength kw.

For all networks without STD \(x^{j}_{mn}=1\) for all synapses and Eq. (33) can be ignored as in the rate based networks. For the positive feedback network with STD and the derivative feedback network with homogeneous STD, STD is the same at all synapses such that \(p^{j}_{mn}=0\) for every synapse in Eq. (33).

Each neuron receives Poisson-distributed background input from 5,000 independent neurons with synaptic strength of 0.2 mV. The firing rates of the background inputs were 1.825 Hz, 3.25 Hz and 1.875 Hz for the positive feedback network, derivative feedback network with homogeneous STD, and derivative feedback network with heterogeneous STD, respectively. Values were chosen to produce a baseline firing rate (for I(t) = 0) of approximately 1 Hz. The boxcar input to each network comes from Nff = 5,000 independent Poisson spiking neurons, each of which randomly connected to the excitatory population with probability ρ = 0.2. The strength of each individual synapse is Jff = wff/(Nffρ) where wff, the strength of the feedforward projection, is given for each simulation. All feedforward inputs are filtered with a τff = 100 ms time constant in order to mimic the temporal integration of the step input caused by upstream populations,
$$\begin{array}{@{}rcl@{}} I(t)&=&\sum S_{ff} \end{array} $$
(37)
$$\begin{array}{@{}rcl@{}} \tau_{ff}\frac{dS_{ff}}{dt}&=&-S_{ff}+\sum\limits_{k}x_{ff}J_{ff}\delta(t-t^{k}) \end{array} $$
(38)
where k indexes each spike at a synapse and xff = 1 if there is no feedforward STD. Equation (37) is the sum over the approximately ρNff poisson inputs to each neuron in the excitatory population. If there is feedforward STD then
$$\begin{array}{@{}rcl@{}} \frac{dx_{ff}}{dt}&=\frac{1-x_{ff}}{\tau_{r}^{ff}}-\sum\limits_{k}u_{ff}x_{ff}\delta(t-t^{k}) \end{array} $$
(39)
for each feedforward synapse. For illustration purposes, all LIF network output shown below is averaged across the excitatory population and then smoothed with a Gaussian filter with σ = 4 ms.

2.4 Summary of model parameters

In Table 1 we summarize the parameters that we vary for all seven model structures in this paper. We do not include membrane time constants, receptor time constants or NMDA/AMPA fractions q, all of which remain constant throughout the paper. Table 1 also applies to the LIF networks which use the same set of parameters, with the exception of LIF networks without feedforward STD. In those, we replace Iff in the rate based network with a set of Poisson spiking neurons with baseline firing rate Rff and projection strength wff. The values of parameters for each set of simulations will be defined in the captions of their respective figures.
Table 1

Summary of model parameters

Model

Parameters

Positive Feedback:

 

no STD

w and Iff

with Recurrent STD

w, u, τr and Iff

Derivative Feedback:

 

no STD

w, k, Δτ and Iff

with homogeneous STD

w, k, Δτ, u, τr and Iff

with homogeneous STD and feedforward STD

w, k, Δτ, u, τr, uff, \(\tau _{r}^{ff}\), wff and Rff

with heterogeneous STD

w, k, Δτ, u, τr, p and Iff

with heterogeneous STD and feedforward STD

w, k, Δτ, u, τr, p, uff, \(\tau _{r}^{ff}\) and Rff

All Matlab and Python code used in this paper is available at: https://github.com/llgigll/FastRiseSlowDecay.

3 Results

3.1 The impact of STD on positive feedback networks

In this section we examine the impact of short-term depression on the decay time of positive feedback networks. The attractor states of such networks have previously been examined in the context of STD and their decay times have also been examined, however without STD (Barak and Tsodyks 2007; Gavornik et al. 2009). Here we examine how transient decay times of positive feedback networks are influenced by STD of their recurrent projections. A positive feedback network can be represented simply by an excitatory population with a single recurrent projection, as in Fig. 1a. We define transient decay as a network returning to baseline activity after the offset of a stimulus. We will show that for positive feedback networks, with STD on the recurrent connections, the time constant of decay is significantly shorter than without STD, and that it is more dependent on the overall strength of STD than on the starting synaptic weights.

The dynamics of a rate-based network with a single excitatory recurrent connection are determined predominantly by the strength of that recurrent connection, x(t)w, see Fig. 1a. As a reminder, w is the maximal strength of the recurrent connection and x is the effect of short-term depression which is parameterized by the usage rate u and the recovery rate τr. We will look at the response of the network to a step input for a range of values of w and u. We hold τr constant but its impact on the network is similar to changing u. We require w ≤ 1 which ensures that the network state decays back to zero after stimulus offset (see Eq. (8) with I = 0).

Figure 1b shows the results of simulations of the network in Fig. 1a for four different strengths of STD but with the same steady state activity. With no STD on the recurrent connection the network has a decay time of 25 s. We chose 25 s arbitrarily to show the significant impact of STD on the decay time. Note that since the stimulus lasts for only 2 s the simulation without STD does not come close to reaching its steady state activity before the offset of the stimulus. Even relatively weak STD yields a dramatic decrease in the decay time of the network. When u = 0.05, which is at the low end of usage rates observed in cortex (Beck et al. 2005), the decay time of the network decreases to a few hundred milliseconds. This is a change of two orders of magnitude and the decrease becomes even larger for usage rates of 0.1 and 0.2 which are more consistent with those observed in early sensory areas (Beck et al. 2005). We note that the rise time of the network also decreases as STD becomes stonger. A reduction in overall response times to both the onset and offset of a stimulus could be one reason that early sensory areas tend to be dominated by STD since fast reaction to changing sensory input is a likely priority. We focus on decay time here but will consider rise time in a subsequent section.

Although this network is nonlinear, the change in the decay time constant of the network can be understood by considering the decay time constant of a linear network (without recurrent STD) as a function of the synaptic strength at each time point in the simulations in Fig. 1b. In this case the linear network would have an instantaneous recurrent strength wlin = x(t)w. Figure 1c shows the decay times for linear networks with recurrent strength wlin at each time point in Fig. 1b. By definition, the network with no STD has a constant decay time. In contrast, networks with STD show a precipitous decrease in decay time after stimulus onset. This decline occurs because the decay time of a linear positive feedback network is proportional to 1/(w − 1) (seen directly from a simplified linear version of Eq. (1)). Figure 1d shows this relationship between decay time and w for the network without STD. Decay time for each value of w is computed directly from a simulation. Irrespective of the starting value of w, STD decreases the effective synaptic strength and thus drives each network toward the left hand side of Fig. 1d, compressing its decay time. When the network dynamics show long time constant decay, then very small changes in w produce large changes in decay time (imagine decreasing w in Fig. 1d when w ≈ 1).

Figure 1e shows that the compression of decay time that we would expect from the linear model does actually occur for networks with recurrent STD. We see that the addition of even weak STD, u = 0.05, reduces the decay time to approximately 700 ms or less for all values of w. Stronger STD reduces network decay times even further. In addition to decreasing the decay time of the network, recurrent STD also makes networks with w > 1 stable. However, when w > 1 in a positive feedback network with STD the network will decay to a non-zero steady state after the offset of a stimulus. Since we are considering networks without attractor like dynamics this provides an effective upper limit on the network decay time at w = 1. Additionally, increasing w past unity does not significantly increase the decay time of the network, even to an attractor state (Barak and Tsodyks 2007). Figure 1f shows the maximum decay time of the network (for w = 1) for a range of usage rates. The figure shows that it is very strongly dependent on u. When u < 0.05 then the network approaches the behavior of the linear system.

We conclude that in networks with STD and physiological usage rates, the positive feedback network has a decay time less than 500 ms. This implies that persistence in early sensory areas, where STD dominates, must be caused by a mechanism other than positive feedback. Figure 1g shows the same dynamics as in Fig. 1a but for a network made up of LIF neurons. The addition of STD has the same impact on decay time as it does in the rate based network implying that these concepts also hold for stochastic spiking networks. In the next section we will consider derivative feedback networks and show that they are far more robust to the addition of STD on their excitatory connections.

3.2 Derivative feedback networks with homogeneous STD

Recurrent networks with negative derivative feedback have been discussed as a model for working memory, mainly in central areas like prefrontal cortex (Lim and Goldman 2013). Recent work has shown that derivative feedback networks with STD can also produce persistent activity that decays to the baseline value over the course of up to several seconds (Gillary and Niebur 2016). Return to the baseline value is a property needed in circuits representing sensory information where, different from short-term memory circuitry, attractor dynamics are undesirable. Figure 2a shows an example of a derivative feedback network where w represents the baseline strength of excitatory connections and kw is the strength of inhibitory projections. In such networks the decay time depends strongly on both the overall strength of the network connections and on the difference in average synaptic time constants between the EE and IE projections. In our model the average synaptic time constant is determined by the relative proportion of AMPA and NMDA receptors on the synapses in a projection. In Fig. 2a we have only one type of synapse on each projection but each projection has a different proportion of AMPA and NMDA receptors. We define projections with relatively more NMDA as slow projections and color those synapses red while projections with relatively more AMPA receptors are termed fast projections and colored green. When the EE projection slows relative to the IE projection then the response time of the network increases. We call this difference in average synaptic time constant between excitatory projections Δτ (see Eqs. (15), (18)) where positive Δτ indicates a relatively slower EE projection when compared to the IE projection. Stability of this and related systems is analyzed by Gillary and Niebur (2016).

As in the positive feedback network we are interested in how STD on the excitatory connections impacts the decay time of the network. In this section we consider derivative feedback networks with homogeneous STD on all excitatory projections (EE and IE), where STD decreases the overall strength of excitatory connections. Figure 2b shows simulations of this network for four different STD strengths. Strikingly, although we selected the parameters of the network with no STD to give the same response times as the positive feedback network with no STD shown in Fig. 1b, the decrease in decay time of the derivate feedback network due to STD is much less pronounced. This can also be seen in the decay times computed for the linear systems as a function of STD at each time point in the simulations, Fig. 2c, to be compared to the much steeper decay in Fig. 1c. Although there is a significant reduction in the decay time of the linear systems the reductions are much smaller than in the positive feedback network for the same STD parameters.

The relative robustness of the derivative feedback network is due to the linear relationship between the strength of the network projections, w, and the decay time of the linear derivative feedback network, Fig. 2d. This is in stark contrast to the approximately geometric relationship between w and the decay time constant in the linear positive feedback network, Fig. 1d. When STD is added to the derivative feedback network the relationship between w and decay time remains linear, Fig. 2e. The strength of STD determines the slope of the relationship between w and decay time. As can be seen from the plot, the derivative feedback network with STD can produce decay times which are more than long enough to replicate the persistence observed in cortex (Shuler and Bear 2006). This is in part due to the fact that there is not an upper limit on the value of w as there is in the positive feedback network where w > 1 results in either instability for the positive feedback network without STD or an attractor state for the positive feedback network with STD. Similar dynamics are also obtained for spiking networks, Fig. 2f.

We have thus shown that derivative feedback networks can develop persistent activity that is far more robust to reductions in synaptic strength due to STD than positive feedback based networks. This robustness to STD implies that, for early sensory systems where STD dominates, derivative feedback networks are significantly more capable of providing the persistence observed in experimental work. Although we have considered only the decay of the derivative feedback network, for a memory system to work, it must also be able to encode information on time scales much shorter than its memory. In the next section we examine how the rise time of the network is impacted by STD on both the recurrent connections as well on the feedforward input to the network.

3.3 Rise time in the derivative feedback network with STD

A defining characteristic of any memory mechanism is that memories are acquired much faster than they decay. However, such dual time constant responses are not possible in linear networks. This is because responses of such linear networks are symmetric to step increases and decreases of equal and opposite magnitude in the input. Therefore, if we change the parameters to yield persistent activity after stimulus offset we also slow down the response of the network to stimulus onset. This is not necessarily the case any more for nonlinear systems, e.g. when we introduce STD. Figure 3a shows three simulations of the derivative feedback network with STD for different values of Δτ (see Eq. (18)). As Δτ increases we see an increase in both the rise time and the decay time of the network, similar in tendency to what we would expect in a linear system. The STD-associated nonlinearity on the excitatory connections, does, however break the exact symmetry between the offset and onset response of the network. Figure 3d shows that for a large range of Δτ and different STD strengths the rise time of the network is only about 30–50 % of the decay time. Therefore, a network with a decay time of ∼1 second will have a rise time of ∼300–500 milliseconds. While the decay times are comparable to those observed in sensory cortex, the rise time is far longer than observed. Furthermore, rise time and decay time are strongly correlated across the range of Δτ’s. In cortex, a correlation between the rise time and the decay time of the network is not observed (Shuler and Bear 2006; O’Herron and von der Heydt 2009).

In order to decouple the response times at the onset of the stimulus from those at the offset we require some additional form of nonlinearity. In previous work this decoupling was obtained through strong input transients to the network on a base level of much lower steady state input (Gavornik et al. 2009; Lim and Goldman 2013). Such a pulse-step approach can be accomplished with STD on the feedforward input to the network. Figure 4a-c depicts how STD produces a step input for three different feedforward usage rates, uff, see Eq. (21). The overall input to the excitatory population is Iin(t) = wffxff(t)Rff(t) where wff is the (constant) starting strength of the feedforward projection, xff is the STD influence, and Rff is the firing rate of the input. The step input to the excitatory population is produced by a step change in Rff which rises at t = 0 s and returns to baseline at t = 2 s. The step input drives STD on the afferent synapses, Fig. 4b. The combination of decreasing synaptic strength and step change in input yield a large initial transient current followed by a smaller steady state input, Fig. 4c. The largest usage rate, uff = 0.5, see Fig. 4c, approaches the tail end of the distribution of usage rates that is observed in cortex (Beck et al. 2005). For this value, the input current attains a peak that is approximately three times the steady state value. Such a ratio is consistent with what has been observed in thalamic input to layer 4 cells in mouse primary visual cortex (Reinhold et al. 2015).

In Fig. 4d we chose Δτ such that the network has a 1 second long decay time. This network receives the three different transient inputs depicted in Fig. 4c. Although the transients do shorten the rise time of the network, all three values produce long rising transients peaking after 350 milliseconds. These transients are much longer than those observed for cortical networks with persistent activity. We conclude that, although STD on the input to the network decreases rise time relative to decay times, it is not sufficient to produce the observed fast onset transients and, at the same time, persistence over the observed range. In the following section, we show that heterogeneous STD on the recurrent connections can change the underlying time constant of the network from fast to slow over the course of a stimulus presentation yielding the observed difference in response time.

3.4 Derivative feedback networks with heterogeneous STD

In the previous sections, we have considered homogeneous STD and a single ratio of AMPA to NMDA receptors on each projection. It is known, however, that most networks have a range of STD values as well as ratios of AMPA to NMDA receptors (Castro-Alamancos and Connors 1997; Myme et al. 2003). Irrespective of the distribution of STD parameters, STD will always decrease the overall strength of a projection. However, STD may either increase or decrease the average synaptic time constant of that projection, depending on how STD parameters and AMPA/NMDA ratios are distributed. For example, if a synapse with a stronger NMDA component has weaker STD while another synapse has both a stronger AMPA component and stronger STD, then the average time constant of the combined synaptic input will increase during stimulus presentation. In Fig. 5a the EE projection is an example such a projection with a fast synapse and a slow synapse. The fast synapse is 25% NMDA while the slow synapse is 75% NMDA. The two synapses have STD with different strength. We define this difference in strength as a change from baseline STD by the parameter p where, for the EE connections, the STD parameters on the fast synapse, xfast, is increased by p and STD on the slow synapse, xslow, is decreased by p with the opposite change being made for the IE connections, see Eq. (14) and discussion following it. The choice of equal but opposite changes on the EE and IE connections ensures that the balance of excitation and inhibition is maintained within the network. When p > 0 this yields stronger STD on the fast EE synapses and weaker STD on the slow EE synapses and vice-versa on the IE synapses. If p = 0 we have homogeneous STD as in previous sections.
Fig. 5

STD can change the average synaptic time constant of projections during a stimulus presentation. a Schematic of the derivative feedback network with heterogeneous STD on the two excitatory projections. Red and green synapses indicate combinations of AMPA and NMDA receptors yielding a longer or shorter average synaptic time constant respectively. Synapses with dotted fills have relatively weak STD while synapses with bars have relatively strong STD. This combination of synapse speed and different STD causes each projection to change its average synaptic time constant, τave, for increasing levels of activity. On the EE connection this causes an overall increase in τave and on the IE projection this causes a decrease in τave making, the EE connection slower than the IE connection. This causes a significant increase in Δτ and therefore a significant increase in the overall response time of the network (see Fig. 3a). A network with this structure slows significantly over the course of a stimulus presentation. b Plot of weak and strong STD for a step increase of the input by 20 Hz at t = 0 s. Each STD has base parameters τr = 500 ms and u = 0.15. We use p = 0.15 giving a 15% increase in the STD parameters for strong STD and a 15% decrease in the STD parameters for weak STD. c Synaptic strength-weighted average time constant of the projection as a function of time. As the system evolves the difference in the average time constant, Δτ, between the projections increases

Figure 5b shows the evolution of strong and weak STD for a step input at t = 0. The STD decreases the overall strength of the input to the population by ≈ 60% over the course of the stimulus presentation while the difference in average synaptic time constant between the projections, Δτ, increases by 8 ms, Fig. 5c. Looking back at Fig. 3c we see that a change Δτ in the derivative feedback network of 8 ms can change the decay time of the network by seconds. Therefore, if STD causes an increase in the average synaptic time constant on the EE projection and a decrease on the IE projection, then Δτ for the network will increase by a large margin. We call networks with STD of this form derivative feedback networks with heterogeneous STD, see Fig. 5a.

Figure 6a shows simulations of the derivative feedback network with heterogeneous STD for four different values of p. As p increases the network yields longer decay times while still maintaining a fast rise time although large p does produce a pronounced dip in activity after the initial peak. Figure 6b shows Δτ of the same network as a function of time. The initial fast rise of the network is caused by a starting small negative value of Δτ which can be seen at t = 0 s. Note that when Δτ is small and negative that derivative feedback networks are close to critically damped (Gillary and Niebur 2016). When p = 0 (i.e. the homogeneous network) then Δτ is constant across the entire simulation. However, when p is positive the initially negative value of Δτ becomes large and positive by stimulus offset, thereby producing the slow decay after stimulus offset. The large difference in rise time versus decay time as a function of p is illustrated in Fig. 6c. The decay time of the network is approximately linearly related to p while the rise time remains close to zero. Since p represents a percentage change from the baseline parameters we also examine how changing the baseline usage rate, u, impacts the network response. Figure 6d shows the network response time as a function of u for p = 0.1. Although the decay time of the network decreases as u increases the network still maintains decay times longer than 1 s even for large values of u. Therefore, we see that the network dynamics are robust across a large range of baseline parameters.
Fig. 6

Heterogeneous STD allows networks to change response time over the course of a stimulus presentation. a Response of derivative feedback network with heterogeneous STD on its excitatory projections to a step input at t = 0 s and ending at t = 2 s for four different values of p. b The difference in average synaptic time constant between the EE and IE projections, Δτ, as a function of time. Δτ is computed using the synaptic strengths from panel a at each time point. c Rise and decay time of the derivative feedback network with heterogeneous STD as a function of p. d Rise and decay time of the derivative feedback network with heterogeneous STD as a function of the base usage rate, u, and with p = 0.1. e Simulations of the derivative feedback network with heterogeneous STD and feedforward STD. f Rise and decay time of the derivative feedback network with heterogeneous STD and feedforward STD. Response times are plotted as a function of p. g-h Fit of the derivative feedback network with heterogeneous and feedforward STD to decaying activity in two different monkeys. Data is taken from previous experimental work (O’Herron and von der Heydt 2009) discounting the onset latency (55 ms) and offset latency (103 ms) of the average neural response. Both fits use Δτ = −0.38 ms, uff = 0.1 and wff set so that each panel has a steady state firing rate of 8 Hz. The only difference in parameters between g and h is p where panel g has p = 0.07 and panel h has p = 0.05. All simulations use base parameters of w = 100, k = 1.1, τr = 500 ms and u = 0.1. Heterogeneous networks without feedforward STD use Δτ = −0.7125 ms and have I(t) chosen to yield a 25 Hz steady state. Heterogeneous networks with feedforward STD use Δτ = 0 ms, uff = 0.2, \(\tau _{r}^{ff}\) = 500 ms, Rff = 15 Hz and have wff chosen to yield a 25 Hz steady state. Any deviations from these values are noted in the panel descriptions

The initial fast response of the network is due to the negative starting value of Δτ. However, the network will also produce a fast response with larger starting values of Δτ if there is STD on the input, as in Fig. 4. Figure 6e shows simulations of the derivative network with heterogeneous STD and, in addition, with STD on the afferent projection. These simulations still show long decay times and short rise times, see also Fig. 6f. Additionally, we see that the dip following stimulus onset seen in Fig. 6a is reduced or removed when there is STD on the afferent synapses.

What Fig. 6 essentially shows is that the combination of STD and derivative feedback networks can reproduce a wide variety of dynamics that are observed in cortex by altering parameters in the local network. This is in contrast to both positive feedback networks and derivative feedback networks with homogeneous STD in which fast rise and slow decay can only be achieved by assuming unphysiologically high rates of STD on the input and virtual absence of STD on the recurrent connections.

In Fig. 6g,h we compare the behavior of our network with neuronal activity in secondary visual cortex (area V2) of two awake behaving monkeys. We use the data in Fig. 2a from O’Herron and von der Heydt (2009) which shows the difference of firing rates of border-ownership selective neurons between responses to an object in their preferred and non-preferred locations. In the experiment, a stimulus is presented either in the preferred or non-preferred border ownership location from t = 0 to t = 0.5 s, after which the object is replaced by an edge in the receptive field of the neuron that is devoid of any border ownership information, see O’Herron and von der Heydt (2009) for details. The blue curves in Fig. 6g,h show this difference for the two monkeys. It rises very fast (within less than 50 ms after stimulus onset), is maintained while the stimulus has border ownership information (until t = 0.5 s), and then decays slowly to zero over the course of about one second. It is believed that the modulation is produced by populations of neurons (“grouping cells”) in a downstream cortical area that represent the grouping of object features (Craft et al. 2007; Mihalas et al. 2011; Sugihara et al. 2011; Russell et al. 2014). We model grouping cell populations as derivative feedback networks with heterogeneous STD and plot their responses (orange lines) along with experimental data (blue lines) in Fig. 6g,h. The only difference in the fits between the two plots is a slightly higher value of p, in Fig. 6g than in Fig. 6h, chosen to match the decay time of each network. The full parameter set is listed in the figure caption. The agreement of simulation results with the experimental data for both monkeys is very good, suggesting that dynamics generated by a derivative network with heterogeneous STD may be responsible for the generation of the observed modulation of neuronal activity.

Up to this point we have chosen specific values for the ratio of AMPA to NMDA and a very specific anti-symmetric structure for STD. However, the dynamics of our network do not depend stringently on either of these choices. In Fig. 7 we show that many different values of q provide fast rise and slow decay. We define two new variables: Δq and qshift.
$$\begin{array}{@{}rcl@{}} &&q_{ee}^{slow}=q_{ie}^{slow}=0.5+{\Delta} q+q_{shift}\\ &&q_{ee}^{fast}=q_{ie}^{fast}=0.5-{\Delta} q+q_{shift} \end{array} $$
(40)
where Δq represents the difference in the amount of NMDA on fast and slow synapses while qshift represents changes in the average amount of NMDA across the two synapses in our model. Under this formulation all of our previous simulations have used Δq = 0.25 and qshift = 0. In Fig. 7 a,b we set qshift = 0 and vary Δq for a range of different values of p. As can be seen, the value of Δq has very little impact on the rise time of our network but does significantly impact the decay time. The greater the difference in the amount of NMDA on fast and slow synapses, the larger the value of Δτ that can be driven by the asymmetric STD on the fast and slow synapses. Conversely, the baseline amount of NMDA on the synapses as represented by qshift has very little impact, Fig. 7 c,d. The black crosses and black plus signs on each panel in Fig. 7 represent the values of q and p used in Fig. 6 g,h. Our fit for q clearly sits within a range of possible values. In Eq. (17) we define an anti-symmetric relationship between the STD parameters on the EE and IE projections. In Fig. 8 we loosen this condition. In panel a we allow STD on the EE projection to change as in previous simulations but hold the STD parameters on the IE projection constant: \(p_{ee}^{fast}=p\), \(p_{ee}^{slow}=-p\) and \(p_{ie}^{fast}=p_{ie}^{slow}=0\). As the value of p increases the network shows fast rise and slow decay. However, it also begins to have an excess of positive feedback. For large values of p it produces an attractor state due to this excess. The reason this occurs is that the removal of the anti-symmetric condition causes EE connection to become stronger than the IE connection when the network is activated. Importantly, the STD parameters have to change by more than 10% before the network approaches this attractor state.
Fig. 7

The rise time and decay time of heterogeneous STD are robust to changes in the AMPA/NMDA ratio. a Rise time of the derivative feedback network with heterogeneous STD as a function of Δq (see text) and p. b Same for the decay time. c Rise time of the derivative feedback network with heterogeneous STD as a function of qshift (see text) and p. d Same for the decay time. All simulations use base parameters of w = 100, k = 1.1, τr = 500 ms, u = 0.1, Δτ = −0.7125 ms and have I(t) chosen to yield a 25 Hz steady state. The black + in the panels represents the parameter values of q and p used in Fig. 6g while the black x represents the values of q and p used in Fig. 6h

Fig. 8

The impact of heterogeneous STD depends on the maintenance of the balance between positive and negative feedback. a Response of the derivative feedback network with heterogeneous STD on its EE projection and homogeneous STD on its IE projection. We use \(p_{ee}^{fast}=p\), \(p_{ee}^{slow}=-p\) and \(p_{ie}^{fast}\) = \(p_{ie}^{slow} =0\). Note the emergence of an attractor state for the largest value, p = 0.15 (purple line). b Response of the derivative feedback network with heterogeneous STD on its EE projection and balanced homogeneous STD on its IE projection. We use \(p_{ee}^{fast}=p\), \(p_{ee}^{slow}=-p\) and \(p_{ie}^{fast}\) = \(p_{ie}^{slow}\) = pbal (see text). This re-balanced network avoids attractor dynamics for all values of p shown, including p = 0.15 (purple line). All simulations use base parameters of w = 100, k = 1.1, τr = 500 ms, u = 0.1, Δτ = −0.7125 ms and have I(t) chosen to yield a 25 Hz steady state

The network can be re-balanced without the anti-symmetric condition. Instead, we can slightly reduce the strength of STD on the IE projection such that the network maintains the balance of positive and negative feedback after activation. In Fig. 8b we set \(p_{ee}^{fast}=p\), \(p_{ee}^{slow}=-p\) and \(p_{ie}^{fast}=p_{ie}^{slow}=p_{bal}\) where pbal is a reduction in the strength of STD on the IE projection that balances the positive and negative feedback but produces no change in the timing of positive and negative feedback. We use values of pbal = {0, −0.0015, −0.0061, −0.0135} for each respective value of p = {0, 0.05, 0.1, 0.15}. The values of pbal were calculated by matching the steady state strength of the EE and IE projections for a firing rate of 25 Hz for each value of p. Figure 8b shows that this way of balancing the network maintains the temporal dynamics while preventing the network from approaching an attractor state.

We note that there is an overall reduction in the range of possible decay times as compared to previous simulations. This reduction occurs because pbal yields no difference in timing on the IE projection thereby reducing Δτ. It should also be noted that the purple trace (p = 0.15) does have a longer decay time than the yellow trace which is lost due to the purple trace failing to reach steady state before the offset of the input. These simulations show that although the anti-symmetric condition simplifies the study of the heterogeneous network, it is not a necessity to get both fast rise and slow decay. The removal of the anti-symmetric condition simply requires that STD parameters are calculated such that positive and negative feedback are balanced. Consequently, Fig. 8 does show that our model requires that the balance between positive and negative feedback is maintained, something to be expected in any network with strong recurrent connections. A balance between excitation and inhibition is also required by other models as well as observed during in-vivo recordings in visual cortex (Rubin et al. 2015; Lim and Goldman 2013; Xue et al. 2014).

Our conclusions about the derivative feedback network also hold for spiking networks with the same structure, as shown in Fig. 9. When the persistence of the network is caused by Δτ then both rise time and decay time increase together, Fig. 9a. The addition of strong STD decreases the rise time but not enough to approach the rise times observed in cortex, Fig. 9b. Conversely, heterogeneous STD is capable of producing fast rise and slow decay but with a pronounced dip, Fig. 9c. The addition of STD on the feedforward connections significantly reduces the dip, Fig. 9d.
Fig. 9

LIF networks have dynamics similar to the rate based models. a Response of derivative feedback network with homogeneous STD for three different values of Δτ. Each run uses w = 10, k = 2, u = 0.1, τr = 500 ms, Rff = 20 Hz and wff = 0.6. b Response of derivative feedback network with homogeneous STD and feedforward STD for three different values of uff. Each run uses w = 10, k = 2, u = 0.1, τr = 500 ms, Δτ = 38 ms, Rff = 20 Hz and wff = 0.6(1 + uffτrffRff). c Response of derivative feedback network with heterogeneous STD for three different values of p. Each run uses w = 20, k = 1, u = 0.1, τr = 500 ms, Rff = 20 Hz and wff = 0.55. For each p we use (0, 0.125 or 0.25) we have a different respective value of Δτ (−2.47 ms, −3.23 ms or −4.18 ms). d Response of derivative feedback network with both heterogeneous STD and feedforward STD for three different values of uff. Each run uses w = 20, k = 1, u = 0.1, τr = 500 ms, p = 0.125, Δτ = −1.14 ms, Rff = 20 Hz and wff = 0.55(1 + uffτrffRff). All simulations in Figure use a step input at t = 0 s and ending at t = 2 s

3.5 Robustness of the heterogeneous network response

The evolution of STD depends not only on the values of the parameters determining synaptic depression but also on the activity of the synapses. This means that for different lengths of stimulus presentation and for different steady state firing rates we expect the overall response of the network to change. Figure 10a shows the response of the derivative feedback network with heterogeneous STD to three different lengths of step input. The onset response of the network is exactly the same in each case, as it has to be for networks with exactly the same initial conditions. The decay time, however, depends on the stimulus length. Their correlation can be clearly seen in Fig. 10b where we show the decay time of the homogeneous network (solid line) and the heterogeneous network (dashed line) as a function of stimulus length. For stimulus lengths above 250 ms in the heterogeneous network, the reduction in decay time from the asymptotic value (i.e. for a stimulus length of 1 s) is less than 15%. Conversely, for the derivative feedback network with homogeneous STD, decreasing the stimulus length to 250 ms from 1 s causes more than a 100% increase in decay time. We choose 250 ms stimulus duration for the comparison because the average fixation duration for monkeys is on the order of 250 ms in free viewing tasks (Guo et al. 2006). The change in decay time for the homogeneous network has the opposite sign and is an order of magnitude larger than for the network with heterogeneous STD. This is an important prediction of our model.
Fig. 10

The impact of stimulus length and steady state firing rate on decay time. a Simulations of the derivative feedback network with heterogeneous STD and feedforward STD for different stimulus lengths. The strengths of the feedforward connections were set to give a steady state firing rate for the population of 25 Hz. b Decay time of the derivative feedback network with homogeneous and heterogeneous STD on the recurrent connections as a function of stimulus length. c Simulations of the derivative feedback network with heterogeneous STD and feedforward STD for different levels of steady state activity. The strengths of the feedforward connections were set to give the steady state firing rates indicated in the legend. d Decay time of the derivative feedback network with homogeneous and heterogeneous STD on the recurrent connections as a function of steady state firing rate, Ress. All simulations of networks with heterogeneous STD in this figure use w = 100, k = 1.1, τr = 500 ms, u = 0.1, p = 0.1, Δτ = 0 ms, uff = 0.2, τrff = 500 ms and Rff = 15 Hz. Networks with homogeneous STD use the same parameters except p = 0 and Δτ = 4.75 ms

While both positive feedback models and derivative feedback network models with homogeneous STD predict strongly decreasing decay times for longer stimulus presentations, the derivative feedback network with heterogeneous STD shows an increasing decay time for longer stimulus lengths, as seen in Fig. 10b. Such behavior has been recently observed in the olfactory bulb (Patterson et al. 2013). Additionally, adaptation aftereffects in perception generally show the same correlation: longer adaptation produces longer lasting aftereffects (Greenlee et al. 1991; Leopold et al. 2005). This prediction could easily be tested more generally in any experimental paradigm yielding persistent activity after stimulus offset.

Figure 10c shows simulations for the same derivative feedback network as Fig. 10a but with different starting feed-forwards weights, wff, to yield a range of steady state firing rates, \(R_{e}^{ss}\), for the excitatory population. Furthemore, now the input duration is the same for all three cases. As in the simulations with different stimulus lengths in panel a the different trials do not show appreciably different rise times. There is, however, an inverse correlation between \(R_{e}^{ss}\) and the decay time of the network, dashed line Fig. 10d. For this choice of parameters the decay time of the network with heterogeneous STD decreases with a slope of 30 ms/Hz between 10 Hz and 50 Hz. However, the homogeneous network, solid line, has an average slope of 75 ms/Hz between 10 Hz and 50 Hz, far larger than for the network with heterogeneous STD. Overall Fig. 10 shows that, in addition to being able to replicate the fast rise and slow decay observed in cortex, the network response, when decay times are driven by heterogeneous STD, is also significantly more robust to changes in input than networks whose decay time is determined by a static Δτ.

4 Discussion

Persistent decaying activity has been observed in a number of early sensory areas. Most previous work on neuronal persistence has been on understanding short-term memory. Substantial progress has been made by the study of networks with attractor dynamics. Such dynamics are, however, not suitable for sensory coding whose task it is to efficiently and faithfully represent environmental variables. In addition (and possibly for this reason), synapses show much stronger short-term depression effects in early sensory cortex than in more central areas (Hempel et al. 2000). Here we examine the impact of STD on the transient dynamics of two classes of circuit models, positive feedback and derivative feedback networks. Both network types have been used to produce either slow integration or slow decay at the onset and offset of neural signals (Lim and Goldman 2014).

We first show that the dynamics for positive feedback models are strongly influenced by the presence of STD on the recurrent excitatory connections. Even for relatively weak STD, such networks can only produce fast onset and offset dynamics but not the observed persistent activity. Conversely, we show that the range of transient dynamics for derivative feedback networks is robust to STD. However, as with positive feedback networks, derivative feedback networks which decay slowly at stimulus offset also respond slowly to stimulus onset. Such networks can only produce a fast onset response while maintaining a level of persistence if they are driven by a large initial transient input. While onset transients are commonly observed in sensory cortex, their magnitude is much lower than what is required to generate the large difference between the time courses of rise and decay of network activity found experimentally. The derivative feedback networks considered at this point therefore do not explain observed neuronal behavior either. Therefore, we look for a way to produce fast rise times and slow decay times by altering the intrinsic time constant of the network rather than through the shape of the input.

It turns out that removing a simplification we (and others) had originally made in the construction of the model solves this problem. We had adopted the standard model of STD (Tsodyks et al. 1998) in which synaptic strength is governed by a single variable (the case \(N=1, p_{mn}^{j}=0\) in Eqs. (11)–(14) and discussed in Section 3.1). In reality, most networks have a range of STD values (Castro-Alamancos and Connors 1997). We approximate this situation in possibly the simplest way by allowing the excitatory projections (EE and IE) to have synapses with different strengths of short-term depression. Different from the homogeneous STD model considered earlier, we now assume that each projection has two types of synapse, one fast-acting and one slow-acting, as an approximation for the more general case in which a whole range of time courses exists. As for the homogeneous model, and in agreement with physiological data (Myme et al. 2003), each of these projections consists of a mixture of NMDA-type and AMPA-type receptors.

The fast and slow components are naturally obtained through synapses with different proportions of fast (AMPA) and slow (NMDA) channel dynamics. We assume, somewhat arbitrarily that the ratio is three-to-one in both cases, Eq. (16) (but show that our results do not depend critically on this choice, Fig. 7). The crucial modification from the homogeneous model is that we assume that these two components have different short term depression behavior. Little quantitative information is available about the details of the distributions of STD characteristics, therefore we make the simplifying assumption that STD is the same on the fast EE connection and the slow IE connection, and equal but opposite on the other two connections, see Eq. (17). While the resulting reduction of free parameters, from four to one, makes the network easier to understand, we emphasize that this constraint can be easily relaxed as soon as more information about the relative and absolute strength of STD in different recurrent connections becomes available. In Fig. 8b we show one particular example of how this constraint might be relaxed. However, in a network with many synaptic connections there would also be many possible ways to maintain the balance between excitation and inhibition while also increasing Δτ. Likewise, the assumptions on the relative strengths of slow and fast synapses, Eq. (16), which result in the same reduction of the number of free parameters, can be easily generalized, as can the number of contributing components, by increasing the value of N in Eqs. (11), (12).

This generalization from the homogeneous network allows us to change the intrinsic time constant of the network dynamically, rather than through the shape of the input, thus generating fast rise times and slow decay times. The reason is that the response times of derivative feedback networks depend on the difference in timing across the excitatory projections. The response time of the network over a stimulus presentation is now changed by the stimulus itself as it changes the effective AMPA to NMDA ratio of synaptic strength differently on the different excitatory projections (EE and IE). We show that in the model with heterogeneous STD parameters on the excitatory projections the average synaptic time constant of those projections changes over the course of the stimulus presentation. Such a network can respond quickly to stimulus onset and decay slowly (over a time course that can reach seconds) after stimulus offset. We also show that the model with STD driven persistence is more robust than the derivative feedback network with homogeneous STD to changes in both the stimulus length and changes in the steady state firing rate of the network. We note that the specific relationship between STD and AMPA/NMDA ratios postulated by our model has not previously been studied in the context of networks in sensory cortex showing long time constant decay. Our model therefore makes the prediction that, in such networks, synapses in the EE projection with a higher ratio of AMPA receptors have stronger STD than synapses with more NMDA receptors, and that the opposite is the case for the IE projection.

A central result of our work is that the addition of STD dramatically alters the transient dynamics of local networks. For networks whose dynamics depend on positive feedback, STD narrows the range of possible response times while, for derivative feedback networks, STD allows a network to show two substantially different response times at the onset and offset of a stimulus. The transient responses of these networks are usually considered in the context of connectivity matrices for linear networks with particular connection strengths between either populations or neurons (Ganguli et al. 2008). STD constantly alters the state of these connectivity matrices, effectively yielding a different network for every point in time. Therefore, STD acts to change the state space of the network and can dramatically alter its dynamics depending on the starting network state and the distribution of STD. This implies that the distribution of STD may be as important to the response of the network as the underlying structure and baseline synaptic strengths.

Although we only consider STD, the addition of short-term facilitation (STF) on the excitatory projections would expand the range of dynamics available to both types of network. For example, positive feedback networks dominated by facilitation can produce persistent activity which decays to baseline level rather than approaching an attractor different from the baseline (Mi et al. 2014). However, STF tends to slow the onset response of a network and prevents the network from producing the fast responses generally observed in early sensory areas (Barak and Tsodyks 2007). In future work it would be interesting to examine how weak facilitation on the excitatory projections may impact the transient responses of both positive feedback and derivative feedback networks. We also note that STF has been observed on inhibitory synapses. While our work has focused on the impact of short-term potentiation on excitatory synapses, the changing strength of inhibitory synapses may also affect the network (Gupta et al. 2000; Reyes 2011).

While stimulus dependent changes in temporal structure have been observed in early sensory areas, their impact on computation is currently unclear. It has been proposed that such activity may be used in networks that learn the temporal structure of stimuli (Johnson et al. 2010). Additionally, in reward timing paradigms cholinergic mechanisms are believed to train networks to represent the temporal relationship between a stimulus and a reward (Chubykin et al. 2013). Such persistence may also act like a transient memory after the offset of a stimulus (Pasternak and Greenlee 2005; O’Herron and von der Heydt 2009). In addition to changing the intrinsic time constant of the network during a stimulus presentation, our network could also change the intrinsic time constant of a network between stimulus presentations. Such a change would be driven by differences in the recovery rate of the synapses and could allow changes in the timing of response to stimuli across presentations.

In addition to the already discussed prediction of the specific distribution of STD strengths in the fast and slow synapses of EE and IE projections in balanced derivative networks, our work makes a number of more general predictions concerning how networks dominated by STD may produce long time constant decay. For positive feedback networks, reducing the strength of STD would be the primary method to increase the network’s decay time. Derivative feedback networks, on the other hand, have a number of different ways that the persistence of decay activity can be altered. An overall increase in w, the strength of recurrent connections, would increase the decay time. As in the positive feedback network, reducing the strength of STD would also have a similar although less drastic effect. Each of these parameters provides a way to change the response time of the network although only heterogeneous STD driven persistence can change the decay time of the network without significantly altering the rise time.

Notes

Acknowledgements

We thank Daniel Jeck and Brian Hu for many useful discussions. We would also like to thank Philip O’Herron for help in accessing the physiological data.

Compliance with Ethical Standards

Conflict of interests

The authors declare that they have no conflict of interest.

References

  1. Barak, O., & Tsodyks, M. (2007). Persistent activity in neural networks with dynamic synapses. PLoS Computational Biology, 3(2), e35.CrossRefPubMedPubMedCentralGoogle Scholar
  2. Beck, O., Chistiakova, M., Obermayer, K., & Volgushev, M. (2005). Adaptation at synaptic connections to layer 2/3 pyramidal cells in rat visual cortex. Journal of Neurophysiology, 94(1), 363–376.CrossRefPubMedGoogle Scholar
  3. Castro-Alamancos, M.A., & Connors, B.W. (1997). Distinct forms of short-term plasticity at excitatory synapses of hippocampus and neocortex. Proceedings of the National Academy of Sciences, 94(8), 4161–4166.CrossRefGoogle Scholar
  4. Chubykin, A.A., Roach, E.B., Bear, M.F., & Shuler, M.G.H. (2013). A cholinergic mechanism for reward timing within primary visual cortex. Neuron, 77(4), 723–735.CrossRefPubMedPubMedCentralGoogle Scholar
  5. Compte, A., Brunel, N., Goldman-Rakic, P.S., & Wang, X.J. (2000). Synaptic mechanisms and network dynamics underlying spatial working memory in a cortical network model. Cerebral Cortex, 10(9), 910–923.CrossRefPubMedGoogle Scholar
  6. Cormier, R., Greenwood, A., & Connor, J. (2001). Bidirectional synaptic plasticity correlated with the magnitude of dendritic calcium transients above a threshold. Journal of Neurophysiology, 85(1), 399–406.PubMedGoogle Scholar
  7. Craft, E., Schütze, H., Niebur, E., & von der Heydt, R. (2007). A neural model of figure-ground organization. Journal of Neurophysiology, 97(6), 4310–26. PMID17442769.CrossRefPubMedGoogle Scholar
  8. Curtis, C.E., & D’Esposito, M. (2003). Persistent activity in the prefrontal cortex during working memory. Trends in cognitive sciences, 7(9), 415–423.CrossRefPubMedGoogle Scholar
  9. Denève, S., & Machens, C.K. (2016). Efficient codes and balanced networks. Nature Neuroscience, 19(3), 375.CrossRefPubMedGoogle Scholar
  10. Ganguli, S., Huh, D., & Sompolinsky, H. (2008). Memory traces in dynamical systems. Proceedings of the National Academy of Sciences, 105(48), 18:970–18:975.CrossRefGoogle Scholar
  11. Gavornik, J.P., & Shouval, H.Z. (2011). A network of spiking neurons that can represent interval timing: mean field analysis. Journal of Computational Neuroscience, 30(2), 501–513.CrossRefPubMedGoogle Scholar
  12. Gavornik, J.P., Shuler, M.G.H., Loewenstein, Y., Bear, M.F., & Shouval, H.Z. (2009). Learning reward timing in cortex through reward dependent expression of synaptic plasticity. Proceedings of the National Academy of Sciences, 106(16), 6826–6831.CrossRefGoogle Scholar
  13. Gillary, G., & Niebur, E. (2016). The edge of stability: Response times and delta oscillations in balanced networks. PLoS Computational Biology, 12(9), e1005,121.CrossRefGoogle Scholar
  14. Greenlee, M.W., Georgeson, M.A., Magnussen, S., & Harris, J.P. (1991). The time course of adaptation to spatial contrast. Vision Research, 31(2), 223–236.CrossRefPubMedGoogle Scholar
  15. Guo, K., Mahmoodi, S., Robertson, R.G., & Young, M.P. (2006). Longer fixation duration while viewing face images. Experimental Brain Research, 171(1), 91–98.CrossRefPubMedGoogle Scholar
  16. Gupta, A., Wang, Y., & Markram, H. (2000). Organizing principles for a diversity of gabaergic interneurons and synapses in the neocortex. Science, 287(5451), 273–278.CrossRefPubMedGoogle Scholar
  17. Hansel C, Artola A, & Singer W (1997). Relation between dendritic Ca2+ levels and the polarity of synaptic long-term modifications in rat visual cortex neurons. European Journal of Neuroscience, 9(11), 2309–2322.CrossRefPubMedGoogle Scholar
  18. Hardy, N.F., & Buonomano, D.V. (2016). Neurocomputational models of interval and pattern timing. Current Opinion in Behavioral Sciences, 8, 250–257.CrossRefPubMedPubMedCentralGoogle Scholar
  19. Hempel, C.M., Hartman, K.H., Wang, X.J., Turrigiano, G.G., & Nelson, S.B. (2000). Multiple forms of short-term plasticity at excitatory synapses in rat medial prefrontal cortex. Journal of Neurophysiology, 83(5), 3031–3041.PubMedGoogle Scholar
  20. Johnson, H.A., Goel, A., & Buonomano, D.V. (2010). Neural dynamics of in vitro cortical networks reflects experienced temporal patterns. Nature Neuroscience, 13(8), 917–919.CrossRefPubMedPubMedCentralGoogle Scholar
  21. Leopold, D.A., Rhodes, G., Müller, K. M., & Jeffery, L. (2005). The dynamics of visual adaptation to faces. Proceedings of the Royal Society of London B: Biological Sciences, 272(1566), 897–904.CrossRefGoogle Scholar
  22. Lim, S., & Goldman, M.S. (2013). Balanced cortical microcircuitry for maintaining information in working memory. Nature Neuroscience, 16(9), 1306–1314.CrossRefPubMedPubMedCentralGoogle Scholar
  23. Lim, S., & Goldman, M.S. (2014). Balanced cortical microcircuitry for spatial working memory based on corrective feedback control. The Journal of Neuroscience, 34(20), 6790–6806.CrossRefPubMedPubMedCentralGoogle Scholar
  24. Major, G., & Tank, D. (2004). Persistent neural activity: prevalence and mechanisms. Current Opinion in Neurobiology, 14(6), 675–684.CrossRefPubMedGoogle Scholar
  25. Mi, Y., Li, L., Wang, D., & Wu, S. (2014). A synaptical story of persistent activity with graded lifetime in a neural system. In Advances in Neural Information Processing Systems (pp. 352–360).Google Scholar
  26. Mihalas, S., Dong, Y., von der Heydt, R., & Niebur, E. (2011). Mechanisms of perceptual organization provide auto-zoom and auto-localization for attention to objects. Proceedings of the National Academy of Sciences, 108(18), 7583–8. PMC3088583.CrossRefGoogle Scholar
  27. Murphy, B., & Miller, K. (2009). Balanced amplification: A new mechanism of selective amplification of neural activity patterns. Neuron, 61(4), 635–648.CrossRefPubMedPubMedCentralGoogle Scholar
  28. Myme, C.I., Sugino, K., Turrigiano, G.G., & Nelson, S.B. (2003). The NMDA-to-AMPA ratio at synapses onto layer 2/3 pyramidal neurons is conserved across prefrontal and visual cortices. Journal of Neurophysiology, 90 (2), 771–779.CrossRefPubMedGoogle Scholar
  29. Nikolić, D, Häusler, S, Singer, W., & Maass, W. (2009). Distributed fading memory for stimulus properties in the primary visual cortex. PLoS Biology, 7(12), e1000, 260.CrossRefGoogle Scholar
  30. O’Herron, P., & von der Heydt, R (2009). Short-term memory for figure-ground organization in the visual cortex. Neuron, 61(5), 801–809. PMC2707495.CrossRefPubMedPubMedCentralGoogle Scholar
  31. O’Herron, P., & von der Heydt, R. (2011). Representation of object continuity in the visual cortex. Journal of Vision, 11(2). PMC3160770.Google Scholar
  32. Pasternak, T., & Greenlee, M.W. (2005). Working memory in primate sensory systems. Nature Reviews Neuroscience, 6(2), 97–107.CrossRefPubMedGoogle Scholar
  33. Patterson, M.A., Lagier, S., & Carleton, A. (2013). Odor representations in the olfactory bulb evolve after the first breath and persist as an odor afterimage. Proceedings of the National Academy of Sciences, 110(35), E3340–E3349.CrossRefGoogle Scholar
  34. Petersen, C.C. (2002). Short-term dynamics of synaptic transmission within the excitatory neuronal network of rat layer 4 barrel cortex. Journal of neurophysiology, 87(6), 2904–2914.PubMedGoogle Scholar
  35. Petreanu, L., Gutnisky, D.A., Huber, D., Xu, N.L., O’Connor, D.H., Tian, L., Looger, L., & Svoboda, K. (2012). Activity in motor-sensory projections reveals distributed coding in somatosensation. Nature, 489(7415), 299–303.CrossRefPubMedPubMedCentralGoogle Scholar
  36. Reinhold, K., Lien, A.D., & Scanziani, M. (2015). Distinct recurrent versus afferent dynamics in cortical visual processing. Nature Neuroscience, 18(12), 1789–1797.CrossRefPubMedGoogle Scholar
  37. Reyes, A.D. (2011). Synaptic short-term plasticity in auditory cortical circuits. Hearing research, 279(1), 60–66.CrossRefPubMedPubMedCentralGoogle Scholar
  38. Rubin, D.B., Van Hooser, S.D., & Miller, K.D. (2015). The stabilized supralinear network: a unifying circuit motif underlying multi-input integration in sensory cortex. Neuron, 85(2), 402–417.CrossRefPubMedPubMedCentralGoogle Scholar
  39. Russell, A.F., Mihalas, S., von der Heydt, R., Niebur, E., & Etienne-Cummings, R. (2014). A model of proto-object based saliency. Vision Research, 94, 1–15.CrossRefPubMedGoogle Scholar
  40. Shuler, M.G., & Bear, M.F. (2006). Reward timing in the primary visual cortex. Science, 311(5767), 1606–1609.CrossRefPubMedGoogle Scholar
  41. Sugihara, T., Qiu, F.T., & von der Heydt, R. (2011). The speed of context integration in the visual cortex. Journal of neurophysiology, 106(1), 374–385. PMC3129740.CrossRefPubMedPubMedCentralGoogle Scholar
  42. Super, H., Spekreijse, H., & Lamme, V. (2001). A neural correlate of working memory in the monkey primary visual cortex. Science, 293, 120–124.CrossRefPubMedGoogle Scholar
  43. Tsodyks, M., Pawelzik, K., & Markram, H. (1998). Neural networks with dynamic synapses. Neural Computation, 10, 821– 835.CrossRefPubMedGoogle Scholar
  44. Tsodyks, M.V., & Markram, H. (1997). The neural code between neocortical pyramidal neurons depends on neurotransmitter release probability. Proceedings of the National Academy of Sciences, 94, 719–23.CrossRefGoogle Scholar
  45. Tsumoto, T., & Yasuda, H. (1996). A switching role of postsynaptic calcium in the induction of long-term potentiation or long-term depression in visual cortex. In Seminars in Neuroscience, (Vol. 8 pp. 311–319): Elsevier.Google Scholar
  46. Wang, X.J. (1999). Synaptic basis of cortical persistent activity: the importance of NMDA receptors to working memory. The Journal of Neuroscience, 19(21), 9587–9603. PMID10531461.PubMedGoogle Scholar
  47. Xue, M., Atallah, B.V., & Scanziani, M. (2014). Equalizing excitation-inhibition ratios across visual cortical neurons. Nature, 511(7511), 596.CrossRefPubMedPubMedCentralGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2017

Authors and Affiliations

  • Grant Gillary
    • 1
  • Rüdiger von der Heydt
    • 1
  • Ernst Niebur
    • 1
  1. 1.Zanvyl Krieger Mind/Brain Institute and Solomon Snyder Department of NeuroscienceJohns Hopkins UniversityBaltimoreUSA

Personalised recommendations