High-capacity embedding of synfire chains in a cortical network model
Authors
- First Online:
- Received:
- Revised:
- Accepted:
DOI: 10.1007/s10827-012-0413-9
- Cite this article as:
- Trengove, C., van Leeuwen, C. & Diesmann, M. J Comput Neurosci (2013) 34: 185. doi:10.1007/s10827-012-0413-9
Abstract
Synfire chains, sequences of pools linked by feedforward connections, support the propagation of precisely timed spike sequences, or synfire waves. An important question remains, how synfire chains can efficiently be embedded in cortical architecture. We present a model of synfire chain embedding in a cortical scale recurrent network using conductance-based synapses, balanced chains, and variable transmission delays. The network attains substantially higher embedding capacities than previous spiking neuron models and allows all its connections to be used for embedding. The number of waves in the model is regulated by recurrent background noise. We computationally explore the embedding capacity limit, and use a mean field analysis to describe the equilibrium state. Simulations confirm the mean field analysis over broad ranges of pool sizes and connectivity levels; the number of pools embedded in the system trades off against the firing rate and the number of waves. An optimal inhibition level balances the conflicting requirements of stable synfire propagation and limited response to background noise. A simplified analysis shows that the present conductance-based synapses achieve higher contrast between the responses to synfire input and background noise compared to current-based synapses, while regulation of wave numbers is traced to the use of variable transmission delays.
Keywords
Recurrent network dynamics Feedforward network Synchrony Synaptic conductance Synfire chain Storage capacity1 Introduction
Evidence of precisely-timed spiking activity suggests that synfire chains play an important role in representing information in the brain (Abeles 1982; Riehle et al. 1997; Prut et al. 1998; Shmiel et al. 2005; Kilavik et al. 2009; Long et al. 2010). Synfire chains, pools of neurons sequentially linked by feed-forward connections, are capable of generating temporally extended and precisely timed patterns of spiking activity. Each such pattern consists of a packet of near-simultaneous spikes within a pool, that triggers a similar packet in the subsequent pool, and so on, giving rise to wave-like propagation of spiking activity down the chain (Abeles 1982). Propagation of synfire waves is robust and efficient because of the properties of single model neurons: a relatively small number of simultaneously arriving inputs can elicit a spike reliably with precise timing (Abeles 1991; Diesmann et al. 1999; Goedeke and Diesmann 2008).
Several authors have presented models of synfire chains as embedded in the detailed structure of the cortex i.e. its synaptic connections and strengths (Bienenstock 1995; Herrmann et al. 1995; Mehring et al. 2003; Aviel et al. 2005; Kumar et al. 2008; Schrader et al. 2008). Modeling of embedding involves assigning individual neurons randomly to pools which are then linked up to form one or more synfire chains. Individual units can belong to more than one pool and thus participate in several different chains, thereby forming a recurrent network.
A system of many embedded synfire chains may provide a form of sparse coding (Földiák 2002), where each chain represents some feature of a cognitive representation. By enabling activation of chains in parallel or in cascaded fashion, embedding can in principle realize combinatorial representation (Bienenstock 1991, 1995; Bienenstock 1996; Abeles et al. 2004; Hayon et al. 2005; Hanuschkin et al. 2010; Schrader et al. 2010).
To permit combinatorial representations, a system needs to, first, embed a sufficient number of pools and, second, allow a sufficient number of these to be activated simultaneously. Let us consider the number of chains or, more specifically, the number of pools they consist of. The number of pools is constrained by factors such as the number of synapses, and the need to avoid excessive overlap between pools. The severity of these constraints is determined by the dynamics of neural activity; the dynamics must enable selective initiation and stable propagation of synfire waves, and avoid modes that would interfere with wave propagation, such as ones characterized by excessively high activity levels or strong oscillations. These dynamical constraints put a limit on how many pools, or more precisely, how many pool-to-pool links can be embedded in the system. This we call the system’s embedding capacity.
A model of cortical embedding of chains should operate at a biologically realistic level of neural connectivity: of the order of 10^{4} synapses per neuron. An early model of chain embedding in cortex achieved an embedding capacity of about 8N pools in a network of N neurons with cortical scale connectivity (Bienenstock 1995). However, the model presented by Bienenstock is rather abstract: it consists of binary state neurons with an r-winners-take-all update scheme that enforces a low level of activity in the network (exactly r/N at each time-step), and a coarse discretization of time that artificially enforces spike packet synchronization. A similarly idealized model using a global inhibitory signal instead of r-winners-take-all gives quantitatively similar embedding capacity results (Herrmann et al. 1995).
Independently, studies using leaky integrate-and-fire neurons with fine temporal resolution have shown that spike packet sequences can propagate while conserving both the number of spikes and narrow temporal dispersion within each packet (Diesmann et al. 1999; Goedeke and Diesmann 2008). However, the high embedding capacity that is found in the idealized models of Bienenstock and Herrmann et al. has proved difficult to realize using integrate-and-fire neurons, due to the appearance of global modes of activity incompatible with wave propagation. The state of the art thus leaves room for improvement. Our first aim is, therefore, to show that such realization is possible.
Like all these models of chain embedding, ours takes as its starting point the balanced random network architecture. This architecture consists of two populations of neurons, inhibitory and excitatory ones, respectively, with sparse random coupling within and between them (van Vreeswijk and Sompolinsky 1996; Amit and Brunel 1997; Brunel 2000), typically driven by external excitatory input. This architecture is an attractive substrate in which to embed synfire chains because it exhibits a stable state in which neurons spike at a low rate, in an asynchronous irregular (AI) manner which is regarded as a good first approximation to the activity of cortical neurons (Brunel 2000). Next, in order to embed synfire chains into this basic architecture, some or all of the random excitatory connections are replaced with connections that make up the links between successive pools. An upper bound on embedding capacity (the combinatorial capacity) is obtained for a given pool size when all the available excitatory connections are replaced with pool-to-pool connections. The smaller the pools, the fewer connections within each pool-to-pool link, and hence the more pools can be embedded (the higher the combinatorial capacity).
Only one published study has systematically investigated the embedding capacity of such systems (Aviel et al. 2005). Like most chain embedding studies to date, Aviel’s study uses current based synapses: each synaptic input generates a fixed-strength pulse of current. The capacity analysis of Aviel et al. assumes the network is in the AI state at a particular mean firing rate and with a characteristic membrane potential distribution. From this the minimum pool size for stable wave propagation is estimated: the minimum number of inputs from one pool to each neuron in the next pool in order for the synfire wave input to shift the membrane potential above threshold with high probability. However, in simulations it was not possible to attain the combinatorial capacity for any pool size. When the fraction of connections used to store pool-to-pool links exceeds a critical level, the AI state spontaneously gives way to a some activity state that prevents synfire propagation. In most of the cases examined, this activity takes the form of strong, periodic bursts of synchronous spikes, but it can also be a state of sustained high firing rate. Even below the critical level for AI state stability, the propagating wave itself drives moderate bursts of synchronized spiking, which tend to destabilize the wave, resulting in a short life time, and the critical level itself is slightly lower in the presence of the wave. The study of Aviel et al. considered the effect of adding inhibitory “shadow” pools to the chains to create a doubly balanced architecture in which each pulse packet delivers balanced input to the rest of the network. With double balance, the undesirable effects of the propagating wave are eliminated. As a result, the embedding capacity in a network of N neurons is 0.07 N pools with double balance as against 0.065 N without. In other words, a marginal improvement.^{1}
The destabilizing effect of spike packets was also observed in a locally connected recurrent network model representing a small area of cortex with a distance-based Gaussian connectivity density (Mehring et al. 2003). Synchronous activity in a single pool of localized neurons triggers a strong reverberation in the surrounding local region similar to that found by Aviel et al. (2005) even without any embedded chains. Embedding a single chain of 10 pools, the reverberation resulting from the spike packet in the first pool typically extinguishes the synfire wave. Mehring et al. (2003) suggested that the inclusion of inhibitory shadow pools could prevent the reverberation. Surprisingly, even without shadow pools, spike packets can propagate without triggering the reverberation when the current-based synapses are replaced by conductance-based ones, in combination with heterogeneity in neuronal properties including the number of connections per neuron (Kumar et al. 2008). This result was attributed chiefly to the reduction of the local response to the stimulated pulse packet due to the integrative properties of conductance based neurons receiving background input. However, the sources of heterogeneity in the model may also play a role, as previously studies have noted their ability to suppress synchronized oscillations in recurrent networks (Denker et al. 2004; Tetzlaff et al. 2005). Variability in transmission delays has also been suggested as a mechanism to stabilize the AI state (Aviel et al. 2005).
Let us now consider the second requirement for a combinatorial architecture based on synfire chains: the capacity to activate a number of chains simultaneously. Aviel et al. (2005) noted that embedding capacity has a dual nature, involving both the number of available pools and the potential number of co-active waves, but did not incorporate the latter in their analysis. They observed that their system is capable of sustaining only 1 to 3 waves simultaneously. It is known that the storage capacity of sparse patterns in networks of binary neurons drops as the size of the retrieved patterns increases (Tsodyks and Feigelman 1988). Correspondingly, for chain embedding models we may expect that a model with more co-active waves will exhibit a lower embedding capacity, because it operates at a higher overall rate of activity.
A theoretical mean-field analysis (Trengove 2006) predicted embedding capacities of αN pools with α > 1, much higher than found in previous models using integrate-and-fire dynamics. This was for a model with conductance-based synapses, inhibitory “shadow” pools and variability in the transmission delays on the connections within each pool-to-pool link. However, the study provided no cortical-scale simulations to verify the predictions of the analysis. This is important in order to demonstrate global network stability. With the simulation technology available today this undertaking has become feasible (Morrison et al. 2005).
The present work provides a more thorough mean field analysis of the model of (Trengove 2006) and cortical-scale simulations showing that over a broad range of cortically realistic parameters, global stability is conserved and the mean field analysis holds. The paper is organized as follows. In Section 2 we define the model for cortical embedding of synfire chains: network structure (Section 2.1), neuron and synapse models (Section 2.2). We present the mean field analysis leading to a self-consistency equation relating the parameters (connectivity, pool size) and dynamical variables (firing rates, number of waves) (Section 2.4). We provide a definition of embedding capacity that accommodate its dual nature, involving as it does both the number of pools and the number of co-active waves (Section 2.5). To test the theory, in the simulations we focus on the upper bound on wave activity, attained by ongoing initiation of new waves (Section 2.3). In Section 3 we exhibit typical behaviors of the model (Section 3.1), numerical solutions of key quantities in the mean field analysis (Section 3.2), and compare network behavior to the predictions of the mean field analysis, demonstrating that over a range of cortical parameters the system exhibits the predicted high-capacity (Section 3.4). We give capacity predictions of the mean field analysis as inhibitory synaptic strength is varied, which allow us to specify an optimal inhibitory strength (Section 3.6), and lastly, we compare conductance and current based models using a simplified mean field analysis in order to explain why the former outperforms the latter (Section 3.7). In Section 4 we summarize the results and the reasons why the present model succeeds in attaining a much higher embedding capacity than previous spiking neuron models. We also discuss the modest departures of the simulations from the mean field analysis, the importance of variability in transmission delays and the absence of synaptic rise times in the model, and outline how the present synfire chain embedding model can be extended to encompass cortical embedding of more general structures such as polychronous spiking.
2 Methods and models
2.1 Network model for synfire chain embedding
2.2 Neuron and synapse model
Neuronal and synaptic parameter values
V _{E} |
0 |
mV |
V _{I} |
− 80 |
mV |
V _{P} |
− 70 |
mV |
V _{R} |
− 70 |
mV |
V _{Θ} |
− 55 |
mV |
τ _{P} |
20 |
ms |
τ _{ref} |
2 |
ms |
g _{E} |
0.005 |
– |
g _{I} |
0.11 |
– |
Transmission delays vary across individual synaptic connections. Identical delays are unrealistic and, moreover, in a model with instantaneous synaptic responses, identical delays would allow the propagation of perfectly synchronized spike packets; there would then be no interval after the arrival of the synchronous inputs during which background input could prevent or modulate the timing of an output spike. Variability of delays introduces such an interval. As a consequence, the model does not require rise times in the post synaptic potentials, which would provide an alternative means for creating such an interval.
The excitatory delays were specified so as to give a realistically broad delay distribution overall, but a narrow distribution of the delays on the synapses from any one pool to the next pool in the sequence. To achieve these two properties, each synapse’s transmission delay was set to be the sum of two components: a component with large variability which is the same for all synapses within the same pool-to-pool link (τ _{A}); and a component with small variability which is different for every synapse (τ _{B}). Thus the transmission delay on the synaptic connection from the jth neuron in the μth E-pool to the ith neuron in the next E/I-pool is given by τ _{A}(μ) + τ _{B}(μ,i,j) where τ _{A}(μ) is drawn from a uniform distribution on \(\left[0.5,4.5\right)\) ms and τ _{B}(μ,i,j) is drawn from a uniform distribution on \(\left[0,0.5\right)\) ms. This specification is complete and self-consistent because every excitatory synapse belongs to exactly one pool-to-pool link. We refer to τ _{A} as the inter-link delay, with a spread of 4.0 ms, and τ _{B} as the intra-link delay, with a spread of 0.5 ms.
The relatively small variability of delays within synfire links is a defining feature of synfire chains: propagation by near-synchronous input. It is also plausible if, as argued by Bienenstock (1991, 1995), synfire chains develop by strengthening synapses that cooperate to deliver synchronous inputs. The magnitude and variability of the transmission delays are realistic for intracortical connectivity on the scale of a few hundred millimeters given typically observed intracortical conduction velocities in the range of 0.1–0.5 m/s (Yger et al. 2011; Bringuier et al. 1999; González-Burgos et al. 2000). This is around the low end of the velocity range observed for cortico-cortical connections (Swadlow 2000). For instance, a synaptic delay of 0.5 ms, an axonal path length of 400 μm, and a velocity of 0.2 m/s would reproduce the mean transmission delay of 2.5 ms used here. The inhibitory connection delays vary according to the same two-component distribution, but in their case both components are drawn independently for every synapse.
2.3 External input
As opposed to most balanced random architecture models, our network does not receive steady, ongoing excitatory input, neither continuous current nor a stochastic (Poisson) input at a constant rate. The model thereby demonstrates that synfire waves alone can drive ongoing activity in a network. By omitting ongoing external excitatory drive, the ratio of excitatory to inhibitory background input becomes invariant with respect to the mean firing rate of the network (see Section 2.4).
The first four wave-initiating stimuli are accompanied by a transient of E/I balanced external Poisson input. This begins at time zero with E/I rates approximately equal to the background input rates generated by 4 synfire waves. The rates are progressively reduced to zero in four equally-sized steps, one at each new stimulus. This was done to ensure that the mean membrane potential is near equilibrium when the first four stimuli arrive, rather than being at the resting potential, in case the latter would be too low to allow a spike packet to initiate a wave. It proved not essential, however, for the working of the model.
2.4 Mean field analysis
Background input does two things. First, it generates stochastic spiking at rate ν _{S}. Second, it modulates spike packet propagation, and thus the rate of spikes participating in synfire waves, ν _{W}. These effects have to be combined with those whereby the spiking rate in turn generates the background input (Eqs. (3)–(5)), in order to obtain a self-consistent description that determines the steady state behaviour for a given number of propagating waves.
There is an analytical expression for the function f _{S}, the so-called Siegert formula (Burkitt 2006), which we use to compare current-based and conductance-based models in Section 3.7. However, this expression can be a poor approximation if the conductances are not small enough and, in a time-step based simulation, if the time-step is not small enough. Hence, for accurate comparison of network simulations and mean field theory, the function f _{S} was computed by single neuron simulations for a closely spaced sequence of λ _{E} values (with λ _{I} = γλ _{E}) and extended to intermediate values of λ _{E} by linear interpolation. The single neuron simulations used a 0.1 ms time step, same as for the network. Each simulation consisted of a set of 100 runs of duration of 5,000 ms. The mean spike rate over the interval \([1\text{,}000\mathrm{\, ms},\,5\text{,}000\,\mathrm{ms}]\) was computed. This procedure was sufficient to obtain an accurate estimate of the spiking rate.
2.5 Embedding capacity
We would like a measure of embedding capacity to allow comparison across ranges of system parameters; in particular, connectivity and inhibitory conductance strength. Embedding capacity is the largest embedding level that the system can incorporate while functioning properly; i.e. allowing stable propagation of a certain number of waves and a stability of the global rate dynamics.
The steady state solution specified by Eq. (8) for an arbitrary number of waves will be at the capacity limit if we add the condition n _{E} = n _{E,min}(λ _{E}).
Alternatively, we can specify the level of activity in terms of the relative number of waves, h/N _{E}, and obtain a different embedding capacity function of connectivity, α _{max}(C _{E};h/N _{E}). For each of these definitions of α _{max}(C _{E}), we optimize embedding capacity with respect to connectivity.
Even below the limit for stochastic spike rate stability, it is necessary to ensure that a reasonable amount of wave activity is present at the capacity limit. As C _{E} increases, holding ν = ν _{W} + ν _{S} constant, the level of stochastic spiking determined by \({\nu_{\mathrm{S}}}=\hat{f}_{\mathrm{S}}({C_{\mathrm{E}}}({\nu_{\mathrm{W}}}+{\nu_{\mathrm{S}}}))\) will increase, while the fraction of spikes due to synfire waves will decrease. We therefore also require that at least 50 % of the spiking should belong to wave activity, which leads to another (usually smaller) connectivity limit, C _{E,max2}.
Since the embedded chain network is not a random network, we rely on simulations to verify both the accuracy and the stability of the mean field solution for the system at capacity, when the connectivity is below both these limits, C _{E,max1} and C _{E,max2}.
2.6 Detection of spike packet sequences
Detecting unitary events from a stream of activity can be done in a variety of ways. Our method is suitable for situations where the spike rate outside a spike packet is low; thus we can use a detection window wide enough to safely cover any typical spike packet while including a negligible number of non-packet spikes. Within such windows, the median spike time gives the precise location of spike packet. We combined all the spike trains of all neurons in the pool to obtain a list of spiking times T _{ s } = t _{1},...,t _{ n }. To every spike time t _{ k } in T _{ s }, a window [t _{ k },t _{ k } + T _{ w }) was applied, with width T _{ w } = 3 ms. The spike times within this window constitute a sublist S _{ k }. Sublists were classified as suprathreshold if they contained more than n _{ θ } spike times (|S _{ k }| > n _{ θ }). n _{ θ } was set to a value midway between an estimated minimum spike packet size and an estimated maximum background spike count, resulting typically in n _{ θ } ≈ 0.4n _{E}. Maximal consecutive sequences of at least 6 suprathreshold sublists were identified; that is, sequences of the form L = (S _{ k + 1},...,S _{ l }) where l − k ≥ 6, |S _{ j }| > n _{ θ } for j ∈ k + 1,...,l, |S _{ k }| ≤ n _{ θ } and |S _{ l + 1}| ≤ n _{ θ }. The minimum sequence length, 6, was chosen to exclude small suprathreshold sequences that can arise when |S _{ j }| ≈ n _{ θ } due to fluctuations in successive |S _{ j }|. Each such sequence L determines a single spike packet, as follows. We identify amongst the sublists in the sequence the ones with maximum activity: L _{max} = (S _{ j } ∈ L : |S _{ j }| = max {|S _{ i }| : S _{ i } ∈ L}). We identify their “middle”, the \(\left\lfloor |L_{\max}|/2\right\rfloor \)’th sublist in L _{ max }, as the spike packet. The time of the spike packet was defined as the median of its spike times. From the extracted spike packets we compute ν _{W}(t), a time-resolved mean rate over the population of spikes belonging to spike packets, using a bin size of 20 ms. By linking up spike packets on consecutive pools, synfire waves were identified. Spike packets on two consecutive pools were linked up if the time interval between the first and second was between 0.5 and 6 ms.
2.7 Simulation tools
Simulations of the chain embedding networks were computationally intensive due to the large size of the networks. We used the NEST simulation software using the Message Passing Protocol (MPI) for parallel processing (Morrison et al. 2005), running on a cluster of 24 Linux PCs with a total of 192 cores and 192 GB of memory. The largest networks we simulated (those with \({C_{\mathrm{E}}}=11\text{,}000\)) had approximately 1.4×10^{5} neurons and 1.9×10^{9} synapses, which is close to the maximum possible with these resources.
3 Results
3.1 Behavior of large scale network simulations
Figure 2 exhibits the behavior of two networks, each with connectivity per neuron \({C_{\mathrm{E}}}=8\text{,}000\). In both cases, the model exhibits stable activity, and shows the emergence of an upper bound on the number of simultaneously propagating spike-packets. In the left side of Fig. 2, pool size is near the low end for allowing viable wave propagation; this results in a high embedding level and a large number of pools (n _{E} = 72, α = 1.54, p = 123457). In the right side of the figure, pool size is about 3 times larger; this leads to a lower embedding level and a smaller number of pools, (n _{E} = 200, α = 0.2, \(p=16\text{,}000\)). Note that even in this case, the number of pools is still high compared to previous studies. Figure 2(a) and (e) give spike rasters for 10 neurons from each pool in a sequence of consecutive pools. The vertical axis indexes pools counted from the stimulated pool (left) and the cumulative count of neurons plotted (right). Since each neuron appears in many pools, its spike train is plotted at many vertical positions. Within a pool, spikes belonging to spike packets are visibly distinguishable from the remaining ’background’ spikes. As discussed in Section 2.4, these are due partly to stochastic spiking, and partly due to the neurons participating in the spike packets of other pools. The raster plots clearly show waves of spike packets, propagating as a consequence of the 25 Hz periodic stimulation. The raster plots of all extracted spike packets (Fig. 2(b) and (f)) verify that all spike packets belong to waves that propagate down the chain from the stimulated pool; none occur spontaneously or out of sequence. In the traces of how the number of co-active waves varies over time (Fig. 2(c) and (g)), for both cases, the number of waves fluctuates around a mean value not far below the maximum. With small-sized pools, the number of waves occasionally drops abruptly to zero or a small fraction of the maximum, from which it gradually recovers as new waves are injected every 40 ms. In contrast, for large-sized pools the number of waves remains relatively close to the mean. This is because with smaller pools there are fewer waves and their survival is more affected by the noise feedback: they have a shorter average life time indicating that they are less robust, as discussed in Section 3.3. The mean firing rate in the population versus time (Fig. 2(d) and (h)) closely tracks the number of waves simultaneously present. With many, small-sized pools, the upper bound is 7 waves (with a mean of 4.9 and a firing rate of 1.75 Hz), while with fewer but larger pools it is 26 waves (with a mean of 23.0 and a firing rate of 26.3 Hz). This apparent trade-off between the number of pools and the maximum number of co-active waves is described systematically in Section 3.4.
3.2 Dependence of pulse-packet propagation properties and stochastic spiking on background input
Note that the existence of a threshold rate above which propagation fails is non-trivial. It is to be expected that there will be a minimum pool size for propagation at a given background rate, but only if n _{E,min}(λ _{E}) is an increasing function at a given rate λ _{E} will that rate λ _{E} be an upper limit for propagation over pools of size n _{E,min}(λ _{E}). The important functional consequence is that background input serves to regulate wave activity; since waves generate background input, with ongoing wave-initiating stimuli the number of waves will increase until background rate reaches the threshold, as seen in Fig. 2. The system equilibrates at the embedding capacity limit where n _{E} = n _{E,min}(λ _{E}).
The probability that a neuron in a pool participates in a given spike packet in that pool, p _{f}(λ _{E},γλ _{E}) and the mean pool-to-pool propagation time T(λ _{E},γλ _{E}) are given in Fig. 3(b, c) respectively. p _{f} and T depend relatively weakly on background input except when λ _{E} exceeds the threshold level. Figure 3(e) shows, for the 3 values of g _{I} given in Table 1, f _{S}(λ _{E},γλ _{E}), the mean spiking rate of a neuron receiving balanced background input. The rate is approximately linear in λ _{E} with a slope that depends strongly on ratio of the synaptic conductances, g _{E}/g _{I}.
3.3 Regulation of wave activity: the equilibrium state of the driven system
Figure 3(a) and (d) demonstrated the existence of a threshold level of background input (λ _{E,max} in Eq. (13)), around which wave survival probability rapidly drops from 1 to 0. This threshold is responsible for the upper bound on the number of waves seen in Fig. 2. Background input is generated as a by-product of the waves (Eq. (8)). Thus, as the number of waves increases with ongoing stimulation, at the same time the background input rises until it reaches the vicinity of the threshold. At this point, the survival time of waves rapidly drops until an equilibrium point is reached, where the wave initiation rate is counterbalanced by the rate of wave failure.
In practice, however, using λ _{E} = λ _{E,max} to evaluate ν _{W}, ν _{S}, ν and h _{eq} is a good enough approximation because of the sharp transition of P _{S} from 1 to 0. This is the method used to define the embedding capacity limit in Section 2.5.
3.4 Comparison of embedding capacity limit by analysis and by network simulations
We compared the equilibrium firing rates and numbers of waves predicted by the mean field analysis with those found in a series of simulated networks, for connectivity \({C_{\mathrm{E}}}\in{5\text{,}000,6\text{,}000,\ldots,11\text{,}000}\) and pool size n _{E} ∈ 20,28,...,220. For each choice of C _{E} and n _{E}, we conducted one network simulation of 10,000 ms duration, extracted spike packets sequences (Section 2.6) and obtained time resolved mean spiking rates ν(t), ν _{W}(t) and number of waves h(t). An initial transient, the time taken for the number of waves to reach equilibrium, was excluded. We set \(t_{\mathrm{start}}=\max(1\text{,}000\,\mathrm{ms},\, t_{\mathrm{mean}})\) where t _{mean} is the time when h(t) first exceeds the mean over the interval [1,000 ms, 10,000 ms]. The mean rate of all spiking activity (ν), the mean rate of spikes belonging to spike packets (ν _{W}) and the mean and maximum number of waves were then determined over the period \([t_{\mathrm{start}},\,10\text{,}000\,\mathrm{ms}]\).
The model reaches stable equilibria in all but a few cases. The few exceptions occurred when both pool size and connectivity was large (n _{E} = 212,220 for \({C_{\mathrm{E}}}=10\text{,}000\) and n _{E} = 204,212,220 for \({C_{\mathrm{E}}}=11\text{,}000\)). In these five cases, network behaviour was “normal” until some apparently arbitrary point at which it rose very rapidly to the saturation rate imposed by the neuronal refractory period. The rate explosion is of course incompatible with wave further propagation. No spontaneous waves were observed to appear before the rate explosion occurred.
Figure 4(d) shows the equilibrium behavior for both simulations and mean field analysis for three connectivity values, \({C_{\mathrm{E}}}=5\text{,}000,\,8\text{,}000,\,11\text{,}000\). The upper plot shows the mean overall rate ν, and the rate of spikes belonging to waves, ν _{W}, versus pool size n _{E}. Although each data point represents only a single network instance, the smoothness of the behavior versus pool size indicates that fluctuations due to different network realizations are small. The lower plot shows the predicted equilibrium number of waves h _{eq} versus n _{E}, as well as the mean and maximum numbers of waves obtained in the simulations. We note the good agreement between the simulations and mean field analysis. The most noticeable quantitative discrepancy is that the analysis overestimates the mean spiking rate by about 10–20 %. The reasons for this discrepancy will be discussed in Section 4.
3.5 Measures of embedding capacity
We have defined embedding capacity relative to a given level of activity: a specified firing rate or number of waves (Section 2.5).
Second, we consider embedding capacity based on equilibration at a particular number of waves (relative to network size). The choice of the number of waves is arbitrary. We consider \(h_{\mathrm{eq}}/{N_{\mathrm{E}}}=(2,5,10,20)/80\text{,}000\) (corresponding to h _{eq} = 2,5,10,20 for the case \({C_{\mathrm{E}}}=8\text{,}000\)). In analogy to Fig. 5(a), the curves in (n _{E},C _{E})-space of h _{eq}/N _{E} along with the two upper limits on connectivity are given in Fig. 5(c). Embedding capacity versus connectivity is given in Fig. 5(d), along with the results obtained by network simulations.
The definitions of embedding capacity allow us to study how it can be optimized with respect to other parameters. In the present work, we use the definition based on the equilibrium firing rate to assess the role of inhibitory conductance.
3.6 Inhibitory conductance governs trade-off between wave propagation stability and stochastic spiking stability
The embedding capacity curves are higher when the inhibitory synaptic conductance is smaller. This is because a higher membrane potential distribution allows stable propagation at smaller pool sizes. This is consistent with Fig. 3(d), which shows that mininimum pool size for stable propagation decreases as inhibitory conductance increases. On the other hand, as seen in Fig. 3(e), the stochastic spiking response is stronger with reduced inhibitory conductance. Hence the upper bound on connectivity for 50 % of spikes to belong to synfire waves (ν _{W}/ν = 0.5) is lowered. The consequence is that there is a trade-off between the stability of wave propagation and the control of stochastic spiking. For ν _{eq} = 5 Hz, Fig. 6(b) indicates the existence of an optimal inhibitory synaptic conductance close to 0.11, with maximum capacity occurring at a connectivity of about 11,000 per neuron. For a lower (higher) choice of equilibrium firing rate, the optimal inhibitory conductance is reduced (increased) and the optimal connectivity is increased (reduced), as indicated by Fig. 6(a) and (c).
3.7 Comparison of conductance-based and current-based models
To explain the high embedding capacity of the present model compared to a current-based model such as that of Aviel et al. (2005), it is desirable to examine the behavior of the two models across suitable slices of parameter space and relate the underlying neuronal response properties of the two models to the global network measure of embedding capacity. We consider two quite different ways of making this comparison: firstly, one that explores conventional parameter choices for the current-based model; and secondly, one that uses choices specifically designed to fit the current-based model behavior to that of the conductance-based model.
In the first method of comparison, the neuronal parameters of the current-based model (τ _{P}, V _{P}, V _{R}, V _{Θ} and τ _{ref}) are set to the same values as used in the conductance-based model (Table 1, Section 2.2). In the second method, however, we allow the current-based model to take different values for τ _{P} and V _{P}, in order to fit the membrane potential distribution and the effective membrane time constant of the current-based model to those of the conductance-based model.
For each particular choice of the neuronal and synaptic parameters of either model, we determine the chain embedding capacity of the network. Following Section 2.5, we assume the system is operating at a given mean firing rate, which we take to be ν = 5 Hz, and require that less than 50 % of the spikes are stochastic. As the connectivity per neuron, C _{E} (with C _{I} = γC _{E}), is varied, we determine (a) the equilibrium stochastic spiking rate, and its stability; (b) the minimum pool size n _{E,min} for stable propagation at this firing rate and connectivity; and (c) the capacity (\({\alpha_{\mathrm{max}}}={C_{\mathrm{E}}}/n_{\mathrm{E,min}}^{2}\)). To simplify the analysis here, we neglect the background-input dependence of the rate of spikes belonging to synfire waves (Eq. (7)). This allows a simpler self-consistency equation for the system: we can specify the spike rate due to waves (ν _{W}) independently of the amount of stochastic spiking (by setting the number of waves, h, appropriately) and solve for a self-consistent stochastic spiking rate ν _{S} such that \({\nu_{\mathrm{S}}}=f\hat{S}({C_{\mathrm{E}}}({\nu_{\mathrm{W}}}+{\nu_{\mathrm{S}}}))\) where \({\nu_{\mathrm{S}}}={\hat{f}_{\mathrm{S}}}({\lambda_{\mathrm{E}}})\equiv{f_{\mathrm{S}}}({\lambda_{\mathrm{E}}},\gamma{\lambda_{\mathrm{I}}})\). Since we require ν _{W} + ν _{S} = ν, the solution is obtained for any connectivity level just by setting \({\nu_{\mathrm{S}}}=\hat{f}_{\mathrm{S}}({C_{\mathrm{E}}}\nu)\) and ν _{W} = ν − ν _{S}, provided ν _{W} is non-negative. This is guaranteed by our requirement that ν _{S} < 0.5ν. As before, stability of this solution is assessed using Eq. (11).
The minimum pool size for stable wave propagation, n _{E,min} is estimated by considering the Gaussian approximation to the membrane potential, and requiring that n _{E,min} synchronous inputs of amplitude a _{E} (current model) or g _{E}(V _{avg} − V) (conductance model) will put 95 % of the membrane potential distribution above threshold, so that the probability of a spike response to the input is 0.95. This leads to the condition n _{E,min} a _{E} = V _{Θ} − (μ − 1.645σ). Importantly, however, this condition is only reasonable on the assumption of a zero intra-link delay spread. In the model specified in Section 2 there is an intra-link delay spread of 0.5 ms. The resulting dispersion of the synfire wave inputs introduces a window of integration during which background input can affect the probability that a spike is produced, so an estimate of minimum pool size based only on the membrane potential distribution prior to the arrival of synfire wave input will only be a good approximation if τ, the time scale at which background input acts on the membrane potential, is large in comparison to the spread of intra-link delays.
To identify the best capacity for each specific choice of neuronal and synaptic parameters, we obtain the behavior as a function of connectivity up to a maximum connectivity, C _{E,max}, the point at which either (a) the fixed point solution for ν _{S} becomes unstable, (b) the condition ν _{S} < 0.5ν = 2.5 Hz breaks down, or (c) α _{max} ceases to increase.
We examine the behavior of the conductance based model for one choice of excitatory conductance (g _{E} = 0.005 as before) while the inhibitory conductance g _{I} is varied over the interval [0.02,0.1]. For the first method of comparison, a corresponding family of current-based models is formed by setting a _{E} to match g _{E} using the relation a _{E} = g _{E}(V _{avg} − V) where V _{avg} = (V _{R} + V _{Θ})/2. Hence a _{E} = 0.312 mV. Choosing a _{E} as a function of g _{E} is appropriate, since the excitatory response in the conductance-based model does not vary much for \(V\in\left[{V_{\mathrm{R}}},{V_{\mathrm{\Theta}}}\right]\). However, this is not true for the inhibitory response, so a one-to-one match between a _{I} and g _{I} is not well founded. Instead we simply vary the inhibitory conductance over the interval a _{I} ∈ [0.25, 1.5] mV and compare the two models over their respective ranges of a _{I} and g _{I}, these ranges being sufficiently representive of the behavior of the two models as inhibition is varied.
In Fig. 7 the horizontal axis can also be intepreted to represent the background input rate, via the conversion λ _{E} = C _{E} ν _{eq}. In particular, we can see the behaviour of n _{E,min} versus λ _{E}. For both conductance and current-based models, at low inhibitory strengths n _{E,min} decreases with background rate, implying that wave activity is not regulated by background input. At higher inhibitory strengths n _{E,min} increases with background rate for the current-based model, and thus wave regulation is restored. However, for the conductance-based model the membrane potential distribution approaches a limit and virtually ceases to change as background input increases, and hence n _{E,min} reaches an asymptotic value at C _{E} values well below C _{E,max}, implying that wave regulation is ineffective. This result is in contrast to our findings for a conductance model with a 0.5 ms spread in intra-link delays: in the network simulations there is effective regulation of the number of waves (Fig. 2) and the comprehensive mean field analysis (Section 3.2, Fig. 3) shows that an increase in background input reduces the probability of a spike in response to synfire wave input and increases the minimum pool size. These effects, being absent in a model with zero intra-link delay spread, must be due to the effect of background input during the window of integration of synfire wave input that exists when there is a 0.5 ms intra-link delay spread. By increasing the minimum pool size, the intra-link delay spread also reduces the embedding capacity, as can be seen by comparing embedding capacity curves in Figs. 5(b) and 6 (0.5 ms delay spread) with those in the first column of Fig. 7 (zero delay spread). With zero delay spread the capacity increases linearly with connectivity because the minimum pool size is asymptotically constant. When the delay spread is introduced, the capacity curves become concave down in shape, due to a minimum pool size that increases almost linearly (with an offset) as connectivity increases (Fig. 3(d)). The large effect of intra-link delay spread in the conductance model is not surprising, since τ drops rapidly with C _{E} and is already within an order of magnitude of the intra-link delay spread at \({C_{\mathrm{E}}}=1\text{,}000\). However, the findings for the current-based model (Fig. 7, second column) should be only slightly changed by a 0.5 ms intra-link delay spread, since τ _{P} is 40 times larger than that.
Turning now to the second method of comparison, here we choose parameters for the current based model that will fit the effective membrane time constant and membrane potential distribution to that of a given conductance model at a given mean firing rate ν and recurrent connectivity C _{E,fit}. We set C _{E,fit} to be the C _{E,max} of the conductance-based model, since that is where, overall, its best capacities are found. ν and C _{E,fit} determine λ _{E} and λ _{I} and hence μ, σ and τ. For the current-based model, we set τ _{P} = τ and constrain a _{E}, a _{I}, and V _{P} to yield the same μ and σ as the conductance model. Since the constraints leave one degree of freedom, they leave a _{E} and a _{I} as functions of V _{P}. Increasing V _{P} decreases a _{E}/|a _{I}| and vice versa, leading to a trade-off between rate stability and wave regulation on the one hand, and capacity on the other. The third column of Fig. 7 shows for a range of V _{P} values the performance of the current-based model fitted to one particular conductance-based model (g _{I} = 0.095). For low V _{P}, the mean membrane potential increases with connectivity and the minimum pool size decreases. Thus, the embedding capacity increases but there is no regulation of wave activity. For high V _{P}, the opposite is the case: there is regulation but capacity is low. There is a narrow range of choices for V _{P} where there is both high capacity and regulation of the number of waves. One such value is V _{P} = μ. This is understandable because V _{P} = μ implies a _{E} λ _{E} + a _{I} λ _{I} = 0. Hence μ remains unchanged as the number of waves and the mean rate increase. Thus the balanced state is preserved (as in the conductance model), while σ increases (unlike the conductance model) resulting in a gradual increase in the minimum pool size (which must shift 95 % of the membrane potential distribution above the firing threshold). Therefore it is charitable to choose V _{P} = μ to fit a particular current model to each conductance model. The fourth column of Fig. 7 shows the performance of a current-based model fitted in this way to each of the conductance-based models shown in the first column, as g _{I} is varied. We observe that the current-based model behaves similarly to the conductance model with respect to the membrane potential parameters (μ, σ) and the capacity. However, unlike the conductance model, the limit C _{E,max} (which determines the ’envelope’ curve) is not due to the 0.5ν limit on the stochastic spiking rate (C _{E,max2}), but to the limit at which stochastic spiking becomes unstable (C _{E,max1}). Thus the drop in performance is not graceful as connectivity increases (with the amount of wave activity decreasing due to regulation), but catastrophic. This could only be remedied by choosing a higher V _{P}, as the third column shows, at the expense of reducing the capacity.
A difference between the conductance and current-based models, the latter with the values of μ, σ and τ fitted to the former, lies in the origin of wave regulation. In both models, zero intra-link delay spread behavior is a poor match to that with 0.5 ms intra-link delay spread, because the effective membrane time constant is small. A strong drift rate − (V − μ)/τ impedes the rise to threshold during the integration of synfire wave input, leading to a larger minimum pool size with the 0.5 ms intra-link delay spread. In the conductance-based model, the small τ is dynamic, and due to background input: 1/τ = 1/τ _{P} + λ _{E}(g _{E} + γg _{I}). In the fitted current models, however, τ is small but static: τ = τ _{P}. Wave regulation requires that the minimum pool size increase with background input. In the conductance-based model this happens because τ decreases even more, while μ and σ remain almost unchanged. In the current-based model, τ cannot change, so regulation must be due to the effects of background input on μ and σ. The conditions for regulation are essentially the same for 0.5 ms intra-link delay spread as for zero intra-link delay spread. We need μ to change only slightly, together with an increase in σ, to ensure that n _{E,min} increases. The conclusion that there is a narrow range of parameters for the current-based model that allow both capacity and wave regulation, including values V _{P} ≈ μ and a _{E} C _{E} + a _{I} C _{I} ≈ 0, is not qualitatively affected by the introduction of 0.5 ms intra-link delay spread.
In conclusion, the comparison of conductance and current-based models has shown that the former outperforms the conventional current-based model because the latter, with a membrane time constant in the order of tens of milliseconds, overestimates the magnitude of membrane potential fluctuations in the high conductance state, which leads to a very poor tradeoff between controlling stochastic spiking and keeping small the minimum pool size for wave propagation. A current-based model with the effective membrane time constant and membrane potential distribution matching those of the conductance model shows similar performance to the latter, both in capacity and regulation of waves. The simple analysis of this section assumed an intra-link delay spread of zero. If the intra-link delay spread is non-zero, the capacity of both the conductance model and the current-based model with the effective time constant fitted to that of the conductance model will be substantially reduced, but they still well-outperform that of the conventional current-based model.
4 Discussion
We presented a cortical network model that allows for the embedding of large numbers of pools linked to form synfire chains. Key features of this model are (i) use of conductance-based synapses, (ii) incorporation of inhibitory pools of neurons into the chains so that synfire waves conserve the balance of excitatory and inhibitory activity, and (iii) variable intra-link transmission delays. The number of pools is at the combinatorial capacity for a given connectivity per neuron: all excitatory connections are used to constitute pool-to-pool links.
We gave a mean-field analysis that determines steady state properties (mean firing rate and components due to stochastic spiking and spikes belonging to synfire waves) as a function of model parameters. Performance is assessed in terms of number of pools and number of co-active waves. There are two dynamical constraints on performance: stability of synfire wave propagation and stability of the low rate AI state of overall network activity. The analysis determines the boundaries for satisfying these two constraints. The constraint of synfire wave stability amounts to a minimum pool size for a given level of background input. For the constraint of AI stability, we provide a “naïve” estimate based on stability of the AI state of the corresponding balanced random network driven by balanced input. This is a necessary condition, but it may not always be sufficient, as in the case for the model studied by Aviel et al. (2005) for instance. Hence we use numerical simulations to confirm AI stability.
The mean field analysis (Section 3.2) shows (for three choices of inhibitory strength) that the minimum pool size increases with background input; equivalently, there is an upper limit on background input for wave propagation at a given pool size. Since wave activity drives background input, regulation of wave activity emerges as a natural outcome of the model: there is an upper bound on the number of propagating waves. In our simulations we drive the system to equilibrium at this upper bound since the pool size is then at the minimum (and the embedding level is at the maximum) for the resulting mean firing rate at the given connectivity per neuron. We verify that over a wide range of pool sizes and connectivity levels the AI state remains stable at this equilibrium and the mean field analysis provides a good quantitative description of the equilibrium behavior. We find a trade-off between embedding capacity on the one hand, and both equilibrium firing rate and equilibrium number of co-active waves on the other hand. Therefore, we define embedding capacity with respect to either the equilibrium firing rate or the equilibrium number of waves. For cortical scale connectivity in the range of 5,000–11,000 excitatory synapses per neuron, we obtain embedding capacities of around 0.5–2.0 for firing rates of 1–10 Hz and for 2–10 waves per 80,000 neurons respectively.
These capacity results are substantially higher than those found in previous models of chain embedding using spiking neurons and current-based synapses (Aviel et al. 2005; Schrader et al. 2008). The higher embedding capacity of the present model is a direct result of two outcomes: a small minimum pool size for a given connectivity, and a high connectivity attainable without a ground state instability. The improvement in both of these outcomes can, according to the analysis made in Section 3.7, be attributed to the use of conductance-based synapses. However, the predicted increase in the attainable connectivity is based on the stability of the mean spike rate of a balanced random network. The naïve prediction will only be valid if the chain-embedding network is no more susceptible to a breakdown of the AI state of spiking activity than the corresponding random network. Our simulation results are consistent with this supposition, but they do not go all the way to limit. Exactly why the present network avoids instabilities in the AI state that occur in previous studies, such as the synfire explosion (Mehring et al. 2003; Aviel et al. 2005), is not clear. The three key ingredients of the model (conductance-based synapses, variability in delays, and balanced pools) are all implicated in this improvement of AI stability but we have not teased apart their relative contributions. Network size is also important for high capacity. Figures 5 and 6 clearly indicate that high capacity is only attained when the connectivity per neuron reaches several thousand.^{8}
The structure of inhibition in the model serves to keep the network in a balanced state, where the ratio of the excitatory and inhibitory input rates and the mean membrane potential remain approximately constant as the number of waves (and the overall firing rate) increase due to external stimulation, by ensuring that excitatory and inhibitory mean firing rates are co-modulated, and indeed remain equal. This is achieved by (a) E–I balanced pools; and (b) the same mean recurrent connectivity to both E and I neurons. (a) and (b) respectively ensure that the mean rate of spikes belonging to synfire waves and the stochastic spiking rate are the same for both E and I neurons. The strength of inhibition affects capacity in two ways. On the one hand, increasing inhibition reduces capacity by increasing the required pool size at a given connectivity, but on the other hand, it allows for more recurrent connectivity by reducing stochastic spiking in response to noise. Overall there is a trade-off between these two effects which determines an optimal inhibition, and this optimum will decrease if the desired equilibrium firing rate is increased.
Our model has structural symmetries that cause the firing rates of excitatory and inhibitory neurons to be the same, for simplicity. This will not be the case if C _{E}, C _{I}, g _{E} or g _{I} differ for the synaptic inputs of the two neuron types, or if the condition for balanced pools, n _{E}/N _{E} = n _{I}/N _{I}, is not satisfied. This will not change the qualitative properties of the model if the ratio of the excitatory and inhibitory spiking rates remains approximately invariant. However, it would be interesting to break the symmetry in such a way that the stochastic spike rate increases more rapidly for inhibitory neurons than for excitatory ones, as this might allow stronger regulation of the number of waves at the same upper bound.
A simplified mean field analysis was used to make an efficient comparison of the conductance-based and current-based models for varying inhibitory strength and connectivity (Section 3.7). Two methods for specifying the current-based model were examined. For the first method, in which conventional parameters for the current-based model are used, the two models differ strikingly in behavior: optimal embedding capacity decreases with respect to inhibition for the current-based model but increases in an asymptotically linear manner for the conductance-based model. The difference stems from the behaviour of the membrane potential distribution. First, at high connectivities in the conductance-based model the neuron is in the high-conductance state in which the effective membrane time constant is substantially reduced (Destexhe and Paré 1999; Meffin et al. 2004; Richardson 2004), leading to a narrower membrane potential distribution than in the current-based model. Second, the inhibitory reversal potential prevents the mean membrane potential from becoming too low. Combined, these effects allow a neuron with conductance-based synapses to maintain both a high response to coincident input and a low response to background input as connectivity increases. This in turn allows cortical scale connectivity to be attained without substantial increase in pool size. For the second method, the current-based model is given a very small membrane time constant to match the effective membrane time constant of the conductance-based model, and other neuronal and synaptic parameters are constrained to fit the membrane potential distribution to that of the conductance-based model. Within these constraints there is a trade-off between capacity and wave regulation, and there exists a small range of parameters which achieve both wave regulation and capacities similar to those of the conductance model.
The model we have presented here uses delta-function synaptic responses and includes variability in the excitatory transmission delays within each synfire link in the form of a 0.5 ms intra-link delay spread. The comprehensive mean field analysis of Sections 2.4 and 2.5 takes this into account. However, the simplified analysis of Section 3.7 is only valid for a model with instantaneous synaptic responses and zero spread of intra-link transmission delays. As discussed in Section 2.2, this is not realistic as it leads to the omission of an integration window during which background input can prevent the spike response to synchronous input. Such a window will be present in a model with a spread of intra-link delays. The intra-link delay spread plays a critical role in determining the embedding capacity. We have shown that intra-link delay variability reduces robustness of wave propagation and hence capacity, but that such variability is necessary in a model with instantaneous synaptic responses in order that waves are not so robust that they cannot be regulated. Section 3.7 shows that regulation is missing in the conductance-based model with zero delay spread. Thus the delay spread governs a trade-off between capacity and regulation. It is also responsible for the existence of an optimal level of inhibition (Section 3.6); without a delay spread, embedding capacity simply continues to increase with increasing inhibition, along with connectivity and network size (Fig. 7, first column). Even without intra-link delay spread, a window of integration of synfire input will be present if synaptic responses are non-instantaneous (ie. there are post-synaptic rise times). Post-synaptic rise times should have the same effects as intra-link delay spread: regulation of wave activity in exchange for reduced capacity, and the existence of an optimal level of inhibition. Being more realistic than instantaneous synaptic responses, they should be included in future studies.
Apart from the role of the intra-link excitatory delay variability, how else do our results depend on the excitatory and inhibitory delay distributions used? The mean excitatory delay excitatory is important as the main component of the pool-to-pool propagation time, and thereby strongly affects how many waves can simultaneously propagate. If the mean delay is increased, the rate of spiking due to a single wave decreases and thus the number of waves will increase while the mean-field solution for the rates (ν _{W}, ν _{S}, etc.) remains invariant. The model would not support as many waves if the mean delay was shorter than the 2.5 ms value used, which we argued was realistic for local cortical connections. The inhibitory delays don’t play any direct role in synfire propagation but differences in E and I delay distributions will affect the nature of the steady state of the balanced random network (Brunel 2000). The asynchronous-irregular state of stochastic spiking is stable for the delay distributions we used, but with different distributions it could give way to synchronous-irregular (SI) oscillations. An interesting question for further study is the extent to which synfire propagation is compatible with SI oscillations.
Some simplifications made in the mean field analysis of Section 2.4 may explain the modest deviations from the simulation results. The analysis treats the stochastic spiking response to background input as that of a neuron receiving solely background input. Likewise the effects of background input on wave propagation are treated by considering a chain in which, apart from the inputs due to connections within the chain itself, the neurons received solely background input. Both treatments do not take into account that synchronous input packets also occur due to the fact that each neuron belongs to many pools in the embedding network. Synchronous inputs lead to additional spikes that cause the membrane potential of each neuron to spend a greater proportion of time in the post-spike recovery-from-reset phase, during which both the propensity to emit stochastic spikes and to respond to synchronous input from a wave propagating down the embedded chain are reduced. Taking these additional spikes into account would lower the mean field prediction for both the rate of stochastic spiking (f _{S}) and wave propagation reliability (P _{S}). Nevertheless, the treatments are still useful approximations: because the effective membrane time-constant of neurons in the high-conductance state is short (a few milliseconds or less) the recovery from reset to equilibrium is quick, minimizing the effect of the additional spikes. The analysis also overestimates the excitatory background input rate by including as background those inputs which are in fact synchronous: the approximation in Eq. (4). This simplification, like the previous one, leads to an overestimate of the stochastic spiking rate (f _{S}) and wave propagation reliability (P _{S}). Nevertheless, the approximation is reasonable, because the relative overestimate n _{E}/(C _{E}(1 + ν _{S}/ν _{W})) is small: ~0.4–4.0 % for the cases shown in Fig. 4(d). These simplifications may explain the quantitative discrepancies between theory and simulations observed in Fig. 4(d). We see, firstly, that total spike rate is lower than predicted. This is consistent with a less-than-predicted wave propagation reliability, implying a lower λ _{E,max}(n _{E}) and equilibration at a lower firing rate. Secondly, the fraction of spikes which are stochastic (ν _{S}/ν) is less than predicted, indicating that at a given firing rate the background input is producing less stochastic spiking. Note that the overestimates of ν and ν _{S} approximately cancel out, in so far as the predicted rate of wave-spiking (ν _{W} = ν − ν _{S}), yields a good quantitative agreement with the model simulations. The predicted number of waves is also in good agreement with the simulations, indicating that the relationship between ν _{W} and h/N _{E}, determined by p _{f}(λ _{E})/T(λ _{E}) evaluated at λ _{E} = λ _{E,max}, is preserved.^{9} From the modest size of the discrepancies observed in Fig. 4(d) and the fact that their origin can explained by the simplifications made, we may conclude that the mean field theory captures the essential behavior of the model, despite the complexity of its dynamics.
The use of independent Poisson input trains for the simulations of the non-embedded chains should also be questioned. Since the inputs actually arise from sampling a common embedding network of finite size (viz. N _{E} = 10C _{E}, N _{I} = 10C _{I}) the neurons will have some common inputs and their background input trains will thus be correlated. Tetzlaff et al. (2004) showed that input correlations based on sampling a common reservoir of Poisson neurons greatly reduced the maximum pool size for ground state stability to a level barely above the minimum pool size for stable wave propagation. However, it was found that this effect surprisingly disappeared when the Poisson reservoir was replaced by a Brunel-type balanced recurrent network. The decorrelating effect of recurrent inhibition has recently been identified as the mechanism responsible for this (Tetzlaff et al. 2012). This may explain why the Poisson approximation for background input turns out to be quite adequate for the present model.
In this paper, because we are concerned with embedding capacity, we have focused on the situation where the number of synfire waves is driven to the maximum level by ongoing spike packet stimulation, reaching an equilibrium (Section 3.3). However, the system need not operate in this marginal regime. It supports ongoing propagation of any number of waves below the maximum. In a more structured system where there is interaction and computation among the chains, the number of waves might vary in the course of this computation, being sometimes below the limit. Furthermore, in a model in which input convergence varies over chains the upper bound on background input rate would not be global, but specfic to individual chains or portions of chains. The equilibrium rate and number of waves would then depend on the specific pattern of activation, and hence could evolve during the course of a computation.
In synfire chains the neurons are organized into discrete pools. This feature, while convenient for study and analysis, can be criticized as artificial. However, it is not essential, as synfire chains can be generalized to a class of feedforward structures known as braids (Bienenstock 1995) or polychrony (Izhikevich 2006). In these structures the neurons need not be grouped into discrete pools, and they support precisely-timed and reproducible sequences of spikes that need not be grouped into synchronous packets. For instance, a uniform braid generates a sequence of spikes distributed uniformly over the interval of propagation, in a steady stream. In braids, the delays on the feedforward connections are specifically related to the spike sequences so as to preserve the key property of synfire chains that, during spike sequence re-activation, each neuron receives synchronously arriving spikes from upstream neurons. In models of the emergence of feedforward structures through synaptic plasticity there is no mechanism that would enforce a pool organization, so braids may be may easier to realize biologically than chains.
The present chain-embedding model can be generalized to a braid-embedding model. To embed uniform braids, instead of assigning neurons group-wise to a sequence of pools, they can be assigned one by one to a sequence of putative spikes distributed uniformly over a time interval. Excitatory feedforward connections can then be assigned appropriate delays so that if upstream neurons fire at the assumed times, then each downstream neuron will receive near-synchronous inputs shortly before its putative spike time. The property of double-balance can be incorporated straightforwardly.
A braid has no pools, so embedding capacity based on the number of pools makes no sense for a braid. To compare the embedding level of a braid embedding to that of a chain embedding, we can define the effective pool size of a braid as that of an “equivalent chain”: one with the same input convergence and mean transmission delay, and for which a propagating wave generates the same mean spike rate. The number of pools is then the number of putative spikes divided by the effective pool size. Note that in this equivalent chain, unlike the chains in present study, the input convergence may not equal to the pool size. However, it is straightforward to generalize the embedding capacity analysis to such chains (Trengove 2006). The mean field theory given here would predict for the braid model and the equivalent chain model quantitatively similar relationships between input convergence, effective pool size, connectivity per neuron, mean firing rate and number of waves, implying similar embedding capacities too.
The present study assumes that there are biological mechanisms for forming synfire chains (or braids) but this has not been established yet. There have been model studies demonstrating the growth of feedforward structures based on spike-timing dependent plasticity (STDP) or other plasticity mechanisms, but these have operated well below the cortical scale and some include artificial constraints (Bienenstock 1991; Doursat and Bienenstock 2006b, a; Jun and Jin 2007; Fiete et al. 2010; Waddington et al. 2011). A recent study using weight-dependent STDP in a cortical-scale balanced random network architecture failed to find the growth of structure (Kunkel et al. 2011), but it is still an open issue whether a more realistic model with the right ingredients to generate structure formation will eventually be found.
An important question is whether there are experimentally accessible features of the present model that can be used to test it against neural data, and thereby infer the presence of synfire activity or challenge the model. One observable consequence of the model is that, because most spikes are due to synfire waves, the spike-triggered average (STA) of the membrane potential will show a sudden rise just before a spike. Positive evidence of this kind was found for synfire chains in the HVC nucleus of the finch (Long et al. 2010). In the cortex, one study (London et al. 2010) found such sharp rises to be vary rare, but that may be due to the anaesthetized state in which the recordings were made. A necessary control for sharp rises as evidence for synfire chains is the conductance-based balanced random network, since in that model, noise-driven spiking with a short effective membrane time constant suggest that most spikes will be due to inputs arriving synchronously by chance; therefore the STA in such a network would also show a sudden rise before the spike. Another feature of the present model is the presence of sharp peaks in the cross-correlograms of pairs of neurons when there is repeated synfire wave transmission over the sequence of pools linking the pair (data not shown). The number of pairs showing such peaks, the size of the peaks and the distribution of offsets depend on the number and duration of the waves and how they are distributed over the chains, which in turn depend on the stimulation protocol. Strong peaks have also been observed experimentally but the time offset of the peak is typically small (Funahashi and Inoue 2000; Mizuseki et al. 2009; Isomura et al. 2009). The absence in experimental data of strong peaks offset by more than a few milliseconds implies that the experiments did not observe two neurons in a frequently re-activated chain. Thus any frequently reactivated chains must include only a small proportion of the neural population being sampled.^{10}
In general, the experimental detection of synfire activity is a difficult task, but methods for its detection have been developed recently (Schrader et al. 2008; Gerstein et al. 2012). These methods require multi-neuron recordings and repeated activation of chains. The number of simultaneously recorded spike trains required for reliable detection has been estimated at several hundred, assuming both the recording sites and the chains are co-localized to a volume of ∼ 10^{5} neurons. With less localization more parallel recordings are required. The methods have been tested on data generated by a variant of the present model, and clearly reveal the presence of synfire activity (Gerstein et al. 2012).
To conclude, we have considered the problem of constructing a model for embedding many entities (synfire chains) in a cortical scale network in which many of them may be activated simultaneously (by waves). With respect to the two aspects of capacity, namely number of pool-to-pool links and number of co-active waves, the present model achieves a performance level that sets a new world record for spiking neuron models. For realistic synaptic strengths, connectivity at the cortical-scale optimizes performance and is achieved while maintaining global stability of rate, without the need for a global mechanism to impose a limit on activity. The model allows this improved performance for two reasons. First, the use of conductance-based synapses compared to current-based increases the contrast between the neuronal response to synchronous input signal and the response to background noise. Second, instability of the global state of asynchronous irregular spiking is avoided by using balanced synfire waves and heterogeneity of transmission delays.
Another, current-based, network model exhibits simultaneous propagation of two waves in a network of 50,000 neurons (Schrader et al. 2008). In this model the combinatorial capacity is reached, but the connectivity is well below the cortical scale and the number of pools is only 0.02N. This number is in fact at the capacity limit, above which the background activity becomes unstable (Diesmann, private communication).
A normalized conductance response is the time-integrated synaptic conductance response divided by the capacitance, and is dimensionless.
Thus the stimulus has the same effect on the target pool as would a packet of n _{E} spikes of standard deviation 0.1 ms occurring in the previous pool.
In Section 3.3 we will describe an equilibrium with ongoing external stimulation of new waves, in which the number of waves is maintained at stable equilibrium level, h _{eq}, because the rate of wave births is balanced by a corresponding rate of wave deaths.
The excitatory background input rate C _{E}((1 − n _{E}/C _{E})ν _{W} + ν _{S}) is the expected value for a neuron. For a specific neuron or a specific time interval the rate will deviate from this. However, the standard deviation of the rate is small relative to the mean. In particular, even if the pools are visited by waves very unequally in frequency (as will be the case with the wave stimulation protocol of Section 2.3) the neurons still receive similar background input rates, because the background input is summing the input of C _{E} neurons, each of which contributes spikes belonging to spike packets in C _{E}/n _{E} − 1 pools. Thus the coefficient of variation (across neurons) of the contribution to background input due to spikes belonging to spike packets is approximately \(1/\sqrt{({C_{\mathrm{E}}}({C_{\mathrm{E}}}/{n_{\mathrm{E}}}-1))}\approx10^{-3}\) times the coefficient of variation (across pools) of the rate at which a pool is visited by waves.
This is an adequate condition for the stability of the stationary rate of a balanced random network driven by external balanced external input at rates C _{E} ν _{W} and C _{I} ν _{W}, except where the latter exhibits an oscillatory instability arising from a Hopf bifurcation. Note, this formulation neglects the dependence of ν _{W} on background input.
For notational brevity, and since λ _{I} = γλ _{E} , the λ _{I}-dependence of T _{S},P _{S},p _{f},... is left implicit in this section.
The connectivity needed for high capacity can be reduced by using larger synaptic strengths (Trengove 2006) but not by very much, as synaptic strengths an order of magnitude larger than used here would be biologically implausible and would cause the diffusion approximation breaks down, leading to more stochastic spiking than predicted by Eq. (20), and hence less increase in capacity.
The theory overestimates p _{f}/T, but since p _{f}/T decreases with λ _{E}, this is compensated for by the overestimate in λ _{E,max}.
An upper bound on how frequently a chain can be re-activated without producing a visible peak could be estimated by the condition that peak be less than twice the background level. Given a binsize δt and a mean firing rate ν this implies a chain activation rate less than ν ^{2} δt. For ν = 10 Hz and δt = 0.1 ms, chain activation rate must be below 0.1 Hz to escape detection. By that criterion, if the waves are uniformly distributed over the chains they will escape detection in the present model, but if a chain’s frequency of activation is an order of magnitude or more above the mean, then a pair of neurons in that chain would produce a detectable peak. (Note, for this discussion we are considering there to be many short chains (of order 100 pools in length) rather than a single long one (of order N _{E} pools) as was used in the simulations.)
Acknowledgements
All numerical experiments were carried out using the NEST simulation software (www.nest-initiative.org). Cees van Leeuwen is aided by an Odysseus grant from the Flemish Organization for Scientific Research FWO. We thank Moritz Helias, Tom Tetzlaff and Alexander Hanuschkin for helpful discussions. We appreciate the technical advice and support of Michael Denker and Peter Jurica. This work was partly funded by the Next-Generation Supercomputer Project of the Japanese Ministry of Education, Culture, Sports, Science and Technology (MEXT), the German-Israeli Project Cooperation (DIP) Grant F1.2, the Helmholtz Alliance on Systems Biology (Germany), and EU Grant 15879 for the Fast Analog Computing with Emergent Transient States (FACETS) Project.
Open Access
This article is distributed under the terms of the Creative Commons Attribution License which permits any use, distribution, and reproduction in any medium, provided the original author(s) and the source are credited.