Storing structured sparse memories in a multi-modular cortical network model

Abstract

We study the memory performance of a class of modular attractor neural networks, where modules are potentially fully-connected networks connected to each other via diluted long-range connections. On this anatomical architecture we store memory patterns of activity using a Willshaw-type learning rule. P patterns are split in categories, such that patterns of the same category activate the same set of modules. We first compute the maximal storage capacity of these networks. We then investigate their error-correction properties through an exhaustive exploration of parameter space, and identify regions where the networks behave as an associative memory device. The crucial parameters that control the retrieval abilities of the network are (1) the ratio between the number of synaptic contacts of long- and short-range origins (2) the number of categories in which a module is activated and (3) the amount of local inhibition. We discuss the relationship between our model and networks of cortical patches that have been observed in different cortical areas.

Appendix

Estimating the storage capacity requires to compute the distributions of the inputs on foreground and background neurons. Because neural activity and synapses are binary, these inputs are sums of binary random variables. We first present general results about such sums, that will be used later to compute the distributions of inputs to neurons.

We consider a random variable h

$$ h = \sum\limits_{k=1}^{K}X_{k} $$

(47)

where the X _k are independent binary random variables described by a parameter q:

$$ X_{k} = \left\{\begin{array}{ll} 1 & \text{with probability} \, q\\ 0 & \text{with probability} \, 1- q \end{array}\right. $$

(48)

The sum h is then distributed according to a binomial distribution

$$ P(h=S) = \binom{K}{S}q^{S}\left( 1-q \right)^{K-S} $$

(49)

Note that to get this binomial distribution, we have to assume the X _k ’s are independent. In our case, this means that two synapses on the same neuron $W_{ij_{1}}^{m,n}$ and $W_{ij_{2}}^{m,n}$ are treated as independent variables. This is a reasonable assumption to make as we have

$$ \mathbb{P} (W_{ij_{1}}^{m,m} = 1) = 1 - (1-f^{2})^{pc} $$

(50)

and

$$\begin{array}{@{}rcl@{}} &&\mathbb{P} (W_{ij_{1}}^{m,m} = 1\vert W_{ij_{2}}^{m,m} = 1)\\ &&= 1 - (1-f^{2})^{pc-1}(1-f) \underset{f\rightarrow 0,pc\rightarrow \infty}{\simeq} \mathbb{P} (W_{ij_{1}}^{m,m} = 1) \end{array} $$

(51)

and similarly for long-range connections $\mathbb {P} (W_{ij_{1}}^{m,n} = 1\vert W_{ij_{2}}^{m,n} = 1) \underset {f\rightarrow 0,pc\rightarrow \infty }{\simeq } \mathbb {P} (W_{ij_{1}}^{m,n} = 1)$.

We will consider cases in which K and S are large. We can then use Stirling formula to express the binomial coefficients and write

$$ P(h=S) = \exp \left( -K{\Phi} \left( \frac{S}{K},q\right) + o(K) \right) $$

(52)

with

$$ {\Phi} = {\Phi}^{fc}\left( \frac{S}{K},q \right) = \frac{S}{K}\ln \left( \frac{S/K}{q}\right) + \left( 1-\frac{S}{K} \right)\ln \left( \frac{1-S/K}{1-q}\right) $$

(53)

We use the superscript fc as this expression will be mainly used to describe fully connected sub-networks. For diluted enough networks, we will have $q, \frac {S}{K} \ll 1$, it is then useful to introduce

$$ {\Phi} = {\Phi}^{dc}\left( \frac{S}{K},q \right) = \frac{S}{K}\ln \left( \frac{S/K}{q} \right) - \frac{S}{K} + q $$

(54)

In our networks, when testing the stability of a given pattern $\boldsymbol {\Xi }^{\mu _{0}}$ it is useful to separate the total input into a local part and an external part. The local part is described by a couple (K, q), where K = f N is the number of neurons active in a given local network, and q = 1 or g depending on whether we are considering the input onto a foreground or a background neuron. The external part can also be described by a couple (K, q) with K = F(M−1)f N or K = F(M−1)(1−f)N and $q=\frac {D}{N}$ or $\frac {D}{N}G$ for foreground or background neurons.

The distribution of the total input on a neuron can be written

$$ \mathbb{P}(h_{i,m} = S) = \sum\limits_{S_{l},S_{e}/{S}_{l} + S_{e} = S}\mathbb{P}_{l}(S_{l})\mathbb{P}_{e}(S_{e}) $$

(55)

To compute it, we first need to express the distribution of the inputs generated by the local module and the distribution of the inputs generated by the other modules. In the asymptotic limits we consider, this sum will be dominated by the most probable term of the sum, we will thus need to find the couple (S _l , S _e ) that maximizes $\mathbb {P}_{l}(S_{l})\mathbb {P}_{e}(S_{e})$.

1.1 Distribution of inputs and probability of no-error in multi-modular network

1.1.1 Case F M → + ∞

We apply the method sketched above, first to compute the distribution of inputs on foreground neurons, and then on background neurons.

Foreground neurons The distribution of local inputs is a delta function $\mathbb {P}_{l}(S_{l}) = \delta (S_{l} - fN)$ as exactly f N neurons are active in each module in each pattern, and because of the fact that each module is a fully-connected network. The external component is the sum of the activity in each of the other F M−1≃F M active modules when their states coincides with the pattern $\boldsymbol {\Xi }^{\mu _{0}}$ we are trying to retrieve. Given the above results on sum of binary variables, it writes

$$\begin{array}{@{}rcl@{}} \mathbb{P}_{e}(S_{e}) &=& \binom{FMfN}{S_{e}}\left( \frac{D}{N}\right)^{S_{e}}\left( 1-\frac{D}{N}\right)^{FMfN-S_{e}} \\ &=& \exp\left[ -fN{\Phi}^{dc}\left( s_{e},\frac{\gamma}{c} \right)\right] \end{array} $$

(56)

with $s_{e} = \frac {S_{e}}{fN}$.

The total input on foreground neurons is then

$$\begin{array}{@{}rcl@{}} \mathbb{P}(h_{i,m} &=& S = fN + S_{e} \vert {\Xi}_{i,m}^{\mu_{0}} = 1)\\ &=& \exp\left[ - fN {\Phi}^{dc}\left( \frac{S_{e}}{fN},\frac{\gamma}{c} \right) + o(fN)\right] \end{array} $$

(57)

Background neurons in active modules The local input now fluctuates because inputs are mediated by synapses that have been potentiated during the presentation of randomly drawn patterns $\boldsymbol {\Xi }^{\mu \neq \mu _{0}}$. It is distributed according to

$$\begin{array}{@{}rcl@{}} P_{l}(S_{l}) &=& \binom{fN}{S_{l}}g^{S_{l}}(1-g)^{fN-S_{l}} \\ &=& \exp \left[-fN {\Phi}^{fc}\left( \frac{S_{l}}{fN},g \right) + o(fN) \right] \end{array} $$

(58)

where g is defined in Eq. (25). Similarly, the external part of the input is distributed according to

$$\begin{array}{@{}rcl@{}} P_{e}(S_{e}) &=& \binom{FMfN}{S_{e}}\left( \frac{D}{N}G\right)^{S_{e}}\left( 1- \frac{D}{N}G\right)^{FMfN-S_{e}} \\ &=& \exp \left[ -fN{\Phi}^{dc}\left( \frac{S_{e}}{fN},\frac{\gamma}{c} G \right) + o(fN)\right] \end{array} $$

(59)

The distribution of the total input is now written

$$\begin{array}{@{}rcl@{}} &&\mathbb{P}(h_{i,m} = S \vert {\Xi}_{i,m}^{\mu_{0}} = 0, {\Xi}_{m}^{\mu_{0}} = 1) \\ &&={\sum}_{S_{l},S_{e}/S_{l}+S_{e} = S} \exp\left[ -fN\left( {\Phi}^{fc}\left( \frac{S_{l}}{fN},g \right)\right.\right.\\ &&+\left.\left.{\Phi}^{dc}\left( \frac{S_{e}}{fN},\frac{\gamma}{c} G \right)\right) + o(fN)\right] \\ &&= \exp\left[ -fN\left( {\Phi}^{fc}\left( s_{l}^{*},g \right)+{\Phi}^{dc}\left( s_{e}^{*},\frac{\gamma}{c}G \right)\right) + o(fN)\right] \end{array} $$

(60)

where $s_{l}^{*} = \frac {S_{l}^{*}}{fN}$ and $s_{e}^{*} = \frac {S_{e}^{*}}{fN} = s - s_{l}^{*}$ (where $s=\frac {S}{fN}$) are given by the condition

$$ \frac{\partial \left( {\Phi}^{fc}(s1,g) + {\Phi}^{dc}(s-s_{l},\gamma FG) \right)}{\partial s_{l}}(s_{l}^{*}) = 0 $$

(61)

Solving this equation yields

$$\begin{array}{@{}rcl@{}} s_{l}^{*} &=& \frac{1}{2}\left( 1+ s + \frac{(1-g)\frac{\gamma}{c}G}{g}\right)\\ &&- \frac{1}{2}\sqrt{\left( 1+ s + \frac{(1-g)\frac{\gamma}{c}G}{g}\right)^{2} - 4s} \\ s_{e}^{*} &=& -\frac{1}{2}\left( 1-s + \frac{(1-g)\frac{\gamma}{c}G}{g} \right)\\ &&+ \frac{1}{2}\sqrt{\left( 1-s + \frac{(1-g)\frac{\gamma}{c}G}{g}\right)^{2} + 4\frac{(1-g)\frac{\gamma}{c}G}{g}s}\\&& \end{array} $$

(62)

Background neurons in silent modules There is only long-range inputs in this case and the fraction of activated long-range synapses G ^′≃α c F is given by (28),

$$\begin{array}{@{}rcl@{}} \mathbb{P}(h_{i,m} &=& S \vert {\Xi}_{i,m}^{\mu_{0}} = 0, {\Xi}_{m}^{\mu_{0}} = 0)\\ &=& \binom{FMfN}{S}\left( \frac{D}{N}G^{\prime}\right)^{S_{e}} \left( 1- \frac{D}{N}G^{\prime}\right)^{FMfN-S} \\ &=& \exp \left[ -fN{\Phi}^{dc}\left( \frac{S}{fN},\frac{\gamma}{c} G^{\prime} \right)+ o(fN)\right] \end{array} $$

(63)

Probability of no errors We have derived the expressions for the distribution of inputs to both foreground and background neurons. In order to compute the probability that there is no error ¶ _{n e} in the retrieval of pattern $\boldsymbol {\Xi }^{\mu _{0}}$, we have to estimate the probability that the inputs are above or below threshold, as written in the main text in (14). To do so we first note that

$$\begin{array}{@{}rcl@{}} \mathbb{P}(h_{i,m} \geq \theta fN \vert {\Xi}_{i,m}^{\mu_{0}}) &=& \mathbb{P}(h_{i,m}= \theta fN \vert {\Xi}_{i,m}^{\mu_{0}})\\ &&\!\sum\limits_{s\geq \theta} \frac{\mathbb{P}(h_{i,m} = s fN \vert {\Xi}_{i,m}^{\mu_{0}})}{\mathbb{P}(h_{i,m} = \theta fN \vert {\Xi}_{i,m}^{\mu_{0}})} \end{array} $$

(64)

where the ‘$\sum$’ term will not contribute to the final expression of $\mathbb {P}_{ne}$ in the large N limit, as has been shown in (Dubreuil et al. 2014). In practice we thus replace the probability to be above threshold by the probability to be at threshold. We can apply the same reasoning for the probability to be above the activation threshold for background neurons. We now have all the elements to express Φ^f, Φ^b and ${\Phi }^{b^{\prime }}$ in formulas (14):

$$ {\Phi}^{f} = {\Phi}^{dc}\left( \theta - 1,\frac{\gamma}{c} \right) $$

(65)

$$ {\Phi}^{b} = {\Phi}^{fc}\left( s_{l}^{*}(\theta),g \right) + {\Phi}^{dc}\left( s_{e}^{*}(\theta),\frac{\gamma}{c}G \right) $$

(66)

and

$$\begin{array}{@{}rcl@{}} {\Phi}^{b^{\prime}} &=& {\Phi}^{dc}\left( \theta,\frac{\gamma}{c}G^{\prime} \right) \\ &\underset{F \rightarrow 0}{\simeq}& \theta \log(1/F) \end{array} $$

(67)

1.1.2 Case F M = O(1)

In this case the microscopic dilution term $\frac {D}{N}$ is finite and we have to use Φ^fc instead of Φ^dc to describe the external inputs. Following the same reasoning as above, the rate functions are

$$\begin{array}{@{}rcl@{}} {\Phi}^{f} &=& (\theta- 1)\ln \left( \frac{\theta - 1}{FM\frac{D}{N}} \right)\\ &+& \left( FM-(\theta-1) \right)\ln \left( \frac{1 - (\theta-1)/FM}{1-D/N}\right) \end{array} $$

(68)

and

$$\begin{array}{@{}rcl@{}} {\Phi}^{b} &=& {\Phi}^{fc}\left( s_{l}^{*}(\theta),g \right) +s_{e}^{*}(\theta)\ln \left( \frac{s_{e}^{*}(\theta)}{FM\frac{D}{N}G} \right)\\ &+& \left( FM-s_{e}^{*}(\theta)\right)\ln \left( \frac{1 - s_{e}^{*}(\theta)/FM}{1-\frac{D}{N}G}\right) \end{array} $$

(69)

with $s_{l}^{*}$ and $s_{e}^{*}$ given by

$$\begin{array}{@{}rcl@{}} s_{l}^{*} &=& \frac{\lambda(FM-s)+1+s}{2(1-\lambda)}\\ &-& \frac{1}{2}\sqrt{\left( \frac{\lambda(FM-s)+1+s}{1-\lambda}\right)^{2} - 4\frac{s}{1-\lambda}}\\ s_{e}^{*} &=& \frac{-\lambda(FM+s)+s-1}{2(1-\lambda)}\\ &+& \frac{1}{2}\sqrt{\left( \frac{-\lambda(FM+s)+s-1}{(1-\lambda)}\right)^{2}+4\frac{\lambda FMs}{1-\lambda} } \\ \end{array} $$

(70)

with

$$ \lambda = \frac{\gamma}{cFM}\frac{G(1-g)}{(1-G\frac{\gamma}{cFM})g} $$

(71)

The expression for ${\Phi }^{b^{\prime }}$ remains unchanged as connectivity between two randomly taken modules is effectively highly diluted.

1.2 Capacity calculation for a single module with diluted connectivity

Here we focus on a model made of a single module with N neurons whose dynamics obey Eq. (1), where we store P patterns ξ ^μ with coding level f (${\sum }_{i} \xi _{i}^{\mu } = fN$). As for modular networks we focus on the limits N → + ∞, $f = \beta \frac {\ln N}{N}$ and $P=\frac {\alpha }{f^{2}}$ with α, β = O(1). Patterns are stored on a diluted connectivity matrix, such that at the end of the learning phase the synaptic matrix is given by W _{i j} = w _{i j} d _{i j} . With w _{i j} = 1 if there exists a pattern such that neurons i and j are co-activated, w _{i j} = 0 otherwise ; and d _{i j} is drawn randomly in {0,1} being 1 with probability d≪1.

To compute the capacity we follow the procedure described in the ‘Methods’ section. We first set the network in a state corresponding to one of the patterns $\boldsymbol {\Xi }^{\mu _{0}}$, and ask whether this a fixed point of Eq. (1). This is done by computing $\mathbb {P}_{ne}$, the probability that all the fields are on the right side of the activation threshold. Similarly to Eq. (13),

$$\begin{array}{@{}rcl@{}} \mathbb{P}_{ne} &=& \left( 1- \mathbb{P}(h_{i}\leq fN\theta \, \vert \, \xi_{i}^{\mu_{0}}=1) \right)^{fN}\\ &&\left( 1- \mathbb{P}(h_{i}\geq fN\theta \, \vert \, \xi_{i}^{\mu_{0}}=0) \right)^{(1-f)N} \end{array} $$

(72)

Using the Eqs. (52), (54) that describe the distributions of inputs and the fact that $\mathbb {P}(h_{i}\leq fN\theta \, \vert \, \xi _{i}^{\mu _{0}}=1)\simeq \mathbb {P}(h_{i}= fN\theta \, \vert \, \xi _{i}^{\mu _{0}}=1)$ (see (Dubreuil et al. 2014)) we can write

$$\begin{array}{@{}rcl@{}} \mathbb{P}(h_{i}\leq fN\theta \, \vert \, \xi_{i}^{\mu_{0}}=1) = \exp(-fN{\Phi}^{dc}(\theta,d)+o(fN)) \end{array} $$

(73)

and

$$\begin{array}{@{}rcl@{}} \mathbb{P}(h_{i}\geq fN\theta \, \vert \, \xi_{i}^{\mu_{0}}=1) = \exp(-fN{\Phi}^{dc}(\theta,dq)+o(fN)) \end{array} $$

(74)

where q is the fraction of synapses on i, w _{i j} , that are 1 after the learning phase, which can be expressed as

$$ q = 1 - \exp(-\alpha) $$

(75)

For $\mathbb {P}_{ne}$ to go 1 in the large N limit, Equations (17), (18) have to be satisfied. Saturating these inequalities leads to a choice of activation threshold θ → d and a coding level with

$$ \beta = \frac{1}{d(-\ln q+1-q)} $$

(76)

In the specific case we are studying the general expression for the storage capacity (20) can be written

$$ I = \frac{1}{\ln 2}\frac{\alpha}{\beta d} $$

(77)

Table 1 Notations

Using the two Eqs. (75)–(76), it can be expressed more simply

$$ I = \frac{\ln(1-q)(\ln q + 1 - q)}{\ln 2} $$

(78)

A maximal storage capacity I = 0.26 is reached at q = 0.24.

Note that for modular networks we can not get such a closed form for I since the $\mathbb {P}(h_{i}=fN\theta )$ needs to be estimated numerically.