On the Number of Limit Cycles in Diluted Neural Networks

Abstract

We consider the storage properties of temporal patterns, i.e. cycles of finite lengths, in neural networks represented by (generally asymmetric) spin glasses defined on random graphs. Inspired by the observation that dynamics on sparse systems has more basins of attractions than the dynamics of densely connected ones, we consider the attractors of a greedy dynamics in sparse topologies, considered as proxy for the stored memories. We enumerate them using numerical simulations and extend the analysis to large systems sizes using belief propagation. We find that the logarithm of the number of such cycles is a non monotonic function of the mean connectivity and we discuss the similarities with biological neural networks describing the memory capacity of the hippocampus.

Appendices

Appendix A: BP Equations

A well-known technique for computing the partition function, see Eq. 4 on a sparse graph is the so-called cavity method [31, 32]. Given the problem under consideration, one can construct a factor graph by introducing factor nodes ${\hat{i}}$’s for each local constraint $\psi _{i}({\underline{\sigma }}_{i}, {\underline{{\varvec{\sigma }}}}_{ {\partial i} } )$, see Fig. 6b. Even if the underlying graph ${\mathcal {G}}$ is tree-like as illustrated in Fig. 6a, the resulting factor graph contains many small loops as can be seen in Fig. 6b. This is notoriously a problem if we want to implement a cavity equation naively since this method is guaranteed to work well only on tree-like factor graphs.

Fig. 6

a Original graph topology ${\mathcal {G}}$ on which we consider a parallel dynamics. b Corresponding factor graph for the counting cycles problem. For each node i, ${\hat{i}}$ corresponds to factor imposing the cycle condition of node i and its neighbors $j_1$, $j_2$ and $j_3$. Because the neighbor $j_1$ also has its own factor ${\hat{j}}_1$ that links to i, we see that the factor graph contains many short loops

Fig. 7

An equivalent super-factor graph representation of the original problem in Fig. 6 is shown. For each node i of the original graph ${\mathcal {G}}$ we assign a super-factor ${\hat{i}}$, which corresponds to the cycle condition each node i has to satisfy. For each link (i, j) we assign a super-variable $S_{ij} = ({\underline{\sigma }}_{ij}, {\underline{\sigma }}_{ji})$ containing both spin configurations node i and j by doubling the dimension of super variables. Then, the resulting graph topology retains the same graph structure as ${\mathcal {G}}$, and thus it allows us to use a cavity method as long as the original graph ${\mathcal {G}}$ is tree-like. The redundancy due to the duplication of spin variables is then handled by implementing additional local constraints such that all the variables ${\underline{\sigma }}_{ik}$ sharing the same first index are constrained to be the same

One way to overcome this problem is to consider a particular form of Generalized Belief Propagation (GBP) [33] proposed precisely to deal with loops. In the original formulation, the difficulty arising from the presence of loops is alleviated considering messages between regions of nodes that have a tree-like structure. In the present case, this comes at the price of doubling the state space and considering the super-factor graph shown in Fig. 7.

In this representation we introduce a super variable for each link (i, j) of ${\mathcal {G}}$ to hold two spin trajectories $S_{ij} = ({\underline{\sigma }}_{ij},{\underline{\sigma }}_{ji})$. Here, the variable ${\underline{\sigma }}_{ij}$ is interpreted as the duplicated copy of spin-trajectories of node i that is introduced due to the presence of link (ij). Then, this redundancy due to doubling the spin dimension is removed by adding delta function constraints to each factor node ${\hat{i}}$ to ensure that all spin variables sharing the same first index i are bound to be the same, i.e, ${\underline{\sigma }}_{ij} = {\underline{\sigma }}_{i}$ for all $j \in {\partial i}$. From the above setting, we achieve a tree-like factor graph without losing the locality of constraints. This trick has been recently used in similar contexts [35, 36].

Having established a locally tree-like factor graph, it is straightforward to write the corresponding BP equations. Following the standard procedure, the message that factor ${\hat{i}}$ sends to node (ij) reads

$$\begin{aligned} \nu _{{\hat{i}}\rightarrow (ij)} ({\underline{\sigma }}_i, {\underline{\sigma }}_j) \propto \sum _{ {\underline{{\varvec{\sigma }}}}_ {\partial i \setminus j} } \psi _{i}({\underline{\sigma }}_{i}, {\underline{{\varvec{\sigma }}}}_{ {\partial i} } ) \prod _{k \in {\partial i \setminus j} } \eta _{(ki)\rightarrow {\hat{i}}} ({\underline{\sigma }}_k , {\underline{\sigma }}_i) \end{aligned}$$

(A.1)

where the first summation runs over all possible sub-trajectories of spins ${\partial i \setminus j}$. Similarly, the messages from the super-variables to the super-factors can be written trivially as there exist only two links for each super-variables, namely

$$\begin{aligned} \eta _{(ij) \rightarrow {\hat{j}}} ({\underline{\sigma }}_i, {\underline{\sigma }}_j ) \propto \nu _{{\hat{i}}\rightarrow (ij)} ({\underline{\sigma }}_i, {\underline{\sigma }}_j ). \end{aligned}$$

(A.2)

By combining Eqs. A.1, A.2, we can define a single set of messages $m_{i \rightarrow j} (\sigma _i, \sigma _j) \equiv \eta _{(ij) \rightarrow {\hat{j}}} (S_{ij}) \propto \nu _{{\hat{i}}\rightarrow (ij)} (S_{ij})$ satisfying Eq. 10.

Once the messages are determined through the iterative process with Eq. 10 or equivalently A.1, A.2, the partition function in Eq. 4 can be viewed as the sum of three different contributions coming from super-nodes, super-factors, and super-links, namely, the partition function reads

$$\begin{aligned} \log Z_{L} = \sum _{(ij)} \log z_{(ij)} + \sum _{{\hat{i}}} \log z_{{\hat{i}}} - \sum _{<{\hat{i}},(ij)>} z_{{\hat{i}}, (ij)}\,, \end{aligned}$$

(A.3)

where the first sum is made on all super-nodes, the second one on all super-factors and the third one on all the links between super-nodes and super-factors. For a general factor graph, these factors are given in terms of $\nu _{{\hat{i}}\rightarrow (ij)}$ and $\eta _{(ij) \rightarrow {\hat{i}}}$, i.e.,

$$\begin{aligned} z_{(ij)}= & {} \sum _{S_{ij}} \nu _{{\hat{i}}\rightarrow (ij)} (S_{ij}) \nu _{{\hat{j}}\rightarrow (ij)} (S_{ji})\,, \end{aligned}$$

(A.4)

$$\begin{aligned} z_{{\hat{i}}}= & {} \sum _{{\underline{S}}_{\partial {\hat{i}}}} \psi _{{\hat{i}}}^S ({\varvec{S}}_{\partial {\hat{i}}}) \prod _{k \in \partial i } \eta _{(ki)\rightarrow {\hat{i}}} (S_{ki})\,, \end{aligned}$$

(A.5)

$$\begin{aligned} z_{{\hat{i}}, (ij)}= & {} \sum _{S_{ij}} \nu _{{\hat{i}}\rightarrow (ij)} (S_{ij}) \eta _{(ij) \rightarrow {\hat{i}}} (S_{ij})\,, \end{aligned}$$

(A.6)

where $\psi _{{\hat{i}}}^S ({\varvec{S}}_{\partial {\hat{i}}})$ denotes the constraint on the super graph,

$$\begin{aligned} \psi _{{\hat{i}}}^S ({\varvec{S}}_{\partial {\hat{i}}}) = \psi _{i}({\underline{\sigma }}_{i}, {\underline{{\varvec{\sigma }}}}_{ {\partial i} } ) \prod _{j \in \partial i } \delta ({\underline{\sigma }}_{ij} - {\underline{\sigma }}_{i} ) \end{aligned}$$

(A.7)

In our setting, these equations are compactly written in terms of the messages in Eq. 10:

$$\begin{aligned} z_{(ij)} = \sum _{\sigma _i, \sigma _j} m_{i \rightarrow j} ({\underline{\sigma }}_{i}, {\underline{\sigma }}_{j}) m_{j \rightarrow i} ({\underline{\sigma }}_{j}, {\underline{\sigma }}_{i}) \end{aligned}$$

(A.8)

and

$$\begin{aligned} z_i = z_{{\hat{i}}} = \sum _{\sigma _i} \sum _{{\underline{\sigma }}_{\partial i}} \psi _{i}({\underline{\sigma }}_{i}, {\underline{{\varvec{\sigma }}}}_{ {\partial i} } ) \prod _{k \in \partial i } m_{k \rightarrow i} ({\underline{\sigma }}_{k}, {\underline{\sigma }}_{i}) \end{aligned}$$

(A.9)

where we have introduced another representation $z_i = z_{{\hat{i}}}$ to drop the hat from ${\hat{i}}$, which is possible because factor nodes and original nodes are in one-to-one correspondence ${\hat{i}}\leftrightarrow i$. Additionally, one can immediately establish the relations $z_{{\hat{i}},(ij)} = z_{{\hat{j}},(ji)}$ and $z_{(ij)}=z_{{\hat{i}},(ij)}$. After making changes with these simplifications to Eq. A.3, one establishes Eq. 7.

Appendix B: Motifs Preventing Cycles of Lengths $L <4$

Fig. 8

Illustration of graph motifs that forbid cycles with $L<4$. If the coupling matrix satisfies both conditions in Eqs. B.1, B.2, the states of i and j are completely independent of the rest of the system. At the same time, the states can only form a periodic orbit of length 4, implying that there exist only cycles of length a multiple of 4 in the system

There exist certain motifs that forbid the existence of cycles of length $L<4$. Let us consider a subset of graph ${\mathcal {G}}$ given by Fig. 8 where two nodes i and j are connected. Now, let us suppose that the couplings satisfy the following condition

$$\begin{aligned} J_{ij} > \sum _{k \in {\partial i \setminus j} } | J_{ik} |. \end{aligned}$$

(B.1)

If this condition is met, the state of spin $\sigma _i^t$ is solely determined to be $\sigma _j^{t-1}$ by the state of the spin j at time step $(t-1)$ according to Eq. 1. Additionally, let us impose a similar condition for the node j but with the negative sign:

$$\begin{aligned} - J_{ji} > \sum _{l \in {\partial j \setminus i} } | J_{jl} |. \end{aligned}$$

(B.2)

Thus, it similarly follows that $\sigma _j^{t} = - \sigma _i^{t-1}$. The existence of such conditions implies that even though the network is connected, there can be subsets of a network that are independent of the rest of the network. In this particular case, because of the imposed conditions, the pair $(\sigma _i^t, \sigma _j^t)$ can only form a periodic orbit of length 4, thus the length of the cycles in this system should be a multiple of 4. Considering that all the graph ensembles discussed in this paper have the finite probability per node of having such motifs as long as $\epsilon >0$ and the connectivity is finite, the cycles of length shorter than $L<4$ cannot exist in the thermodynamic limit. This result becomes more and more likely as $\epsilon$ increases. In fact, for small values of $\epsilon$ the couplings tend to be more symmetric and the occurrences of this motif is suppressed. In this case, the existence of cycles of length smaller than $L=4$ is possible at finite sizes, as shown in Fig. 9.

Fig. 9

Complexity of cycles of length $L=1$ and $L=2$ for a RR graph with $c=6$ as a function of $\epsilon$ for $N=100$. As mentioned in the text, these cycles disappears as N increases. However, at finite N and small $\epsilon$, the probability of the existence of the motif discussed in the text is small and numerically it is not found

Fig. 10

Histogram of the values of Q for $N=10$, $\epsilon =1$ and connectivity $c=1$ (top) or $c=4$ (bottom) in the RR ensemble

If the two conditions defined in Eqs.  B.1 and B.2 holds for each couple of link, cycles are skew symmetric. A generic 4-cycle

$$\begin{aligned} {\varvec{\sigma }}_1 \rightarrow {\varvec{\sigma }}_2 \rightarrow {\varvec{\sigma }}_3 \rightarrow {\varvec{\sigma }}_4 \rightarrow {\varvec{\sigma }}_1 \rightarrow \cdots \end{aligned}$$

is skew symmetric if

$$\begin{aligned} {\varvec{\sigma }}_1 \rightarrow {\varvec{\sigma }}_2 \rightarrow -{\varvec{\sigma }}_1 \rightarrow -{\varvec{\sigma }}_2 \rightarrow {\varvec{\sigma }}_1 \rightarrow \cdots \end{aligned}$$

Such symmetry may be conveniently described by the quantity Q:

$$\begin{aligned} Q=\frac{1}{2N} \left( {\varvec{\sigma }}_{1} \cdot {\varvec{\sigma }}_{3}+ {\varvec{\sigma }}_{2} \cdot {\varvec{\sigma }}_{4} \right) . \end{aligned}$$

(B.3)

Because $\sigma = \{-1,1\}$, which means that $\vert {\varvec{\sigma }}\vert ^2=N$, we have $-1 \le Q< 1$. If a cycle is skew symmetric ($\sigma _1=-\sigma _3$ and $\sigma _2=-\sigma _4$), then $Q=-1$. Additionally, if we consider only limit cycles of length 4, Q cannot be equal to 1. At $\epsilon =2$, coupling are perfectly antisymmetric and our numerical results confirm that the limit cycles are of length $L=4$, and all of them have $Q=-1$. We then ran simulations and checked the skew symmetry of limit cycles of length 4 to study the Q values with decreasing connectivity parameter at $\epsilon =1$ (from $N=6$ to $N=15$ and c ranging from 0.4 to 4). As an example, in Fig.  10 we report two histograms of the obtained Q values for $N=10$, in the case of $c=1$ and $c=4$ ($\epsilon =1$). It is evident that the fraction of limit cycles with $Q=-1$, which dominates in the case of $c=4$, tends to disappear when considering the case of $c=1$. To confirm this data, Fig. 11 shows the fraction of limit cycles of length 4 for which $Q = -1$, as a function of c. In this case, all curves decrease on decreasing connectivity, consistently with the results of the previous figure. It is worth noticing that for large N the occurrences of situations like the one outlined in Fig. 8, which can be found only at sufficiently small values of c, increase, as well as the probability that ${{\,\mathrm{sign}\,}}(J_{ij})={{\,\mathrm{sign}\,}}(J_{ji})$ (given that $\epsilon =1$) in at least one pair of nodes, which is sufficient to break the skew-symmetry of the limit cycles of length 4. The second effect dominates over the first one at low connectivities and skew symmetricity disappears.

Fig. 11

Fraction of limit cycles of length 4 with $Q=-1$ as a function of the connectivity c for various N, ranging from 7 to 16, at $\epsilon =1$ obtained for the RR ensemble

Appendix C: Additivity of Complexity in the Presence of Disconnected Clusters

In this appendix, we construct a simple relation for $Z_L$ for the case of graphs with a finite number of connected clusters. Since each connected cluster is independent of the other ones, we have the following additivity property:

$$\begin{aligned} \ln Z_L = N_C \sum _C P(C) \ln Z_C, \end{aligned}$$

(C.1)

where $N_C$ is the number of connected clusters and P(C) is the cluster size distribution. From the normalization condition, we have the following relation

$$\begin{aligned} N_C \sum _C P(C) C = N. \end{aligned}$$

(C.2)

If the clusters possess the same statistical properties and their sizes are sufficiently large, it is reasonable to assume that $\ln Z_C$ follows the same asymptotic expansion of $\ln Z_L$ i.e., $\ln Z_C \sim C \Sigma _{L} + B_L + O(C^{-1})$. Under these conditions, the overall partition function reads

$$\begin{aligned} \ln Z_L\sim & {} N \Sigma _L + B_L + O(N^{-1}) \nonumber \\= & {} N_C \sum _C P(C) \ln Z_C \nonumber \\\sim & {} N \Sigma _L + N_C B_L + O(N_C C^{-1}), \end{aligned}$$

(C.3)

where we have used the normalization condition. Additionally, using Eq. (11), we reach a mismatch in the constant order, thus implying $B_L =0$. This relation can potentially become wrong if $N_C$ is extensive. In this case all the higher corrections in Eq. C.3 become of the same order as $\Sigma _L$. Nevertheless, we found numerically that $B_L$ is distributed close to our prediction $B_L =0$.