Spin Glass Systems as Collective Active Inference

Abstract

An open question in the study of emergent behaviour in multi-agent Bayesian systems is the relationship, if any, between individual and collective inference. In this paper we explore the correspondence between generative models that exist at two distinct scales, using spin glass models as a sandbox system to investigate this question. We show that the collective dynamics of a specific type of active inference agent is equivalent to sampling from the stationary distribution of a spin glass system. A collective of specifically-designed active inference agents can thus be described as implementing a form of sampling-based inference (namely, from a Boltzmann machine) at the higher level. However, this equivalence is very fragile, breaking upon simple modifications to the generative models of the individual agents or the nature of their interactions. We discuss the implications of this correspondence and its fragility for the study of multiscale systems composed of Bayesian agents.

Appendices

A Bayesian Inference for a Single Agent

In this appendix we derive the exact Bayesian inference update for the posterior over the latent state z, taking the perspective of a single agent.

We begin by rehearsing the component likelihood and prior distributions of the generative model in more detail.

1.1 A.1 Likelihood

The likelihood model relates the hidden state z to the observed spin state of a particular neighbour $\sigma _j$ as an exponential distribution parameterized by a sensory precision parameter $\gamma$:

$$\begin{aligned} P(\sigma _j|z; \gamma ) = \frac{\exp (\gamma \sigma _j z)}{2 \cosh (\gamma z)} \end{aligned}$$

(A.1)

The sign and magnitude of $\gamma$ determines the nature of the expected mapping between hidden states z and neighbouring spin observations $\sigma _j$. For $\gamma > 0$, then the observed spin is expected to reflect the latent state z, and with $\gamma < 0$, then the observed spin is expected to be opposite to the latent state z. The magnitude of $\gamma$ then determines how deterministic this mapping is.

Equation (A.1) can alternatively be seen as a collection of two conditional Bernoulli distributions over $\sigma _j$, one for each setting of z. This can be visualized as a symmetric matrix mapping from the two settings of z (the columns, corresponding to $z = -1, +1$) to the values of $\sigma _j$ (the rows $\sigma _j = -1, +1$):

$$\begin{aligned} P(\sigma _j | z ; \gamma )&= \begin{bmatrix} \frac{1}{1 + \exp (-2\gamma )} &{} \frac{1}{1 + \exp (2\gamma )} \\ \frac{1}{1 + \exp (2\gamma )} &{} \frac{1}{1 + \exp (-2\gamma )} \end{bmatrix} \end{aligned}$$

(A.2)

where this mapping approaches the identity matrix as $\gamma \rightarrow \infty$.

Now we can move onto writing down the likelihood over the observed spins of multiple neighbours: $\tilde{\sigma } = \{\sigma _j : j \in M\}$ where M denotes the set of the focal agent’s neighbours. We build in a conditional independence assumption into the focal agent’s generative model, whereby the full likelihood model over all observed spins factorizes across the agent’s neighbours. This means we can write the likelihood as a product of the single-neighbour likelihoods shown in Eq. (A.1):

$$\begin{aligned} P(\tilde{\sigma }|z; \gamma )&= \prod _{j \in M} \frac{\exp (\gamma \sigma _j z)}{2 \cosh (\gamma z)} \nonumber \\&= \exp \left( z \gamma \sum _{j \in M} \sigma _j - K \log (2 \cosh (\gamma z))\right) \end{aligned}$$

(A.3)

where K is the number of the focal agent’s neighbours (i.e. the size of the set M). We can easily generalize this likelihood to heterogeneous precisions by instead parameterizing it with a precision vector $\tilde{\gamma } = \{\gamma _j : j \in M\}$ that assigns a different precision to observations coming from each of the focal agent’s neighbours:

$$\begin{aligned} P(\tilde{\sigma }|z;\tilde{\gamma })&= \prod _{j \in M} \frac{\exp (\gamma _j \sigma _j z)}{2 \cosh (\gamma _j z)} \nonumber \\&= \exp \left( z \sum _{j \in M}\gamma _j \sigma _j - \sum _{j \in M} \log (2 \cosh (\gamma _j z))\right) \end{aligned}$$

(A.4)

1.2 A.2 Prior

We parameterize the focal agent’s prior beliefs about the latent spin state z as a simple Bernoulli distribution, and similarly to the likelihood model, we will express it as an exponential function parameterized by a ‘prior precision’ parameter $\zeta$:

$$\begin{aligned} P(z;\zeta )&= \frac{\exp (\zeta z)}{2 \cosh (\zeta z)}\nonumber = \exp (\zeta z - \log ( 2 \cosh (\zeta ))) \end{aligned}$$

As with the sensory precision $\gamma$, the prior precision also scales the strength of the focal agent’s prior belief that the spin state z is $+1$.³

1.3 A.3 Bayesian Inference of Hidden States

Now we ask the question: how would a focal agent (i.e., the agent that occupies a single lattice site) optimally compute a belief over z, that is most consistent with a set of observed spins $\tilde{\sigma }$? This is a problem of Bayesian inference, which can be expressed as calculation of the posterior distribution over z via Bayes Rule:

$$\begin{aligned} P(z | \tilde{\sigma }; \gamma , \zeta ) = \frac{P(\tilde{\sigma }, z; \gamma , \zeta )}{P(\tilde{\sigma }; \gamma , \zeta )} \end{aligned}$$

(A.5)

Since we are dealing with a conjugate exponential model⁴, we can derive an analytic form for the posterior: $P(z | \tilde{\sigma }, \gamma , \zeta )$:

$$\begin{aligned} P(z|\tilde{\sigma }; \gamma , \zeta ) = \frac{\exp \left( z (\zeta + \gamma \sum _{j} \sigma _j)\right) }{2\cosh \left( \zeta + \gamma \sum _{j} \sigma _j \right) } \end{aligned}$$

(A.6)

where the sum over neighbouring spins j only includes the neighbours of the focal agent, i.e., $\sum _{j \in M} \sigma _j$. If we fix the hidden state z to a particular value (e.g. $z = +1$), then we arrive at a simple expression for the posterior probability that the hidden spin state is in the ‘UP’ state, given the observations and the generative model parameters $\gamma , \zeta$. This probability reduces to a logistic or sigmoid function of sensory input, which is simply the sum of neighbouring spin values $\varDelta \sigma = \sum _{j} \sigma _j$. This can also be seen as the ‘spin difference’, or the number of neighbouring spins that are in the ‘UP’ position, minus those that are in the ‘DOWN’ position. The steepness and horizontal shift of this logistic function are intuitively given by likelihood and prior precisions, respectively:

$$\begin{aligned} P(z = +1|\tilde{\sigma }, \gamma , \zeta )&= \frac{\exp (\zeta + \gamma \varDelta \sigma )}{\exp (\zeta + \gamma \varDelta \sigma ) + \exp (- (\zeta + \gamma \varDelta \sigma ))} \nonumber \\&= \left( 1 + \frac{\exp (-(\zeta + \gamma \varDelta \sigma ))}{\exp (\zeta + \gamma \varDelta \sigma )}\right) ^{-1} \nonumber \\&= \frac{1}{1 + \exp (-2(\zeta + \gamma \varDelta \sigma ))} \end{aligned}$$

(A.7)

The denominator in the first line of (A.7) follows from the identity $\cosh (x) = \frac{1}{2}(\exp (x) + \exp (-x))$.

B Active Inference Derivations

In this section we provide the additional derivations needed to equip each agent with the ability to infer a posterior over control states and sample from this posterior to generate actions. This achieved through the framework of active inference.

Active inference casts the selection of control states or actions as an inference problem, whereby actions u are sampled or drawn from posterior belief about controllable hidden states. The posterior over actions is computed as the softmax transform of a quantity called the expected free energy [9]. This is the critical objective function for actions that enables active inference agents to plan actions into the future, since the expected free energy scores the utility of the anticipated consequences of actions.

1.1 B.1 Predictive Generative Model

We begin by writing a so-called ‘predictive’ generative model that crucially includes probability distributions over the agent’s own control states $u \in \{-1, +1\}$ and how those control states relate to future (anticipated) observations. In other words, we consider a generative model over two timesteps: the current timestep t and one timestep in the future, $t+1$. This will endow our agents with a shallow form of ‘planning’, where they choose actions in order to maximize some (pseudo-) reward function defined with respect to expected outcomes. This can be expressed as follows:

$$\begin{aligned} P(\tilde{\sigma }_t, z_{t}, u_{t}, \mathcal {O}_{t+1}; \gamma , \zeta ) = \tilde{P}( \mathcal {O}_{t+1}|z_t, u_t, \tilde{\sigma }_{t})P(\tilde{\sigma }_t, z_t, u_t; \gamma , \zeta ) \end{aligned}$$

(B.8)

where the generative model at the second timestep $\tilde{P}( \mathcal {O}_{t+1}|z_t, u_t, \tilde{\sigma }_{t})$ we hereafter refer to as the ‘predictive’ generative model, defined over a single future timestep.

Active inference consists in sampling a belief from the posterior distribution over control states u—this sampled control state or action is then fixed to be the spin state of the agent under consideration. Thus the action of one agent is fed in as the observations for those spin sites that it’s connected to. In order to imbue active inference agents with a sense of goal-directedness or purpose, we encode a prior distribution over actions P(u) that is proportional to the negative of the expected free energy, via the softmax relationship:

$$\begin{aligned} P(u) = \frac{\exp (-\textbf{G}(u))}{ \sum _u \exp (-\textbf{G}(u))} \end{aligned}$$

(B.9)

Crucially, the expected free energy of an action $\textbf{G}$ is a function of outcomes expected under a particular control state u, where beliefs about future outcomes are ‘biased‘ by prior beliefs about encountering particular states of affairs. In order to optimistically ‘bend’ these future beliefs towards certain outcomes, and thus make some actions more probable than others, we supplement the predictive generative model $\tilde{P}$ with a binary ‘optimality’ variable $\mathcal {O} \pm 1$ that the agent has an inherent belief that it will observe. This is encoded via a ‘goal prior’ or preference vector, which is a Bernoulli distribution over seeing a particular value of $\mathcal {O}$ with some precision parameter $\omega$:

$$\begin{aligned} \tilde{P}(\mathcal {O}_{t+1}; \omega )&= \frac{\exp (\omega \mathcal {O})}{2 \cosh (\omega \mathcal {O})} \end{aligned}$$

(B.10)

Hereafter we assume an infinitely high precision, i.e. $\omega \rightarrow \infty$. This renders the preference an ‘all-or-nothing’ distribution over observing the optimality variable being in the ‘positive’ state $\mathcal {O} = +1$:

$$\begin{aligned}&=\begin{bmatrix} \tilde{P}(\mathcal {O}_{t+1} = -1) \\ \tilde{P}(\mathcal {O}_{t+1} = +1)\end{bmatrix} = \begin{bmatrix} 0.0 \\ 1.0 \end{bmatrix} \end{aligned}$$

(B.11)

To allow an agent the ability to predict the relationship between their actions and expected observations, it’s important to include an additional likelihood distribution, what we might call the ‘forward model’ of actions $P(\mathcal {O}_{t+1}|z_t, u_t; \xi )$. This additional likelihood encodes the focal agent’s assumptions about the relationship between hidden states, actions, and the (expected) optimality variable. By encoding a deterministic conditional dependence relationship into this likelihood, we motivate the agent (via the expected free energy) to honestly signal its own estimate of the hidden state via its spin action u. To achieve this, we explicitly design this likelihood to have the following structure, wherein the optimality variable is only expected to take its ‘desired value’ of $\mathcal {O} = +1$ when $z = u$. This can be written as a set of conditional Bernoulli distributions over $\mathcal {O}$, and each of which jointly depends on z and u and is parameterized by a (infinitely high) precision $\xi$:

$$\begin{aligned} P(\mathcal {O}_{t+1} | z_{t} , u_{t} ; \xi )&= \frac{\exp (\xi \mathcal {O}_{t+1} z_t u_t)}{2\cosh (\xi z_t u_t)} \end{aligned}$$

(B.12)

When we assume $\xi \rightarrow \infty$, then we arrive at a form for this likelihood which can be alternatively expressed as a set of Bernoulli distributions that conjunctively depend on z and u, and can be visualized as follows:

$$\begin{aligned} P(\mathcal {O}_{t+1} | z_t, u_t = -1)&= \begin{bmatrix} 0 &{} 1 \\ 1 &{} 0 \end{bmatrix} \nonumber \\ P(\mathcal {O}_{t+1} | z_t, u_t = +1)&= \begin{bmatrix} 1 &{} 0 \\ 0 &{} 1 \end{bmatrix} \end{aligned}$$

(B.13)

where the columns of the matrices above correspond to settings of $z \in \{-1, +1\}$. Therefore, the agent only expects to see $\mathcal {O} = +1$ (the desired outcome) when the value of the hidden state and the value of the control variable are equal, i.e. $z = u$; otherwise $\mathcal {O} = -1$ is expected. For the purposes of the present study, we assume both the optimality prior $\tilde{P}(\mathcal {O};\omega )$ and the optimality variable likelihood $P(\mathcal {O}|z,u; \xi )$ are parameterized by infinitely high precisions $\omega = \xi = \infty$, and hereafter will exclude them when referring to these distributions for notational convenience.

Having specified these addition priors and likelihoods, we can write down the new (predictive) generative model as follows:

$$\begin{aligned} \tilde{P}(\mathcal {O}_{t+1}, u_t , z_t) = P(\mathcal {O}_{t+1} | z_t, u_t) P(u_t)\tilde{P}(\mathcal {O}_{t+1})P(z_t) \end{aligned}$$

(B.14)

1.2 B.2 Active Inference

Under active inference, both state estimation and action are consequences of the optimization of an approximate posterior belief over hidden states and actions $Q(z, u;\phi )$. This approximate posterior is optimized in order to minimize a variational free energy (or alternatively maximize an evidence lower bound). This is the critical concept for a type of approximate Bayesian inference known as variational Bayesian inference [4]. This can be described as finding the optimal set of variational parameters $\phi$ that minimizes the following quantity:

$$\begin{aligned} \phi ^{*}&= \underset{\phi }{\arg \min }\ \, \, \mathcal {F} \nonumber \\ {}&= \mathbb {E}_Q[\ln Q(z_t, u_t;\phi ) - \ln \tilde{P}(\tilde{\sigma }_t, z_t, u_t, \mathcal {O}_{t+1}; \gamma , \zeta )] \end{aligned}$$

(B.15)

In practice, because of the factorization of the generative model into a generative model of the current and future timesteps, we can split state-estimation and action inference into two separate optimization procedures. To do this we also need to factorize the posterior as follows:

$$\begin{aligned} Q(z, u;\phi ) = Q(z;\phi _z)Q(u;\phi _u) \end{aligned}$$

(B.16)

where we have also separated the variational parameters $\phi = \{\phi _z, \phi _u\}$ into those that parameterize the belief about hidden states $\phi _z$, and those that parameterize the belief about actions $\phi _u$.

When considering state-estimation (i.e. optimization of $Q(z_t;\phi _z)$), we only have to consider the generative model of the current timestep $P(\tilde{\sigma }_t, z_t; \gamma , \zeta )$. The optimal posterior parameters $\phi ^{*}_z$ are found as the minimum of the variational free energy, re-written using only those terms that depend on $\phi _z$:

$$\begin{aligned} \phi ^{*}_z&= \underset{\phi _z}{\arg \min }\ \,\, \mathcal {F}(\phi _z) \nonumber \\ \mathcal {F}(\phi _z)&= \mathbb {E}_{Q(z_t ; \phi _z)}[\ln Q(z_t;\phi _z) - \ln P(\tilde{\sigma }_t, z_t; \gamma , \zeta )] \end{aligned}$$

(B.17)

To solve this, we also need to decide on a parameterization of the approximate posterior over hidden states $z_t$. We parameterize $Q(z_t;\phi _z)$ as a Bernoulli distribution with parameter $\phi _z$, that can be interpreted as the posterior probability that $z_t$ is in the ‘UP’ ($+1$) state:

$$\begin{aligned} Q(z_t;\phi _z) = (1-\phi _z)^{1 - \frac{z_t + 1}{2}} \phi _z^{ \frac{z_t + 1}{2}} \end{aligned}$$

(B.18)

By minimizing the variational free energy with respect to $\phi _z$, we can obtain an expression for the optimal posterior $Q(z;\phi ^{*}_z)$ that sits at the variational free energy minimum. Due to the exponential and conjugate form of the generative model, $Q(z_t;\phi _z)$ is the exact posterior and thus variational inference reduces to exact Bayesian inference. This means we can simply re-use the posterior update equation of Eq. (A.7) to yield an analytic expression for $\phi ^{*}_z$:

$$\begin{aligned} \phi ^{*}_z = \frac{1}{1 + \exp \left( -2(\zeta + \gamma \varDelta \sigma )\right) } \end{aligned}$$

(B.19)

When considering inference of actions, we now consider the generative model of the future timestep, which crucially depends on the current control state $u_t$ and the optimality variable $\mathcal {O}_{t+1}$. We can then write the variational problem as finding the setting of $\phi _u$ that minimizes the variational free energy, now re-written in terms of its dependence on $\phi _u$:

$$\begin{aligned} \phi ^{*}_u&= \underset{\phi _u}{\arg \min }\ \,\, \mathcal {F}(\phi _u) \nonumber \\ \mathcal {F}(\phi _u)&= \mathbb {E}_{Q(u_t ; \phi _u)}[\ln Q(u_t;\phi _u) - \ln \tilde{P}(\mathcal {O}_{t+1}, u_t, z_t)] \end{aligned}$$

(B.20)

As we did for the posterior over hidden states, we need to decide on a parameterization for the posterior over actions $Q(u_t ; \phi _u)$; we also parameterize this as a Bernoulli distribution with parameter $\phi _u$ that represents the probability of taking the ‘UP’ ($+1$) action:

$$\begin{aligned} Q(u_t;\phi _u) = (1-\phi _z)^{1 - \frac{u_t + 1}{2}} \phi _z^{ \frac{u_t + 1}{2}} \end{aligned}$$

(B.21)

From Eq. (B.20) it follows that the optimal $\phi _u$ is that which minimizes the Kullback-Leibler divergence between the approximate posterior $Q(u_t; \phi _u)$ and the prior $P(u_t)$, which is a softmax function of the expected free energy of actions $\textbf{G}(u_t)$. In this particular generative model, the expected free energy can be written as a single term that scores the ‘expected utility’ of each action [7, 9]:

$$\begin{aligned} \textbf{G}(u_t)&= -\mathbb {E}_{Q(\mathcal {O}_{t+1}|u_t)}[\ln \tilde{P}(\mathcal {O}_{t+1})] \end{aligned}$$

(B.22)

To compute this, we need to compute the ‘variational marginal’ over $\mathcal {O}_{t+1}$, denoted $Q(\mathcal {O}_{t+1}|u_t)$:

$$\begin{aligned} Q(\mathcal {O}_{t+1}|u_t)&= \mathbb {E}_{Q(z_t;\phi ^{*}_z)}[P(\mathcal {O}_{t+1}|z_t, u_t)] \end{aligned}$$

(B.23)

We can simplify the expression for $Q(\mathcal {O}_{t+1}|u_t)$ when we take advantage of the Bernoulli-parameterization of the posterior over hidden states $Q(z;\phi ^{*}_z)$. This allows us to then write the variational marginals, conditioned on different actions as a matrix, with one column for each setting of $u_t$:

$$\begin{aligned} Q(\mathcal {O}_{t+1}|u_t)&= \begin{bmatrix}\phi ^{*}_z &{} 1 - \phi ^{*}_z \\ 1 - \phi ^{*}_z &{} \phi ^{*}_z \end{bmatrix} \end{aligned}$$

(B.24)

The expected utility (and thus the negative expected free energy) is then computed as the dot-product of each column of the matrix expressed in Eq. (B.24) with the log of the prior preferences $\tilde{P}(\mathcal {O}_{t+1})$:

$$\begin{aligned} \mathbb {E}_{Q(\mathcal {O}_{t+1}|u_t)}[\ln \tilde{P}(\mathcal {O}_{t+1})]&= \begin{bmatrix} -\infty \phi ^{*}_z \\ -\infty (1 - \phi ^{*}_z)\end{bmatrix} \nonumber \\ \implies \textbf{G}(u_t)&= \begin{bmatrix} \infty \phi ^{*}_z \\ \infty (1 - \phi ^{*}_z) \end{bmatrix} \end{aligned}$$

(B.25)

Because the probability of an action is proportional to its negative expected free energy, this allows us to write the Bernoulli parameter $\phi ^{*}_u$ of the posterior over actions directly in terms of the parameter of the state posterior

$$\begin{aligned} \phi ^{*}_u&= \frac{1}{1 + \exp (\beta (\infty ( 1- \phi ^{*}_z)))} \nonumber \\&= \frac{1}{1 + C\exp (-\phi ^{*}_z)))} \end{aligned}$$

(B.26)

The inverse temperature parameter $\beta$ is an arbitrary re-scaling factor that can be used to linearize the sigmoid function in (B.26) over the range [0, 1] such that

$$\begin{aligned} \phi ^{*}_u&\approx \phi ^{*}_z \end{aligned}$$

(B.27)

Note that the equivalence relation in Eq. (B.27) is only possible due to the infinite precisions $\omega$ and $\xi$ of the likelihood and prior distributions over the ‘optimality’ variable $P(\mathcal {O}_{t+1}|u_t, z_t)$ and $\tilde{P}(\mathcal {O}_{t+1})$, and from an appropriately re-scaled $\beta$ parameter that linearizes the sigmoid relationship in Eq. (B.26).

1.3 B.3 Action Sampling as Probability Matching

Now that we have an expression for the parameter $\phi ^{*}_u$ of the posterior over control states $Q(u_t; \phi ^{*}_u)$, an agent can generate a spin state by simply sampling from this posterior over actions:

$$\begin{aligned} \sigma&\sim Q(u_t;\phi ^{*}_u) \nonumber \\&\sim Q(z_t;\phi ^{*}_z) \triangleq P(z_t|\tilde{\sigma };\gamma , \zeta ) \end{aligned}$$

(B.28)

In short, each agent samples its spin state from a posterior belief over the state of the latent variable $z_t$, rendering their action-selection a type of probability matching [11, 29, 31], whereby actions (whether to spin ‘UP’ or ‘DOWN’) are proportional to the probability they are assigned in the agent’s posterior belief. Each agent’s sampled spin state also serves as an observation ($\sigma _j$ for some j) for the other agents that the focal agent is a neighbour of. This collective active inference scheme corresponds to a particular form of sampling from the stationary distribution of a spin glass model known as Glauber dynamics [12]. Crucially, however, the temporal scheduling of the action-updating across the group determines which stationary distribution the system samples from. We explore this distinction in the next section.

C Temporal Scheduling of Action Sampling

In this appendix we examine how the stationary distribution from which the collective active inference system samples depends on the order in which actions are updated across all agents in the network. First, we consider the case of synchronous action updates (all agents update their actions in parallel and only observe the- spin states of their neighbours from the last timestep), and show how this system samples from a different stationary distribution than the one defined by the standard Ising energy provided in Eq. (7). We then derive the more ‘classical’ case of asynchronous updates, where agents update their spins one at a time, and show how in this case the system converges to the standard statioanry distribution of the Ising model. This Appendix thus explains one of the ‘fragilities’ mentioned in the main text, that threaten the unique equivalence between local active inference dynamics and a unique interpretation at the global level in terms of inference.

We denote some agent’s spin using $\sigma _i$ and its set of neighbours as $M_i$. The local sum of spins or spin difference $\sum _{j \in M} \sigma _j$ for agent i we denote $\varDelta _i \sigma = \sum _{j \in M_i} \sigma _j$.

1.1 C.1 Synchronous Updates

To derive the stationary distribution in case of synchronous updates, we can take advantage of the following detailed balance relation, which obtains in the case of systems at thermodynamic equilibrium:

$$\begin{aligned} \frac{P(\tilde{\sigma })}{P(\tilde{\sigma }')}&= \frac{P(\tilde{\sigma }|\tilde{\sigma }')}{P(\tilde{\sigma }'|\tilde{\sigma })} \nonumber \\ \implies P(\tilde{\sigma })&= \frac{P(\tilde{\sigma }|\tilde{\sigma }')P(\tilde{\sigma }')}{P(\tilde{\sigma }'|\tilde{\sigma })} \end{aligned}$$

(C.29)

where $\tilde{\sigma }$ and $\tilde{\sigma }'$ are spin configurations at two adjacent times $\tau$ and $\tau +1$. In the case of synchronous updates (all spins are sampled simultaneously, given the spins at the last timestep), then the spin action of each agent $\sigma _i'$ at time $\tau +1$ is conditionally independent of all other spins, given the vector of spins $\tilde{\sigma }$ at the previous timestep $\tau$. We can therefore expand the ‘forward’ transition distribution $P(\tilde{\sigma }'|\tilde{\sigma })$ as a product over the action posteriors of each agent:

$$\begin{aligned} P(\tilde{\sigma }'|\tilde{\sigma })&= P(\sigma _1|\tilde{\sigma })P(\sigma _1|\tilde{\sigma })...P(\sigma _N|\tilde{\sigma }) \nonumber \\&= \prod _i Q(u_t; \phi ^{*}_{u, i}) \nonumber \\&= \prod _{i} \frac{\exp \left( \sigma '_i \left( \zeta + \gamma \sum _{j \in M_i} \sigma _j \right) \right) }{2\cosh \left( \zeta + \gamma \sum _{j \in M_i} \sigma _j \right) } \nonumber \\&= \exp \left( \sum _i \sigma '_i (\zeta + \gamma \sum _{j \in M_i} \sigma _j ) - \sum _i\log \left( 2 \cosh (\zeta + \gamma \sum _{j \in M_i} \sigma _j )\right) \right) \end{aligned}$$

(C.30)

Note we have replaced each latent variable in the posterior z with the agent’s own spin state $\sigma _i$, because there is a one-to-one mapping between the posterior over $z_t$ and the posterior over actions $\sigma _i$.

The reverse transition distribution, yielding the probability of transitioning from configuration $\tilde{\sigma }' \rightarrow \tilde{\sigma }$ is the same expression as for the forward transition, except that $\sigma _i'$ and $\sigma _i$ are swapped:

$$\begin{aligned} P(\tilde{\sigma }|\tilde{\sigma }')&= \exp \left( \sum _i \sigma _i (\zeta + \gamma \sum _{j \in M_i} \sigma '_j ) - \sum _i\log \left( 2 \cosh (\zeta + \gamma \sum _{j \in M_i} \sigma '_j)\right) \right) \end{aligned}$$

(C.31)

The detailed balance equation in (C.29) then tells us that the stationary probability distribution over $\tilde{\sigma }$ is proportional to the ratio of the backwards transition to the forwards transition:

$$\begin{aligned} \frac{P(\tilde{\sigma })}{P(\tilde{\sigma '})}&= \frac{\exp \left( \sum _i \sigma _i (\zeta + \gamma \sum _{j \in M_i} \sigma '_j ) - \sum _i\log \left( 2 \cosh (\zeta + \gamma \sum _{j \in M_i} \sigma '_j)\right) \right) }{\exp \left( \sum _i \sigma '_i (\zeta + \gamma \sum _{j \in M_i} \sigma _j ) - \sum _i\log \left( 2 \cosh (\zeta + \gamma \sum _{j \in M_i} \sigma _j )\right) \right) } \nonumber \\&= \frac{\exp \left( \zeta \sum _i \sigma _i + \gamma \sum _{\langle i, j\rangle } \sigma _i \sigma _j'\right) \exp \left( -\sum _i \log \left( 2 \cosh (\zeta + \gamma \sum _{j \in M_i} \sigma _j'\right) \right) }{\exp \left( \zeta \sum _i \sigma _i' + \gamma \sum _{\langle i, j\rangle } \sigma _i' \sigma _j\right) \exp \left( -\sum _i \log \left( 2 \cosh (\zeta + \gamma \sum _{j \in M_i} \sigma _j\right) \right) } \nonumber \\&= \frac{\exp \left( \zeta \sum _i \sigma _i +\sum _i \log \left( 2 \cosh (\zeta + \gamma \sum _{j \in M_i} \sigma _j\right) \right) }{\exp \left( \zeta \sum _i \sigma _i' + \sum _i \log \left( 2 \cosh (\zeta + \gamma \sum _{j \in M_i} \sigma _j'\right) \right) } \end{aligned}$$

(C.32)

Therefore, we can write down the stationary distribution in the case of synchronous updates as an exponential term normalized by a partition function:

$$\begin{aligned} P(\tilde{\sigma })&= Z^{-1}\exp \left( \zeta \sum _i \sigma _i + \sum _i \log \left( 2 \cosh (\zeta + \gamma \sum _{j \in M_i} \sigma _j)\right) \right) \nonumber \\ Z&= \sum _{\tilde{\sigma }} \exp \left( \zeta \sum _i \sigma _i + \sum _i \log \left( 2 \cosh (\zeta + \gamma \sum _{j \in M_i} \sigma _j)\right) \right) \end{aligned}$$

(C.33)

Note that the action update for an individual agent can still be written in terms of the local energy difference $\varDelta _i E$, where the energy is defined using the standard Hamiltonian function given by Eq. (7) in the main text. However, due to the temporal sampling of each agent’s action with respect to the others, the system collectively sample from a system with a different energy function and Gibbs measure, given by Eq. (C.33). This energy function is therefore nonlinear and can be written:

$$\begin{aligned} E_{sync}(\tilde{\sigma })&= -\zeta \sum _i \sigma - \sum _i \log (2\cosh (\zeta + \gamma \sum _{j \in M_i} \sigma _j)) \end{aligned}$$

(C.34)

1.2 C.2 Asynchronous Updates

Now we treat the case where agents update their agents one-by-one or asynchronously. This means that at each timestep only one agent is updated, and that particular agent uses the spin states of all the other agents at the last timestep as inputs for its posterior inference.

We can write down the forward transition as follows, using the notation $\sigma _{\setminus i}$ to denote all the spins except for $\sigma _i$:

$$\begin{aligned} p(\sigma _i',\tilde{\sigma }_{\setminus i}| \tilde{\sigma })&= \frac{\exp (\sigma _i'(\zeta + \gamma \sum _{j \in M_i} \sigma _j))}{2 \cosh (\zeta + \gamma \sum _{j \in M_i} \sigma _j)} \end{aligned}$$

(C.35)

which indicates that only agent i is updated at the current timestep. The detailed balance condition implies that

$$\begin{aligned} p(\sigma _i',\tilde{\sigma }_{\setminus i}| \tilde{\sigma }) p(\tilde{\sigma }) = p(\tilde{\sigma }|\sigma _i',\tilde{\sigma }_{\setminus i}) p(\sigma _i',\tilde{\sigma }_{\setminus i}) \end{aligned}$$

(C.36)

Then

$$\begin{aligned} \frac{p(\tilde{\sigma })}{p(\sigma _i',\tilde{\sigma }_{\setminus i})} =&\frac{p(\tilde{\sigma }|\sigma _i',\tilde{\sigma }_{\setminus i})}{p(\sigma _i',\tilde{\sigma }_{\setminus i}| \tilde{\sigma })} = \frac{\exp (\sigma _i(\zeta + \gamma \sum _{j \in M_i} \sigma _j) - \log (2\cosh (\zeta + \gamma \sum _{j \in M_i} \sigma _j)))}{\exp (\sigma _i'(\zeta + \gamma \sum _{j\in M_i} \sigma _j) - \log (2\cosh (\zeta + \gamma \sum _{j\in M_i}\sigma _j)))} \end{aligned}$$

(C.37)

$$\begin{aligned} =&\frac{\exp (\sigma _i(\zeta + \gamma \sum _{j\in M_i} \sigma _j))}{\exp (\sigma _i'(\zeta + \gamma \sum _{j\in M_ii} \sigma _j ))} \end{aligned}$$

(C.38)

By repeating this operation for every agent (i.e. $N-1$ more times), then we arrive at:

$$\begin{aligned} \frac{ p(\tilde{\sigma })}{p(\tilde{\sigma }')} = \frac{ p(\tilde{\sigma })}{p(\sigma _i',\tilde{\sigma }_{\setminus i})} \frac{p(\sigma _i',\tilde{\sigma }_{\setminus i})}{p(\sigma _i',\sigma _j',\tilde{\sigma }_{\setminus i,j})} \ldots \frac{p(\tilde{\sigma }'_{\setminus i},\sigma _i)}{p(\tilde{\sigma }')} =&\frac{\exp (\zeta \sum _i \sigma _i + \gamma \sum _{i<j} \sigma _i \sigma _j )}{\exp (\zeta \sum _i \sigma '_i + \gamma \sum _{i<j}\sigma '_i \sigma '_j)} \end{aligned}$$

(C.39)

We can therefore write the marginal distributions $p(\tilde{\sigma })$ as proportional to the numerator of the last term in Eq. (C.39)⁵:

$$\begin{aligned} p(\tilde{\sigma })&\propto \exp (\zeta \sum _i \sigma _i + \gamma \sum _{\langle i, j\rangle } \sigma _i \sigma _j) \nonumber \\ \implies p(x) =&Z^{-1} \exp (\zeta \sum _i \sigma _i + \gamma \sum _{\langle i, j\rangle } \sigma _i \sigma _j) \end{aligned}$$

(C.40)

We thus recover the original stationary distribution with the standard, linear energy function as given by Eq. (7) in the main text, written now in terms of generative model parameters $\gamma , \zeta$ instead of the standard ‘couplings’ and ‘biases’ J, h:

$$\begin{aligned} E_{async}(\tilde{\sigma })&= -\gamma \sum _{\langle i, j \rangle }\sigma _i \sigma _j - \zeta \sum _i \sigma _i \end{aligned}$$

(C.41)