Thermodynamics of Restricted Boltzmann Machines and Related Learning Dynamics

Abstract

We investigate the thermodynamic properties of a restricted Boltzmann machine (RBM), a simple energy-based generative model used in the context of unsupervised learning. Assuming the information content of this model to be mainly reflected by the spectral properties of its weight matrix W, we try to make a realistic analysis by averaging over an appropriate statistical ensemble of RBMs. First, a phase diagram is derived. Otherwise similar to that of the Sherrington–Kirkpatrick (SK) model with ferromagnetic couplings, the RBM’s phase diagram presents a ferromagnetic phase which may or may not be of compositional type depending on the kurtosis of the distribution of the components of the singular vectors of W. Subsequently, the learning dynamics of the RBM is studied in the thermodynamic limit. A “typical” learning trajectory is shown to solve an effective dynamical equation, based on the aforementioned ensemble average and explicitly involving order parameters obtained from the thermodynamic analysis. In particular, this let us show how the evolution of the dominant singular values of W, and thus of the unstable modes, is driven by the input data. At the beginning of the training, in which the RBM is found to operate in the linear regime, the unstable modes reflect the dominant covariance modes of the data. In the non-linear regime, instead, the selected modes interact and eventually impose a matching of the order parameters to their empirical counterparts estimated from the data. Finally, we illustrate our considerations by performing experiments on both artificial and real data, showing in particular how the RBM operates in the ferromagnetic compositional phase.

Appendices

Appendix A: AT Line

The stability of the RS solution to the mean-field equations is studied along the lines of [33] by looking at the Hessian of the replicated version of the free energy and identifying eigenmodes from symmetry arguments. Before taking the limit $p\rightarrow 0$ the free energy reads

$$\begin{aligned} f[m,\bar{m},Q,\bar{Q}] = \sum _{a,\alpha }w_\alpha m_\alpha ^a\bar{m}_\alpha ^a +\frac{\sigma ^2}{2}\sum _{a\ne b} Q_{ab}\bar{Q}_{ab} -\frac{1}{\sqrt{\kappa }} A_p[m,Q]-\sqrt{\kappa }B_p[\bar{m},\bar{Q}], \end{aligned}$$

with $A_p$ and $B_p$ given in (10,11). Assuming the small perturbations

$$\begin{aligned} m_\alpha ^a = m_\alpha +\epsilon _\alpha ^a\qquad \qquad \bar{m}_\alpha ^a = \bar{m}_\alpha +\bar{\epsilon }_\alpha ^a\\ Q_{ab} = q +\eta _{ab}\qquad \qquad \bar{Q}_{ab} = \bar{q} + \bar{\eta }_{ab}, \end{aligned}$$

around the saddle point $(m_\alpha ,\bar{m}_\alpha ,q,\bar{q})$, the perturbed free energy reads

$$\begin{aligned} \Delta f =&\sum _{a,\alpha }w_\alpha \bar{\epsilon }_\alpha ^a\epsilon _\alpha ^a+\frac{\sigma ^2}{2}\sum _{a\ne b}\bar{\eta }_{ab}\eta _{ab} +\sum _{a,b,\alpha ,\beta }\bigl [\bigl (\delta _{ab}\bar{A}_{\alpha \beta }+\bar{\delta }_{ab}\bar{B}_{\alpha \beta }\bigr )\epsilon _\alpha ^a\epsilon _\beta ^b +CT\bigr ]\\&+\sum _{a\ne b,c,\alpha }\bigl [\bigl ((\delta _{ab}+\delta _{ac})\bar{C}_{\alpha }+(1-\delta _{ac}-\delta _{bc})\bar{D}_{\alpha }\bigr )\epsilon _\alpha ^c\eta _{ab} +CT\bigr ]\\&+\sum _{a\ne b,c\ne d}\bigl [\bigl (\delta _{(ab)(cd)}\bar{E}_0+\mathbb {1}_{\{a\in (cd)\oplus b\in (cd)\}}\bar{E}_1+\mathbb {1}_{\{(ab)\cap (cd)=\emptyset \}}\bar{E}_2\bigr )\eta _{ab}\eta _{cd} +CT\bigr ], \end{aligned}$$

where CT means “conjugate term” in the sense $\epsilon \leftrightarrow \bar{\epsilon }$, $A_{\alpha \beta } \leftrightarrow \bar{A}_{\alpha \beta }$..., where $\bar{\delta }_{ab} {\mathop {=}\limits ^{\text{ def }}}1-\delta _{ab}$ and the operators are given by

$$\begin{aligned} A_{\alpha \beta }&{\mathop {=}\limits ^{\text{ def }}}(\delta _{\alpha \beta }-m_\alpha m_\beta )w_\alpha w_\beta \qquad \qquad B_{\alpha \beta } {\mathop {=}\limits ^{\text{ def }}}\Bigl (\mathsf {E}_{x,v}\bigl (v^\alpha v^\beta \tanh ^2(\bar{h}(x,v))\bigr )-m_\alpha m_\beta \Bigr )w_\alpha w_\beta \\ C_\alpha&{\mathop {=}\limits ^{\text{ def }}}\frac{\kappa ^{1/4}\sigma ^2}{2}m_\alpha (1-q)w_\alpha \qquad \qquad D_\alpha {\mathop {=}\limits ^{\text{ def }}}\frac{\kappa ^{1/4}\sigma ^2}{2} \Bigl (\mathsf {E}_{x,v}\bigl (v^\alpha \tanh ^3(\bar{h}(x,v))\bigr )-m_\alpha q\Bigr )w_\alpha \\ E_0&{\mathop {=}\limits ^{\text{ def }}}\frac{\sqrt{\kappa }\sigma ^4}{4}(1-q^2)\qquad E_1 {\mathop {=}\limits ^{\text{ def }}}\frac{\sqrt{\kappa }\sigma ^4}{4}q(1-q)\qquad E_2 {\mathop {=}\limits ^{\text{ def }}}\frac{\sqrt{\kappa }\sigma ^4}{4}\Bigl (\mathsf {E}_{x,v}\bigl (\tanh ^4(\bar{h}(x,v))\bigr )-q^2\Bigr ) \end{aligned}$$

with

$$\begin{aligned} h(x,u) {\mathop {=}\limits ^{\text{ def }}}\kappa ^{1/4}\left( \sqrt{q}\sigma x + \sum _\alpha (m_\alpha w_\alpha - \eta _\alpha )u^\alpha \right) , \end{aligned}$$

Conjugate quantities are obtained by replacing $m_\alpha$ by $\bar{m}_\alpha$, q by $\bar{q}$, $u^\alpha$ by $v^\alpha$, $\eta _\alpha$ by $\theta _\alpha$ and $\kappa$ by $1/\kappa$. As for the SK model, the $2Kp\times 2Kp$ Hessian thereby defined can be diagonalized with the help of three similar sets of eigenmodes corresponding to different permutation symmetries in replica space.

The first set corresponds to $2K+2$ replica symmetric modes defined by $\eta _\alpha ^a = \eta _\alpha$ and $\eta _{ab} = \eta$ solving the linear system

$$\begin{aligned}&\left( \frac{w_\alpha }{2}-\lambda \right) \bar{\epsilon }_\alpha -\frac{1}{2}\bar{A}_{\alpha \alpha }\epsilon _\alpha +\sum _\beta \bigl (\bar{A}_{\alpha \beta } +(p-1)\bar{B}_{\alpha \beta }\bigr )\epsilon _\beta \\&\quad +\left( (p-1)\bar{C}_\alpha +\frac{(p-1)(p-2)}{2}\bar{D}_\alpha \right) \eta = 0\\&\left( \frac{w_\alpha }{2}-\lambda \right) \epsilon _\alpha -\frac{1}{2}A_{\alpha \alpha }\bar{\epsilon }_\alpha +\sum _\beta \bigl (A_{\alpha \beta } +(p-1)B_{\alpha \beta }\bigr )\bar{\epsilon }_\beta \\&\quad +\left( (p-1)C_\alpha +\frac{(p-1)(p-2)}{2}D_\alpha \right) \bar{\eta }= 0\\&\left( \frac{\sigma ^2}{2}-\lambda \right) \bar{\eta }+\sum _\alpha \left( \bar{C}_\alpha +\frac{p-2}{2}\bar{D}_\alpha \right) \epsilon _\alpha +2\left( \bar{E}_0+2(p-2)\bar{E}_1+\frac{(p-2)(p-3)}{2}\bar{E}_2\right) \eta = 0\\&\left( \frac{\sigma ^2}{2}-\lambda \right) \eta +\sum _\alpha \left( C_\alpha +\frac{p-2}{2}D_\alpha \right) \bar{\epsilon }_\alpha +2\left( E_0+2(p-2)E_1+\frac{(p-2)(p-3)}{2}E_2\right) \bar{\eta }= 0 \end{aligned}$$

with eigenvalue $\lambda$ solving a polynomial equation of degree $2K+2$ corresponding to a vanishing determinant in the above system.

The second set corresponds to a broken replica symmetry where one replica $a_0$ is different from the others

$$\begin{aligned} (\epsilon _\alpha ^a,\bar{\epsilon }_\alpha ^a)= & {} {\left\{ \begin{array}{ll} (\epsilon _\alpha ,\bar{\epsilon }_\alpha )\qquad \text {for}\ a\ne a_0\\ (1-p)(\epsilon _\alpha ,\bar{\epsilon }_\alpha )\qquad \text {for}\ a=a_0 \end{array}\right. } \\ (\eta _{ab},\bar{\eta }_{ab})= & {} {\left\{ \begin{array}{ll} (\eta ,\bar{\eta })\qquad \text {for}\ a,b\ne a_0\\ (1-\frac{p}{2})(\eta ,\bar{\eta })\qquad \text {for}\ a=a_0\ or\ b=a_0 \end{array}\right. } \end{aligned}$$

This set has dimension $(2K+2)(p-1)$. Its parameterization is obtained by imposing orthogonality with the previous one. The corresponding system reads

$$\begin{aligned}&\left( \frac{w_\alpha }{2}-\lambda \right) \bar{\epsilon }_\alpha -\frac{1}{2}\bar{A}_{\alpha \alpha }\epsilon _\alpha +\sum _\beta (\bar{A}_{\alpha \beta }-\bar{B}_{\alpha \beta })\epsilon _\beta +\frac{p-2}{2}\bigl (\bar{C}_\alpha -\bar{D}_\alpha \bigr )\eta = 0\\&\left( \frac{w_\alpha }{2}-\lambda \right) \epsilon _\alpha -\frac{1}{2}A_{\alpha \alpha }\bar{\epsilon }_\alpha +\sum _\beta (A_{\alpha \beta }-B_{\alpha \beta })\bar{\epsilon }_\beta +\frac{p-2}{2}\bigl (C_\alpha -D_\alpha \bigr )\bar{\eta }= 0\\&\left( \frac{\sigma ^2}{2}-\lambda \right) \bar{\eta }+\sum _\alpha (\bar{C}_\alpha -\bar{D}_\alpha )\epsilon _\alpha +2\bigl (\bar{E}_0+(p-4)\bar{E}_1-(p-3)\bar{E}_2\bigr )\eta = 0\\&\left( \frac{\sigma ^2}{2}-\lambda \right) \eta +\sum _\alpha (C_\alpha -D_\alpha )\bar{\epsilon }_\alpha +2\bigl (E_0+(p-4)E_1-(p-3)E_2\bigr )\bar{\eta }= 0 \end{aligned}$$

Finally the eigenmodes of the Hessian are made complete by considering a broken symmetry where two replicas $a_0$ and $a_1$ are different from the others, with the following parameterization dictated again by orthogonality constraints with the previous sets:

$$\begin{aligned} (\epsilon _\alpha ^a,\bar{\epsilon }_\alpha ^a) = 0, \qquad (\eta _{ab},\bar{\eta }_{ab}) = {\left\{ \begin{array}{ll} (\eta ,\bar{\eta })\qquad \text {for}\ a,b\ne a_0\\ \frac{3-p}{2}(\eta ,\bar{\eta })\qquad \text {for}\ a\in {a_0,a_1}\ or\ b\in {a_0,a_1}\\ \frac{(p-2)(p-3)}{2}(\eta ,\bar{\eta })\qquad \text {for}\ (a,b)=(a_0,a_1). \end{array}\right. } \end{aligned}$$

The dimension of this set is now $p(p-3)$, and it represents eigenvectors iff the following system of equations is satisfied

$$\begin{aligned}&\left( \frac{\sigma ^2}{2}-\lambda \right) \bar{\eta }+2(\bar{E}_0-2\bar{E}_1+\bar{E}_2)\eta = 0\\&\left( \frac{\sigma ^2}{2}-\lambda \right) \eta +2(E_0-2E_1+E_2)\bar{\eta }= 0 \end{aligned}$$

The corresponding eigenvalues read

$$\begin{aligned} \lambda = \frac{\sigma ^2}{2}\pm 2\sqrt{(\bar{E}_0-2\bar{E}_1+\bar{E}_2)(E_0-2E_1+E_2)}, \end{aligned}$$

with degeneracy $p(p-3)/2$. Finally the RS stability condition reads

$$\begin{aligned} \frac{1}{\sigma ^2} > \sqrt{\mathsf {E}_{x,u}\Bigl ({\text {sech}}^4\bigl (h(x,u)\bigr )\Bigr )\mathsf {E}_{x,v}\Bigl ({\text {sech}}^4\bigl (\bar{h}(x,v)\bigr )\Bigr )}, \end{aligned}$$

which reduces to the same form of the AT line for the SK model when $\kappa =1$, except for the u and v averages that are specific to our model. As seen in Fig. 2 the influence of $\kappa$ is very limited.

Appendix B: Synthetic Dataset

The multimodal distribution modeling the N-dimensional synthetic data is

$$\begin{aligned} P(s) = \sum _{c=1}^C p_c\prod _{i=1}^N \frac{e^{h_i^c s_i}}{2\cosh (h_i^c)}, \end{aligned}$$

(45)

where $C$ is the number of clusters, $p_c$ is a weight and ${\varvec{h}}^c$ is a hidden field for cluster $c$. The values for $p_c$ are taken at random and normalized, while to compute $h_i^c$ we take into account the magnetizations $m_i^c = \tanh (h_i^c)$. Expanding over the spectral modes, we can set an effective dimension $d$ by constraining the sum to the range $\alpha = 1, \dots , d$

$$\begin{aligned} m_i^c = \sum _{\alpha = 1}^d m_{\alpha }^c u_i^\alpha \end{aligned}$$

(46)

Clusters’ magnetizations $m_{\alpha }^c$ are drawn at random between $[-1, 1]$ and normalized with the factor

$$\begin{aligned} Z = \sqrt{\frac{\sum _{\alpha } m_{\alpha }^2}{d \cdot r}}, \quad r = \tanh (\eta ) \end{aligned}$$

(47)

where $r$ is introduced to decrease the clusters’ polarizations (in our simulations, we used $\eta = 0.3$). The spectral basis $u_i^\alpha$ is obtained by drawing at random $d$ N-dimensional vectors and applying the Gram-Schmidt process (which can be safely employed as N is supposedly big and thus the initial vectors are nearly orthogonal). The hidden fields are then obtained from the magnetizations

$$\begin{aligned} h_i^c = \tanh ^{-1}(m_i^c) \end{aligned}$$

(48)

and the samples are generated by choosing a cluster according to $p_c$ and setting the visible variables to $\pm 1$ according to

$$\begin{aligned} p(s_i = 1) = \frac{1}{1 + e^{-2 h_i^c}} \end{aligned}$$

(49)