Bio-inspired machine learning: programmed death and replication

Abstract

We analyze algorithmic and computational aspects of biological phenomena, such as replication and programmed death, in the context of machine learning. We use two different measures of neuron efficiency to develop machine learning algorithms for adding neurons to the system (i.e., replication algorithm) and removing neurons from the system (i.e., programmed death algorithm). We argue that the programmed death algorithm can be used for compression of neural networks and the replication algorithm can be used for improving performance of the already trained neural networks. We also show that a combined algorithm of programmed death and replication can improve the learning efficiency of arbitrary machine learning systems. The computational advantages of the bio-inspired algorithms are demonstrated by training feedforward neural networks on the MNIST dataset of handwritten images.

A Appendix

Here, we list the pruning algorithms customized for the feedforward neural network with n neurons $\{x_{1}(t),...x_{n} (t)\}=\{x_{1},...x_{n}\}$ in an hidden layer t such that

$$\begin{aligned} x_{i}=f\left(\sum _kw_{ik}^{t-1}x_{k}(t-1)+b_{i}^{t-1}\right),\quad x_{j}(t+1)=f\left(\sum _kw_{jk}^{t} x_{k}+b_{j}^{t}\right). \end{aligned}$$

(70)

1.1 A.1 Connection cut algorithm

• 1. Measure variances of neurons on level t and level $t+1$

  $$\begin{aligned} C^t_{kk}=\langle \Delta x_k(t)^2\rangle ,\quad C^{t+1}_{ii}=\langle \Delta x_i(t+1)^2\rangle . \end{aligned}$$

  (71)

• 2. Find the neuron l with minimal efficiency (28)

  $$\begin{aligned} E_l=\min _k E_k =\min _k C^t_{kk} \sum _i \frac{(w_{ik}^t)^2}{C_{ii}^{t+1}}f'_i (\sum _jw_{ij}^{t} \langle x_{j}\rangle +b_{i}^{t} )^2. \end{aligned}$$

  (72)

• 3. Use $x_l=\langle x_l\rangle$ as linear dependence equation (39) with

  $$\begin{aligned} a_{k\ne l}=0,\quad a_l=1,\quad a_0=\langle x_l\rangle . \end{aligned}$$

  (73)

• 4. Remove neuron l from the net according to (41)

  $$\begin{aligned} \sum _kw_{jk}^{t}x_{k}+b_{j}^{t}&\simeq \sum _{k\ne l} w_{jk}^{t}x_{k}+{\tilde{b}}_{j}^{t},\quad {\tilde{b}}_{j}^{t}=w_{jl}^{t} \langle x_l\rangle +b_{j}^{t}. \end{aligned}$$

  (74)

• 5. Do so while there are neurons with efficiency less than a cutoff or while accuracy or loss stays acceptable.

1.2 A.2 Probability algorithm

• 1. Measure variances of neurons on level t and level $t+1$

  $$\begin{aligned} C^t_{kk}=\langle \Delta x_k(t)^2\rangle ,\quad C^{t+1}_{ii}=\langle \Delta x_i(t+1)^2\rangle . \end{aligned}$$

  (75)

• 2. Find the neuron l with minimal efficiency (28)

  $$\begin{aligned} E_l=\min _k E_k =\min _k C^t_{kk} \sum _i \frac{(w_{ik}^t)^2}{C_{ii}^{t+1}}f' \left(\sum _jw_{ij}^{t} \langle x_{j}\rangle +b_{i}^{t} \right)^2. \end{aligned}$$

  (76)

• 3. Use linear dependence equation (39) $\sum _j a_j x_j=a_0$ with

  $$\begin{aligned} a_j=\sum _i \frac{w_{il}^tw_{ij}^t}{C_{ii}^{t+1}}f' \left(\sum _kw_{ik}^{t} \langle x_{k}\rangle +b_{i}^{t} \right)^2,\quad a_0=\sum _ja_j\langle x_j\rangle . \end{aligned}$$

  (77)

• 4. Remove neuron l from the net according to (41)

  $$\begin{aligned} \sum _kw_{jk}^{t}x_{k}+b_{j}^{t}&\simeq \sum _{k\ne l} {\tilde{w}}_{jk}^{t}x_{k}+{\tilde{b}}_{j}^{t}, \end{aligned}$$

  (78)

  where

  $$\begin{aligned} {\tilde{w}}_{jk}^{t}=w_{jk}^{t}-w_{jl}^{t}\frac{a_{k}}{a_{l}},\quad \tilde{b}_{j}^{t}=w_{jl}^{t}\frac{ a_0}{a_{l}}+b_{j}^{t}. \end{aligned}$$

  (79)

• 5. Do so while there are neurons with efficiency less than a cutoff or while accuracy or loss stays acceptable.

1.3 A.3 Covariance algorithm

• 1. Measure covariance matrix $C^t_{kj}=\langle \Delta x_k\Delta x_j\rangle$ (10) of the neurons on hidden layer t and find its eigenvectors $\textbf{v}$ and eigenvalues $\lambda$ (11).

• 2. Find the neuron l and the eigenvalue $\lambda _p$ with minimal efficiency (17)

  $$\begin{aligned} E'_l = \min _k E'_k = \min _{i,k} \frac{ \lambda _i }{\left( \textbf{v}^{(i)}_k\right) ^2}=\frac{ \lambda _p }{\left( \textbf{v}^{(p)}_l\right) ^2}. \end{aligned}$$

  (80)

• 3. Use $\sum _{k}{} \textbf{v}^{(p)}_k x_{k}=\lambda _{p}$ as linear dependence equation (39) with

  $$\begin{aligned} a_k=\textbf{v}^{(p)}_k,\quad a_0=\lambda _{p}. \end{aligned}$$

  (81)

• 4. Remove neuron l from the net according to (41)

  $$\begin{aligned} \sum _kw_{jk}^{t}x_{k}+b_{j}^{t}&\simeq \sum _{k\ne l} {\tilde{w}}_{jk}^{t}x_{k}+{\tilde{b}}_{j}^{t}, \end{aligned}$$

  (82)

  where

  $$\begin{aligned} {\tilde{w}}_{jk}^{t}=w_{jk}^{t}-w_{jl}^{t}\frac{a_{k}}{a_{l}},\quad \tilde{b}_{j}^{t}=w_{jl}^{t}\frac{ a_0}{a_{l}}+b_{j}^{t}. \end{aligned}$$

  (83)

• 5. Do so while there are neurons with efficiency less than a cutoff or while accuracy or loss stays acceptable.