Basics of Neural Networks

Abstract

In this chapter, we derive neural networks from the viewpoint of physical models. A neural network is a nonlinear function that maps an input to an output, and giving the network is equivalent to giving a function called an error function in the case of supervised learning. By considering the output as dynamical degrees of freedom and the input as an external field, various neural networks and their deepened versions are born from simple Hamiltonians. Training (learning) is a procedure for reducing the value of the error function, and we will learn the specific method of backpropagation using the bra-ket notation popular in quantum mechanics. And we will look at how the “universal approximation theorem” works, which is why neural networks can express connections between various types of data.

Appendices

Column: Statistical Mechanics and Quantum Mechanics

Canonical Distribution in Statistical Mechanics

The statistical mechanics used in this book is called canonical distribution for a given temperature. The physical situation is as follows. Suppose we have some physical degrees of freedom, such as a spin or a particle position. Denote it as d. Since this is a physical degree of freedom, it should have energy which depends on its value. Let it be H(d). When this system is immersed in an environment with a temperature of T (called a heat-bath), the probability of achieving d with the energy H(d) is known to be

$$\displaystyle \begin{aligned} P(d) = \frac{e^{- \frac{H(d)}{k_B T}}}{Z} \, . \end{aligned} $$

(3.92)

Here, k _B is an important constant called the Boltzmann constant , and Z is called the partition function , defined as

$$\displaystyle \begin{aligned} Z = \sum_{d} e^{- \frac{H(d)}{k_B T}} \, . \end{aligned} $$

(3.93)

In the main text, we put k _B T = 1.

Simple example: law of equipartition of energy

As an example, consider d as the position x and momentum p of a particle in a box of length L, with the standard kinetic energy function as its energy. The expectation value of energy is

$$\displaystyle \begin{aligned} \langle E \rangle = \int_0^L dx \int_{-\infty}^{+\infty} dp \ \frac{p^2}{2m} \frac{e^{ - \frac{p^2}{2 m k_B T } } }{Z} \, . \end{aligned} $$

(3.94)

Here

$$\displaystyle \begin{aligned} Z = \int_0^L dx \int_{-\infty}^{+\infty} dp \ e^{ - \frac{p^2}{2 m k T } } = L (2 \pi m k_B T)^{1/2} \, . \end{aligned} $$

(3.95)

The calculation of 〈E〉 looks a bit difficult, but using $\beta = \frac {1}{k_B T}$ we find

$$\displaystyle \begin{aligned} &Z = L \Big( \frac{2\pi m}{\beta} \Big)^{1/2}, \end{aligned} $$

(3.96)

$$\displaystyle \begin{aligned} &\langle E \rangle = - \frac{\partial}{\partial \beta} \log Z = \frac{1}{2} \frac{1}{\beta} = \frac{1}{2} k_BT \, . \end{aligned} $$

(3.97)

In other words, when the system temperature T is high (i.e. when the system is hot), the expectation value of energy is high, and when the temperature is low, the expectation value of energy is low. This is consistent with our intuition. In addition, if we consider the case of three spatial dimensions, we can obtain the famous formula $\frac {3}{2}k_BT$ as the expectation value of energy.

Bracket Notation in Quantum Mechanics

In the derivation of the backpropagation method, bracket notation has been introduced as a simple method. This is nothing more than just writing a vector as a ket ,

$$\displaystyle \begin{aligned} \mathbf{v} = | v \rangle \, . \end{aligned} $$

(3.98)

There are some notational conveniences. For example, the inner bracket is written conventionally as

$$\displaystyle \begin{aligned} \mathbf{w} \cdot \mathbf{v} = \langle w | v \rangle, \end{aligned} $$

(3.99)

which means that the inner product is regarded as the “product of the matrices,”

$$\displaystyle \begin{aligned} \mathbf{w} \cdot \mathbf{v} = \begin{pmatrix} w_1, & w_2, &\cdots \end{pmatrix} \begin{pmatrix} v_1 \\ v_2 \\ \vdots \end{pmatrix} \, . \end{aligned} $$

(3.100)

Also in the main text we have had

$$\displaystyle \begin{aligned} |v \rangle \langle w | \end{aligned} $$

(3.101)

and also this can be regarded as a “matrix,” made by a matrix product,

$$\displaystyle \begin{aligned} |v \rangle \langle w | &= \begin{pmatrix} v_1 \\ v_2 \\ \vdots \end{pmatrix} \begin{pmatrix} w_1, & w_2, &\cdots \end{pmatrix} = \begin{pmatrix} v_1 w_1& v_1 w_2&\cdots \\ v_2 w_1& v_2 w_2&\cdots \\ \dots \end{pmatrix} \, . \end{aligned} $$

(3.102)