Solution Structure of Multi-layer Neural Networks with Initial Condition

Abstract

This paper studies the initial value problem of multi-layer cellular neural networks. We demonstrate that the mosaic solutions of such system is topologically conjugated to a new class in symbolic dynamical systems called the path set (Abram and Lagarias in Adv Appl Math 56:109–134, 2014). The topological entropies of the solution, output, and hidden spaces of a multi-layer cellular neural network with initial condition are formulated explicitly. Also, a sufficient condition for whether the mosaic solution space of a multi-layer cellular neural network is independent of initial conditions is addressed. Furthermore, two spaces exhibit identical topological entropy if and only if they are finitely equivalent.

Appendix: Equivalent Relation for Templates

The essential study of (1) is investigating two-layer cellular neural networks with the nearest neighborhood; namely, $n = 2$ and $\mathcal {N} = \{-1, 0, 1\}$. For the clarification of the investigation, we assume that $a_{-1}^{(1)} = a_{-1}^{(2)} = 0$. Under such condition, (1) is represented as

$$\begin{aligned} \left\{ \begin{aligned} \dfrac{d}{dt} x_i^{(2)}(t)&= - x_i^{(2)}(t) + z^{(2)} + a^{(2)} y_i^{(2)}(t) + a_r^{(2)} y_{i+1}^{(2)}(t) + b^{(2)} y_i^{(1)}(t) + b_r^{(2)} y_{i+1}^{(1)}(t), \\ \dfrac{d}{dt} x_i^{(1)}(t)&= - x_i^{(1)}(t) + z^{(1)} + a^{(1)} y_i^{(1)}(t) + a_r^{(1)} y_{i+1}^{(1)}(t), \end{aligned} \right. \end{aligned}$$

(23)

where $i \in \mathbb {N}$, $t \ge 0$, and $y^{(j)}_i = f(x^{(j)}_i)$ for $j = 1, 2$. Since, for all i, j, $|x^{(j)}_i(t)| > 1$ provided t is large enough, the output $y^{(j)}_i(t)$ is either 1 or $-1$ after finite time. Hence we omit the time factor in the following discussion.

Suppose $\mathbf {y} = \left( \begin{array}{ll} y_1^{(2)} y_2^{(2)} y_3^{(2)} \cdots \\ y_1^{(1)} y_2^{(1)} y_3^{(1)} \cdots \end{array} \right) \in \mathbf {Y}$ is a mosaic pattern. For $i \in \mathbb {N}$, $y_i^{(1)} = 1$ if and only if $x_i^{(1)} > 1$. This derives

$$\begin{aligned} a^{(1)} + z^{(1)} - 1 > -a^{(1)}_r y^{(1)}_{i+1}. \end{aligned}$$

(24)

Similarly, $y_i^{(1)} = -1$ if and only if $x_i^{(1)}< -1$. This implies $y_i^{(1)} = -1$ if and only if

$$\begin{aligned} a^{(1)} - z^{(1)} - 1 > a^{(1)}_r y^{(1)}_{i+1}. \end{aligned}$$

(25)

The same argument asserts

$$\begin{aligned} a^{(2)} + z^{(2)} - 1&> -a^{(2)}_r y^{(2)}_{i+1} - (b^{(2)} y^{(1)}_i + b^{(2)}_r y^{(1)}_{i+1}) \end{aligned}$$

(26)

and

$$\begin{aligned} a^{(2)} - z^{(2)} - 1&> a^{(2)}_r y^{(2)}_{i+1} + (b^{(2)} y^{(1)}_i + b^{(2)}_r y^{(1)}_{i+1}) \end{aligned}$$

(27)

are the necessary and sufficient condition for $y_i^{(2)} = -1$ and $y_i^{(2)} = 1$, respectively. Define $\xi _1: \{-1, 1\} \rightarrow \mathbb {R}$ and $\xi _2: \{-1, 1\}^{\mathbb {Z}_{3 \times 1}} \rightarrow \mathbb {R}$ by

$$\begin{aligned} \xi _1(w) = a^{(1)}_r w, \qquad \xi _2(w_1, w_2, w_3) = a^{(2)}_r w_1 + b^{(2)} w_2 + b^{(2)}_r w_3. \end{aligned}$$

Set

That is,

Since two-layer cellular neural networks are locally coupled systems, $\mathcal {B}^{(1)}$ and $\mathcal {B}^{(2)}$ represents the basic sets of admissible local patterns of the first and second layer of (23), respectively. The set of admissible local patterns $\mathcal {B}$ of (23) is then

This is to say, the investigation of the equivalent relation for the templates is identical to the discussion of the equivalent relation for the basic sets of admissible local patterns. The fact that $y^{(1)}, y_r^{(1)} \in \{-1, 1\}$ indicates $a^{(1)} + z^{(1)} - 1 = - \xi _1(y_r^{(1)})$ and $a^{(1)} + z^{(1)} - 1 = \xi _1(y_r^{(1)})$ partition $a^{(1)}$-$z^{(1)}$ plane into 9 regions. More precisely, $a^{(1)}$-$z^{(1)}$ plane is partitioned by

$$\begin{aligned} a^{(1)} + z^{(1)} - 1 = a^{(1)}_r, \quad a^{(1)} + z^{(1)} - 1 > - a^{(1)}_r, \end{aligned}$$

and

$$\begin{aligned} a^{(1)} - z^{(1)} - 1 = a^{(1)}_r, \quad a^{(1)} - z^{(1)} - 1 = - a^{(1)}_r. \end{aligned}$$

Encode these nine regions as [p, q] for $0 \le p, q \le 2$, then a pair of parameters $(a^{(1)}, z^{(1)}) \in [p, q]$ infers that $(a^{(1)}, z^{(1)})$ satisfies m inequalities in (24) and n inequalities in (25). The relative positions of

$$\begin{aligned} a^{(1)} + z^{(1)} - 1 = a^{(1)}_r \quad \text {and} \quad a^{(1)} + z^{(1)} - 1 > - a^{(1)}_r \end{aligned}$$

and the relative positions of

$$\begin{aligned} a^{(1)} - z^{(1)} - 1 = a^{(1)}_r \quad \text {and} \quad a^{(1)} - z^{(1)} - 1 = - a^{(1)}_r \end{aligned}$$

remain to be determined. The “order” of lines $a^{(1)} + z^{(1)} - 1 = (-1)^{\ell } \xi _1(y_r^{(1)})$, $\ell = 0, 1$, in the plane come from the sign of $a_r^{(1)}$, this demonstrates that the parameter space $\{(a^{(1)}, a^{(1)}_r, z^{(1)})\}$ is partitioned into $2 \times 9 = 18$ equivalent regions. (Notably, the order of lines $a^{(1)} - z^{(1)} - 1 = \xi _1(y_r^{(1)})$ and $a^{(1)} - z^{(1)} - 1 = - \xi _1(y_r^{(1)})$ is determined seamlessly once the order of $a^{(1)} + z^{(1)} - 1 = (-1)^{\ell } \xi _1(y_r^{(1)})$ is given.) Namely, any two sets of parameters located in the same region determine the identical basic set of admissible local patterns ${{\mathcal {B}}}^{(1)}$. See Fig. 12a.

In an analogous manner, $y^{(2)}, y_r^{(2)}, u^{(2)}, u_r^{(2)} \in \{-1, 1\}$ indicates that $a^{(2)} + z^{(2)} - 1 > - \xi _2(y_r^{(2)}, u^{(2)}, u_r^{(2)})$ and $a^{(2)} + z^{(2)} - 1 > \xi _2(y_r^{(2)}, u^{(2)}, u_r^{(2)})$ partition $a^{(2)}$-$z^{(2)}$ plane into 81 regions. Encode these regions as [p, q] for $0 \le p, q \le 8$, then a pair of parameters $(a^{(2)}, z^{(2)}) \in [p, q]$ infers that $(a^{(2)}, z^{(2)})$ satisfies m inequalities in (26) and n inequalities in (27). Furthermore, the order of $a^{(2)} + z^{(2)} - 1 = \xi _2(y_r^{(2)}, u^{(2)}, u_r^{(2)})$ can be uniquely determined according to the following procedures.

• (1) The signs of $a_r^{(2)}, b^{(2)}, b_r^{(2)}$.

• (2) The magnitude of $a_r^{(2)}, b^{(2)}, b_r^{(2)}$.

• (3) The competition between the parameter with the largest magnitude and the others. In other words, suppose $m_1 > m_2 > m_3$ represent $|a_r^{(2)}|, |b^{(2)}|, |b_r^{(2)}|$. We need to determine whether $m_1 > m_2 + m_3$ or $m_1 < m_2 + m_3$.

This partitions the parameter space $\{(a^{(2)}, a^{(2)}_r, b^{(2)}, b^{(2)}_r, z^{(2)})\}$ into $8 \times 6 \times 2 \times 81 = 7776$ regions and each region is associated with a basic set of admissible local patterns (cf. Fig. 12b).

Fig. 12

The partition of $a^{(1)}$-$z^{(1)}$ and $a^{(2)}$-$z^{(2)}$ planes. In (a), $\ell ^{+}_i$ and $\ell ^-_i$, $i = 1, 2$, that represent the lines in (24) and (25) have partitioned the $a^{(1)}$-$z^{(1)}$ plane into nine regions. In (b), $\ell ^{+}_i$ and $\ell ^-_i$, represent the lines in (26) and (27), ave partitioned the $a^{(2)}$-$z^{(2)}$ plane into nine regions, where $1 \le i \le 8$

The above discussion demonstrates the following proposition.

Proposition A.1

Let $\mathcal {P}_8 = \{(a^{(1)}, a^{(1)}_r, a^{(2)}, a^{(2)}_r, b^{(2)}, b^{(2)}_r, z^{(1)}, z^{(2)})\}$ be the parameter space of (23). There is a positive integer K and unique set of open subregions $\{P_k\}_{k=1}^K$ satisfying

• (1) $\mathcal {P}_8 = \bigcup _{k=1}^K \overline{P}_k$, where $\overline{U}$ refers to the closure of U;

• (2) $P_i \bigcap P_j = \varnothing$ if $i \ne j$;

• (3) Templates $\mathbb {T}, \mathbb {T}^{\prime } \in P_k$ for some k if and only if $\mathbf {Y}_{\mathbb {T}} = \mathbf {Y}_{\mathbb {T}^{\prime }}$.

Example A.2

Suppose $a^{(1)}_r < 0$ and $a^{(2)}_r, b^{(2)}, b^{(2)}_r$ are all positive. Moreover, choose $a^{(2)}_r, b^{(2)}, b^{(2)}_r$ such that

$$\begin{aligned} a^{(2)}_r > b^{(2)} > b^{(2)}_r \quad \text {and} \quad a^{(2)}_r > b^{(2)} + b^{(2)}_r. \end{aligned}$$

For instance, $a^{(1)}_r = -1, a^{(2)}_r = 6, b^{(2)} = 3$, and $b^{(2)}_r = 2$. Then the position of each line is settled. We number the partitions of $a^{(1)}$-$z^{(1)}$ and $a^{(2)}$-$z^{(2)}$ planes by a pair $[m_{\ell }, n_{\ell }]$, $\ell = 1, 2$, where $m_{\ell }, n_{\ell }$ illustrate how many inequalities

$$\begin{aligned} a^{(\ell )} + z^{(\ell )} - 1 > - \xi _{\ell }(\cdot ) \quad \text {and} \quad a^{(\ell )} - z^{(\ell )} - 1 > \xi _{\ell }(\cdot ) \end{aligned}$$

are satisfied, respectively. Thus $0 \le m_1, n_1 \le 2$ and $0 \le m_2, n_2 \le 8$. Pick $[m_1, n_1] = [1,2]$ and $[m_2, n_2] = [6,4]$, for instance, $a^{(1)} = 2, z^{(1)} = -0.3, a^{(2)} = 4$, and $z^{(2)} = 2.5$. It is easy to check that the basic set of admissible local patterns is