Simulation of retinal ganglion cell response using fast independent component analysis

Abstract

Advances in neurobiology suggest that neuronal response of the primary visual cortex to natural stimuli may be attributed to sparse approximation of images, encoding stimuli to activate specific neurons although the underlying mechanisms are still unclear. The responses of retinal ganglion cells (RGCs) to natural and random checkerboard stimuli were simulated using fast independent component analysis. The neuronal response to stimuli was measured using kurtosis and Treves–Rolls sparseness, and the kurtosis, lifetime and population sparseness were analyzed. RGCs exhibited significant lifetime sparseness in response to natural stimuli and random checkerboard stimuli. About 65 and 72% of RGCs do not fire all the time in response to natural and random checkerboard stimuli, respectively. Both kurtosis of single neurons and lifetime response of single neurons values were larger in the case of natural than in random checkerboard stimuli. The population of RGCs fire much less in response to random checkerboard stimuli than natural stimuli. However, kurtosis of population sparseness and population response of the entire neurons were larger with natural than random checkerboard stimuli. RGCs fire more sparsely in response to natural stimuli. Individual neurons fire at a low rate, while the occasional “burst” of neuronal population transmits information efficiently.

Appendix

Sparse coding

Sparse coding includes a class of unsupervised methods for learning sets of complete bases for efficient data representation. The aim of sparse coding is to develop a set of basic vectors that represent an input vector as a linear combination of the basic vectors:

$$x = As = \sum\limits_{i = 1}^{n} {a_{i} s_{i} }$$

(7)

where $x = (x_{1} ,x_{2} , \ldots ,x_{n} )^{T}$ represents input data, $A = (a_{1} ,a_{2} , \ldots ,a_{m} )^{T}$ is base matrix, $a_{i}$ is column i in $A$, which represents the basic functions. $S = (s_{1} ,s_{2} , \ldots ,s_{m} )^{T}$ denotes coefficient matrix. With a complete basis, $S$ is no longer uniquely determined by the input vector $x$. Therefore, we introduced the additional criterion of sparsity in sparse coding. We define sparsity in terms of few non-zero components or few components not close to zero. The choice of sparsity as a desired characteristic in our representation of the input data is motivated by the observation that most sensory data such as natural images may be described as the superposition of a small number of atomic elements such as surfaces or edges. Other justifications such as comparisons of the properties of the primary visual cortex have also been advanced.

We define the sparse coding cost function using a set of n input vectors as follows:

$$F(A,S) = \min_{{a_{j} ,s_{j} }} \sum\limits_{i = 1}^{n} {\left\| {x_{i} - \sum\limits_{j = 1}^{m} {a_{j} s_{j} } } \right\|^{2} } + \lambda \sum\limits_{j = 1}^{m} {H(s_{j} )}$$

(8)

where $a_{j}$ represents basic function, $s_{j}$ is coefficient, $x_{i}$ is input data, $\lambda$ is a constant, $H(s_{j} )$ denotes a sparsity cost function, which penalizes $s_{j}$ for being far from zero. Usually a common choice for the sparsity cost is the L1 penalty $H(s_{j} ) = \left| {s_{j} } \right|_{1}$, but it is non-differentiable when basic function equals 0, therefore, we selected sparsity cost $H(s_{j} ) = \sqrt {s_{j}^{2} + \varepsilon }$, wherein $\varepsilon$ is a constant.

We interpret the first term of the sparse coding objective as a reconstruction term, which uses the algorithm to provide a good representation of x and the second term as a sparsity penalty, which is a sparce representation of $x$. The constant $\lambda$ is a scaling constant determining the relative importance of these two contributions.

In addition, it is possible to make the sparsity penalty arbitrarily small by scaling down $s_{j}$ and scaling $a_{j}$ up using a large constant. To prevent this event, we constrain $\left\| {a_{j} } \right\|^{2} \le C,\quad \forall j = 1,2, \ldots m$ to be less than the constant C.

The full sparse coding cost function including our constraint is as follows:

$$\begin{aligned} F(A,S) &= \min_{{a_{j} ,s_{j} }} \sum\limits_{i = 1}^{n} {\left\| {x_{i} - \sum\limits_{j = 1}^{m} {a_{j} s_{j} } } \right\|^{2} } + \lambda \sum\limits_{j = 1}^{m} {H(s_{j} )} \\ &\hbox{Subject to} \,\,\left\| {a_{j} } \right\|^{2} \le C,\quad \forall j = 1,2, \ldots m \end{aligned}$$

(9)

However, the constraint of $\left\| {a_{j} } \right\|^{2} \le C,\quad \forall j = 1,2, \ldots m$ cannot be enforced using simple gradient-based methods. This constraint is weakened to a “weight decay” term designed to keep the entries of $A$ small. Therefore, we added the constraints to the objective function to provide a new objective function:

$$F(A,S) = \left\| {X - AS} \right\|^{2} + \lambda \sum {\sqrt {S^{2} + \varepsilon } } + \gamma \left\| A \right\|^{2}$$

(10)

where $X$ is input data,$A = (a_{1} ,a_{2} , \ldots a_{m} )^{T}$ is base matrix, $S = (s_{1} ,s_{2} , \ldots s_{m} )^{T}$ is coefficient matrix, $\lambda$ and $\gamma$ are constants.

The objective function is non-convex, and hence impossible to optimize well using gradient-based methods. However, given $A$, the problem of finding $S$ that minimizes $F(A,S)$ is convex. Similarly, given $S$, the problem of finding $A$ that minimizes $F(A,S)$ is also convex suggesting an alternative to optimize $A$ for a fixed $S$, and then optimizing $S$ with a fixed $A$.

The analytic solution of $A$ is obtained as follows:

$$\frac{\partial F(A,S)}{\partial A} = xS^{T} - ASS^{T} - \gamma A$$

(11)

The analytic solution of ${\text{S}}$ is provided by:

$$\frac{\partial F(A,S)}{\partial S} = A^{T} x - A^{T} AS - \lambda S./\sqrt {S^{2} + \varepsilon }$$

(12)

Therefore, the learning equation of basic function $a_{i}$ is represented by:

$$\Delta a_{i} = a_{i} (t + 1) - a_{i} (t) = x_{i} s_{i}^{T} - a_{i} s_{i} s_{i}^{T} - \gamma a_{i}$$

(13)

The learning equation of coefficient $s_{i}$ is as follows:

$$\Delta s_{i} = s_{i} (t + 1) - s_{i} (t) = a_{i}^{T} x_{i} - a_{i}^{T} a_{i} s_{i} - \lambda s_{i} ./\sqrt {s_{i}^{T} s_{i} + \varepsilon }$$

(14)

Using the simple iterative algorithm on a large dataset (including 10,000 patches) results in prolonged iterations and convergence of the algorithm. To increase the rate of convergence by accelerating the iteration, the algorithm may be run on mini-patch selecting a mini-patch random subset of 1000 patches from the 10,000 patches.

A faster and better convergence may be obtained via initialization of the feature matrix $S$ before using gradient descent (or other methods) to optimize the objective function for $S$ given $A$. In practice, initializing $S$ randomly at each iteration results in poor convergence unless a good optimum is found for $S$ before optimizing for $A$. A better way to initialize $S$ involves the following steps:

• 1. Random initialization of  $A$

• 2. Repetition until convergence

  • 1. Selection of a mini-patch random subset.

  • 2. Initialization of $S$ with $S = A^{T} X$, dividing the feature by the the corresponding basic vector in $A$.

  • 3. Finding $S$ that minimizes  $F(A,S)$ for the $A$ in the previous step.

  • 4. Determination of $A$ that minimizes $F(A,S)$ for the $S$ found in the previous step. Using this method, good local optima can be reached relatively quickly.

We obtained the base matrix $A$ trained by FASTICA. Using the FASTICA method, we derived the coefficient matrix and the objective function values through traditional sparse coding. We selected a convex function $H(s_{j} ) = \sqrt {s_{j}^{2} + \varepsilon }$ for the sparsity cost in objective function, where $\varepsilon$ = 0.01 and $\lambda$ = 0.3. Since the base matrix $\left\| A \right\|^{2}$ in objective function was obtained and normalized by FASTICA, $\left\| A \right\|^{2}$ equals 1. $\left\| A \right\|^{2}$ does not affect the optimization of the objective function, and therefore, $\gamma$ = 0.