Robustness of neural codes and its implication on natural image processing

Abstract

In this study, based on the view of statistical inference, we investigate the robustness of neural codes, i.e., the sensitivity of neural responses to noise, and its implication on the construction of neural coding. We first identify the key factors that influence the sensitivity of neural responses, and find that the overlap between neural receptive fields plays a critical role. We then construct a robust coding scheme, which enforces the neural responses not only to encode external inputs well, but also to have small variability. Based on this scheme, we find that the optimal basis functions for encoding natural images resemble the receptive fields of simple cells in the striate cortex. We also apply this scheme to identify the important features in the representation of face images and Chinese characters.

Appendices

Appendix A: The Fisher information and the performance of LSE

The Fisher information

Since noise is independent Gaussian, the conditional probability of observing data I(x) given a is written as

$$ p(I|{\varvec a})={\frac {1} {(\sqrt{2\pi}\sigma)^{N}}}\hbox{exp}\{-{\frac {1} {2\sigma^{2}}} \sum_{i}^{N}[I(x^i)-a_{1}\phi_{1}(x^{i})-a_{2}\phi_{2}(x^{i})]^{2}\}, $$

(17)

where N is the number of data points and σ² the noise strength.

The Fisher information matrix F is calculated to be

$$ F_{mn}=-\int {\frac {\partial^{2}\hbox{ln}\,P(I|{\varvec a})} {\partial a_{m}\partial a_{n}}}P(I|{\varvec a})dI,\quad\hbox{for}\quad m,n=1,2, $$

(18)

where F _mn is the element in the mth row and nth column of F.

It is straightforward to check that

$$ {\varvec F}={\frac {1} {\sigma^{2}}} \left(\begin{array}{ll} \quad\sum_{i}\phi_{1}(x^{i})^{2} & \sum_{i}\phi_{1}(x^{i})\phi_{2}(x^{i}) \\ \sum_{i}\phi_{1}(x^{i})\phi_{2}(x^{i}) & \quad\sum_{i}\phi_{2}(x^{i})^{2} \end{array} \right) $$

(19)

According to the Cramér-Rao bound, the inverse of the Fisher information defines the lower bound for decoding errors of un-biased estimators. Consider the covariance matrix for estimation errors of an un-biased estimator is given by $\Omega_{mn}=\langle (\hat{a}_{m}-a_{m}) (\hat{a}_{n}-a_{n})\rangle$ . The Cramér-Rao bound states that $\Omega \ge {\varvec F}^{-1}$ , or more exactly, the matrix $(\Omega-{\varvec F}^{-1}$) is semi-positive definite. Intuitively, this means that the inverse of the Fisher information quantifies the minimum inferential sensitivity of un-biased estimators.

The asymptotical performance of LSE

It is straightforward to check that for independent Gaussian noise, LSE is equivalent to maximum likelihood inference, i.e., its solution is obtained through maximizing the log likelihood, $\hbox{ln}\,\,p(I|{\varvec a})$ (comparing Eq. (3) with Eq. (17)). This implies the solution of LSE satisfies the condition,

$$ {\frac {\partial \hbox{ln}\,p(I|\hat{{\varvec a}})} {\partial a_{l}}}=0, \quad\hbox{for}\quad l=1,2. $$

(20)

Consider $\hat{{\varvec a}}$ is sufficiently close to the true value a, the above equations can be approximated as (the first-order Taylor expansion at the point a)

$$ {\frac {\partial \,\hbox{ln}\,p(I|{\varvec a})} {\partial a_{l}}} +{\frac {\partial^{2}\,\hbox{ln}\,p(I|{\varvec a})} {\partial a_{l}^{2}}}(\hat{a}_{l}-a_{l}) + {\frac {\partial^{2}\,\hbox{ln}\, p(I|{\varvec a})} {\partial a_{l} \partial a_{m\neq l }}}(\hat{a}_{m}-a_{m}) \approx 0,\quad\hbox{for}\quad l,m=1,2. $$

(21)

By using Eq. (17), the above equations can be simplified as

$$ {\frac {\partial\,\hbox{ln}\,p(I|{\varvec a})} {\partial a_{l}}} +\sum_{i}{\frac {\phi_{l}(x^{i})^{2}} {\sigma^{2}}}(\hat{a}_{l}-a_{l}) + \sum_{i}{\frac { \phi_{l}(x^{i})\phi_{m\neq l}(x^{i})} {\sigma^{2}}} (\hat{a}_{m}-a_{m}) \approx 0,\quad\hbox{for}\quad l,m=1,2. $$

(22)

Since noise are independent Gaussian, we have

$$ {\frac {\partial\,\hbox{ln}\,p(I|{\varvec a})} {\partial a_{l}}}=\sum_{i}{\frac {\epsilon_{i}} {\sigma^{2}}}\phi_{l}(x^{i}),\quad\hbox{for}\quad l=1,2, $$

(23)

where $\epsilon_{i}$, for i = 1,...,N, are independent Gaussian random numbers of zero mean and variance σ².

Combining Eqs. (22) and (23), we obtain the estimation error of LSE. Here, we only show the result for $\hat{a}_{1}$ (the case for $\hat{a}_{2}$ is similar), which is given by,

$$ \hat{a}_{1}-a_{1} = -{\frac {\sum_{i}\varepsilon_{i}\phi_{1}(x^{i})\sum_{j}\phi_{2}(x^{j})^{2} -\sum_{i}\varepsilon_{i}\phi_{2}(x^{i})\sum_{j}\phi_{1}(x^{j})\phi_{2}(x^{j})} {\sum_{i}\phi_{1}(x^{i})^{2}\sum_{j}\phi_{2}(x^{j})^{2}-(\sum_{i}\phi_{1}(x^{i})\phi_{2}(x^{i}))^{2}}}. $$

(24)

It is easy to check LSE is un-biased, i.e.,

$$ \langle (\hat{a}_{1}-a_{1}) \rangle =0. $$

(25)

The variance of $\hat{a}_{1}$ is calculated to be

$$ \langle (\hat{a}_{1}-a_{1})^{2} \rangle = {\frac {\sigma^{2}\sum_{i}\phi_{2}(x^{i})^{2}} {\sum_{i}\phi_{1}(x^{i})^{2}\sum_{i}\phi_{2}(x^{i})^{2} -(\sum_{i}\phi_{1}(x^{i})\phi_{2}(x^{i}))^{2}}}. $$

(26)

According to the Central Limiting Theorem, when the number of data points N is sufficiently large, the random variable $(\hat{a}_{1}-a_{1})$ will satisfy a normal distribution with the variance given by Eq. (26).

The covariance between the estimation errors of the two components can also be calculated, which is given by

$$ \langle (\hat{a}_{1}-a_{1})(\hat{a}_{2}-a_{2}) \rangle = {\frac {-\sigma^{2}\sum_{i}\phi_{1}(x^{i})\phi_{2}(x^{i})} {\sum_{i}\phi_{1}(x^{i})^{2}\sum_{i}\phi_{2}(x^{i})^{2} -(\sum_{i}\phi_{1}(x^{i})\phi_{2}(x^{i}))^{2}}}. $$

(27)

It is straightforward to check that the covariance matrix of estimation errors of LSE, given by $\Omega_{mn}=\langle (\hat{a}_{m}-a_{m})(\hat{a}_{n}-a_{n}) \rangle$, for m,n = 1,2, is the inverse of the Fisher information matrix F, i.e., $\Omega{\varvec F}=1$. This implies LSE is asymptotically efficient.

Appendix B: Optimizing the basis functions of robust coding

The sensitivity measure H(a)

We choose the Renyi’s quadratic entropy to measure the variability of neural responses when natural images are presented, which is given by

$$ H\left(a|{\varvec I}\right)=-\hbox{ln}\int p\left(a|{\varvec I}\right)^{2}da. $$

(28)

Here for simplicity, we use a to replace a _l , for $l=1,\ldots,M$.

Suppose we have K sampled values of a which are obtained when K natural images are presented, then according to the Parzen window approximation (with the Gaussian kernel), $p(a|{\varvec I})$ can be approximated as

$$ p\left(a|{\varvec I}\right)={\frac {1} {\sqrt{2\pi}dK}} \sum_{k=1}^{K}e^{-\frac{(a-a^{k})^{2}}{2d^{2}}}, $$

(29)

where $\{a^k\}$, for k = 1,...,K, represents the sampled values, and d is the width of Gaussian kernel.

Note that

$$ \begin{array}{lll}\int p\left(a|{\varvec I}\right)^{2}da &=&\int{\frac{1}{\sqrt{2\pi}dK}}\sum\limits_{k=1}^{K}e^{-\left(a-a^k\right)^{2}/2d^{2}} \cdot{\frac{1}{\sqrt{2\pi}dK}}\sum\limits_{m=1}^{K}e^{-\left(a-a^m\right)^{2}/2d^{2}}da,\\&= &\frac{1}{\sqrt{2\pi}dK^{2}}\sum\limits_{k=1}^{K}\sum\limits_{m=1}^{K}e^{-\frac{\left(a^{k}-a^{m}\right)^{2}}{4d^{2}}}.\end{array} $$

(30)

Thus, we have

$$ H\left(a|{\varvec I}\right)=-\hbox{ln}\left[\frac{1}{\sqrt{2\pi}dK^{2}} \sum\limits_{k=1}^{K}\sum\limits_{m=1}^{K}e^{-\frac{\left(a^{k}-a^{m}\right)^{2}}{4d^{2}}}\right], $$

(31)

which fully depends on the sampled values.

The training procedure

Minimizing Eq. (15) is carried out by using the gradient descent method in two alternative steps, namely, (1) updating a while fixing ϕ and (2) updating ϕ while fixing a.

(1) Updating a

To apply the gradient descent method, the key is to calculate the gradient of E with respect to $a_{l}^{k}$, for $l=1,\ldots,M$ and k = 1,...,K.

For the first term in Eq. (15), we have

$$ \frac{\partial}{\partial a_{l}^{k}}\frac{1}{2K} \sum_{k=1}^{K}\left|I^{k} \left({\varvec x}\right)-{\varvec a}^{k}\phi\left({\varvec x})\right)\right|^{2}=-{\frac 1 K}\left[I^{k}({\varvec x}) -{\varvec a}^{k}\phi({\varvec x})\right]\phi_{l}({\varvec x}). $$

(32)

For the second term, we have

$$ \frac{\partial}{\partial a_{l}^{k}}\lambda H\left({\varvec a}\right) =\lambda\sum_{m=1}^{K}e^{-(a_{l}^{k}-a_{l}^{m})^{2}/4d^{2}} (a_{l}^{k}-a_{l}^{m}) /\left[d^{2}\sum_{j=1}^{K}\sum_{m=1}^{K}e^{-(a_{l}^{j}-a_{l}^{m})^{2}/4d^{2}}\right]. $$

(33)

Combining Eqs. (32) and (33), we obtain the update rule for a:

$$ a_{l}^{k}(new)=a_{l}^{k}(old)+\eta \Delta a_{l}^{k},\quad\hbox{for}\quad l=1,\ldots,M, \,\hbox{and}\,k=1,\ldots,K, $$

(34)

where η is the learning rate, and $\Delta a_{l}^{k}$ is given by

$$ \begin{array}{lll} \Delta a_{l}^{k} & =& {\frac 1 K} \left[I^{k}({\varvec x}) -{\varvec a}^{k}\phi({\varvec x})\right] \phi_{l}({\varvec x})\\ &-&\lambda\sum\limits_{m=1}^{K}e^{-(a_{l}^{k}-a_{l}^{m})^{2}/4d^{2}} (a_{l}^{k}-a_{l}^{m}) /\left[d^{2}\sum\limits_{j=1}^{K} \sum\limits_{m=1}^{K}e^{-(a_{l}^{j}-a_{l}^{m})^{2}/4d^{2}}\right]. \end{array} $$

(35)

(2) Updating ϕ

Similarly, the update rule for ϕ is given by

$$ \phi_{l}({\varvec x})(new)=\phi_{l}({\varvec x})(old)+{\frac {\eta} {K}} \sum_{k=1}^{K}a_{l}^{k}\left[I\left({\varvec x}\right)- {\varvec a}^{k}\phi\left({\varvec x}\right)\right], \quad\hbox{for}\quad l=1,\ldots,K. $$

(36)