Skip to main content
SpringerLink
Log in

Fusion of ANNs as decoder of retinal spike trains for scene reconstruction

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

The retina is one of the most developed sensing organs in the human body. However, the knowledge on the coding and decoding of the retinal neurons are still rather limited. Compared with coding (i.e., transforming visual scenes to retinal spike trains), the decoding (i.e., reconstructing visual scenes from spike trains, especially those of complex stimuli) is more complex and receives less attention. In this paper, we focus on the accurate reconstruction of visual scenes from their spike trains by designing a retinal spike train decoder based on the combination of the Fully Connected Network (FCN), Capsule Network (CapsNet) and Convolutional Neural Network (CNN), and a loss function incorporating the structural similarity index measure (SSIM) and L1 loss. CapsNet is used to extract the features from the spike trains, that are fused with the original spike trains and used as the inputs to FCN and CNN to facilitate the scene reconstruction. The feasibility and superiority of our model are evaluated on five datasets (i.e., MNIST, Fashion-MNIST, Cifar-10, Celeba-HQ and COCO). The model is evaluated quantitatively with four image evaluation indices, i.e., SSIM, MSE, PSNR and Intra-SSIM. The results show that the model provides a new means for decoding visual scene stimuli from retinal spike trains, and promotes the development of brain-machine interfaces.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  1. Schwemmer MA, Skomrock ND, Sederberg PB, Ting JE, Sharma G, Bockbrader MA, Friedenberg DA (2018) Meeting brain-computer interface user performance expectations using a deep neural network decoding framework. Nat Med 24(11):1669–76. https://doi.org/10.1038/s41591-018-0171-y

    Article  Google Scholar 

  2. Hutmacher F (2019) Why is there so much more research on vision than on any other sensory modality? Frontiers in psychology. https://doi.org/10.3389/fpsyg.2019.02246

  3. Epstein Russell A, Baker Chris I (2019) Scene perception in the human brain. Annu Rev Vis Sci 5:373–97. https://doi.org/10.1146/annurev-vision-091718-014809

    Article  Google Scholar 

  4. Langer Kirstin B, Ohlemacher SK, Phillips MJ, Fligor CM, Jiang P, Gamm DM, Meyer JS (2018) Retinal ganglion cell diversity and subtype specification from human pluripotent stem cells. Stem Cell Rep 10(4):1282–1293. https://doi.org/10.1016/j.stemcr.2018.02.010

    Article  Google Scholar 

  5. Zhang Y, Jia S, Zheng Y, Yu Z, Tian Y, Ma S, Huang T, Liu JK (2020) Reconstruction of natural visual scenes from neural spikes with deep neural networks. Neural Netw 125:19–30. https://doi.org/10.1016/j.neunet.2020.01.033

    Article  Google Scholar 

  6. Grimes WN, Songco-Aguas A, Rieke F (2018) Parallel processing of rod and cone signals: retinal function and human perception. Annual Rev Vis Sci 4:123–41. https://doi.org/10.1146/annurev-vision-091517-034055

    Article  Google Scholar 

  7. O’Brien J, Bloomfield SA (2018) Plasticity of retinal gap junctions: roles in synaptic physiology and disease. Annual review of vision science 4:79–100. https://doi.org/10.1146/annurev-vision-091517-034133

    Article  Google Scholar 

  8. Rivlin-Etzion M, Grimes WN, Rieke F (2018) Flexible neural hardware supports dynamic computations in retina. Trends Neurosci 41(4):224–37. https://doi.org/10.1016/j.tins.2018.01.009

    Article  Google Scholar 

  9. Demb JB, Singer JH (2015) Functional circuitry of the retina. Network:, Computation in Neural Systems 12(2):199. https://doi.org/10.1146/annurev-vision-082114-035334

    Google Scholar 

  10. Chichilnisky EJ (2001) A simple white noise analysis of neuronal light responses. Front Syst Neurosci 10:109

    MATH  Google Scholar 

  11. Pillow JW, Shlens J, Paninski L, Sher A, Litke AM, Chichilnisky EJ, Simoncelli EP (2008) Spatio-temporal correlations and visual signalling in a complete neuronal population. Nature 454(7207):995–9. https://doi.org/10.1038/nature07140

    Article  Google Scholar 

  12. Cessac B, Kornprobst P, Kraria S, Nasser H, Pamplona D, Portelli G, Viéville T (2017) PRANAS: A new platform for retinal analysis and simulation. Front Neuroinform 11:49. https://doi.org/10.3389/fninf.2017.00049

    Article  Google Scholar 

  13. Meyer AF, Williamson RS, Linden JF, Sahani M (2017) Models of neuronal stimulus-response functions: elaboration, estimation, and evaluation. Front Systems Neurosci 10:109. https://doi.org/10.3389/fnsys.2016.00109

    Article  Google Scholar 

  14. Botella-Soler V, Deny S, Martius G, Marre O, Tkačik G (2018) Nonlinear decoding of a complex movie from the mammalian retina. PLoS Comput Biol 14(5):e1006057. https://doi.org/10.1371/journal.pcbi.1006057

    Article  Google Scholar 

  15. Pillow JW, Paninski L, Uzzell VJ, Simoncelli EP, Chichilnisky EJ (2005) Prediction and decoding of retinal ganglion cell responses with a probabilistic spiking model. J Neurosci 25(47):11003–13. https://doi.org/10.1523/JNEUROSCI.3305-05.2005

    Article  Google Scholar 

  16. Parthasarathy N, Batty E, Falcon W, Rutten T, Rajpal M, Chichilnisky EJ, Paninski L (2017) Neural networks for efficient bayesian decoding of natural images from retinal neurons. Adv Neural Inf Process Syst 30:6434–45. https://doi.org/10.1101/153759

    Google Scholar 

  17. Sabour S, Frosst N, Hinton GE (2017) Dynamic routing between capsules. Adv Neural Inf Process Syst, pp 3859–3869

  18. Roy K, Roy P, Chaudhuri SS (2021) Capsule Neural Network Architecture Based Multi-class Fruit Image Classification. In: Advances in smart communication technology and information processing:, OPTRONIX, vol 2020, pp 171–180

  19. Wang Z, Simoncelli EP, Bovik AC (2003) Multiscale structural similarity for image quality assessment. In: The thrity-Seventh asilomar conference on signals, systems and computers, vol 2003, pp 1398–1402

  20. Pathak D, Krahenbuhl P, Donahue J, Darrell T, Efros AA (2016) Context encoders: Feature learning by inpainting. Inproceedings of the IEEE conference on computer vision and pattern recognition, pp 2536–2544

  21. Isola P, Zhu JY, Zhou T, Efros AA (2017) Image-to-image translation with conditional adversarial networks. Inproceedings of the IEEE conference on computer vision and pattern recognition, pp 1125–1134

  22. Gadirov H (2018) Capsule architecture as a discriminator in generative adversarial networks. MS thesis

  23. Li W, Raj AN, Tjahjadi T, Zhuang Z (2021) Digital hair removal by deep learning for skin lesion segmentation. Pattern Recogn 117:107994. https://doi.org/10.1016/j.patcog.2021.107994

    Article  Google Scholar 

  24. Onken A, Liu JK, Karunasekara PC, Delis I, Gollisch T, Panzeri S (2016) Using matrix and tensor factorizations for the single-trial analysis of population spike trains. PLoS comput biol 12(11):e1005189. https://doi.org/10.1371/journal.pcbi.1005189

    Article  Google Scholar 

  25. Brackbill N, Rhoades C, Kling A, Shah NP, Sher A (2020) Reconstruction of natural images from responses of primate retinal ganglion cells. Elife 9:e58516. https://doi.org/10.7554/eLife.58516

    Article  Google Scholar 

  26. Kim YJ, Brackbill N, Batty E, Lee J, Mitelut C, Tong W, Chichilnisky EJ, Paninski L (2021) Nonlinear decoding of natural images from large-scale primate retinal ganglion recordings. Neural Comput 33(7):1719–50. https://doi.org/10.1162/neco_a_01395

    Article  Google Scholar 

  27. Odermatt B, Nikolaev A, Lagnado L (2012) Encoding of luminance and contrast by linear and nonlinear synapses in the retina. Neuron 73(4):758–73. https://doi.org/10.1016/j.neuron.2011.12.023

    Article  Google Scholar 

  28. Benoit A, Caplier A, Durette B, Hérault J (2010) Using human visual system modeling for bio-inspired low level image processing. Comput Vis Image Underst 114(7):758–73. https://doi.org/10.1016/j.cviu.2010.01.011

    Article  Google Scholar 

  29. Martinez-Alvarez A, Olmedo-Payá A, Cuenca-Asensi S, Ferrández JM, Fernandez E (2013) Retinastudio: A bioinspired framework to encode visual information. Neurocomputing 114:45–53. https://doi.org/10.1016/j.neucom.2012.07.035

    Article  Google Scholar 

  30. Nasser H, Kraria S, Cessac B (2013) Enas: a new software for neural population analysis in large scale spiking networks. BMC Neurosci 14(1):1–2. https://doi.org/10.1186/1471-2202-14-S1-P57

    Google Scholar 

  31. Cofre R, Cessac B (2014) Exact computation of the maximum-entropy potential of spiking neural-network models. Physical Review E 89(5):052117. https://doi.org/10.1103/PhysRevE.89.052117

    Article  Google Scholar 

  32. Gollisch T, Meister M (2010) Eye smarter than scientists believed: neural computations in circuits of the retina. Neuron 65(2):150–64. https://doi.org/10.1016/j.neuron.2009.12.009

    Article  Google Scholar 

  33. Wohrer A, Kornprobst P (2009) Virtual retina: a biological retina model and simulator, with contrast gain control. J Comput Neurosci 26(2):219–249. https://doi.org/10.1007/s10827-008-0108-4

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgements

This research was financially supported by the Scientific Research Grant of Shantou University, China, Grant (No. NTF17016), National Natural Science Foundation of China (No. 82071992) and Basic and Applied Basic Research Foundation of Guangdong Province [No. 2020B1515120061].

Author information

Authors and Affiliations

  1. Key Laboratory of Digital Signal and Image Processing of Guangdong Province, Department of Electronic Engineering, College of Engineering, Shantou University, Shantou, China

    Wei Li, Alex Noel Joseph Raj & Zhemin Zhuang

  2. School of Engineering, University of Warwick, Coventry, UK

    Tardi Tjahjadi

Authors
  1. Wei Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Alex Noel Joseph Raj
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tardi Tjahjadi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zhemin Zhuang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Alex Noel Joseph Raj.

Ethics declarations

Conflict of Interests

The authors declare that they have no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, W., Joseph Raj, A.N., Tjahjadi, T. et al. Fusion of ANNs as decoder of retinal spike trains for scene reconstruction. Appl Intell 52, 15164–15176 (2022). https://doi.org/10.1007/s10489-022-03402-w

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-022-03402-w

Keywords

Search

Navigation