Abstract
Recent development in artificial neural networks has led to an increase in performance, but also in complexity and size. This poses a significant challenge for the exploration and analysis of the spatial structure and temporal behaviour of such networks. Several projects for the 3D visualisation of neural networks exist, but they focus largely on the exploration of the spatial structure alone, and are using standard 2D screens as output and mouse and keyboard as input devices. In this article, we present NeuVis, a framework for an intuitive and immersive 3D visualisation of spiking neural networks in virtual reality, allowing for a larger variety of input and output devices. We apply NeuVis to NeuCube, a 3-dimensional spiking neural network learning framework, significantly improving the user’s abilities to explore, analyse, and also debug the network. Finally, we discuss further venues of development and alternative render methods that are currently under development and will increase the visual accuracy and realism of the visualisation, as well as further extending its analysis and exploration capabilities.
This is a preview of subscription content, log in via an institution to check access.
Notes
Available from the Knowledge Engineering and Discovery Research Institute (KEDRI), Auckland University of Technology, http://www.kedri.aut.ac.nz/neucube.
References
Armstrong JD, van Hemert JI (2009) Towards a virtual fly brain. Philos Transact A Math Phys Eng Sci 367:2387–2397
Bruckner S, Šoltészová V, Gröller ME, Hladuvka J, Buhler K, Yu JY, Dickson BJ (2009) BrainGazer—visual queries for neurobiology research. IEEE Trans Vis Comput Graph 15(6):1497–1504. doi:10.1109/TVCG.2009.121
Foottit J, Brown D, Marks S, Connor A (2014) An intuitive tangible game controller. In: Nesbitt K, Blackmore K (eds) The 10th Australasian conference on interactive entertainment (IE 2014), Newcastle, NSW, Australia. doi:10.1145/2677758.2677774
Furber S, Galluppi F, Temple S, Plana L (2014) The SpiNNaker Project. Proceedings of the IEEE 102(5):652–665. doi:10.1109/JPROC.2014.2304638
HTC Corporation (2016) Vive. https://www.vive.com/anz/
von Kapri A, Rick T, Potjans TC, Diesmann M, Kuhlen T (2011) Towards the visualization of spiking neurons in virtual reality. Stud Health Technol Inf 163:685–687
Kasabov N (2012) NeuCube EvoSpike architecture for spatio-temporal modelling and pattern recognition of brain signals. In: Artificial neural networks in pattern recognition, Springer, pp 225–243. doi:10.1007/978-3-642-33212-8_21
Kasabov N, Scott NM, Tu E, Marks S, Sengupta N, Capecci E, Othman M, Doborjeh MG, Murli N, Hartono R, Espinosa-Ramos JI, Zhou L, Alvi FB, Wang G, Taylor D, Feigin V, Gulyaev S, Mahmoud M, Hou ZG, Yang J (2016) Evolving spatio-temporal data machines based on the neucube neuromorphic framework: design methodology and selected applications. Neural Netw 78:1–14. doi:10.1016/j.neunet.2015.09.011, http://www.sciencedirect.com/science/article/pii/S0893608015001860 (special issue on neural network learning in big data)
Kolasinski EM (1995) Simulator sickness in virtual environments. Tech. rep., Defense Technical Information Center. www.dtic.mil/cgi-bin/GetTRDoc?AD=ADA295861
Lancaster J, Rainey L, Summerlin J, Freitas C, Fox P, Evans A, Toga A, Mazziotta J (1997) Automated labeling of the human brain. Hum Brain Mapp 5(4):238–242. doi:10.1002/(SICI)1097-0193(1997)5:4<238::AID-HBM6>3.0.CO;2-4, http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2860189/
Lancaster JL, Woldorff MG, Parsons LM, Liotti M, Freitas CS, Rainey L, Kochunov PV, Nickerson D, Mikiten SA, Fox PT (2000) Automated Talairach atlas labels for functional brain mapping. Hum Brain Mapp 10(3):120–131
Lin CY, Tsai KL, Wang SC, Hsieh CH, Chang HM, Chiang AS (2011) The neuron navigator: exploring the information pathway through the neural maze. In: 2011 IEEE Pacific visualization symposium (PacificVis), pp 35–42. doi:10.1109/PACIFICVIS.2011.5742370
Maciel PWC, Shirley P (1995) Visual navigation of large environments using textured clusters. In: Proceedings of the 1995 symposium on interactive 3D graphics, ACM, New York, NY, USA, I3D ’95, pp 95–ff. doi:10.1145/199404.199420, http://doi.acm.org/10.1145/199404.199420
Oculus VR (2015) Best practices guide. Tech. rep., Oculus VR. http://developer.oculus.com/best-practices
Oculus VR (2016) Oculus rift. https://www3.oculus.com/en-us/rift/
Ridder Md, Jung Y, Huang R, Kim J, Feng DD (2015) Exploration of virtual and augmented reality for visual analytics and 3d volume rendering of functional magnetic resonance imaging (fMRI) data. In: Big data visual analytics (BDVA), 2015, pp 1–8. doi:10.1109/BDVA.2015.7314293
Scott N, Kasabov N, Indiveri G (2013) NeuCube neuromorphic framework for spatio-temporal brain data and its python implementation. In: Neural information processing, Springer, pp 78–84. doi:10.1007/978-3-642-42051-1_11
Sherif T, Kassis N, Rousseau ME, Adalat R, Evans AC (2015) BrainBrowser: distributed, web-based neurological data visualization. Front Neuroinf 8:89. doi:10.3389/fninf.2014.00089, http://journal.frontiersin.org/article/10.3389/fninf.2014.00089/full
Wejchert J, Tesauro G (1990) Neural network visualization. In: Touretzky DS (ed) Advances in neural information processing systems 2, Morgan Kaufmann Publishers Inc., San Francisco, pp 465–472. http://dl.acm.org/citation.cfm?id=109230.109287
Xia M, Wang J, He Y (2013) BrainNet Viewer: a network visualization tool for human brain connectomics. PloS One 8(7):e68,910. doi:10.1371/journal.pone.0068910, http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0068910
Acknowledgements
Thanks to the staff at KEDRI and specifically Nathan Scott and Prof. Nikola Kasabov for his support in the development of this visualisation. Also thanks to Javier Estevez, MoCap technician at AUT, for his support in developing the Sentience Lab framework.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Marks, S. Immersive visualisation of 3-dimensional spiking neural networks. Evolving Systems 8, 193–201 (2017). https://doi.org/10.1007/s12530-016-9170-8
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12530-016-9170-8