International Journal of Parallel Programming

, Volume 40, Issue 6, pp 553–582 | Cite as

Managing Burstiness and Scalability in Event-Driven Models on the SpiNNaker Neuromimetic System

  • Alexander D. Rast
  • Javier Navaridas
  • Xin Jin
  • Francesco Galluppi
  • Luis A. Plana
  • Jose Miguel-Alonso
  • Cameron Patterson
  • Mikel Luján
  • Steve Furber
Open Access
Article

Abstract

Neural networks present a fundamentally different model of computation from the conventional sequential digital model, for which conventional hardware is typically poorly matched. However, a combination of model and scalability limitations has meant that neither dedicated neural chips nor FPGA’s have offered an entirely satisfactory solution. SpiNNaker introduces a different approach, the “neuromimetic” architecture, that maintains the neural optimisation of dedicated chips while offering FPGA-like universal configurability. This parallel multiprocessor employs an asynchronous event-driven model that uses interrupt-generating dedicated hardware on the chip to support real-time neural simulation. Nonetheless, event handling, particularly packet servicing, requires careful and innovative design in order to avoid local processor congestion and possible deadlock. We explore the impact that spatial locality, temporal causality and burstiness of traffic have on network performance, using tunable, biologically similar synthetic traffic patterns. Having established the viability of the system for real-time operation, we use two exemplar neural models to illustrate how to implement efficient event-handling service routines that mitigate the problem of burstiness in the traffic. Extending work published in ACM Computing Frontiers 2010 with on-chip testing, simulation results indicate the viability of SpiNNaker for large-scale neural modelling, while emphasizing the need for effective burst management and network mapping. Ultimately, the goal is the creation of a library-based development system that can translate a high-level neural model from any description environment into an efficient SpiNNaker instantiation. The complete system represents a general-purpose platform that can generate an arbitrary neural network and run it with hardware speed and scale.

Keywords

Asynchronous Burst Network Event-driven Universal Neural Multiprocessor Interconnection Real-time Traffic Characterisation 

Notes

Acknowledgments

The SpiNNaker project is supported by the Engineering and Physical Sciences Research Council, partly through the Advanced Processor Technologies Portfolio Partnership at the University of Manchester, and through Grants EP/D07908X/1 and GR/S61270/01; and also by ARM and Silistix. When this research was performed Dr. Javier Navaridas was supported by a post-doctoral grant of the University of the Basque Country and is now a Newton International Fellow with the University of Manchester. Prof. Jose Miguel-Alonso is supported by the Spanish Ministry of Education and Science, grant TIN2010-14931, and by Basque Government grant IT-242-07. Dr.Mikel Luján holds a Royal Society University Research Fellowship. We appreciate the support of these sponsors and industrial partners.

Open Access

This article is distributed under the terms of the Creative Commons Attribution Noncommercial License which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.

References

  1. 1.
    Abeles M.: Local Cortical Circuits: An Electrophysiological Study. Springer, Berlin (1982)CrossRefGoogle Scholar
  2. 2.
    Ananthanarayanan, R., Modha, D.S.: Anatomy of a cortical simulator. In: Proceedings of 2007 ACM/IEEE International Conference on Supercomputing (SC’07), pp. 1–12 (2007)Google Scholar
  3. 3.
    Boahen K.A.: Point-to-point connectivity between neuromorphic chips using address events. IEEE Trans. Circuits Syst. 2 Analog Digit. Signal Process. 47(5), 416–434 (2000)MATHCrossRefGoogle Scholar
  4. 4.
    Cauwenberghs G.: An analog VLSI recurrent neural network learning a continuous-time trajectory. IEEE Trans. Neural Netw. 7(2), 346–361 (1996)CrossRefGoogle Scholar
  5. 5.
    Dally W.J., Seitz C.L.: Deadlock-free message routing in multiprocessor interconnection networks. IEEE Trans. Comput. C-36(5), 547–553 (1987)CrossRefGoogle Scholar
  6. 6.
    Davison, A.P., Brüderle, D., Eppler, J., Kremkow, J., Muller, E., Pecevski, D., Perrinet, L., Yger, P.: PyNN: a common interface for neuronal network simulators. Frontiers Neuroinform. 2(11) (2009)Google Scholar
  7. 7.
    Delorme A., Thorpe S.J.: SpikeNET: an event-driven simulation package for modelling large networks of spiking neurons. Network Comput. Neural Syst. 14(4), 613–627 (2003)CrossRefGoogle Scholar
  8. 8.
    Fieres, J., Schemmel, J., Meier, K.: Realizing biological spiking network models in a configurable wafer-scale hardware system. In: Proceedings of 2008 International Joint Conference on Neural Networks (IJCNN2008), pp. 969–976. IEEE Press, NJ (2008)Google Scholar
  9. 9.
    Furman B., White J., Abidi A.A.: CMOS analog IC implementing the back propagation algorithm. Neural Netw. 1(Supplement 1), 381 (1988)Google Scholar
  10. 10.
    Galluppi, F., Rast, A., Davies, S., Furber, S.: A General-purpose Model Translation System for a Universal Neural Chip. In: Proceedings of 2010 International Conference on Neural Information Processing (ICONIP 2010). Springer, Berlin (2010)Google Scholar
  11. 11.
    Gewaltig M.O., Diesmann M.: NEST (NEural Simulation Tool). Scholarpedia 2(4), 1430 (2007)CrossRefGoogle Scholar
  12. 12.
    Glackin, B., McGinnity, T.M., Maguire, L.P., Wu, Q.X., Belatreche, A.: A novel approach for the implementation of large scale spiking neural networks on FPGA hardware. In: Proceedings of 8th Internatioal Work Conference on Artificial Neural Networks (IWANN 2005), pp. 552–563. Springer, Berlin (2005)Google Scholar
  13. 13.
    Goodman, D., Brette, R.: Brian: a simulator for spiking neural networks in Python. Frontiers Neuroinform. 2(5) (2008)Google Scholar
  14. 14.
    Graf, H.P., Hubbard, W., Jackel, L.D., de Vegvar, P.G.N.: A CMOS Associative Memory Chip. In: Proceedings of IEEE First International Conference on Neural Networks, pp. 461–468 (1987)Google Scholar
  15. 15.
    Hagmann P., Cammoun L., Gigandet X., Meuli R., Honey C.J., Wedeen V.J., Sporns O.: Mapping the structural core of human cerebral cortex. PLoS Biology 6(7), 1479–1493 (2008)CrossRefGoogle Scholar
  16. 16.
    Harkin, J., Morgan, F., Hall, S., Dudek, P., Dowrick, T., McDaid, L.: Reconfigurable platforms and the challenges for large-scale implementations of spiking neural networks. In: Proceedings of 2008 International Conference on Field Programmable Logic and Applications (FPL 2008), pp. 483–486 (2008)Google Scholar
  17. 17.
    Hayon G., Abeles M., Lehmann D.: A model for representing the dynamics of a system of synfire chains. J. Comput. Sci. 18(1), 41–53 (2005)MathSciNetGoogle Scholar
  18. 18.
    Hines M.L., Carnevale N.T.: The NEURON simulation environment. Neural Comput. 9(6), 1179–1209 (1997)CrossRefGoogle Scholar
  19. 19.
    Holler, M., Tam, S., Castro, H., Benson, R.: An electrically trainable artificial neural network (ETANN) with 10240 “Floating Gate” Synapses. In: Proceedings of 1989 Internatiional Joint Conference on Neural Networks (IJCNN1989), pp. 191–196 (1989)Google Scholar
  20. 20.
    Indiveri G., Chicca E., Douglas R.: A VLSI array of low-power spiking neurons and bistable synapses with spike-timing dependent plasticity. IEEE Trans. Neural Netw. 17(1), 211–221 (2006)CrossRefGoogle Scholar
  21. 21.
    Izhikevich E.: Simple model of spiking neurons. IEEE Trans. Neural Netw. 14, 1569–1572 (2003)CrossRefGoogle Scholar
  22. 22.
    Izhikevich E., Edelman G.M.: Large-scale model of mammalian thalamocortical systems. Proc. Natl. Acad. Sci. USA 105(9), 3593–3598 (2008)CrossRefGoogle Scholar
  23. 23.
    Izhikevich E.M.: Which model to use for cortical spiking neurons. IEEE Trans. Neural Netw. 15(5), 1063–1070 (2004)CrossRefGoogle Scholar
  24. 24.
    James M., Hoang D.: Design of low-cost, real-time simulation systems for large neural networks. J. Parallel Distrib. Comput. 14(3), 221–235 (1992)CrossRefGoogle Scholar
  25. 25.
    Jin, X., , Luján, M., Khan, M.M., Plana, L.A., Rast, A.D., Welbourne, S.R., Furber, S.B.: Efficient parallel implementation of multilayer backpropagation network on torus-connected CMPs. In: Proceedings of 2010 ACM International Conference on Computing Frontiers (CF’10), pp. 89–90 (2010)Google Scholar
  26. 26.
    Jin, X., Furber, S., Woods, J.: Efficient modelling of spiking neural networks on a scalable chip multiprocessor. In: Proceedings of 2008 International Joint Conference on Neural Networks (IJCNN2008) (2008)Google Scholar
  27. 27.
    Jin, X., Rast, A., Galluppi, F., Khan, M.M., Furber, S.: Implementing learning on the SpiNNaker universal neural chip multiprocessor. In: Proceedings of 2009 International Conference on Neural Information Processing (ICONIP 2009). Springer, Berlin (2009)Google Scholar
  28. 28.
    Johansson C., Lansner A.: Towards cortex sized artificial neural systems. Neural Netw. 20(1), 48–61 (2007)MATHCrossRefGoogle Scholar
  29. 29.
    Khan, M., Lester, D., Plana, L., Rast, A., Jin, X., Painkras, E., Furber, S.: SpiNNaker: Mapping Neural Networks onto a Massively-Parallel Chip Multiprocessor. In: Proceedings of 2008 International Joint Conference on Neural Networks (IJCNN2008) (2008)Google Scholar
  30. 30.
    Lazzaro J., Wawrzynek J., Mahowald M., Silviotti M., Gillespie D.: Silicon auditory processors as computer peripherals. IEEE Trans. Neural Netw. 4(3), 523–528 (1993)CrossRefGoogle Scholar
  31. 31.
    Lee B.J., Sheu B.W.: General-purpose neural chips with electrically programmable synapses and gain-adjustable neurons. IEEE J. Solid-State Circuits 27(9), 1299–1302 (1992)CrossRefGoogle Scholar
  32. 32.
    Lytton W.H., Omurtag A., Neymotin S.A., Hines M.L.: Just-in-Time connectivity for large spiking networks. Neural Comput. 20(11), 2745–2756 (2008)MATHCrossRefGoogle Scholar
  33. 33.
    Maguire L., McGinnity T.M., Glackin B., Ghani A., Belatreche A., Harkin J.: Challenges for large-scale implementations of spiking neural networks on FPGAs. Neurocomputing 71(1–3), 13–29 (2007)CrossRefGoogle Scholar
  34. 34.
    Mattia M., Guidice P.D.: Efficient event-driven simulation of large networks of spiking neurons and dynamical synapses. Neural Comput. 12(10), 2305–2329 (2000)CrossRefGoogle Scholar
  35. 35.
    Mehrtash N., Jung D., Hellmich H., Schönauer T., Lu V.T., Klar H.: Synaptic plasticity in spiking neural networks (SP2INN): a System Approach. IEEE Trans. Neural Netw. 14(5), 980–992 (2003)CrossRefGoogle Scholar
  36. 36.
    Migliore M., Cannia C., Lytton W.W., Markram H., Hines M.L.: Parallel network simulations with NEURON. J. Comput. Neurosci. 21(2), 119–129 (2006)MathSciNetMATHCrossRefGoogle Scholar
  37. 37.
    Mouraud, A., Paugam-Moisy, H., Puzenat, D.: A distributed and multithreaded neural event driven simulation framework. In: Proceedings of IASTED International Conference on Parallel and Distributed Computing and Networks, pp. 212–217 (2006)Google Scholar
  38. 38.
    Nageswaran, J.M., Dutt, N., Krichmar, J.L., Nicolau, A.: A configurable simulation environment for the efficient simulation of large-scale spiking neural networks on graphics processors. Neural Netw. 22(5–6) (2007)Google Scholar
  39. 39.
    Navaridas, J., Luján, M., Miguel-Alonso, J., Plana, L.A., Furber, S.B.: Understanding the interconnection network of SpiNNaker. In: Proceedings of 23rd International Conference on Supercomputing (ICS’09), pp. 286–295 (2009)Google Scholar
  40. 40.
    Navaridas J., Miguel-Alonso J., Pascual J.A., Ridruejo F.J.: Simulating and evaluating interconnection networks with INSEE. Simul. Modell. Pract. Theory 19(1), 494–515 (2011)CrossRefGoogle Scholar
  41. 41.
    Navaridas, J., Plana, L.A., Miguel-Alonso, J., Luján, M., Furber, S.B.: SpiNNaker: impact of traffic locality, causality and burstiness on the performance of the interconnection Network. In: Proceedings of 2010 ACM Conference on Computing Frontiers (CF’10), pp. 11–19 (2010)Google Scholar
  42. 42.
    Orellana, C.G., Caballero, R.G., Velasco, H.M.G., Aligue, F.J.L.: NeuSim: a modular neural networks simulator for Beowulf clusters. In: Proceedings of 6th International Work-Conference on Artifical and Natural Neural Networks (IWANN 2001), Part II, pp. 72–79. Springer, Berlin (2001)Google Scholar
  43. 43.
    Pelayo F.J., Ros E., Arreguit X., Prieto A.: VLSI implementation of a neural model using spikes. Analog Integr. Circuits Signal Process. 13(1–2), 111–121 (1997)CrossRefGoogle Scholar
  44. 44.
    Plana L., Furber S., Temple S., Khan M., Shi Y., Wu J., Yang S.: A GALS infrastructure for a massively parallel multiprocessor. IEEE Design Test Comput. 24(5), 454–463 (2007)CrossRefGoogle Scholar
  45. 45.
    Porrmann, M., Witkowski, U., Kalte, H., Rückert, U.: Implementation of artificial neural networks on a reconfigurable hardware accelerator. In: Proceedings of 2002 Euromicro Conference on Parallel, Distributed, and Network-based processing, pp. 243–250 (2002)Google Scholar
  46. 46.
    Rast, A., Jin, X., Khan, M., Furber, S.: The deferred event model for hardware-oriented spiking neural networks. In: Proceedings of 2008 International Conference on Neural Information Processing (ICONIP 2008). Springer, Berlin (2009)Google Scholar
  47. 47.
    Rast, A., Khan, M.M., Jin, X., Plana, L.A., Furber, S.: A universal abstract-time platform for real-time neural networks. In: Proceedings of 2009 International Joint Conference on Neural Networks (IJCNN2009), pp. 2611–2618 (2009)Google Scholar
  48. 48.
    Rast, A., Welbourne, S., Jin, X., Furber, S.: Optimal connectivity in hardware-targetted MLP networks. In: Proceedings of 2009 International Joint Conference on Neural Networks (IJCNN2009), pp. 2619–2626 (2009)Google Scholar
  49. 49.
    Rast, A., Yang, S., Khan, M., Furber, S.: Virtual synaptic interconnect using an asynchronous network-on-chip. In: Proceedings of 2008 International Joint Conference on Neural Networks (IJCNN2008) (2008)Google Scholar
  50. 50.
    Rast, A.D., Galluppi, F., Jin, X., Furber, S.B.: The Leaky Integrate-and-Fire Neuron: A Platform for Synaptic Model Exploration on the SpiNNaker Chip. In: Proceedings of 2010 International Joint Conference Neural Networks (IJCNN2010), pp. 3959–3966 (2010)Google Scholar
  51. 51.
    Rast, A.D., Jin, X., Galluppi, F., Plana, L.A., Patterson, C., Furber, S.B.: Scalable event-driven native parallel processing: the SpiNNaker neuromimetic System. In: Proceedings of 2010 ACM Conference on Computing Frontiers (CF’10), pp. 20–29 (2010)Google Scholar
  52. 52.
    Rice, K.L., Vutsinas, C.N., Taha, T.M.: A preliminary investigation of a neocortex model implementation on the cray XD1. In: Proceedings of 2007 ACM/IEEE International Conference on Supercomputing (SC’07), pp. 1–8 (2007)Google Scholar
  53. 53.
    Ros E., Carrillo R., Ortigosa E.M.: Event-driven simulation scheme for spiking neural networks using lookup tables to characterize neuronal activity. Neural Comput. 18(12), 2959–2993 (2006)MATHCrossRefGoogle Scholar
  54. 54.
    Rückert, U.: ULSI architectures for artificial neural networks. In: Proceedings of 9th Euromicro Workshop on Parallel and Distributed Processing, pp. 436–442 (2001)Google Scholar
  55. 55.
    Steinkraus, D., Buck, I., Simard, P.Y.: Using GPUs for machine learning algorithms. In: Proceedings 8th International Conference on Document Analysis and Recognition, pp. 1115–1120 (2005)Google Scholar
  56. 56.
    Upegui A., Peña-Reyes C.A., Sanchez E.: An FPGA platform for on-line topology exploration of spiking neural networks. Microprocess. Microsyst. 29(5), 211–223 (2005)CrossRefGoogle Scholar
  57. 57.
    Vogelstein R.J., Mallik U., Vogelstein J.T., Cauwenberghs G.: Dynamically reconfigurable silicon array of spiking neurons with conductance-based synapses. IEEE Trans. Neural Netw. 18(1), 253–265 (2007)CrossRefGoogle Scholar
  58. 58.
    Watts, L.: Event-driven simulation of networks of spiking neurons. In: Advances in Neural Information Processing (NIPS) 6, pp. 927–934. Morgan Kaufmann Publishers, Waltham (1994)Google Scholar
  59. 59.
    Yasunaga, M., Masuda, N., Yagyu, M., Asai, M., Yamada, M., Masaki, A.: Design, Fabrication and Evaluation of a 5-inch Wafer Scale Neural Network LSI Composed of 576 Digital Neurons. In: Proceedings of 1990 International Joint Conference Neural Networks (IJCNN1990), pp. 527–535 (1990)Google Scholar
  60. 60.
    Yu T., Cauwenberghs G.: Analog VLSI biophysical neurons and synapses with programmable membrane channel kinetics. IEEE Trans. Biomed. Circuits Syst. 4(3), 139–148 (2010)CrossRefGoogle Scholar

Copyright information

© The Author(s) 2011

Authors and Affiliations

  • Alexander D. Rast
    • 1
  • Javier Navaridas
    • 1
  • Xin Jin
    • 1
  • Francesco Galluppi
    • 1
  • Luis A. Plana
    • 1
  • Jose Miguel-Alonso
    • 2
  • Cameron Patterson
    • 1
  • Mikel Luján
    • 1
  • Steve Furber
    • 1
  1. 1.School of Computer ScienceUniversity of ManchesterManchesterUK
  2. 2.School of Computer ScienceUniversity of the Basque CountrySan SebastianSpain

Personalised recommendations