Advertisement

Special-purpose digital hardware for neural networks: An architectural survey

  • Paolo Ienne
  • Thierry Cornu
  • Gary Kuhn
Article

Abstract

This paper presents a survey of digital systems to implement neural networks. We consider two basic options for designing these systems: parallel systems with standard digital components and parallel systems with custom processors. We describe many examples under each option, with an emphasis on commercially available systems. We report a first trend toward more general architectures and a second trend toward simple and fast structures. We discuss our experience in running a small ANN problem on two of these machines. After a reasonable programming effort, we obtain good convergence, but most of the training times are actually slower or moderately faster than on a serial workstation. We conclude that it is important to chose one's problems carefully, and that support software and in general, system integration, is only beginning to reach the level of versatility that many researchers will require.

Keywords

Neural Network Processing Element Systolic Array Custom Processor Arithmetic Unit 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    M. Glesner and W. Pöchmüller,An Overview of Neural Networks in VLSI, Chapman & Hall. London, 1994.zbMATHGoogle Scholar
  2. 2.
    C.S. Lindsey and T. Lindblad, “Review of hardware neural networks: A user's perspective,” inProceedings of the Third Workshop on Neural Networks: From Biology to High Energy Physics, Isola d'Elba, Italy, Sept. 1993.Google Scholar
  3. 3.
    A. König, ‘Survey and current status of neural network hard-ware,” inProceedings of the International Conference on Artificial Neural Networks, F. Fogelman-Soulie and P. Gallinari, (Eds.), Paris, Oct. 1995, pp. 391–410.Google Scholar
  4. 4.
    S. Oteki, A. Hashimoto, T. Furuta, T. Watanabe, D.G. Stork, and H. Eguchi, “A digital neural network VLSI with on-chip learning using stochastic pulse encoding,” inProceedings of the International Joint Conference on Neural Networks, Nagoya, Japan, Oct. 1993, Vol. III, pp. 3039–45.Google Scholar
  5. 5.
    N. Mauduit, M. Duranton, J. Gobert, and J.-A. Sirat, “Lneuro 1.0: A piece of hardware LEGO for building neural network systems,”IEEE Transactions on Neural Networks, Vol. NN-3, No. 3, pp. 414–22, May 1992.CrossRefGoogle Scholar
  6. 6.
    M. Yasunaga, N. Masuda, M. Yagyu, M. Asai, K. Shibata, M. Ooyama, M. Yamada, T. Sakaguchi, and M. Hashimoto, “A self-learning neural network composed of 1152 digital neurons in wafer-scale LSIs,” inProceedings of the International Joint Conference on Neural Networks, Seattle, Wash., July 1991, pp. 1844–49.Google Scholar
  7. 7.
    C. Park, K. Buckmann, J. Diamond, U. Santoni, S.C. The, M. Holler, M. Glier, C.L. Scofield, and L. Nunez, “A radial basis function neural network with on-chip learning,” inProceedings of the International Joint Conference on Neural Networks, Nagoya, Japan, Oct. 1993, Vol. III, pp. 3035–38.Google Scholar
  8. 8.
    P. Ienne and M.A. Viredaz, “GENES IV: A bit-serial processing element for a multi-model neural-network accelerator,”Journal of VLSI Signal Processing, Vol. 9, No. 3, pp. 257–73, Apr. 1995.CrossRefGoogle Scholar
  9. 9.
    N. Brüls, “Programmable VLSI array processor for neural networks and matrix-based signal processing—User description,” Technical Report, Siemens AG, Corporate Research and Development Division, Munich, Version 1.3, Oct. 1993.Google Scholar
  10. 10.
    M.A. Glover and W.T. Miller, III, “A massively-parallel SIMD processor for neural network and machine vision applications,” inAdvances in Neural Information Processing Systems, J.D Cowan, G. Tesauro, and J. Alspector (Eds.), Morgan Kaufmann, San Mateo, Calif., Vol. 6, pp. 843–49, 1994.Google Scholar
  11. 11.
    D. Hammerstrom, “A highly parallel digital architecture for neural network emulation,”VLSI for Artificial Intelligence and Neural Networks, J.G. Delgado-Frias and W.R. Moore (Eds.), Chap. 5.1, pp. 357–66, Plenum Press, New York, 1991.CrossRefGoogle Scholar
  12. 12.
    R.W. Means and L. Lisenbee, “Floating-point SIMD neurocomputer array processor,”Parallel Implementations of Neural Networks, K.W. Prztula and V.K. Prasanna (Eds.), Prentice Hall, New York, 1993.Google Scholar
  13. 13.
    J. Wawrzynek, K. Asanovié, B. Kingsbury, J. Beck, D. Johnson, and N. Morgan, “SPERT-II: A vector microprocessor system,”Computer, Vol. 29, No. 3, Mar. 1996.Google Scholar
  14. 14.
    N. Morgan, J. Beck, P. Kohn, J. Bilmes, E. Allman, and J. Beer, “The Ring Array Processor: A multiprocessing peripheral for connectionist applications,”Journal of Parallel and Distributed Computing, Vol. 14, No. 3, pp. 248–59, Mar. 1992.CrossRefGoogle Scholar
  15. 15.
    U.A. Müller, A. Gunzinger, and W. Guggenbühl, “Fast neural net simulation with a DSP processor array,”IEEE Transactions on Neural Networks, Vol. NN-6, No. 1, pp. 203–13, Jan. 1995.CrossRefGoogle Scholar
  16. 16.
    D.E. Rumelhart, G.E. Hinton, and R.J. Williams, “Learning internal representations by errorpropagation,”Parallel Distributed Processing: Explorations in the Microstructure of Cognition, D.E. Rumelhart and J.L. McClelland (Eds.), Vol. 1. Foundations, MIT Press, Cambridge, Mass., Chap. 8, 1986.Google Scholar
  17. 17.
    G. Kuhn and N. Herzberg, “Some variations on training of recurrent networks,”Neural Networks: Theory and Applications, R. Mammone and Y. Zeevi (Eds.), Academic Press, San Diego, Calif., Chap. 11, pp. 233–44, 1991.Google Scholar
  18. 18.
    R. Battiti, “First- and second-order methods for learning: Between steepest descent and Newton's method,”Neural Computation, Vol. 4, pp. 141–66, 1992.CrossRefGoogle Scholar
  19. 19.
    E.M. Johansson, F.U. Dowla, and D.M. Goodman, “Backpropagation learning for multilayer feed-forward neural networks using the conjugate gradient method,”International Journal of Neural Systems, Vol. 2, No. 4, pp. 291–301, 1992.CrossRefGoogle Scholar
  20. 20.
    A.S. Weigend, B.A. Huberman, and D.E. Rumelhart, “Predicting the future: A connectionist approach,”International Journal of Neural Systems, Vol. 1, pp. 193–209, 1990.CrossRefGoogle Scholar
  21. 21.
    K. Asanović, K. Beck, T. Callahan, J. Feldman, B. Irissou, B. Kingsbury, P. Kohn, J. Lazzaro, K. Morgan, D. Stoutamire, and J. Wawrzynek, “CNS-1 architecture specification: A connectionist network supercomputer,” Technical Report, TR-93-021, International Computer Science Institute, Berkeley, Calif., Apr. 1993.Google Scholar
  22. 22.
    A.J. De Groot and S.R. Parker, “Systolic implementation of neural networks,” inHigh Speed Computing II, K. Bromley (Ed.), SPIE—The International Society for Optical Engineering, Los Angeles, Calif., Vol. 1058. pp. 182–90, Jan. 1989.CrossRefGoogle Scholar
  23. 23.
    D.A. Pomerleau, G.L. Gusciora, D.S. Touretzky, and H.T. Kung, “Neural network simulation at warp speed: How we got 17 million connections per second,” inProceedings of the IEEE Conference on Neural Networks, San Diego, Calif., 1988, Vol. II, pp. 143–50.Google Scholar
  24. 24.
    K. Asanović and N. Morgan, “Experimental determination of precision requirements for back-propagation training of artificial neural networks,” inProceedings of the Second International Conference on Microelectronics for Neural Networks, Munich, 1991, pp. 9–15.Google Scholar
  25. 25.
    J.L. Holt and T.E. Baker, “Back propagation simulations using limited precision calculation,” inProceedings of the International Joint Conference on Neural Networks, Seattle, Wash., July 1991.Google Scholar
  26. 26.
    P. Thiran, V. Peiris, P. Heim, and B. Hochet, “Quantization effects in digitally behaving circuit implementations of Kohonen networks,”IEEE Transactions on Neural Networks, Vol. NN-5, No. 3, pp. 450–58, May 1994.CrossRefGoogle Scholar
  27. 27.
    F. Castillo, J.A. García, J.M. Moreno, and J. Cabestany, “A coprocessor card for fast neural network emulation,” inFrom Natural to Artificial Neural Computation, J. Mira and F. Sandoval (Eds.),Lecture Notes in Computer Science, Vol. 930, pp. 752–60, Springer, Berlin, 1995.CrossRefGoogle Scholar
  28. 28.
    M. Griffin, G. Tahara, K. Knorpp, and B. Riley, “An 11-million transistor neural network execution engine,”IEEE International Conference on Solid-State Circuits, pp. 180–81, 1991.Google Scholar
  29. 29.
    inova microelectronics corporation,N64000 Digital Neural Network Processor (Preliminary Data), Santa Clara, Calif.Google Scholar
  30. 30.
    H. McCartor, “Back propagation implementation on the adaptive solutions neurocomputer chip,”Advances in Neural Information Processing Systems, D.S. Touretzky (Ed.), Morgan Kaufmann, San Mateo, Calif., Vol. 3, 1991.Google Scholar
  31. 31.
    E. Franzi, “Neural accelerator for parallelization of backpropagation algorithm,”Microprocessing and Microprogramming, Vol. 38, pp. 689–96, 1993.CrossRefGoogle Scholar
  32. 32.
    P. Murtagn and A.C. Tsoi, “Implementation issues of sigmoid function and its derivative for VLSI digital neural networks,”IEEE Proceedings, Vol. 139, No. 3, pp. 207–14, May 1992.Google Scholar
  33. 33.
    D. Hammerstrom and N. Nguyen, “An implementation of Kohonen's self-organizing map on the adaptive solutions neurocomputer,” inProceedings of the International Conference on Artificial Neural Networks, Helsinki, June 1991, Vol. I, pp. 715–20.Google Scholar
  34. 34.
    Y. Sato, K. Shibata, M. Asai, M. Ohki, M. Sugie, T. Sakaguchi, M. Hashimoto, and Y. Kuwabara, Yoshihiro, “Development of a high-performance general purpose neuro-computer composed of 512 digital neurons,” inProceedings of the International Joint Conference on Neural Networks, Nagoya, Japan, Oct. 1993, Vol. II, pp. 1967–70.Google Scholar
  35. 35.
    U. Ramacher, J. Beichter, W. Raab, J. Anlauf, N. Brüls, U. Hachmann, and M. Wesseling, “Design of a 1st generation neurocomputer,” inVLSI Design of Neural Networks, Kluwer Academic, Norwell, Mass., pp. 271–310, 1991.CrossRefGoogle Scholar
  36. 36.
    U. Ramacher and J. Beichter, “Systolic architectures for fast emulation of artificial neural networks,” inSystolic Array Processors, J. McCanny, J. McWhirter, and E. Swartzlander, Jr. (Eds.), Prentice Hall, New York, pp. 277–86, 1989.Google Scholar
  37. 37.
    U. Ramacher and P. Nachbar, “Hamiltonian dynamics of neural networks,” inProceedings of the Second International Conference on Microelectronics for Neural Networks, Munich, 1991, pp. 95–102.Google Scholar
  38. 38.
    U. Ramacher, W. Raab, J. Anlauf, U. Hachmann, and M. Weßeling, “SYNAPSE-1—A general-purpose neurocomputer,” Technical Report, Munich, Siemens AG, Corporate Research and Development Division, Feb. 1993.Google Scholar
  39. 39.
    M. Viredaz, “Design and analysis of a systolic array for neural computation,” École Polytechnique Fédérale de Lausanne, Lausanne, Ph.D. Thesis No 1264, 1994.Google Scholar
  40. 40.
    M.A. Viredaz and P. Ienne, “MANTRA I: A systolic neurocomputer,” inProceedings of the International Joint Conference on Neural Networks, Nagoya, Japan, Oct. 1993, Vol. III, pp. 3054–57.Google Scholar
  41. 41.
    Y.I. Fet, “Vertical processing systems: A survey,”IEEE Micro, Vol. M-15, No. 1, pp. 65–75, Feb. 1995.CrossRefGoogle Scholar
  42. 42.
    T.G. Clarkson, D. Gorse, J.G. Taylor, and C.K. Ng, “Learning probabilistic RAM nets using VLSI structures,”IEEE Transactions on Computers, Vol. C-41, No. 12, pp. 1552–61, Dec. 1992.CrossRefGoogle Scholar
  43. 43.
    M. Yasunaga, N. Masuda, M. Yagyu, M. Asai, M. Yamada, and A. Masaki, “Design, fabrication and evaluation of a 5-inch wafer scale neural network LSI composed of 576 digital neurons,” inProceedings of the International Joint Conference on Neural Networks, San Diego, Calif., June 1990, Vol. II, pp. pp527–36.Google Scholar
  44. 44.
    M. Fujita, Y. Kobayashi, K. Shiozawa, T. Takahashi, F. Mizuno, H. Hayakawa, M. Kato, S. Mori, T. Kase, and M. Yamada, “Development and fabrication of digital neural network WSIs,”IEICE Transactions on Electronics, Vol. E 76-C, No. 7, pp. 1182–90, July 1993.Google Scholar
  45. 45.
    J.B. Theeten, M. Duranton, N. Mauduit, and J.A. Sirat, “The Lneuro chip: A digital VLSI with on-chip learning mechanism,”Proceedings of the International Neural Networks Conference, pp. 593–96, 1990.Google Scholar
  46. 46.
    Nestor Inc., R.I. Providence,Ni1000 Recognition Accelerator Datasheet, 1994.Google Scholar
  47. 47.
    Intel Corp.,Ni1000 Datasheet (Preliminary), Santa Clara, Calif., 1993.Google Scholar
  48. 48.
    C.L. Scofield and D.L. Reilly, “Into Silicon: Real time learning in a high density RBF neural network,” inProceedings of the International Joint Conference on Neural Networks, Seattle, Wash., July 1991, Vol. I, pp. 551–56.Google Scholar
  49. 49.
    D.F. Spec, “Enhancements to probabilistic neural networks,” inProceedings of the International Joint Conference on Neural Networks, Baltimore, Md., June 1992, Vol. I, pp. 761–68.Google Scholar
  50. 50.
    IBM.ZISC036 Data Book (Preliminary), Version 2.1, Nov. 1994.Google Scholar
  51. 51.
    M.P. Perrone and L.N. Cooper, “The Ni1000: High speed parallel VLSI for implementing multilayer perceptrons,” inAdvances in Neural Information Processing Systems, J.D. Cowan, G. Tesauro, and J. Alspector (Eds.), Vol. 8, 1995.Google Scholar
  52. 52.
    S.R. Jones and K.M. Sammut, “Learning in linear systolic neural network engines: Analysis and implementation,”IEEE Transactions on Neural Networks, Vol. NN-5, No. 4, pp. 584–93, July 1994.CrossRefGoogle Scholar
  53. 53.
    M. Duranton, “L-Neuro 2.3: A VLSI for image processing by neural networks,” inProceedings of the Fifth International Conference on Microelectronics for Neural Networks and Fuzzy Systems, Lausanne, Feb. 1996.Google Scholar
  54. 54.
    Cray Research Inc.,CRAY T3D System Architecture Overview, HR-04033 edition, Sept. 1993.Google Scholar
  55. 55.
    T. Kohonen,Self-Organization and Associative Memory, Springer Series in Information Sciences, Springer, Berlin, third edition, Vol. 8, 1989.CrossRefGoogle Scholar
  56. 56.
    T. Cornu and P. Ienne, “Performance of digital neuro-computers,” inProceedings of the Fourth International Conference on Microelectronics for Neural Networks and Fuzzy Systems, Turin, Sept. 1994, pp. 87–93.Google Scholar
  57. 57.
    P. Ienne, P. Thiran, and N. Vassilas, “Modified self-organising feature map algorithms for efficient digital hardware implementation,”IEEE Transactions on Neural Networks, Submitted paper, 1995.Google Scholar
  58. 58.
    N. Vassilas, P. Thiran, and P. Lenne, “On modifications of Kohonen's feature map algorithm for an efficient parallel implementation,” inProceedings of the International Conference on Neural Networks, Washington, D.C., June 1996.Google Scholar
  59. 59.
    A. Grama, A. Gupta, and V. Kumar, “Isoefficiency function: A scalability metric for parallel algorithms and architectures,”IEEE Parallel and Distributed Technology, Vol. 1, No. 3, pp. 12–21, 1993.CrossRefGoogle Scholar

Copyright information

© Kluwer Academic Publishers 1996

Authors and Affiliations

  • Paolo Ienne
    • 1
  • Thierry Cornu
    • 2
  • Gary Kuhn
    • 3
  1. 1.Microcomputing Laboratory, IN-F EcublensSwiss Federal Institute of TechnologyLausanneSwitzerland
  2. 2.Mantra Centre for Neuromimetic Systems, IN-J EcublensSwiss Federal Institute of TechnologyLausanneSwitzerland
  3. 3.SCR AISP DepartmentPrincetonUSA

Personalised recommendations