An Efficient Hardware Architecture for a Neural Network Activation Function Generator

  • Daniel Larkin
  • Andrew Kinane
  • Valentin Muresan
  • Noel O’Connor
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3973)


This paper proposes an efficient hardware architecture for a function generator suitable for an artificial neural network (ANN). A spline-based approximation function is designed that provides a good trade-off between accuracy and silicon area, whilst also being inherently scalable and adaptable for numerous activation functions. This has been achieved by using a minimax polynomial and through optimal placement of the approximating polynomials based on the results of a genetic algorithm. The approximation error of the proposed method compares favourably to all related research in this field. Efficient hardware multiplication circuitry is used in the implementation, which reduces the area overhead and increases the throughput.


Activation Function Hardware Implementation Hardware Architecture Area Overhead Range Reduction 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Widrow, B., Rumelhart, D.E., Lehr, M.A.: Neural Networks: Applications in Industry, Business and Science. Communications of the ACM 37, 93–105 (1994)CrossRefGoogle Scholar
  2. 2.
    Cauwenberghs, G., Bayoumi, M.: Learning on Silicon - Adaptive VLSI Neural Systems. Kluwer Academic, Dordrecht (1999)MATHGoogle Scholar
  3. 3.
    Zhang, D., Pal, S.K.: Neural Networks and Systolic Array Design. World Scientific, New Jersey (2002)MATHCrossRefGoogle Scholar
  4. 4.
    Ruckert, U.: ULSI Architectures for Artificial Neural Aetworks. IEEE Micro 22, 10–19 (2002)CrossRefGoogle Scholar
  5. 5.
    Draghici, S.: On the Capabilities of Neural Networks using Limited Precision Weights. Neural Networks 15, 395–414 (2002)CrossRefGoogle Scholar
  6. 6.
    Holt, J., Hwang, J.: Finite Precision Error Analysis of Neural Network Hardware Implementations. IEEE Transactions on Computers 42, 280–291 (1993)Google Scholar
  7. 7.
    Haykins, S.: Neural Networks - A Comprehensive Foundation. Prentice-Hall, Englewood Cliffs (1999)Google Scholar
  8. 8.
    Myers, D.J., Hutchinson, R.: Efficient Implementation of Piecewise Linear Activation Function for Digital VLSI Neural Networks. Electronics Letters 25, 1662–1663 (1989)CrossRefGoogle Scholar
  9. 9.
    Alippi, C., Storti-Gajani, G.: Simple Approximation of Sigmoid Functions: Realistic Design of Digital VLSI Neural Networks. In: Proceedings of the IEEE Int’l Symp. Circuits and Systems, pp. 1505–1508 (1991)Google Scholar
  10. 10.
    Amin, H., Curtis, K., Hayes Gill, B.: Piecewise Linear Approximation applied to Non-Linear Function of a Neural Network. IEE Proceedings Circuits, Devices and Systems 144, 313–317 (1997)CrossRefGoogle Scholar
  11. 11.
    Vassiliadis, S., Zhang, M., Delgado-Frias, J.: Elementary Function Generators for Neural-Network Emulators. IEEE Transactions on Neural Networks 11, 1438–1449 (2000)CrossRefGoogle Scholar
  12. 12.
    Faiedh, H., Gafsi, Z., Besbes, K.: Digital Hardware Implementation of Sigmoid Function and its Derivative for Artificial Neural Networks. In: Proceedings of the 13th International Conference on Microelectronics, pp. 189–192. Rabat, Morocco (2001)Google Scholar
  13. 13.
    Basterretxea, K., Tarela, J., del Campo, I.: Approximation of Sigmoid Function and the Derivative for Hardware Implementation of Artificial Neurons. IEE Proceedings of Circuits, Devices and Systems 151, 18–24 (2004)CrossRefGoogle Scholar
  14. 14.
    Tommiska, M.: Efficient Digital Implementation of the Sigmoid Function for Reprogrammable Logic. IEE Proceedings Computers and Digital Techniques 150, 403–411 (2003)CrossRefGoogle Scholar
  15. 15.
    Pineiro, J., Oberman, S., Muller, J., Bruguera, J.: High-Speed Function Approximation Using a Minimax Quadratic Interpolator. IEEE Transactions on Computers 54, 304–318 (2005)CrossRefGoogle Scholar
  16. 16.
    The MathWorks: MATLAB and Simulink for Technical Computing (2006),
  17. 17.
    Koren, I.: Computer Arithmetic Algorithms, 2nd edn. A K Peters Ltd, Wellesley (2001)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Daniel Larkin
    • 1
  • Andrew Kinane
    • 1
  • Valentin Muresan
    • 1
  • Noel O’Connor
    • 1
  1. 1.Centre for Digital Video ProcessingDublin City UniversityDublin 9Ireland

Personalised recommendations