Advertisement

Springer Nature is making SARS-CoV-2 and COVID-19 research free. View research | View latest news | Sign up for updates

Low-error, High-speed Approximation of the Sigmoid Function for Large FPGA Implementations

Abstract

There has been much study of ASIC neurocomputers but, in comparison, relatively little for FPGA neurocomputers. Nevertheless, with current (and future) dense, high-speed FPGAs, the latter are now viable and will be more successful than the former. In this paper, we discuss a technique for low-error, high-speed implementations of the sigmoid function in such FPGAs. This function is commonly used as an activation function in artificial neural networks, but it also has applications in many other areas. Our results compare very favourably with others that have been reported in the published literature.

This is a preview of subscription content, log in to check access.

References

  1. 1.

    A. R. Omondi and J. Rajapakse, “Neural Networks in FPGAs”, in Proc. of the 9th International Conference on Neural Information Processing (ICONIP), Singapore, 18–22 November, 2002, pp. 954–959.

  2. 2.

    A. R. Omondi, “Neurocomputers: A Dead End?”, Int. J. Neural. Syst., vol. 10, 2000, pp. 475–481.

  3. 3.

    M. Zhang, S. Vassiliadis, and J. G. Delgado-Frias, “Sigmoid Generators for Neural Computing Using Piecewise Approximations,” IEEE Trans. Comput., vol. 45, 1996, pp. 1045–1049.

  4. 4.

    L. Fang and N. Nagarajan, “A back-up onboard orbit propagator using neural networks for small satellite missions”, in Proc. of the 23rd International Symposium on Space Technology and Science, Matsue, Japan, May 26–June 2, 2002.

  5. 5.

    J. Wawrzynek et al., “SPERT II: A Vector Microprocessor System”, IEEE Computer, vol. 29, 1996, pp. 79–86.

  6. 6.

    J. L. Ayala, A. G. Lomeña, M. López-Vallejo, and A. Fernández, “Design of a pipelined hardware architecture for real-time neural network applications”, in Proc. of the IEEE Midwest Symposium on Circuits and Systems, Tulsa, USA, 2002.

  7. 7.

    J. M. Muller, “Elementary Functions: Algorithms and Implementation”, Birkhauser, Boston, USA, 1997.

  8. 8.

    A. R. Omondi, “Computer Arithmetic Systems: Algorithms, Architecture, and Implementations”, Prentice-Hall, UK, 1994.

  9. 9.

    H. Amin, M. K. Curtis, and R. B. Hayes-Gill, “Piecewise Linear Approximation Applied to Non-linear Activation Functions of a Neural Network”, IEE Proc. Circ. Devices Syst., vol. 144, 1997, pp. 313–317.

  10. 10.

    C. Alippi and G. Storti-Gajani, “Simple approximations of sigmoidal functions: realistic design of digital neural networks capable of learning”, in Proceedings IEEE Int. Symp. on Circuits and Systems, Singapore, 11–14 June, 1991, pp. 1505–1508.

  11. 11.

    K. M. Sammut and S. R. Jones, “Implementing Non-linear Activation Functions in Neural Network Emulators”, Electron. Lett., vol. 27, 1991, pp. 1037–1038.

  12. 12.

    K. Basterretxea, J. M. Tarela, and I. del Campo, “Approximation of Sigmoid Function for Hardware Implementation of Artificial Neurons”, IEE Proc. Circ. Devices Syst., vol. 151, 2004, pp. 18–24.

  13. 13.

    D. J. Myers and R. A. Hutchison, “Efficient Implementation of Piecewise Linear Activation for Digital VLSI Neural Networks”, Electron. Lett., vol. 25, 1989, pp. 1662–1663.

  14. 14.

    M. Al-Nsour and H. S. Abdel-Aty-Zahdy, “Implementation of programmable digital sigmoid function circuit for neuro-computing”, in Proc. of the IEEE Midwest Symposium on Circuits and Systems, 1998, pp. 571–574.

  15. 15.

    Virtex-4 User Guide, Xilinx, 2004.

  16. 16.

    O. Mencer and W. Luk, “Parameterized High Throughput Function Evaluation for FPGAs”, J. VLSI Signal Process., vol. 36, 2004, pp. 17–25.

  17. 17.

    S. M. Pizer and V. L. Wallace, “To Compute Numerically: Concepts and Strategies”, Little, Brown, Boston, USA, 1983.

  18. 18.

    S. Vassiliadis, M. Zhang and J. G. Delgado-Frias, “Elementary Function Generators for Neural-Network Emulators”, IEEE Trans. Neural Netw., vol. 11, 2000, pp. 1438–1449.

  19. 19.

    M. T. Tommiska, “Efficient Digital Implementation of Sigmoid Function for Reprogrammable Logic”, IEE Proc. Comput. Digit. Tech., vol. 150, 2003, pp. 403–411.

  20. 20.

    H. K. Kwan, “Simple Sigmoid-Like Activation Function Suitable for Digital Hardware Implementation”, Electron. Lett., vol. 29, 1992, pp. 1379–1380.

  21. 21.

    XtremeDSP Design Considerations: User Guide, Xilinx, 2004.

  22. 22.

    M. Bajger and A.R. Omondi, “Implementations of the square-root and exponential functions for large FPGAs”, in Proc. of the 11th Asia-Pacific Computer Systems Architecture Conference, Shanghai, 6–8 September, 2006, LNCS 4186, pp. 6–23, Springer-Verlag.

  23. 23.

    Maple 8 Programming Guide, Waterloo Maple, 2002.

  24. 24.

    R. M. Corless, G. H. Gonnet, D. E. G. Hare, D. J. Jeffrey, and D. E. Knuth, “On the Lambert W Function”, Adv. Comput. Math., vol. 12, 1996, pp. 329–359.

  25. 25.

    M.J. Flynn and S.F. Oberman. “Advanced Computer Arithmetic Design”, Wiley, New York, 2001.

  26. 26.

    A. Peng and A.R. Omondi. “FPGA implementations of CORDIC algorithms”, in 11th International Conference on Signal Processing Applications and Technology, Dallas, Texas, 2000.

Download references

Author information

Correspondence to Amos Omondi.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Bajger, M., Omondi, A. Low-error, High-speed Approximation of the Sigmoid Function for Large FPGA Implementations. J Sign Process Syst Sign Image Video Technol 52, 137–151 (2008). https://doi.org/10.1007/s11265-007-0140-z

Download citation

Keywords

  • artificial neural network
  • FPGA
  • sigmoid
  • arithmetic