A Quantitative Comparison of Different MLP Activation Functions in Classification

  • Emad A. M. Andrews Shenouda
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3971)


Multilayer perceptrons (MLP) has been proven to be very successful in many applications including classification. The activation function is the source of the MLP power. Careful selection of the activation function has a huge impact on the network performance. This paper gives a quantitative comparison of the four most commonly used activation functions, including the Gaussian RBF network, over ten real different datasets. Results show that the sigmoid activation function substantially outperforms the other activation functions. Also, using only the needed number of hidden units in the MLP, we improved its conversion time to be competitive with the RBF networks most of the time.


Activation Function Extreme Learn Machine Training Time Online Learning Radial Basis Function Neural Network 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Lu, B., Evans, B.L.: Channel Equalization by Feedforward Neural Networks. In: IEEE Int. Symposium on Circuits and Systems, Orlando, FL, vol. 5, pp. 587–590 (1999)Google Scholar
  2. 2.
    Michie, D., Spiegelhalter, D.J., Taylor, C.C.: Machine Learning, Neural and Statistical Classification. Elis Horwood, London (1994)MATHGoogle Scholar
  3. 3.
    Piekniewski, F., Rybicki, L.: Visual Comparison of Performance for Different Activation Functions in MLP Networks. In: IJCNN 2004 & FUZZ-IEEE, Budapest, vol. 4, pp. 2947–2953 (2004)Google Scholar
  4. 4.
    Dorffner, G.: A Unified Framework for MLPs and RBFNs: Introducing Conic Section Function Networks. Cybernetics and Systems 25(4), 511–554 (1994)CrossRefMathSciNetGoogle Scholar
  5. 5.
    Haykin, S.: Neural Networks A Comprehensive Introduction. Prentice Hall, New Jersey (1999)Google Scholar
  6. 6.
    Huang, G., Chen, Y., Babri, H.A.: Classification Ability of Single Hidden Layer Feedforward Neural Networks. IEEE Transactions on Neural Networks 11(3), 799–801 (2000)CrossRefGoogle Scholar
  7. 7.
    Le Cun, Y., Touresky, D., Hinton, G., Sejnowski, T.: A Theoretical Framework for Backpropagation. The Connectionist Models Summer School, 21–28 (1988)Google Scholar
  8. 8.
    Li, Y., Pont, M.J., Jones, N.B.: A Comparison of the Performance of Radial Basis Function and Multi-layer Perceptron Networks in Condition Monitoring and Fault Diagnosis. In: The International Conference on Condition Monitoring, Swansea, pp. 577–592 (1999)Google Scholar
  9. 9.
    Arahal, M.R., Camacho, E.F.: Application of the Ran Algorithm to the Problem of Short Term Load Forecasting. Technical Report, University of Sevilla, Sevilla (1996)Google Scholar
  10. 10.
    Finan, R.A., Sapeluk, A.T., Damper, R.I.: Comparison of Multilayer and Radial Basis Function Neural Networks for Text-Dependent Speaker Recognition. In: IEEE Int. Conf. on Neural Networks, Washington DC, vol. 4, pp. 1992–1997 (1996)Google Scholar
  11. 11.
    Karkkainen, T.: MLP in Layer-Wise Form with Applications to Weight Decay. Neural Computation 14(6), 1451–1480 (2002)CrossRefGoogle Scholar
  12. 12.
    Werbos, P.J.: Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences. Doctoral Thesis, Applied Mathematics. Harvard University. Boston (1974)Google Scholar
  13. 13.
    Wang, D., Huang, G.: Protein Sequence Classification Using Extreme Learning Machine. In: IJCNN 2005, Montréal, vol. 3, pp. 1406–1411 (2005)Google Scholar
  14. 14.
    Duch, W., Jankowski, N.: Survey of Neural Transfer Functions. Neural Computing Surveys 2, 163–212 (1999)Google Scholar
  15. 15.
    Duch, W., Jankowski, N.: Transfer functions: Hidden Possibilities for Better Neural Networks. In: 9th European Symposium on Artificial Neural Network, Bruges, pp. 81–94 (2001)Google Scholar
  16. 16.
    Hu, Y., Hwang, J.: Handbook of Neural Network Signal Processing, 3rd edn. CRC-Press, Florida (2002)Google Scholar
  17. 17.
    Zurada, J.M.: Introduction to Artificial Neural Systems. PWS Publishing, Boston (1999)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Emad A. M. Andrews Shenouda
    • 1
  1. 1.Department of Computer ScienceUniversity of TorontoTorontoCanada

Personalised recommendations