Generalized Entropy Cost Function in Neural Networks

  • Krzysztof GajowniczekEmail author
  • Leszek J. Chmielewski
  • Arkadiusz Orłowski
  • Tomasz Ząbkowski
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10614)


Artificial neural networks are capable of constructing complex decision boundaries and over the recent years they have been widely used in many practical applications ranging from business to medical diagnosis and technical problems. A large number of error functions have been proposed in the literature to achieve a better predictive power. However, only a few works employ Tsallis statistics, which has successfully been applied in other fields. This paper undertakes the effort to examine the \( q \)-generalized function based on Tsallis statistics as an alternative error measure in neural networks. The results indicate that Tsallis entropy error function can be successfully applied in the neural networks yielding satisfactory results.


Neural networks Tsallis entropy error function Classification 


  1. 1.
    Paliwal, M., Kumar, U.A.: Neural networks and statistical techniques: a review of applications. Expert Syst. Appl. 36(1), 2–17 (2009)CrossRefGoogle Scholar
  2. 2.
    Bishop, C.M.: Neural Networks for Pattern Recognition. Clarendon Press, Oxford (1995)zbMATHGoogle Scholar
  3. 3.
    Szupiluk, R., Wojewnik, P., Ząbkowski, T.: Prediction improvement via smooth component analysis and neural network mixing. In: Kollias, S., Stafylopatis, A., Duch, W., Oja, E. (eds.) ICANN 2006. LNCS, vol. 4132, pp. 133–140. Springer, Heidelberg (2006). doi: 10.1007/11840930_14 CrossRefGoogle Scholar
  4. 4.
    Gajowniczek, K., Ząbkowski, T.: Data mining techniques for detecting household characteristics based on smart meter data. Energies 8(7), 7407–7427 (2015)CrossRefGoogle Scholar
  5. 5.
    Ząbkowski, T., Szczesny, W.: Insolvency modeling in the cellular telecommunication industry. Expert Syst. Appl. 39, 6879–6886 (2012)CrossRefGoogle Scholar
  6. 6.
    Kalman, B.L., Kwasny, S.C.: A superior error function for training neural networks. In: International Joint Conference on Neural Networks, pp. 49–52 (1991)Google Scholar
  7. 7.
    White, H.: Artificial Neural Networks: Approximation and Learning Theory. Blackwell, Cambridge (1992)Google Scholar
  8. 8.
    Golik, P., Doetsch, P., Ney, H.: Cross-entropy vs squared error training: a theoretical and experimental comparison. In: 14th Annual Conference of the International Speech Communication Association “Interspeech-2013”, pp. 1756–1760, France (2013)Google Scholar
  9. 9.
    Waghmare, L.M., Bidwai, N.N., Bhogle, P.P.: Neural network weight initialization. In: Proceedings of IEEE International Conference on Mechatronics and Automation, pp. 679–681 (2007)Google Scholar
  10. 10.
    Ramos, E.Z., Nakakuni, M., Yfantis, E.: Quantitative measures to evaluate neural network weight initialization strategies. In: IEEE Computing and Communication Workshop and Conference (CCWC), pp. 1–7 (2017)Google Scholar
  11. 11.
    Falas, T., Stafylopatis, A.G.: The impact of the error function selection in neural network-based classifiers. In: International Joint Conference on Neural Networks, pp. 1799–1804 (1999)Google Scholar
  12. 12.
    Shamsuddin, S.M., Sulaiman, M.N., Darus, M.: An improved error signal for the backpropagation model for classification problems. Int. J. Comput. Math. 76(3), 297–305 (2001)CrossRefzbMATHMathSciNetGoogle Scholar
  13. 13.
    Narayan, S.: The generalized sigmoid activation function: competitive supervised learning. Inf. Sci. 99(1–2), 69–82 (1997)CrossRefMathSciNetGoogle Scholar
  14. 14.
    Kamruzzaman, J., Aziz, S.M.: A note on activation function in multilayer feedforward learning, In: Proceedings of the 2002 International Joint Conference on Neural Networks, IJCNN 2002, pp. 519–523 (2002)Google Scholar
  15. 15.
    Kline, D.M., Berardi, V.L.: Revisiting squared-error and cross-entropy functions for training neural network classifiers. Neural Comput. Appl. 14(4), 310–318 (2005)CrossRefGoogle Scholar
  16. 16.
    Picoli, S., Mendes, R.S., Malacarne, L.C., Santos, R.P.B.: Q-distributions in complex systems: a brief review. Braz. J. Phys. 39(2A), 468–474 (2009)CrossRefGoogle Scholar
  17. 17.
    Stosic, D., Stosic, D., Zanchettin, C., Ludermir, T., Stosic, B.: QRNN: q-generalized random neural network. IEEE Trans. Neural Netw. Learn. Syst. 28(2), 383–390 (2016)CrossRefGoogle Scholar
  18. 18.
    Tsallis, C.: Introduction to Nonextensive Statistical Mechanics. Springer, New York (2009)zbMATHGoogle Scholar
  19. 19.
    Cover, T.M., Thomas, J.A.: Elements of Information Theory. Wiley, New York (1991)CrossRefzbMATHGoogle Scholar
  20. 20.
    Dai, Y.H.: Convergence properties of the BFGS algorithm. SIAM J. Optim. 13(3), 693–701 (2002)CrossRefzbMATHMathSciNetGoogle Scholar
  21. 21.
    R Core Team: A language and environment for statistical computing, R Foundation for Statistical Computing, Vienna, Austria (2015)Google Scholar
  22. 22.
    Gajowniczek, K., Ząbkowski, T., Szupiluk, R.: Estimating the ROC curve and its significance for classification models’ assessment. Quant. Methods Econ. 15(2), 382–391 (2014)Google Scholar
  23. 23.
    Chrzanowska, M., Alfaro, E., Witkowska, D.: The individual borrowers recognition: single and ensemble trees. Expert Syst. Appl. 36(3), 6409–6414 (2009)CrossRefGoogle Scholar
  24. 24.
    Churn dataset. Last Accessed 12 Jan 2017
  25. 25.
    Hepatitis dataset. Last Accessed 12 Jan 2017
  26. 26.
    Gajowniczek, K., Ząbkowski, T., Orłowski, A.: Entropy based trees to support decision making for customer churn management. Acta Phys. Pol., A 129(5), 971–979 (2016)CrossRefGoogle Scholar
  27. 27.
    Gajowniczek, K., Karpio, K., Łukasiewicz, P., Orłowski, A., Ząbkowski, T.: Q-entropy approach to selecting high income households. Acta Phys. Pol., A 127(3A), 38–44 (2015)CrossRefGoogle Scholar
  28. 28.
    Gajowniczek, K., Ząbkowski, T., Orłowski, A.: Comparison of decision trees with renyi and tsallis entropy applied for imbalanced churn dataset. Ann. Comput. Sci. Inf. Syst. 5, 39–43 (2015)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Krzysztof Gajowniczek
    • 1
    Email author
  • Leszek J. Chmielewski
    • 1
  • Arkadiusz Orłowski
    • 1
  • Tomasz Ząbkowski
    • 1
  1. 1.Faculty of Applied Informatics and Mathematics – WZIMWarsaw University of Life Sciences – SGGWWarsawPoland

Personalised recommendations