Abstract
Artificial neural networks represent an important class of methods for fitting nonlinear regression to data with an unknown regression function. However, usual ways of training of the most common types of neural networks applied to nonlinear regression tasks suffer from the presence of outlying measurements (outliers) in the data. So far, only a few robust alternatives for training common forms of neural networks have been proposed. In this work, we robustify two common types of neural networks by considering robust versions of their loss functions, which have turned out to be successful in linear regression. Particularly, we extend the idea of using the loss of the least trimmed squares estimator to radial basis function networks. We also propose multilayer perceptrons and radial basis function networks based on the loss of the least weighted squares estimator. The performance of these novel methods is compared with that of standard neural networks on 4 datasets. The results bring arguments in favor of the novel robust approach based on the least weighted squares estimator with trimmed linear weights in terms of yielding the smallest robust prediction error in a variety of situations. Robust neural networks are even able to outperform the prediction ability of support vector regression.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Alnafessah, A., Casale, G.: Artificial neural networks based techniques for anomaly detection in Apache Spark. Cluster Computing (2020) (online first)
Borş, A.G., Pitas, I.: Robust RBF networks. In: Howlett, R.J., Jain, L.C., Kacprzyk, J. (eds.), Radial basis function networks 1. Recent developments in theory and applications, pp. 123–133. Physica Verlag Rudolf Liebing KG, Vienna (2001)
Chollet, F.: Keras. Github repository (2015). https://github.com/fchollet/keras
Čížek, P.: Semiparametrically weighted robust estimation of regression models. Comput. Stat. Data Anal. 55, 774–788 (2011)
Čížek, P.: Reweighted least trimmed squares: an alternative to one-step estimators. Test 22, 514–533 (2013)
Davies, L.: Data analysis and approximate models. In: Nonparametric Regression and Image Analysis. CRC Press, Boca Raton, Model Choice, Location-scale, Analysis of Variance (2014)
Du, K.L., Swamy, M.N.S.: Neural Networks and Statistical Learning. Springer, London (2014)
Frank, A., Asuncion, A.: UCI Machine Learning Repository. University of California, Irvine (2010). http://archive.ics.uci.edu/ml
Hastie, T., Tibshirani, R., Friedman, J.: The Elements of Statistical Learning, 2nd edn. Springer, New York (2009)
Haykin, S.O.: Neural Networks and Learning Machines: A Comprehensive Foundation, 3rd edn. Prentice Hall, Upper Saddle River (2009)
Hubert, M., Rousseeuw, P.J., van Aelst, S.: High-breakdown robust multivariate methods. Stat. Sci. 23, 92–119 (2008)
Jurečková, J., Picek, J., Schindler, M.: Robust Statistical Methods with R, 2nd edn. Chapman & Hall/CRC, Boca Raton (2019)
Kalina, J.: Implicitly weighted methods in robust image analysis. J. Math. Imag. Vis. 44, 449–462 (2012)
Kalina, J.: A robust supervised variable selection for noisy high-dimensional data. BioMed. Res. Int. Article 320385 (2015)
Kalina, J., Tichavský, J.: On robust estimation of error variance in (highly) robust regression. Meas. Sci. Rev. 20, 1–9 (2020)
Kalina, J., Vidnerová, P.: Robust training of radial basis function neural networks. In: Proceedings 18th International Conference ICAISC 2019, pp. 113–124 (2019)
Kordos, M., Rusiecki, A.: Reducing noise impact on MLP training—techniques and algorithms to provide noise-robustness in MLP network training. Soft Comput. 20, 46–65 (2016)
Lee, C.C., Chung, P.C., Tsai, J.R., Chang, C.I.: Robust radial basis function neural networks. IEEE Trans. Syst. Man Cybern. B 29, 674–685 (1999)
Mašíček, L.: Optimality of the least weighted squares estimator. Kybernetika 40, 715–734 (2004)
R Core Team: R: A language and environment for statistical computing. In: R Foundation for Statistical Computing, Vienna (2019). https://www.R-project.org/
Roelant, E., van Aelst, S., Willems, G.: The minimum weighted covariance determinant estimator. Metrika 70, 177–204 (2009)
Rousseeuw, P.J., Hubert, M.: Robust statistics for outlier detection. WIREs Data Mining Knowl. Discov. 1, 73–79 (2011)
Rousseeuw, P.J., Van Driessen, K.: Computing LTS regression for large data sets. Data Mining Knowl. Discov. 12, 29–45 (2006)
Rusiecki, A.: Robust learning algorithm based on LTA estimator. Neurocomputing 120, 624–632 (2013)
Rusiecki, A., Kordos, M., Kamiński, T., Greń, K.: Training neural networks on noisy data. Lect. Notes Artif. Intel. 8467, 131–142 (2014)
Su, M.J., Deng, W.: A fast robust learning algorithm for RBF network against outliers. Lect. Notes Comput. Sci. 4113, 280–285 (2006)
Víšek, J.Á.: The least trimmed squares. Part I: consistency. Kybernetika 42, 1–36 (2006)
Víšek, J.Á.: Consistency of the least weighted squares under heteroscedasticity. Kybernetika 47, 179–206 (2011)
Wilcox, R.R.: Introduction to Robust Estimation and Hypothesis Testing, 2nd edn. Elsevier, Burlington (2005)
Acknowledgements
The work is supported by the projects GA19-05704S and GA18-23827S of the Czech Science Foundation. The authors are grateful to Jan Tichavský and Jiří Tumpach for technical help.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Kalina, J., Vidnerová, P. (2020). Regression Neural Networks with a Highly Robust Loss Function. In: Maciak, M., Pešta, M., Schindler, M. (eds) Analytical Methods in Statistics. AMISTAT 2019. Springer Proceedings in Mathematics & Statistics, vol 329. Springer, Cham. https://doi.org/10.1007/978-3-030-48814-7_2
Download citation
DOI: https://doi.org/10.1007/978-3-030-48814-7_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-48813-0
Online ISBN: 978-3-030-48814-7
eBook Packages: Mathematics and StatisticsMathematics and Statistics (R0)