Skip to main content
SpringerLink
Log in

An outlier-robust kernel RLS algorithm for nonlinear system identification

  • Original Paper
  • Published:
Nonlinear Dynamics Aims and scope Submit manuscript

Abstract

The kernel recursive least squares (KRLS), a nonlinear counterpart of the famed RLS algorithm, performs linear regression in a high-dimensional feature space induced by a Mercer kernel. Despite the growing interest in the KRLS for nonlinear signal processing, the presence of outliers in the estimation data causes the resulting predictor’s performance to deteriorate considerably. Bearing this in mind, we introduce an approach to amalgamate the kernel-based learning framework that gives rise to the KRLS algorithm with the robust regression framework of M-estimators with the aim of building an outlier-robust variant for the KRLS. Initially, we develop the theoretical aspects of the proposed algorithm and then analyze its behavior in nonlinear system identification problems using synthetic and real-world datasets (including a large-scale one) contaminated with outliers. The obtained results indicate that the robust variant of the KRLS algorithm consistently outperforms the state-of-the-art in robust adaptive filtering algorithms, as the amount of outliers in the data increases.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Notes

  1. A signal processing task in which an inverse model of a communication channel (e.g., in cellular systems) is built and its parameters are adapted in an online fashion to compensate for “intersymbol interference”, a phenomenon caused by signal pulse smearing in dispersive media.

  2. A collection of atoms which can represent each training data by a linear combination of the atoms.

  3. http://www.iau.dtu.dk/nnbook/systems.html.

References

  1. Bakker, R., Schouten, J.C., Coppens, M.O., Takens, F., Giles, C.L., van den Bleek, M.: Robust learning of chaotic attractors. In: Solla, S.A., Leen, T.K., Müller, K.R. (eds.) Advances in Neural Information Processing Systems, vol. 12, pp. 879–885. MIT Press (2000)

  2. Bhotto, M.Z.A., Antoniou, A.: Robust recursive least-squares adaptive-filtering algorithm for impulsive-noise environments. IEEE Signal Process. Lett. 18(3), 185–188 (2011)

    Article  Google Scholar 

  3. Billings, S.A.: Nonlinear System Identification NARMAX Methods in the Time, Frequency and Spatio-Temporal Domains, 1st edn. Wiley, London (2013)

    Book  MATH  Google Scholar 

  4. Billings, S.A., Voon, W.S.F.: Correlation based model validity tests for non-linear models. Int. J. Control 44(1), 235–244 (1986)

    Article  MATH  Google Scholar 

  5. Cai, Y., Wang, H., Ye, X., Fan, Q.: A multiple-kernel LSSVR method for separable nonlinear system identification. J. Control Theory Appl. 11(4), 651–655 (2013)

    Article  MathSciNet  Google Scholar 

  6. Campbell, C.: Kernel methods: a survey of current techniques. Neurocomputing 48(1–4), 63–84 (2002)

    Article  MATH  Google Scholar 

  7. Chan, S.C., Zou, Y.X.: A recursive least M-estimate algorithm for robust adaptive filtering in impulsive noise: fast algorithm and convergence performance analysis. IEEE Trans. Signal Process. 52(4), 975–991 (2004)

    Article  MathSciNet  MATH  Google Scholar 

  8. Chaudhary, N.I., Raja, M.A.Z.: Identification of Hammerstein nonlinear ARMAX systems using nonlinear adaptive algorithms. Nonlinear Dyn. 79(2), 1385–1397 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  9. De Brabanter, K., De Brabanter, J., Suykens, J.A., Vandewalle, J., De Moor, B.: Robustness of kernel based regression: influence and weight functions. In: The 2012 International Joint Conference on Neural Networks (IJCNN), pp. 1–8 (2012)

  10. De Brabanter, K., Pelckmans, K., De Brabanter, J., Debruyne, M., Suykens, J.A., Hubert, M., De Moor, B.: Robustness of kernel based regression: a comparison of iterative weighting schemes. In: Artificial Neural Networks–ICANN 2009, pp. 100–110. Springer (2009)

  11. Debruyne, M., Christmann, A., Hubert, M., Suykens, J.A.K.: Robustness and stability of reweighted kernel based regression. Technical Report 06-09, Department of Mathematics, Katholiek University of Leuven, Leuven, Belgium (2008)

  12. Engel, Y., Mannor, S., Meir, R.: Sparse online greedy support vector regression. In: Elomaa, T., Mannila, H., Toivonen, H. (eds.) Proceedings of the 13th European Conference on Machine Learning (ECML’2002), pp. 84–96 (2002)

  13. Engel, Y., Mannor, S., Meir, R.: The kernel recursive least-squares algorithm. IEEE Trans. Signal Process. 52(8), 2275–2285 (2004)

    Article  MathSciNet  MATH  Google Scholar 

  14. Espinoza, M., Pelckmans, K., Hoegaerts, L., Suykens, J.A., De Moor, B.: A comparative study of LS-SVMs applied to the silverbox identification problem. In: Proceedings of the 6th IFAC Symposium on Nonlinear Control Systems (NOLCOS’2004) (2004)

  15. Falck, T., Dreesen, P., De Brabanter, K., Pelckmans, K., De Moor, B., Suykens, J.A.K.: Least-squares support vector machines for the identification of Wiener–Hammerstein systems. Control Eng. Pract. 20(11), 1165–1174 (2012)

    Article  Google Scholar 

  16. Fan, H., Song, Q.: A sparse kernel algorithm for online time series data prediction. Expert Syst. Appl. 40, 2174–2181 (2013)

    Article  Google Scholar 

  17. Farahmand, S., Giannakis, G.B.: Robust RLS in the presence of correlated noise using outlier sparsity. IEEE Trans. Signal Process. 60(6), 3308–3313 (2012)

    Article  MathSciNet  Google Scholar 

  18. Fox, J.: An R and S-Plus Companion to Applied Regression, 1st edn. SAGE (2002)

  19. Golub, G.H., Van Loan, C.F.: Matrix Computations, vol. 3. JHU Press, Baltimore (2012)

    MATH  Google Scholar 

  20. Haykin, S.: Adaptive Filter Theory, 5th edn. Prentice Hall, Englewood Cliffs (2013)

    MATH  Google Scholar 

  21. Huber, P.J., Ronchetti, E.M.: Robust Statistics. Wiley, London (2009)

    Book  MATH  Google Scholar 

  22. Huber, P.J., et al.: Robust estimation of a location parameter. Ann. Math. Stat. 35(1), 73–101 (1964)

    Article  MathSciNet  MATH  Google Scholar 

  23. Jäkel, F., Schölkopf, B., Wichmann, F.A.: A tutorial on kernel methods for categorization. J. Math. Psychol. 51(6), 343–358 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  24. Jang, J.S.: ANFIS: adaptive-network-based fuzzy inference system. IEEE Trans. Syst. Man Cybern. 23(3), 665–685 (1993)

    Article  Google Scholar 

  25. Kocijan, J., Girard, A., Banko, B., Murray-Smith, R.: Dynamic systems identification with Gaussian processes. Math. Comput. Model. Dyn. Syst. 11(4), 411–424 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  26. Li, X., Zhou, L., Sheng, J., Ding, R.: Recursive least squares parameter estimation algorithm for dual-rate sampled-data nonlinear systems. Nonlinear Dyn. 76(2), 1327–1334 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  27. Liu, B., Wang, Z.: On system identification based on online least squares support vector machine. In: International Conference on Intelligent Systems and Knowledge Engineering. Atlantis Press (2007)

  28. Liu, W., Park, I., Principe, J.: An information theoretic approach of designing sparse kernel adaptive filters. IEEE Trans. Neural Netw. 20(12), 1950–1961 (2009)

    Article  Google Scholar 

  29. Liu, W., Park, I., Wang, Y., Principe, J.: Extended kernel recursive least squares algorithm. IEEE Trans. Signal Process. 57(10), 3801–3804 (2009)

    Article  MathSciNet  Google Scholar 

  30. Liu, W., Pokharel, P., Principe, J.: The kernel least mean square algorithm. IEEE Trans. Signal Process. 56(2), 543–554 (2008)

    Article  MathSciNet  Google Scholar 

  31. Liu, Y., Chen, J.: Correntropy kernel learning for nonlinear system identification with outliers. Ind. Eng. Chem. Res. 53, 5248–5260 (2014)

    Article  Google Scholar 

  32. Liu, Y., Wang, H., Yu, J., Li, P.: Selective recursive kernel learning for online identification of nonlinear systems with NARX form. J. Process Control 20(2), 181–194 (2010)

    Article  Google Scholar 

  33. Ljung, L.: System Identification: Theory for the User, 2nd edn. Prentice Hall, Englewood Cliffs (1999)

    MATH  Google Scholar 

  34. Ljung, L.: Recursive identification algorithms. Circuits Syst. Signal Process. 21(1), 57–68 (2002)

    Article  MathSciNet  Google Scholar 

  35. Mattos, C.L.C., Damianou, A., Barreto, G.A., Lawrence, N.D.: Latent autoregressive Gaussian processes models for robust system identification. In: Proceedings of the 11th IFAC Symposium on Dynamics and Control of Process Systems (DYCOPS’2016) (2016, to appear)

  36. Naderahmadian, Y., Tinati, M.A., Beheshti, S.: Generalized adaptive weighted recursive least squares dictionary learning. Sig. Process. 118, 89–96 (2016)

    Article  Google Scholar 

  37. Narendra, K.S., Parthasarathy, K.: Identification and control of dynamical systems using neural networks. IEEE Trans. Neural Netw. 1(1), 4–27 (1990)

    Article  Google Scholar 

  38. Papageorgiou, G., Bouboulis, P., Theodoridis, S.: Robust linear regression analysis—a greedy approach. IEEE Trans. Signal Process. 63(15), 3872–3887 (2015)

    Article  MathSciNet  Google Scholar 

  39. Pillonetto, G., Dinuzzob, F., Chen, T., De Nicolao, G., Ljung, L.: Kernel methods in system identification, machine learning and function estimation: a survey. Automatica 50, 657–682 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  40. Pintelon, R., Schoukens, J.: System Identification: A Frequency Domain Approach. Wiley, London (2012)

    Book  MATH  Google Scholar 

  41. Platt, J.: A resource-allocating network for function interpolation. Neural Comput. 3(2), 213–225 (1991)

    Article  MathSciNet  Google Scholar 

  42. Principe, J., Rathie, A., Kuo, J.M.: Prediction of chaotic time series with neural networks and the issue of dynamic modeling. Int. J. Bifurc. Chaos 2(4), 989–996 (1992)

    Article  MATH  Google Scholar 

  43. Quan, T., Liu, X., Liu, Q.: Weighted least squares support vector machine local region method for nonlinear time series prediction. Appl. Soft Comput. 10(2), 562–566 (2010)

    Article  Google Scholar 

  44. Richard, C., Bermudez, J.C.M., Honeine, P.: Online prediction of time series data with kernels. IEEE Trans. Signal Process. 57(3), 1058–1066 (2009)

    Article  MathSciNet  Google Scholar 

  45. Rojo-Alvarez, J.L., Martinez-Ramon, M., Munoz-Maric, J., Camps-Valls, G.: A unified SVM framework for signal estimation. Digit. Signal Proc. 26, 1–20 (2014)

    Article  Google Scholar 

  46. Rousseeum, P.J., Leroy, A.M.: Robust Regression and Outlier Detection, 1st edn. Wiley, London (1987)

    Book  Google Scholar 

  47. Saide, C., Lengellé, R., Honeine, P., Achkar, R.: Online kernel adaptive algorithms with dictionary adaptation for MIMO models. IEEE Signal Process. Lett. 20(5), 535–538 (2013)

    Article  MATH  Google Scholar 

  48. Saide, C., Lengelle, R., Honeine, P., Richard, C., Achkar, R.: Nonlinear adaptive filtering using kernel-based algorithms with dictionary adaptation. Int. J. Adapt. Control Signal Process. 29(11), 1391–1410 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  49. Santos, J.D.A., Mattos, C.L.C., Barreto, G.A.: Performance evaluation of least squares SVR in robust dynamical system identification. In: Proceedings of the 13th International Work-Conference on Artificial Neural Networks (IWANN’2015), pp. 422–435 (2015)

  50. Saunders, C., Gammerman, A., Vovk, V.: Ridge regression learning algorithm in dual variables. In: Proceedings of the 15th International Conference on Machine Learning (ICML’98), pp. 515–521 (1998)

  51. Schölkopf, B., Smola, A.J.: Learning With Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. MIT Press, Cambridge (2002)

    Google Scholar 

  52. Schoukens, J., Németh, J.G., Crama, P., Rolain, Y., Pintelon, R.: Fast approximate identification of nonlinear systems. Automatica 39(7), 1267–1274 (2003)

    Article  MathSciNet  MATH  Google Scholar 

  53. Sjöberg, J., Zhang, Q., Ljung, L., Benveniste, A., Delyon, B., Glorennec, P.Y., Hjalmarsson, H., Juditsky, A.: Nonlinear black-box modeling in system identification: a unified overview. Automatica 31(12), 1691–1724 (1995)

    Article  MathSciNet  MATH  Google Scholar 

  54. Smola, A.J., Schölkopf, B.: A tutorial on support vector regression. Stat. Comput. 14(3), 199–222 (2004)

    Article  MathSciNet  Google Scholar 

  55. Suykens, J.A.K., De Brabanter, J., Lukas, L., Vandewalle, J.: Weighted least squares support vector machines: robustness and sparse approximation. Neurocomputing 48(1), 85–105 (2002)

    Article  MATH  Google Scholar 

  56. Suykens, J.A.K., Van Gestel, T., De Brabanter, J., De Moor, B., Vandewalle, J.: Least Squares Support Vector Machines. World Scientific, Singapore (2002)

    Book  MATH  Google Scholar 

  57. Takagi, T., Sugeno, M.: Fuzzy identification of systems and its application to modeling and control. IEEE Trans. Syst. Man Cybern. 15(1), 116–132 (1985)

    Article  MATH  Google Scholar 

  58. Tang, H.S., Xue, S.T., Chen, R., Sato, T.: Online weighted LS-SVM for hysteretic structural system identification. Eng. Struct. 28(12), 1728–1735 (2006)

    Article  Google Scholar 

  59. Taouali, O., Elaissi, E., Messaoud, H.: Design and comparative study of online kernel methods identification of nonlinear system in RKHS space. Artif. Intell. Rev. 37(4), 289–300 (2012)

    Article  Google Scholar 

  60. Tenreiro Machado, J.A., Babaei, A., Moghaddam, B.P.: Highly accurate scheme for the Cauchy problem of the generalized Burgers–Huxley equation. Acta Polytech. Hung. 13(6), 183–195 (2016)

    Google Scholar 

  61. Van Vaerenbergh, S., Vía, J., Santamaría, I.: Nonlinear system identification using a new sliding-window kernel RLS algorithm. J. Commun. 2(3), 1–8 (2007)

    Article  Google Scholar 

  62. Van Vaerenbergh, S., Vía, J., Santamaría, I.: Kernel recursive least-squares tracker for time-varying regression. IEEE Trans. Neural Netw. Learn. Syst. 23(8), 1313–1326 (2012)

    Article  Google Scholar 

  63. Vapnik, V.: The Nature of Statistical Learning Theory, 1st edn. Springer, Berlin (1995)

    Book  MATH  Google Scholar 

  64. Vapnik, V.: Statistical Learning Theory, 1st edn. Wiley, London (1998)

    MATH  Google Scholar 

  65. Wedderburn, R.W.: Quasi-likelihood functions, generalized linear models, and the Gauss–Newton method. Biometrika 61(3), 439–447 (1974)

    MathSciNet  MATH  Google Scholar 

  66. Widrow, B., Lehr, M.: Noise canceling and channel equalization. In: Arbib, M.A. (ed.) Handbook of Brain Theory and Neural Networks, pp. 648–650. MIT Press, Cambridge (1995)

    Google Scholar 

  67. Widrow, B., Walach, E.: Adaptive Inverse Control: A Signal Processing Approach. Wiley-IEEE Press, London (2007)

    Book  Google Scholar 

  68. Wills, A., Schön, T.B., Ljung, L., Ninness, B.: Identification of Hammerstein–Wiener models. Automatica 49(1), 70–81 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  69. Wu, Z., Shi, J., Zhang, X., Ma, W., Chen, B.: Kernel recursive maximum correntropy. Signal Process. 117, 11–16 (2015)

    Article  Google Scholar 

  70. Yang, X.J.: A new integral transform operator for solving the heat-diffusion problem. Appl. Math. Lett. 64, 193–197 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  71. Yang, X.J., Gao, F.: A new technology for solving diffusion and heat equations. Therm. Sci. 21, 133–140 (2017)

  72. Young, P.C.: Recursive Estimation and Time-series Analysis: An Introduction for the Student and Practitioner. Springer, Berlin (2011)

    Book  MATH  Google Scholar 

  73. Zhang, B., Billings, S.A.: Identification of continuous-time nonlinear systems: the nonlinear difference equation with moving average noise (NDEMA) framework. Mech. Syst. Signal Process. 60, 810–835 (2015)

    Article  Google Scholar 

  74. Zhang, B., Billings, S.A.: Volterra series truncation and kernel estimation of nonlinear systems in the frequency domain. Mech. Syst. Signal Process. 84–A, 39–57 (2017)

    Article  Google Scholar 

  75. Zhang, J., Pang, Y.: Pipelined robust \(m\)-estimate adaptive second-order volterra filter against impulsive noise. Digit. Signal Process. 26, 71–80 (2014)

    Article  Google Scholar 

  76. Zhang, K., Fei, M., Wu, J., Zhang, P.: Fast prediction model based big data system identification. In: Chinese Automation Congress (CAC), pp. 465–469. IEEE (2013)

  77. Zhang, Y.Q., Wang, X.Y.: Spatiotemporal chaos in mixed linear-nonlinear coupled logistic map lattice. Physica A 402, 104–118 (2014)

    Article  MathSciNet  Google Scholar 

  78. Zhang, Y.Q., Wang, X.Y., Liu, L.Y., He, Y., Liu, J.: Spatiotemporal chaos of fractional order logistic equation in nonlinear coupled lattices. Commun. Nonlinear Sci. Numer. Simul. 52, 52–61 (2017)

    Article  MathSciNet  Google Scholar 

  79. Zhao, S., Chen, B., Principe, J.C.: Kernel adaptive filtering with maximum correntropy criterion. In: Proceedings of the 2011 International Joint Conference on Networks (IJCNN’2011), pp. 2012–2017 (2011)

  80. Zhu, P.P., Chen, B., Principe, J.C.: A novel extended kernel recursive least squares algorithm. Neural Netw. 32, 349–357 (2012)

    Article  MATH  Google Scholar 

  81. Zou, Y., Chan, S., Ng, T.: A recursive least M-estimate (RLM) adaptive filter for robust filtering in impulse noise. IEEE Signal Process. Lett. 7(11), 324–326 (2000)

    Article  Google Scholar 

  82. Zoubir, A.M., Koivunen, V., Chakhchoukh, Y., Muma, M.: Robust estimation in signal processing: a tutorial-style treatment of fundamental concepts. IEEE Signal Process. Mag. 29(4), 61–80 (2012)

    Article  Google Scholar 

Download references

Acknowledgements

The authors thank the financial support of Federal Institute of Ceará (IFCE), NUTEC and CNPq (Grant 309451/2015-9).

Author information

Authors and Affiliations

  1. Department of Industry, Federal Institute of Ceará, Maracanaú, Ceará, Brazil

    José Daniel A. Santos

  2. Department of Teleinformatics Engineering, Federal University of Ceará, Campus of Pici, Fortaleza, Ceará, Brazil

    Guilherme A. Barreto

Authors
  1. José Daniel A. Santos
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Guilherme A. Barreto
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Guilherme A. Barreto.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Santos, J.D.A., Barreto, G.A. An outlier-robust kernel RLS algorithm for nonlinear system identification. Nonlinear Dyn 90, 1707–1726 (2017). https://doi.org/10.1007/s11071-017-3760-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11071-017-3760-2

Keywords

Search

Navigation