Skip to main content
SpringerLink
Log in

Regularization and validation of neural network models of nonlinear systems

Regelung und Zuverlässigkeitsprüfung von Neuromodellen nichtlinearer Systeme

  • Reviewed Original Papers
  • Published:
e & i Elektrotechnik und Informationstechnik Aims and scope Submit manuscript

Abstract

A characteristic feature of the neural network models is the large number of parameters. A model offering many parameters usually gives rise to problems, and the variance contribution to the modeling error might be very high. Therefore, it is crucial to find the model with the optimal number of parameters. In this paper two techniques of selection of the optimal number of model parameters are described and compared: explicit and implicit regularization techniques. Model validation forms the final stage of an identification procedure with the aim of assessing objectively whether the identified model agrees sufficiently well with the observed data. In this paper the reliability of the correlation-based validation tests and the χ2-test is analyzed.

Zusammenfassung

Große Parameteranzahl ist ein typisches Merkmal von Neuromodellen. Ein Neuromodell mit großer Parameteranzahl ist gewöhnlich mit vielen Problemen belastet weil in diesem Fall der Einfluss der Varianz auf den Modellfehler erheblich ansteigt. Deshalb ist es entscheidend, ein Neuromodell mit optimaler Parameteranzahl zu erstellen. In dem vorliegenden Beitrag werden zwei Techniken für die Auswahl der optimalen Modellparameteranzahl untersucht und verglichen: explizite und implizite Regularisationstechniken. Die Zuverlässigkeitsprüfung des Modells bildet den letzten Schritt eines Identifikationsverfahrens, dessen Ziel es ist, die Übereinstimmung des identifizierten Modells mit den durch Beobachtung gewonnenen Daten objektiv zu beurteilen. In diesem Beitrag wird die Zuverlässigkeit von Korrelationstests sowie des χ2-Tests analysiert.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Hornik, K., Stinchcombe, M., White, H.: Multilayer feedforward networks are universal approximators. Neural Networks, Vol. 2 (1989), pp. 359–366.

    Article  Google Scholar 

  2. Fahlman, S. E., Lebiere, C.: The cascade-correlation learning architecture. Advances in Neural Information Processing Systems 2 (1990), pp. 524–532.

    Google Scholar 

  3. Petrović, I., Baotić, M., Perić, N.: A cascade-correlation learning network with smoothing. Proc. of ICSC/IFAC NC ’98 Symposium (1998), pp. 1023–1029.

  4. Zhao, Y., Atkeson, C. G.: Implementing projection pursuit learning. IEEE Transaction on neural networks. Vol. 7, No. 2 (1996), pp.362–373.

    Article  Google Scholar 

  5. Reed, R.: Pruning algorithms — a survey. IEEE Transactions on neural networks, Vol. 4, No. 5 (1993), pp. 740–747.

    Article  Google Scholar 

  6. Moody, J.: The effective number of parameters: An analysis of generalization and regularization in nonlinear learning systems. Advances in Neural Information Processing Systems 4 (1992).

  7. Sjöberg, J., Ljung, L.: Overtraining, regularization, and searching for minimum in neural networks. Proc. of 4th IFAC Symposium on Adaptive Systems and Signal Processing (1992), pp. 669–674.

  8. Leontaritis, I. J., Billings, S. A.: Input-output parameteric models for non-linear systems. Part I: Deterministic non-linear systems. International Journal of Control, Vol. 41, No. 2 (1985), pp. 303–328.

    Article  MathSciNet  MATH  Google Scholar 

  9. Chen, S., Billings, S. A., Grant, P. M.: Non-linear system identification using neural networks. International Journal of Control, Vol. 51, No. 6 (1990), pp. 1191–1214.

    Article  MathSciNet  MATH  Google Scholar 

  10. Sjöberg, J., Zhang, Q., Ljung, L., Benveniste, A., Deylon, B., Glorennec, P. Y., Hjalmarsson, H., Juditsky, A.: Non-linear black-box modeling in system identification: a unified overview. Automatica, Vol. 31, No. 12 (1995), pp. 1691–1724.

    Article  MathSciNet  MATH  Google Scholar 

  11. Bazaraa, S. M., Sherali, D. H., Shety, C. M.: Nonlinear programming — theory and algorithms. New York: John Wiley & Sons, 1993.

    Google Scholar 

  12. Ljung, L.: System identification: Theory for the user. Englewood Cliffs, New Jersey: Prentice-Hall, 1987.

    MATH  Google Scholar 

  13. Billings, S. A., Voon, W. S. F.: Correlation based model validity tests for non-linear models. International Journal of Control, Vol. 44, No. 1 (1986), pp. 235–244.

    Article  MathSciNet  MATH  Google Scholar 

  14. Leontaritis, I. J., Billings, S. A.: Model selection and validation methods for non-linear systems. International Journal of Control, Vol. 45, No. 1 (1987), pp. 311–341.

    Article  MATH  Google Scholar 

  15. Petrović, I., Baotić, M., Perić, N.: An efficient Newton-type learning algorithm for MLP neural networks. Proc. of ICSC/IFAC NC ’98 Symposium (1998), pp. 551–557.

  16. Nguyen, D., Widrow, B.: Improving the learning speed of 2-layer neural networks by choosing initial values of the adaptive weights. International Joint Conference on Neural Networks, Vol. 3 (1990), pp. 21–26.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Faculty of Electrical Engineering and Computing, Department of Control and Computer Engineering in Automation, University of Zagreb, Unska 3, Zagreb, Croatia

    I. Petrović Ph. D., M. Baotić B. Sc. & N. Perić Ph. D.

Authors
  1. I. Petrović Ph. D.
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. M. Baotić B. Sc.
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. N. Perić Ph. D.
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

About this article

Cite this article

Petrović, I., Baotić, M. & Perić, N. Regularization and validation of neural network models of nonlinear systems. Elektrotech. Inftech. 117, 24–31 (2000). https://doi.org/10.1007/BF03161395

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/BF03161395

Keywords

Schlüsselwörter

Search

Navigation