Modifications of the Least Squares Parameter Estimation

  • Rolf Isermann
  • Marco Münchhof


In order to obtain bias-free estimates of linear dynamic processes by the method of least squares, the error signal e(k) may not be correlated. This requirement is only satisfied if the disturbance n(k) that is acting on the system is a colored noise that is generated from a white noise v(k) filtered by a form filter with the transfer function 1/A(z -1). Since this prerequisite is hardly ever met in practice, the method of least squares typically works on a correlated error signal and hence yields biased estimates. The bias can be so high for larger noise levels that the results are unusable. To avoid this problem, in the following, methods are presented which yield bias-free estimates for larger classes of dynamic processes.


Instrumental Variable Stochastic Approximation Total Little Square IEEE Trans Autom Control Auxiliary Model 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Albert AE, Gardner LA Jr (1967) Stochastic approximation and non-linear regression. MIT Press, Cambridge, MAGoogle Scholar
  2. Baur U (1976) On-Line Parameterschätzverfahren zur Identifikation linearer, dynamischer Prozesse mit Prozeßrechnern: Entwicklung, Vergleich, Erprobung: KfK-PDV-Bericht Nr. 65. Kernforschungszentrum Karlsruhe, KarlsruheGoogle Scholar
  3. Blum J (1954) Multidimensional stochastic approximation procedures. Ann Math Statist 25(4):737–744CrossRefzbMATHGoogle Scholar
  4. Brown M, Harris C (1994) Neurofuzzy adaptive modelling and control. Prentice-Hall international series in systems and control engineering, Prentice Hall, New YorkGoogle Scholar
  5. Clarke DW (1967) Generalized least squares estimation of the parameters of a dynamic model. In: Preprints of the IFAC Symposium on Identification, PragGoogle Scholar
  6. Durbin J (1954) Errors in variables. Revue de l’Institut International de Statistique / Review of the International Statistical Institute 22(1/3):23–32CrossRefMathSciNetGoogle Scholar
  7. Dvoretzky A (1956) On stochastic approximation. In: Proceedings of the 3rd Berkeley Symposium on Mathematical Statistics and Probability, Berkeley, CA, USAGoogle Scholar
  8. Goedecke W (1987) Fehlererkennung an einem thermischen Prozeß mit Methoden der Parameterschätzung: Fortschr.-Ber. VDI Reihe 8 Nr. 130. VDI Verlag, DüsseldorfGoogle Scholar
  9. Golub GH, van Loan C (1980) An analysis of the total least squares problems. SIAM J Numer Anal 17(6):883–893CrossRefMathSciNetzbMATHGoogle Scholar
  10. Golub GH, Reinsch C (1970) Singular value decomposition and least squares solutions. Numer Math 14(5):403–420CrossRefMathSciNetzbMATHGoogle Scholar
  11. Goodwin GC, Sin KS (1984) Adaptive filtering, prediction and control. Prentice-Hall information and system sciences series, Prentice-Hall, Englewood Cliffs, NJzbMATHGoogle Scholar
  12. Hastings-James R, Sage MW (1969) Recursive generalized least-squares procedure for on-line identification of process parameters. Proc IEE 116(12):2057–2062Google Scholar
  13. Haykin S, Widrow B (2003) Least-mean-square adaptive filters. Wiley series in adaptive and learning systems for signal processing, communication and control, Wiley-Interscience, Hoboken, NJGoogle Scholar
  14. van Huffel S, Vandewalle J (1991) The total least squares problem: Computational aspects and analysis, Frontiers in applied mathematics, vol 9. SIAM, Philadelphia, PAGoogle Scholar
  15. Isermann R (1974) Prozessidentifikation: Identifikation und Parameterschätzung dynamischer Prozesse mit diskreten Signalen. Springer, HeidelbergGoogle Scholar
  16. Isermann R, Baur U (1973) Results of testcase A. In: Proceedings of the 3rd IFAC Symposium on System Identification, North Holland Publ. Co., Amsterdam, NetherlandsGoogle Scholar
  17. Jolliffe IT (2002) Principal component analysis, 2nd edn. Springer series in statistics, Springer, New YorkzbMATHGoogle Scholar
  18. Joseph P, Lewis J, Tou J (1961) Plant identification in the presence of disturbances and application to digital control systems. Trans AIEE (Appl and Ind) 80:18–24Google Scholar
  19. Kendall MG, Stuart A (1961) The advanced theory of statistics. Volume 2. Griffin, London, UKGoogle Scholar
  20. Kiefer J, Wolfowitz J (1952) Statistical estimation of the maximum of a regression function. Ann Math Stat 23(3):462–466CrossRefMathSciNetzbMATHGoogle Scholar
  21. Kumar PR, Moore JB (1979) Towards bias elimination in least squares system identification via detection techniques. In: Proceedings of the 5th IFAC Symposium on Identification and System Parameter Estimation Darmstadt, Pergamon Press, Darmstadt, GermanyGoogle Scholar
  22. Kushner HJ, Yin GG (2003) Stochastic approximation and recursive algorithms and applications, Applications of mathematics, vol 35, 2nd edn. Springer, New York, NYGoogle Scholar
  23. Ljung L (1999) System identification: Theory for the user, 2nd edn. Prentice Hall Information and System Sciences Series, Prentice Hall PTR, Upper Saddle River, NJGoogle Scholar
  24. Markovsky I, van Huffel S (2007) Overview of total least-squares methods. Signal Proc 87(10):2283–2302CrossRefzbMATHGoogle Scholar
  25. Markovsky I, Willems JC, van Huffel S, de Bart M, Pintelon R (2005) Application of structured total least squares for system identification and model reduction. IEEE Trans Autom Control 50(10):1490–1500CrossRefGoogle Scholar
  26. Panuska V (1969) An adaptive recursive least squares identification algorithm. In: Proceedings of the IEEE Symposium in Adaptive Processes, Decision and ControlGoogle Scholar
  27. Pintelon R, Schoukens J (2001) System identification: A frequency domain approach. IEEE Press, Piscataway, NJCrossRefGoogle Scholar
  28. Reiersøl O (1941) Confluence analysis by means of lag moments and other methods of confluence analysis. Econometrica 9(1):1–24CrossRefMathSciNetzbMATHGoogle Scholar
  29. Robbins H, Monro S (1951) A stochastic approximation method. Ann Math Statist 22(3):400–407CrossRefMathSciNetzbMATHGoogle Scholar
  30. Sage AP, Melsa JL (1971) System identification. Academic Press, New YorkzbMATHGoogle Scholar
  31. Sakrison DJ (1966) Stochastic approximation. Adv Commun Syst 2:51–106Google Scholar
  32. Saridis GN, Stein G (1968) Stochastic approximation algorithms for linear discretetime system identification. IEEE Trans Autom Control 13(5):515–523CrossRefGoogle Scholar
  33. Söderström T (2007) Errors-in-variables methods in system identification. Automatica 43(6):939–958CrossRefMathSciNetzbMATHGoogle Scholar
  34. Söderström T, Stoica PG (1983) Instrumental variable methods for system identification, Lecture notes in control and information sciences, vol 57. Springer, BerlinCrossRefGoogle Scholar
  35. Steiglitz K, McBride LE (1965) A technique for the identification of linear systems. IEEE Trans Autom Control 10:461–464CrossRefGoogle Scholar
  36. Stoica P, Jansson M (2000) On the estimation of optimal weights for instrumental variable system identification methods. In: Proccedings of the 12th IFAC Symposium on System Identification, Santa Barbara, CA, USAGoogle Scholar
  37. Stoica PG, Söderström T (1977) A method for the identification of linear systems using the generalized least squares principle. IEEE Trans Autom Control 22(4):631–634CrossRefzbMATHGoogle Scholar
  38. Stoica PG, Söderström T (1982) Bias correction in least squares identification. Int J Control 35(3):449–457CrossRefzbMATHGoogle Scholar
  39. Wong K, Polak E (1967) Identification of linear discrete time systems using the instrumental variable method. IEEE Trans Autom Control 12(6):707–718CrossRefGoogle Scholar
  40. Young P (1968) The use of linear regression and related procedures for the identification of dynamic processes. In: Proceedings of the 7th IEEE Symposium in Adaptive Processes, Los Angeles, CA, USAGoogle Scholar
  41. Young P (1970) An instrumental variable method for real-time identification of a noisy process. Automatica 6(2):271–287CrossRefGoogle Scholar
  42. Zimmerschied R (2008) Identifikation nichtlinearer Prozesse mit dynamischen lokalaffinen Modellen : Maßnahmen zur Reduktion von Bias und Varianz. Fortschr.-Ber. VDI Reihe 8 Nr. 1150. VDI Verlag, DüsseldorfGoogle Scholar

Copyright information

© Springer Berlin Heidelberg 2011

Authors and Affiliations

  1. 1.Technische Universität Darmstadt, Institut für AutomatisierungstechnikDarmstadtGermany

Personalised recommendations