Automation and Remote Control

, Volume 80, Issue 9, pp 1591–1606 | Cite as

On Convexification of System Identification Criteria

  • L. LjungEmail author
Topical Issue


System Identification is about estimating models of dynamical systems from measured input-output data. Its traditional foundation is basic statistical techniques, such as maximum likelihood estimation and asymptotic analysis of bias and variance and the like. Maximum likelihood estimation relies on minimization of criterion functions that typically are non-convex, and may cause numerical search problems and estimates trapped in local minima. Recent interest in identification algorithms has focused on techniques that are centered around convex formulations. This is partly the result of developments in semidefinite programming, machine learning and statistical learning theory. The development concerns issues of regular-ization for sparsity and for better tuned bias/variance trade-offs. It also involves the use of subspace methods as well as nuclear norms as proxies to rank constraints. A special approach is to look for difference-of-convex programming (DCP) formulations, in case a pure convex criterion is not found. Other techniques are based on Lagrangian relaxation and contraction theory. A quite different route to convexity is to use algebraic techniques to manipulate the model parameterizations. This article will illustrate all this recent development.


maximimum likelihood regularization convexification 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Ljung, L., System Identification—Theory for the User, Upper Saddle River: Prentice Hall, 1999, 2nd ed.zbMATHGoogle Scholar
  2. 2.
    Soderstrom, T. and Stoica, P., System Identification, London: Prentice Hall, 1989.zbMATHGoogle Scholar
  3. 3.
    Pintelon, R. and Schoukens, J., System Identification—A Frequency Domain Approach, New York: Wiley-IEEE Press, 2012, 2nd ed.zbMATHGoogle Scholar
  4. 4.
    Tao, P.D. and An, L.T.H., Convex Analysis Approach to D.C. Programming: Theory, Algorithms and Applications, ACTA Math. Vietnamica, 1997, vol. 22, no. 1, pp. 289–355.MathSciNetzbMATHGoogle Scholar
  5. 5.
    Horst, R. and Thoai, N.V., DC Programming: Overview, J. Optim. Theory Appl., 1999, vol. 103, no. 1, pp. 1–43.MathSciNetzbMATHGoogle Scholar
  6. 6.
    Thoai, R.H.N., DC Programming: An Overview, J. Optim. Theory Appl., 1999, vol. 193(1), pp. 1–43.MathSciNetzbMATHGoogle Scholar
  7. 7.
    Ljung, L. and Glad, T., On Global Identifiability of Arbitrary Model Parameterizations, Automatica, 1994, vol. 30, no. 2, pp. 265–276.MathSciNetzbMATHGoogle Scholar
  8. 8.
    Tobenkin, M.M., Manchester, I.R., and Megretski, A., Convex Paramterizations and Fidelity Bounds for Nonlinear Identification and Reduced-Order Modelling, IEEE Trans. Autom. Control, 2017, vol. AC-62, no. 7, pp. 3679–3686.zbMATHGoogle Scholar
  9. 9.
    Manchester, I.R., Tobenkin, M.M., and Megretski, A., Stable Nonliner System Identifiction: Convexity, Model Class and Consistency, Proc. 16 IFAC Sympos. Syst. Identificat., Brussels, Belgium, July 2012, pp. 328–333.Google Scholar
  10. 10.
    Umenberger, J. and Manchester, I.R., Convex Bounds for Equation Error in Stable Nonlinear Identification, IEEE Control Syst. Lett., 2019, vol. 3, no. 1, pp. 73–79.zbMATHGoogle Scholar
  11. 11.
    Van Overschee, P. and De Moor, B., N4SID: Subspace Algorithms for the Identification of Combined Deterministic-Stochastic Systems, Automatica (Special Issue), 1994, vol. 30, no. 1, pp. 75–93.zbMATHGoogle Scholar
  12. 12.
    Verhaegen, M. and Dewilde, P., The Output-Error State-Space Model Identification Class of Algorithms. Int J. Control, 1992, vol. 56, no. 5, pp. 1187–1210.zbMATHGoogle Scholar
  13. 13.
    Ljung, L. and Wahlberg, B., Asymptotic Properties of the Least-Squares Method for Estimating Transfer Functions and Disturbance Spectra, Adv. Appl. Prob., 1992, vol. 24, no. 2, pp. 412–440.MathSciNetzbMATHGoogle Scholar
  14. 14.
    Tikhonov, A.N. and Arsenin, V.Y., Solutions of Ill-posed Problems, Washington: Winston/Wiley, 1977.zbMATHGoogle Scholar
  15. 15.
    Chen, T., Ohlsson, H., and Ljung, L., On the Estimation of Transfer Functions, Regularizations and Gaussian Processes—Revisited, Automatica, 2012, vol. 48, no. 8, pp. 1525–1535.MathSciNetzbMATHGoogle Scholar
  16. 16.
    Carlin, B.P. and Lewis, T.A., Bayes and Empirical Bayes Methods for Data Analysis, London: Chapman and Hall, 1996.zbMATHGoogle Scholar
  17. 17.
    Chen, T. and Ljung, L., Implementation of Algorithms for Tuning Parameters in Regularized Least Squares Problems in System Identification, Automatica, 2013, vol. 49, no. 7, pp. 2213–2220.MathSciNetzbMATHGoogle Scholar
  18. 18.
    Carli, F.P., Chiuso, A., and Pillonetto, C., Efficient Algorithms for Large Scale Linear System Identification Using Stable Spline Estimators, Proc. 16 IFAC Sympos. Syst. Identificat (SYSID 2012), 2012, pp. 119–124.Google Scholar
  19. 19.
    Ljung, L., The System Identification Toolbox: The Manual, Natick: The MathWorks Inc., USA, 1986, 1st ed., 2012, 8th ed.Google Scholar
  20. 20.
    Chen, T., Ljung, L., Andersen, M., Chiuso, A., Carli, P.F., and Pillonetto, C., Sparse Multiple Kernels for Impulse Response Estimation with Majorization Minimization Algorithms, IEEE Conf. Decision and Control, Hawaii, 2012, pp. 1500–1505.Google Scholar
  21. 21.
    Chen, T., Andersen, M.S., Ljung, L., Chiuso, A., and Pillonetto, C., System Identification via Sparse Multiple Kernel-Based Regularization Using Sequential Convex Optimization Techniques, Autom. Control. IEEE Transact, 2014, vol. 59, no. 11, pp. 2933–2945.MathSciNetzbMATHGoogle Scholar
  22. 22.
    Dinuzzo, F., Kernels for Linear Time Invariant System Identification, Manuscript, Max Planck Institute for Intelligent Systems, Tubingen, Germany, 2012.zbMATHGoogle Scholar
  23. 23.
    Pillonetto, G. and De Nicolao, G., A New Kernel-Based Approach for Linear System Identification, Automatica, 2010, vol. 46, no. 1, pp. 81–93.MathSciNetzbMATHGoogle Scholar
  24. 24.
    Pillonetto, G., Chiuso, A., and De Nicolao, G., Prediction Error Identification of Linear Systems: A Non-parametric Gaussian Regression Approach, Automatica, 2011, vol. 47, no. 2, pp. 291–305.MathSciNetzbMATHGoogle Scholar
  25. 25.
    Pillonetto, G., Dinuzzo, F., Chen, T., De Nicolao, G., and Ljung, L., Kernel Methods in System Identification, Machine Learning and Function Estimation: A Survey, Automatica, 2014, vol. 50, no. 3, pp. 657–682.MathSciNetzbMATHGoogle Scholar
  26. 26.
    Mu, B., Chen, T., and Ljung, L., On Asymptotic Properties of Hyperparameter Estimators for Kernel-Based Regularization Methods, Automatica, 2018, vol. 94, no. 8, pp. 381–395.MathSciNetzbMATHGoogle Scholar
  27. 27.
    Parrilo, P. and Ljung, L., Initialization of Physical Parameter Estimates, in Proc. 13 IFAC Sympos. Syst. Identificat., van der Hof, P., Wahlberg, B., and Weiland, S., Eds., Rotterdam, The Netherlands, 2003, pp. 1524–1529.Google Scholar
  28. 28.
    Xie, L.L. and Ljung, L., Estimate Physical Parameters by Black Box Modeling, Proc. 21 Chinese Control Conf, Hangzhou, China, 2002, pp. 673–677.Google Scholar
  29. 29.
    Yu, C., Ljung, L., and Verhaegen, M., Identification of Structured State-Space Models, Automatica, 2018, vol. 90, no. 4, pp. 54–61.MathSciNetzbMATHGoogle Scholar
  30. 30.
    Yu, C., Ljung, L., and Verhaegen, M., Gray Box Identification Using Difference of Convex Programming, Proc. IFAC World Congress, Toulouse, France, 2017.Google Scholar
  31. 31.
    Wills, A., Yu, C., Ljung, L., and Verhaegen, M., Affinely Parametrized State-Space Models: Ways to Maximize the Likelihood Function, Proc. IFAC Sympos. Syst. Identificat. SYSID18, Stockholm, July 2018.Google Scholar
  32. 32.
    Ritt, J.F., Differential Algebra, Providence: Am. Math. Soc., 1950.zbMATHGoogle Scholar
  33. 33.
    Glad, S.T., Implementing Ritt's Algorithm of Differential Algebra, IFAC Sympos. Control Systems Design, NOLCOS'92, Bordeaux, France, 1992, pp. 398–401.Google Scholar
  34. 34.
    Ljung, L. and Chen, T., Convexity Issues in System Identification, Proc. 10 IEEE Int. Conf. Control Automat. (ICCA), Hangzhou, China, 2013.Google Scholar
  35. 35.
    Ljung, L., Pespectives on System Identification, IFAC Annual Reviews, Spring Issue, 2010.Google Scholar
  36. 36.
    Ljung, L., Hjalmarsson, H., and Ohlsson, H., Four Encounters with System Identification, Eur. J. Control, 2011, vol. 17, no. 5–6, pp. 449–471.MathSciNetzbMATHGoogle Scholar

Copyright information

© Pleiades Publishing, Ltd. 2019

Authors and Affiliations

  1. 1.Linköping UniversityLinköpingSweden

Personalised recommendations