Gauss–Newton Methods for Robust Parameter Estimation

  • Tanja Binder
  • Ekaterina Kostina
Part of the Contributions in Mathematical and Computational Sciences book series (CMCS, volume 4)


In this paper we treat robust parameter estimation procedures for problems constrained by differential equations. Our focus is on the l 1 norm estimator and Huber’s M-estimator. Both of the estimators are briefly characterized and the corresponding optimality conditions are given. We describe the solution of the resulting minimization problems using the Gauss–Newton method and present local convergence results for both nonlinear constrained l 1 norm and Huber optimization. An approach for the efficient solution of the linearized problems of the Gauss–Newton iterations is also sketched as well as globalization strategies using line search methods. Two numerical examples are exercised to demonstrate the superiority of the two presented robust estimators over standard least squares estimation in case of outliers in the measurement data.


Newton Method Constraint Qualification Complementarity Condition Stationarity Condition Strict Complementarity 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



This research was supported by the German Federal Ministry for Education and Research (BMBF) through the Programme “Mathematics for Innovations in Industry and Public Services”, as well as by the LOEWE Center for Synthetic Microbiology (SYNMIKRO).


  1. 1.
    D.A. Andrews, F. Bickel, F. Hampel, P. Huber, W. Rogers, and J. Tukey, Robust Estimates of Location: Survey and Advanves, Princeton University Press, 1972.Google Scholar
  2. 2.
    U. Ascher, Collocation for two-point boundary value problems revisited, SIAM Journal on Numerical Analysis, 23(3), pp. 596–609, 1986.MathSciNetzbMATHCrossRefGoogle Scholar
  3. 3.
    I. Barrodale and F.D.K. Roberts, An improved algorithm for discrete l 1 linear approximation, SIAM Journal on Numerical Analysis 10, pp. 839–848, 1973.MathSciNetzbMATHCrossRefGoogle Scholar
  4. 4.
    I. Barrodale and F.D.K. Roberts, Algorithm 478: Solution of an overdetermined system of linear equations in the l 1 norm, Communications of the ACM 17, pp. 319–320, 1974.CrossRefGoogle Scholar
  5. 5.
    I. Bauer, H. G. Bock, S. Körkel, and J. P. Schlöder, Numerical methods for initial value problems and derivative generation for DAE models with application to optimum experimental design of chemical processes, in F. Keil, W. Mackens, H. Voss, and J. Werther, eds., Scientific Computing in Chemical Engineering II, 2, pp. 282–289, Springer, Berlin-Heidelberg, 1999.Google Scholar
  6. 6.
    I. Bauer, H. G. Bock, S. Körkel, and J. P. Schlöder, Numerical methods for optimum experimental design in DAE systems, Journal of Computational and Applied Mathematics 120, pp. 1–25, 2000.MathSciNetzbMATHCrossRefGoogle Scholar
  7. 7.
    A.E. Beaton and J.W. Tukey, The Fitting of Power Series, Meaning Polynomials, Illustrated on Band-Spectroscopic Data, Technometrics 16, pp. 147–185, 1974.zbMATHCrossRefGoogle Scholar
  8. 8.
    A. Ben-Tal, Second-Order and Related Extremality Conditions in Nonlinear Programming, Journal of Optimization Theory and Applications 31(2), pp. 143–165, 1980.MathSciNetzbMATHCrossRefGoogle Scholar
  9. 9.
    H.G. Bock, Numerical treatment of inverse problems in chemical reaction kinetics, in K.-H. Ebert, P. Deuflhard, and W. Jäger, eds., Modelling of Chemical Reaction Systems, volume 18 of Springer Series in Chemical Physics, Springer Verlag, pp. 102–125, 1981.Google Scholar
  10. 10.
    H.G. Bock, Randwertproblemmethoden zur Parameteridentifizierung in Systemen nichtlinearer Differentialgleichungen, Bonner Mathematische Schriften, Nr. 183, Bonn, 1987.Google Scholar
  11. 11.
    H.G. Bock, E.A. Kostina, and J.P. Schlöder, Numerical Methods for Parameter Estimation in Nonlinear Differential Algebraic Equations, GAMM Mitteilungen 30(2), pp. 376–408, 2007.MathSciNetzbMATHCrossRefGoogle Scholar
  12. 12.
    H.G. Bock, E. Kostina, and J.P. Schlöder, On the Role of Natural Level Functions to Achieve Global Convergence for Damped Newton Methods, in M.J.D. Powell and S. Scholtes, eds., System Modelling and Optimization. Methods, Theory and Applications, Kluwer, Boston, pp. 51–74, 2000.Google Scholar
  13. 13.
    A.M. Bruckstein, D.L. Donoho, and M. Elad, From Sparse Solutions of Systems of Equations to Sparse Modeling of Signals and Images, SIAM Review 51(1), pp. 34–81, 2009.MathSciNetzbMATHCrossRefGoogle Scholar
  14. 14.
    R. Bulirsch, Die Mehrzielmethode zur numerischen Lösung von nichtlinearen Randwertproblemen und Aufgaben der optimalen Steuerung, Technical report, Carl-Cranz-Gesellschaft, 1971.Google Scholar
  15. 15.
    A. Cervantes and L.T. Biegler, Large-scale DAE optimization using a simultaneous NLP formulation, AIChE Journal 44(5), pp. 1038–1050, 2004.CrossRefGoogle Scholar
  16. 16.
    W. Chauvenet, A Manual of Spherical and Practical Astronomy Vol. II - Theory and Use of Astronomical Instruments, J.B. Lippincott & Co., 1868.Google Scholar
  17. 17.
    J.E. Dennis and R.E. Welsch, Techniques for nonlinear least squares and robust regression, Communications in Statistics - Simulation and Computation 7, pp. 345–359, 1978.CrossRefGoogle Scholar
  18. 18.
    P. Deuflhard, A Modified Newton Method for the Solution of Ill-Conditioned Systems of Nonlinear Equations with Application to Multiple Shooting, Numerical Mathematics 22, pp. 289–315, 1974.MathSciNetzbMATHCrossRefGoogle Scholar
  19. 19.
    R.C. Fair, On the robust estimation of econometric models, Annals of Economic and Social Measurement 3, pp. 667–677, 1974.Google Scholar
  20. 20.
    A.V. Fiacco and G.P. McCormick, Nonlinear Programming, Sequential Unconstrained Minimization Techniques, SIAM, 1990.zbMATHGoogle Scholar
  21. 21.
    C.F. Gauss, Theoria Motus Corporum Coelestium in Sectionibus Conicis Solem Ambientium, F. Perthes & J.H. Besser, 1809.Google Scholar
  22. 22.
    J.E. Gentle, Least absolute value estimation: an introduction, Communications in Statistics - Simulation and Computation 6, pp. 313–328, 1977.MathSciNetCrossRefGoogle Scholar
  23. 23.
    F.R. Hampel, Robust Estimation: A Condensed Partial Survey, Zeitschrift für Wahrscheinlichkeitstheorie und verwandte Gebiete 27, pp. 87–104, 1973.MathSciNetzbMATHCrossRefGoogle Scholar
  24. 24.
    M.J. Hinich and P.P. Talwar, A Simple Method for Robust Regression, Journal of the American Statistical Association 70, pp. 113–119, 1975.zbMATHCrossRefGoogle Scholar
  25. 25.
    P.E. Holland and R.E. Welsch, Robust regression using iteratively reweighted least-squares, Communications in Statistics - Theory and Methods 6, pp. 813–827, 1977.CrossRefGoogle Scholar
  26. 26.
    P.J. Huber, Robust Estimation of a Location Parameter, The Annals of Mathematical Statistics 35, pp. 73–101, 1964.zbMATHCrossRefGoogle Scholar
  27. 27.
    P.J. Huber and E.M. Ronchetti, Robust Statistics - Second Edition, John Wiley & Sons, 2009.Google Scholar
  28. 28.
    A. Koberstein, The Dual Simplex method, Techniques for a Fast and Stable Implementation, PhD Thesis, U Paderborn, 2005.Google Scholar
  29. 29.
    S. Körkel and E. A. Kostina, Numerical Methods for Nonlinear Experimental Design, in H. G. Bock, E. A. Kostina, H. X. Phu, and R. Rannacher, eds., Modeling, Simulation and Optimization of Complex Processes, Proceedings of the International Conference on High Performance Scientific Computing, 2003, Hanoi, Vietnam. Springer, 2004.Google Scholar
  30. 30.
    S. Körkel, Numerische Methoden für Optimale Versuchsplanungsprobleme bei nichtlinearen DAE-Modellen, PhD thesis, U Heidelberg, 2002.Google Scholar
  31. 31.
    E. Kostina, The Long Step Rule in the Bounded-Variable Dual Simplex Method: Numerical Experiments, Mathematical Methods of Operations Research 3(55), pp. 413–429, 2002.MathSciNetCrossRefGoogle Scholar
  32. 32.
    E. Kostina, Robust Parameter Estimation in Dynamic Systems, Optimization and Engineering 5(4), pp. 461–484, 2004.MathSciNetzbMATHCrossRefGoogle Scholar
  33. 33.
    E. Kostina and O. I. Kostyukova, A Primal-Dual Active-Set Method for Convex Quadratic Programming, Technical Report, IWR, U Heidelberg, 2003.Google Scholar
  34. 34.
    S. Krishna, M.H. Jensen, and K. Sneppen, Minimal model of spiky oscillations in NF-κB signaling, Proceedings of the National Academy of Sciences of the United States of America 103(29), pp. 10840–10845, 2006.CrossRefGoogle Scholar
  35. 35.
    P.S. Laplace, Théorie Analytique des Probabilités, V. Courcier, 1814.Google Scholar
  36. 36.
    A.M. Legendre, Nouvelles méthodes pour la détermination des orbites des comètes, Firmin Didot, 1805.Google Scholar
  37. 37.
    O.L. Mangasarian and D.R. Musicant, Robust Linear and Vector Support Regression, IEEE Transactions on Pattern Analysis and Machine Intelligence 22(9), pp. 1–6, 2000.CrossRefGoogle Scholar
  38. 38.
    M.R. Osborne, B. Presnell, and B.A. Turlach, On the LASSO and Its Dual, Journal of Computational and Graphical Statistics 9(2), pp. 319–337, 2000.MathSciNetGoogle Scholar
  39. 39.
    B. Peirce, Criterion for the rejection of doubtful observations, The Astronomical Journal 45, pp. 161–163, 1852.CrossRefGoogle Scholar
  40. 40.
    J.R. Rice and J.S. White, Norms for Smoothing and Estimation, SIAM Review 6, pp. 243–256, 1964.MathSciNetzbMATHCrossRefGoogle Scholar
  41. 41.
    J.A. Scales and A. Gersztenkorn, Robust methods in inverse theory, Inverse Problems 4, pp. 1071–1091, 1988.MathSciNetzbMATHCrossRefGoogle Scholar
  42. 42.
    V.H. Schulz, Ein effizientes Kollokationsverfahren zur numerischen Behandlung von Mehrpunktrandwertaufgaben in der Parameteridentifizierung und Optimalen Steuerung, Diploma thesis, U Augsburg, 1990.Google Scholar
  43. 43.
    S. Waldherr, Uncertainty and robustness analysis of biochemical reaction networks via convex optimisation and robust control theory, PhD thesis, University of Stuttgart, 2009.Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  1. 1.Department of Mathematics and Computer ScienceUniversity of MarburgMarburgGermany

Personalised recommendations