Regression on the basis of nonstationary Gaussian processes with Bayesian regularization

  • E. V. Burnaev
  • M. E. Panov
  • A. A. Zaytsev
Mathematical Models and Computational Methods


We consider the regression problem, i.e. prediction of a real valued function. A Gaussian process prior is imposed on the function, and is combined with the training data to obtain predictions for new points. We introduce a Bayesian regularization on parameters of a covariance function of the process, which increases quality of approximation and robustness of the estimation. Also an approach to modeling nonstationary covariance function of a Gaussian process on basis of linear expansion in parametric functional dictionary is proposed. Introducing such a covariance function allows to model functions, which have non-homogeneous behaviour. Combining above features with careful optimization of covariance function parameters results in unified approach, which can be easily implemented and applied. The resulting algorithm is an out of the box solution to regression problems, with no need to tune parameters manually. The effectiveness of the method is demonstrated on various datasets.


Gaussian processes regression Bayesian regularization a priori distribution Bayesian regression 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    A. I. J. Forrester, A. Sóbester, and A. J. Keane, “Engineering design via surrogate modelling: a practical guide,” in Progress in Astronautics and Aeronautics (Wiley, New York, 2008).Google Scholar
  2. 2.
    A. A. Zaytsev, E. V. Burnaev, and V. G. Spokoiny, “Properties of the Bayesian parameter estimation of a regression based on Gaussian processes,” J. Math. Sci. 203, 789–798 (2014).MathSciNetCrossRefzbMATHGoogle Scholar
  3. 3.
    E. V. Burnaev, A. A. Zaytsev, and V. G. Spokoiny, “The Bernstein-Von Mises theorem for regression based on Gaussian processes,” Rus. Math. Surv. 68, 954–956 (2013).MathSciNetCrossRefzbMATHGoogle Scholar
  4. 4.
    E. Burnaev and M. Panov, “Adaptive design of experiments based on Gaussian processes,” in Lecture Notes in Artificial Intelligence (Proc. SLDS, April 20–23, London, 2015), Ed. by A. Gammerman et al. (UK, Springer, 2015), Vol. 9047, p. 116–126.Google Scholar
  5. 5.
    M. Belyaev, E. Burnaev, and Y. Kapushev, “Gaussian process regression for structured data sets,” in Lecture Notes in Artificial Intelligence (Proc. SLDS, April 20–23, London, 2015, Ed. by A. Gammerman et al. (UK, Springer, 2015), Vol. 9047, p. 106–115.Google Scholar
  6. 6.
    A. Giunta and L. T. A. Watson, “Comparison of approximation modeling technique: Polynomial versus interpolating models,” in Proc. 7th AIAA/USAF/ NASA/ISSMO Symp. on Multidisciplinary Analysis and Optimization, 1998 (AIAA, 1998), Vol. 1, p. 392–404 (1998).Google Scholar
  7. 7.
    T. W. Simpson, A. J. Booker, S. Ghosh, A. Giunta, P. N. Koch, and R. J. Yang, “Approximation methods in multidisciplinary analysis and optimization: a panel discussion,” Structur. Multidisciplin. Optimiz. 27, 302–313 (2004).CrossRefGoogle Scholar
  8. 8.
    S. M. Batill, J. E. Renaud, and X. Gu, “Modeling and simulation uncertainty in multidisciplinary design optimization,” in Proc. 8th AIAA/USAF/NASA/ISSMO Symp. on Multidisciplinary Analysis and Optimization, 2000 (AIAA, 2000), p. 1–11 (2000).Google Scholar
  9. 9.
    J. E. Pacheco, C. H. Amon, and S. Finger, “Bayesian surrogates applied to conceptual stages of the engineering design process,” J. Mechan. Design 125, 664–672 (2003).CrossRefGoogle Scholar
  10. 10.
    A. J. Booker, J. E. Dennis, P. D. Frank, D. B. Serafini, V. Torczon, and M. Trosset, “A rigorous framework for optimization of expensive functions by surrogates,” Struct. Optimiz. 17, 1–13 (1999).CrossRefGoogle Scholar
  11. 11.
    P. N. Koch, B. A. Wujek, O. Golovidov, and T. W. Simpson, “Facilitating probabilistic multidisciplinary design optimization using kriging approximation models,” in Proc. 9th AIAA/ISSMO Symp. on Multidisciplinary Analysis and Optimization, 2002 (AIAA, 2002).Google Scholar
  12. 12.
    T. W. Simpson, T. M. Maurey, J. J. Korte, and F. Mistree, “Kriging Metamodels for Global Approximation in Simulation-Based Multidisciplinary Design Optimization,” AIAA J. 39, 2223–2241 (2001).CrossRefGoogle Scholar
  13. 13.
    T. W. Simpson, T. M. Maurey, J. J. Korte, and F. Mistree, “Metamodeling development for vehicle frontal impact simulation,” in Proc. American Society of Mechanical Engineers (ASME) Design Engineering Technical Conf.-Design Automation Conf., DETC2001/DAC-21012, Sept., 2001 (ASME, 2001).Google Scholar
  14. 14.
    C. E. Rasmussen and C. K. I. Williams, Gaussian Processes for Machine Learning (MIT Press, Cambridge, MA, 2006).zbMATHGoogle Scholar
  15. 15.
    S. N. Lophaven, H. B. Nielsen, and J. Sondergaard, Aspects of the Matlab Toolbox Dace. Tech. Rep. IMMREP-2002-13, 2002 (Techn. Univ. Denmark, Lyngby, 2002).Google Scholar
  16. 16.
    A. Bhattacharya, D. Pati, and D. Dunson, “Anisotropic function estimation using multi-bandwidth Gaussian processes,” Ann. Statist. 42, 352–381 (2014).MathSciNetCrossRefzbMATHGoogle Scholar
  17. 17.
    Ying Xiong, Wei Chen, Daniel Apley, and Xuru Ding, “A non-stationary covariance-based kriging method for metamodelling in engineering design,” Int. J. Numer. Methods Eng. 71, 733–756 (2007).CrossRefzbMATHGoogle Scholar
  18. 18.
    B. Nagy, J. L. Loeppky, and W. J. Welch, Correlation parameterization in random function models to improve normal approximation of the likelihood or posterior. Tech. Rep., 229 (Department of Statistics, Univ. British Columbia, 2007).Google Scholar
  19. 19.
    B. MacDonald, P. Ranjan, and H. Chipman, “Gpfit: an r package for fitting a Gaussian process model to deterministic simulator outputs,” J. Statist. Software 64 (12), 1–23 (2015).CrossRefGoogle Scholar
  20. 20.
    E. V. Burnaev, M. G. Belyaev, and P. V. Prihodko, “About hybrid algorithm for tuning of parameters in approximation based on linear expansions in parametric functions,” in Proc. 8th Int. Conf. on Intelligent Information Processing (IIP-2010), Paphos, Cyprus, Oct. 17–24, 2010 (MAKS, Moscow, 2011), pp. 200–203.Google Scholar
  21. 21.
    M. Belyaev and E. Burnaev, “Approximation of a multidimensional dependency based on a linear expansion in a dictionary of parametric functions,” Inform. Appl. 7, 114–125 (2013).Google Scholar
  22. 22.
    E. Burnaev and P. Prikhodko, “On a method for constructing ensembles of regression models,” Avtom. Telemekh. 74, 1630–1644 (2013).MathSciNetzbMATHGoogle Scholar
  23. 23.
    S. Grihon, E. Burnaev, M. Belyaev, and P. Prikhodko, Surrogate Modeling of Stability Constraints for Optimization of Composite Structures in Springer Series in Computational Science & Engineering (Springer, New York, 2013), pp. 359–391.Google Scholar
  24. 24.
    N. A. C. Cressie, Statistics for Spatial Data (Wiley, New York, 1993).zbMATHGoogle Scholar
  25. 25.
    E. Burnaev, E. Zaytsev, M. Panov, P. Prihodko, and Yu. Yanovich, “Modeling of nonstationary covariance function of Gaussian process using decomposition in dictionary of nonlinear functions,” in Information Technologies and Systems-2011, (Proc. Conf., Moscow, Russia, Oct. 2–7, 2011) (Instit. Problem Pered. Inf., Moscow, 2011), pp. 355–362.Google Scholar
  26. 26.
    C. M. Bishop, Pattern Recognition and Machine Learning (Springer, New York, 2006).zbMATHGoogle Scholar
  27. 27.
    A. Torn and A. Zilinskas, Global Optimization in Lecture Notes in Computer Science (Berlin, Springer-Verlag, 1989), Vol. 350.zbMATHGoogle Scholar
  28. 28.
    E. D. Dolan and J. J. More, “Benchmarking optimization software with performance profiles,” Math. Program. 91, 201–213 (2002).MathSciNetCrossRefzbMATHGoogle Scholar
  29. 29.
    A. Saltelli and I. M. Sobol, “About the use of rank transformation in sensitivity analysis of model output,” Reliab. Eng. Syst. Safety 50, 225–239 (1995).CrossRefGoogle Scholar
  30. 30.
    J. Ronkkonen and J. Lampinen, “An extended mutation concept for the local selection based differential evolution algorithm,” in Genetic and Evolutionary Computation Conf. (GECCO’2007), London, England, July 7–11, 2007 (ACM Press, New York, 2007).Google Scholar
  31. 31.
    T. Ishigami and T. Homma, “An importance qualification technique in uncertainty analysis for computer models,” in Proc. ISUMA’90, 1st Int. Symp. on Uncertainty Modelling and Analysis, (ISUMA’90), Univ. Maryland, USA, December 3–5, 1990 (IEEE, New York, 1990), pp. 398–403.Google Scholar

Copyright information

© Pleiades Publishing, Inc. 2016

Authors and Affiliations

  1. 1.Institute for Information Transmission ProblemsRussian Academy of SciencesMoscowRussia

Personalised recommendations