Gaussian Process Emulators for Computer Experiments with Inequality Constraints

Abstract

Physical phenomena are observed in many fields (science and engineering) and are often studied by time-consuming computer codes. These codes are analyzed with statistical models, often called emulators. In many situations, the physical system (computer model output) may be known to satisfy inequality constraints with respect to some or all input variables. The aim is to build a model capable of incorporating both data interpolation and inequality constraints into a Gaussian process emulator. By using a functional decomposition, a finite-dimensional approximation of Gaussian processes such that all conditional simulations satisfy the inequality constraints in the entire domain is proposed. To show the performance of the proposed model, some conditional simulations with inequality constraints such as boundedness, monotonicity or convexity conditions in one and two dimensions are given. A simulation study to investigate the efficiency of the method in terms of prediction is included.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

References

  1. Abrahamsen P, Benth FE (2001) Kriging with inequality constraints. Math Geol 33(6):719–744

    Article  Google Scholar 

  2. Aronszajn N (1950) Theory of reproducing kernels. Trans Am Math Soc 68(3):337–404

    Article  Google Scholar 

  3. Bay X, Grammont L, Maatouk H (2015) A new method for interpolating in a convex subset of a Hilbert space Hal-01136466. https://hal.archives-ouvertes.fr/hal-01136466

  4. Bay X, Grammont L, Maatouk H (2016) Generalization of the Kimeldorf–Wahba correspondence for constrained interpolation. Electron J Stat 10(1):1580–1595

    Article  Google Scholar 

  5. Botts C (2013) An accept–reject algorithm for the positive multivariate normal distribution. Comput Stat 28(4):1749–1773

    Article  Google Scholar 

  6. Boyd S, Vandenberghe L (2004) Convex optimization. Cambridge University Press, New York

    Google Scholar 

  7. Chopin N (2011) Fast simulation of truncated Gaussian distributions. Stat Comput 21(2):275–288

    Article  Google Scholar 

  8. Cousin A, Maatouk H, Rullière D (2016) Kriging of financial term-structures. Eur J Oper Res 255(2):631–648

    Article  Google Scholar 

  9. Cramer H, Leadbetter R (1967) Stationary and related stochastic processes: sample function properties and their applications. Wiley series in probability and mathematical statistics. Tracts on probability and statistics. Wiley, Hoboken

    Google Scholar 

  10. Cressie N, Johannesson G (2008) Fixed rank kriging for very large spatial data sets. J R Stat Soc Ser B Stat Methodol 70(1):209–226

    Article  Google Scholar 

  11. Da Veiga S, Marrel A (2012) Gaussian process modeling with inequality constraints. Ann Fac Sci Toulouse 21(3):529–555

    Article  Google Scholar 

  12. Dole D (1999) CoSmo: a constrained scatterplot smoother for estimating convex, monotonic transformations. J Bus Econ Stat 17(4):444–455

    Google Scholar 

  13. Emery X, Arroyo D, Peláez M (2014) Simulating large Gaussian random vectors subject to inequality constraints by gibbs sampling. Math Geosci 46(3):265–283

  14. Freulon X, de Fouquet C (1993) Conditioning a Gaussian model with inequalities. In: Geostatistics Troia92. Springer, Netherlands, pp 201–212

  15. Fritsch FN, Carlson RE (1980) Monotone piecewise cubic interpolation. SIAM J Numer Anal 17(2):238–246

    Article  Google Scholar 

  16. Geweke J (1991) Efficient simulation from the multivariate normal and student-t distributions subject to linear constraints and the evaluation of constraint probabilities. In: Computing science and statistics: proceedings of the 23rd symposium on the interface, pp 571–578

  17. Golchi S, Bingham D, Chipman H, Campbell DA (2015) Monotone emulation of computer experiments. SIAM/ASA J Uncertain Quantif 3(1):370–392

    Article  Google Scholar 

  18. Goldfarb D, Idnani A (1982) Dual and primal-dual methods for solving strictly convex quadratic programs. In: Numerical analysis. Springer, Berlin, pp 226–239

  19. Goldfarb D, Idnani A (1983) A numerically stable dual method for solving strictly convex quadratic programs. Math Program 27(1):1–33

    Article  Google Scholar 

  20. Jones DR, Schonlau M, Welch W (1998) Efficient global optimization of expensive black-box functions. J Glob Optim 13(4):455–492

    Article  Google Scholar 

  21. Kimeldorf GS, Wahba G (1970) A correspondence between Bayesian estimation on stochastic processes and smoothing by splines. Ann Math Stat 41(2):495–502

    Article  Google Scholar 

  22. Kleijnen JP, Van Beers WC (2013) Monotonicity-preserving bootstrapped Kriging metamodels for expensive simulations. J Oper Res Soc 64(5):708–717

    Article  Google Scholar 

  23. Kotecha JH, Djuric PM (1999) Gibbs sampling approach for generation of truncated multivariate Gaussian random variables. In: Proceedings on 1999 IEEE International Conference on Acoustics, Speech, and Signal Processing, 1999, vol 3, pp 1757–1760

  24. Maatouk H (2015) Correspondence between Gaussian process regression and interpolation splines under linear inequality constraints. Theory and applications. Ph.D. thesis, École des Mines de St-Étienne

  25. Maatouk H, Bay X (2016) A new rejection sampling method for truncated multivariate Gaussian random variables restricted to convex sets. In: Nuyens R, Cools R (ed) Monte carlo and quasi-monte carlo methods, vol 163. Springer International Publishing, Cham, pp 521–530

  26. Maatouk H, Richet Y (2015) constrKriging R package. https://github.com/maatouk/constrKriging

  27. Maatouk H, Roustant O, Richet Y (2015) Cross-validation estimations of hyper-parameters of Gaussian processes with inequality constraints. Procedia Environ Sci 27:38–44

    Article  Google Scholar 

  28. Micchelli C, Utreras F (1988) Smoothing and interpolation in a convex subset of a Hilbert space. SIAM J Sci Stat Comput 9(4):728–746

    Article  Google Scholar 

  29. Parzen E (1962) Stochastic processes. Holden-Day series in probability and statistics. Holden-Day, San Francisco

    Google Scholar 

  30. Philippe A, Robert CP (2003) Perfect simulation of positive Gaussian distributions. Stat Comput 13(2):179–186

    Article  Google Scholar 

  31. Ramsay JO (1988) Monotone regression splines in action. Stat Sci 3(4):425–441

    Article  Google Scholar 

  32. Rasmussen CE, Williams CK (2005) Gaussian processes for machine learning (adaptive computation and machine learning). The MIT Press, Cambridge

    Google Scholar 

  33. Riihimaki J, Vehtari A (2010) Gaussian processes with monotonicity information. JMLR Proc AISTATS 9:645–652

    Google Scholar 

  34. Robert CP (1995) Simulation of truncated normal variables. Stat Comput 5(2):121–125

    Article  Google Scholar 

  35. Roustant O, Ginsbourger D, Deville Y (2012) DiceKriging, DiceOptim: two R packages for the analysis of computer experiments by kriging-based metamodeling and optimization. J Stat Softw 51(1):1–55

    Article  Google Scholar 

  36. Sacks J, Welch WJ, Mitchell TJ, Wynn HP (1989) Design and analysis of computer experiments. Stat Sci 4(4):409–423

    Article  Google Scholar 

  37. Villalobos M, Wahba G (1987) Inequality-constrained multivariate smoothing splines with application to the estimation of posterior probabilities. J Am Stat Assoc 82(397):239–248

    Article  Google Scholar 

  38. Wolberg G, Alfy I (2002) An energy-minimization framework for monotonic cubic spline interpolation. J Comput Appl Math 143(2):145–188

    Article  Google Scholar 

  39. Wright IW, Wegman EJ (1980) Isotonic, convex and related splines. Ann Stat 8(5):1023–1035

    Article  Google Scholar 

  40. Xiaojing W (2012) Bayesian modeling using latent structures. Ph.D. thesis, Duke University, Department of Statistical Science

  41. Xuming H, Peide S (1996) Monotone B-spline smoothing. J Am Stat Assoc 93:643–650

    Google Scholar 

Download references

Acknowledgements

The authors would like to thank the Associate Editor and the two anonymous referees for their helpful comments. Part of this work has been conducted within the frame of the ReDice Consortium, gathering industrial (CEA, EDF, IFPEN, IRSN, Renault) and academic (École des Mines de Saint-Étienne, INRIA, and the University of Bern) partners around advanced methods for Computer Experiments. The authors also thank Olivier Roustant (ENSM-SE), Laurence Grammont (ICJ, Lyon1) and Yann Richet (IRSN) for helpful discussions as well as the participants of the UCM2014 conference and GRF-Sim2014 workshop.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Hassan Maatouk.

Appendix: Proofs of Propositions

Appendix: Proofs of Propositions

Proof

(Proofs of Propositions 1 and 2) Let \(Y^N\) be defined as in (6) (respectively, in 8). Since \(Y(u_j), \ j=0,\ldots ,N\) are Gaussian variables, then \(Y^N\) is a GP with dimension equal to \(N+1\) and covariance function

$$\begin{aligned} {\mathrm {Cov}}\left( Y^N(x),Y^N(x')\right)= & {} \sum _{i,j=0}^N{\mathrm {Cov}}\left( Y(u_i),Y(u_j)\right) h_i(x)h_j(x')\\= & {} \sum _{i,j=0}^NK(u_i,u_j)h_i(x)h_j(x'). \end{aligned}$$

The derivative of a GP is also a GP, respectively. For all \(x,x'\in [0,1]\),

$$\begin{aligned} K_N(x,x')= & {} {\mathrm {Cov}}\left( Y^N(x),Y^N(x')\right) ={\mathrm {Var}}\left( Y(0)\right) +\sum _{i=0}^N\frac{\partial K}{\partial x}(u_i,0)\phi _i(x)\\&+\sum _{j=0}^N\frac{\partial K}{\partial x'}(0,u_j)\phi _j(x)+\sum _{i,j=0}^N\frac{\partial ^2 K}{\partial x\partial x'}(u_i,u_j)\phi _i(x)\phi _j(x'). \end{aligned}$$

To prove the pathwise convergence of \(Y^N\) to Y, write more explicitly, for any \(\omega \in \varOmega \)

$$\begin{aligned} Y^N(x;\omega )=\sum _{j=0}^NY(u_j;\omega )h_j(x). \end{aligned}$$

Hence, the sample paths of the approximating process \(Y^N\) are piecewise linear approximations of the sample paths of the original process Y. From \(h_j\ge 0\) and \(\sum _{j=0}^Nh_j(x)=1\), for all \(x\in [0,1]\),

$$\begin{aligned} \left| Y^N(x;\omega )-Y(x;\omega )\right|= & {} \left| \sum _{j=0}^N(Y(u_j;\omega )-Y(x;\omega )h_j(x)\right| \nonumber \\\le & {} \sum _{j=0}^N \sup _{|x-x'|\le \varDelta _N}\left| Y(x';\omega )-Y(x;\omega )\right| h_j(x)\nonumber \\= & {} \sup _{|x-x'|\le \varDelta _N}\left| Y(x';\omega )-Y(x;\omega )\right| . \end{aligned}$$
(13)

By uniformly continuity of sample paths of the process Y on the compact interval [0, 1], this last inequality (13) shows that

$$\begin{aligned} \sup _{x\in [0,1]}\left| Y^N(x;\omega )-Y(x;\omega )\right| \underset{N\rightarrow +\infty }{\longrightarrow }0, \end{aligned}$$

with probability one. Let us write, respectively, that for any \(\omega \in \varOmega \)

$$\begin{aligned} Y^N(x;\omega )=Y(0;\omega )+\int _0^x\left( \sum _{j=0}^NY'(u_j;\omega )h_j(t)\right) {\hbox {d}}t. \end{aligned}$$

From Proposition 1, \(\sum _{j=0}^NY'(u_j;\omega )h_j(x)\) converges uniformly pathwise to \(Y'(x;\omega )\) since the realizations of the process are almost surely continuously differentiable. One can conclude that \(Y^N\) converges uniformly to Y for almost all \(\omega \in \varOmega \).

The last property done by the fact that \(Y^N\) is a piecewise linear approximation and

$$\begin{aligned} Y^N(u_i)=\sum \limits _{j=0}^NY(u_j)h_j(u_i)=\sum \limits _{j=0}^NY(u_j)\delta _{ij}=Y(u_i)\in [a,b], \end{aligned}$$

\(i=0,\ldots ,N\), (respectively, the fact that the basis functions are non-decreasing and

$$\begin{aligned} 0\le \left( Y^N\right) '(u_i)=\sum _{j=0}^NY'(u_j)h_j(u_i)=Y'(u_j), \end{aligned}$$

\(i=0,\ldots ,N\)). \(\square \)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Maatouk, H., Bay, X. Gaussian Process Emulators for Computer Experiments with Inequality Constraints. Math Geosci 49, 557–582 (2017). https://doi.org/10.1007/s11004-017-9673-2

Download citation

Keywords

  • Gaussian process emulator
  • Inequality constraints
  • Finite-dimensional approximation
  • Uncertainty quantification
  • Design and modeling of computer experiments