Statistics and Computing

, Volume 22, Issue 3, pp 773–793 | Cite as

Sequential design of computer experiments for the estimation of a probability of failure

  • Julien BectEmail author
  • David Ginsbourger
  • Ling Li
  • Victor Picheny
  • Emmanuel VazquezEmail author


This paper deals with the problem of estimating the volume of the excursion set of a function f:ℝ d →ℝ above a given threshold, under a probability measure on ℝ d that is assumed to be known. In the industrial world, this corresponds to the problem of estimating a probability of failure of a system. When only an expensive-to-simulate model of the system is available, the budget for simulations is usually severely limited and therefore classical Monte Carlo methods ought to be avoided. One of the main contributions of this article is to derive SUR (stepwise uncertainty reduction) strategies from a Bayesian formulation of the problem of estimating a probability of failure. These sequential strategies use a Gaussian process model of f and aim at performing evaluations of f as efficiently as possible to infer the value of the probability of failure. We compare these strategies to other strategies also based on a Gaussian process model for estimating a probability of failure.


Computer experiments Sequential design Gaussian processes Probability of failure Stepwise uncertainty reduction 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Arnaud, A., Bect, J., Couplet, M., Pasanisi, A., Vazquez, E.: Évaluation d’un risque d’inondation fluviale par planification séquentielle d’expériences. In: 42èmes Journées de Statistique (2010) Google Scholar
  2. Au, S.K., Beck, J.: Estimation of small failure probabilities in high dimensions by subset simulation. Probab. Eng. Mech. 16(4), 263–277 (2001) CrossRefGoogle Scholar
  3. Bayarri, M.J., Berger, J.O., Paulo, R., Sacks, J., Cafeo, J.A., Cavendish, J., Lin, C.H., Tu, J.: A framework for validation of computer models. Technometrics 49(2), 138–154 (2007) MathSciNetCrossRefGoogle Scholar
  4. Berry, D.A., Fristedt, B.: Bandit Problems: Sequential Allocation of Experiments. Chapman & Hall, London (1985) zbMATHGoogle Scholar
  5. Bertsekas, D.P.: Dynamic Programming and Optimal Control, vol. 1. Athena Scientific, Nashua (1995) zbMATHGoogle Scholar
  6. Bichon, B.J., Eldred, M.S., Swiler, L.P., Mahadevan, S., McFarland, J.M.: Efficient global reliability analysis for nonlinear implicit performance functions. AIAA J. 46(10), 2459–2468 (2008) CrossRefGoogle Scholar
  7. Bjerager, P.: On computational methods for structural reliability analysis. Struct. Saf. 9, 76–96 (1990) CrossRefGoogle Scholar
  8. Borri, A., Speranzini, E.: Structural reliability analysis using a standard deterministic finite element code. Struct. Saf. 19(4), 361–382 (1997) CrossRefGoogle Scholar
  9. Box, G.E.P., Draper, N.R.: Empirical Model-Building and Response Surfaces. Wiley, New York (1987) zbMATHGoogle Scholar
  10. Bucher, C.G., Bourgund, U.: A fast and efficient response surface approach for structural reliability problems. Struct. Saf. 7(1), 57–66 (1990) CrossRefGoogle Scholar
  11. Chilès, J.P., Delfiner, P.: Geostatistics: Modeling Spatial Uncertainty. Wiley, New York (1999) zbMATHCrossRefGoogle Scholar
  12. Currin, C., Mitchell, T., Morris, M., Ylvisaker, D.: Bayesian prediction of deterministic functions, with applications to the design and analysis of computer experiments. J. Am. Stat. Assoc. 86(416), 953–963 (1991) MathSciNetCrossRefGoogle Scholar
  13. Deheeger, F.: Couplage mécano-fiabiliste : 2SMART—méthodologie d’apprentissage stochastique en fiabilité. Ph.D. thesis, Université Blaise Pascal, Clermont II (2008) Google Scholar
  14. Deheeger, F., Lemaire, M.: Support vector machine for efficient subset simulations: 2SMART method. In: Kanda, J., Takada, T., Furuta, H. (eds.) 10th International Conference on Application of Statistics and Probability in Civil Engineering, Proceedings and Monographs in Engineering, Water and Earth Sciences, pp. 259–260. Taylor & Francis, London (2007) Google Scholar
  15. Echard, B., Gayton, N., Lemaire, M.: Kriging-based Monte Carlo simulation to compute the probability of failure efficiently: AK-MCS method. In: 6èmes Journées Nationales de Fiabilité, Toulouse, France, 24–26 mars 2010a Google Scholar
  16. Echard, B., Gayton, N., Lemaire, M.: Structural reliability assessment using kriging metamodel and active learning. In: IFIP WG 7.5 Working Conference on Reliability and Optimization of Structural Systems (2010b) Google Scholar
  17. Fleuret, F., Geman, D.: Graded learning for object detection. In: Proceedings of the Workshop on Statistical and Computational Theories of Vision of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR/SCTV) (1999) Google Scholar
  18. Frazier, P.I., Powell, W.B., Dayanik, S.: A knowledge-gradient policy for sequential information collection. SIAM J. Control Optim. 47(5), 2410–2439 (2008) MathSciNetzbMATHCrossRefGoogle Scholar
  19. Ginsbourger, D.: Métamodèles multiples pour l’approximation et l’optimisation de fonctions numériques multivariables. Ph.D. thesis, Ecole nationale supérieure des Mines de Saint-Etienne (2009) Google Scholar
  20. Ginsbourger, D., Le Riche, R., Carraro, L.: Kriging is well-suited to parallelize optimization. In: Hiot, L.M., Ong, Y.S., Tenne, Y., Goh, C.K. (eds.) Computational Intelligence in Expensive Optimization Problems. Adaptation Learning and Optimization, vol. 2, pp. 131–162. Springer, Berlin (2010) CrossRefGoogle Scholar
  21. Handcock, M.S., Stein, M.L.: A Bayesian analysis of kriging. Technometrics 35(4), 403–410 (1993) CrossRefGoogle Scholar
  22. Hornik, K.: The R FAQ (2010). ISBN 3-900051-08-9
  23. Hurtado, J.E.: An examination of methods for approximating implicit limit state functions from the viewpoint of statistical learning theory. Struct. Saf. 26(3), 271–293 (2004) CrossRefGoogle Scholar
  24. Hurtado, J.E.: Filtered importance sampling with support vector margin: a powerful method for structural reliability analysis. Struct. Saf. 29(1), 2–15 (2007) CrossRefGoogle Scholar
  25. Jones, D.R., Schonlau, M., William, J.: Efficient global optimization of expensive black-box functions. J. Glob. Optim. 13(4), 455–492 (1998) zbMATHCrossRefGoogle Scholar
  26. Kennedy, M., O’Hagan, A.: Bayesian calibration of computer models. J. R. Stat. Soc., Ser. B, Stat. Methodol. 63(3), 425–464 (2001) MathSciNetzbMATHCrossRefGoogle Scholar
  27. Kimeldorf, G.S., Wahba, G.: A correspondence between Bayesian estimation on stochastic processes and smoothing by splines. Ann. Math. Stat. 41(2), 495–502 (1970) MathSciNetzbMATHCrossRefGoogle Scholar
  28. Kushner, H.J.: A new method of locating the maximum point of an arbitrary multipeak curve in the presence of noise. J. Basic Eng. 86, 97–106 (1964) CrossRefGoogle Scholar
  29. Loeppky, J.L., Sacks, J., Welch, W.J.: Choosing the sample size of a computer experiment: a practical guide. Technometrics 51(4), 366–376 (2009) MathSciNetCrossRefGoogle Scholar
  30. Mockus, J.: Bayesian Approach to Global Optimization. Theory and Applications. Kluwer Academic, Dordrecht (1989) zbMATHCrossRefGoogle Scholar
  31. Mockus, J., Tiesis, V., Zilinskas, A.: The application of Bayesian methods for seeking the extremum. In: Dixon, L., Szego, E.G. (eds.) Towards Global Optimization, vol. 2, pp. 117–129. Elsevier, Amsterdam (1978) Google Scholar
  32. Oakley, J.: Estimating percentiles of uncertain computer code outputs. J. R. Stat. Soc., Ser. C 53(1), 83–93 (2004) MathSciNetzbMATHCrossRefGoogle Scholar
  33. Oakley, J., O’Hagan, A.: Bayesian inference for the uncertainty distribution of computer model outputs. Biometrika 89(4) (2002) Google Scholar
  34. Oakley, J., O’Hagan, A.: Probabilistic sensitivity analysis of complex models: a Bayesian approach. J. R. Stat. Soc., Ser. B, Stat. Methodol. 66(3), 751–769 (2004) MathSciNetzbMATHCrossRefGoogle Scholar
  35. O’Hagan, A.: Curve fitting and optimal design for prediction. J. R. Stat. Soc., Ser. B, Methodol. 40(1), 1–42 (1978) MathSciNetzbMATHGoogle Scholar
  36. Parzen, E.: An approach to time series analysis. Ann. Math. Stat. 32, 951–989 (1962) MathSciNetCrossRefGoogle Scholar
  37. Paulo, R.: Default priors for Gaussian processes. Ann. Stat. 33(2), 556–582 (2005) MathSciNetzbMATHCrossRefGoogle Scholar
  38. Picheny, V., Ginsbourger, D.: KrigInv: Kriging-based inversion for deterministic and noisy computer experiments, version 1.1 (2011).
  39. Picheny, V., Ginsbourger, D., Roustant, O., Haftka, R.T., Kim, N.H.: Adaptive designs of experiments for accurate approximation of target regions. J. Mech. Des. 132(7), 1–9 (2010) CrossRefGoogle Scholar
  40. Piera-Martinez, M.: Modélisation des comportements extrêmes en ingénierie. Ph.D. thesis, Université Paris Sud, Paris XI (2008) Google Scholar
  41. Pradlwarter, H., Schuëller, G., Koutsourelakis, P., Charmpis, D.: Application of line sampling simulation method to reliability benchmark problems. Struct. Saf. 29(3), 208–221 (2007). A Benchmark Study on Reliability in High Dimensions CrossRefGoogle Scholar
  42. Press, W.H., Teukolsky, S.A., Vetterling, W.T., Flannery, B.P.: Numerical Recipes in C. The Art of Scientific Computing, 2nd edn. Cambridge University Press, Cambridge (1992) zbMATHGoogle Scholar
  43. Rajashekhar, M.R., Ellingwood, B.R.: A new look at the response surface approach for reliability analysis. Struct. Saf. 12(3), 205–220 (1993) CrossRefGoogle Scholar
  44. Ranjan, P., Bingham, D., Michailidis, G.: Sequential experiment design for contour estimation from complex computer codes. Technometrics 50(4), 527–541 (2008) MathSciNetCrossRefGoogle Scholar
  45. Rubinstein, R., Kroese, D.: The Cross-Entropy Method. Springer, Berlin (2004) zbMATHGoogle Scholar
  46. Sacks, J., Welch, W.J., Mitchell, T.J., Wynn, H.P.: Design and analysis of computer experiments. Stat. Sci. 4(4), 409–435 (1989) MathSciNetzbMATHCrossRefGoogle Scholar
  47. Santner, T.J., Williams, B.J., Notz, W.: The Design and Analysis of Computer Experiments. Springer, Berlin (2003) zbMATHGoogle Scholar
  48. Schueremans, L.: Probabilistic evaluation of structural unreinforced masonry. Ph.D. thesis, Catholic University of Leuven (2001) Google Scholar
  49. Schueremans, L., Gemert, D.V.: Benefit of splines and neural networks in simulation based structural reliability analysis. Struct. Saf. 27(3), 246–261 (2005) CrossRefGoogle Scholar
  50. Stein, M.L.: Interpolation of Spatial Data: Some Theory for Kriging. Springer, New York (1999) zbMATHCrossRefGoogle Scholar
  51. Vazquez, E., Bect, J.: A sequential Bayesian algorithm to estimate a probability of failure. In: Proceedings of the 15th IFAC Symposium on System Identification, SYSID 2009, Saint-Malo, France (2009) Google Scholar
  52. Vazquez, E., Piera-Martinez, M.: Estimation du volume des ensembles d’excursion d’un processus gaussien par krigeage intrinsèque. In: 39ème Journées de Statistiques Conférence Journée de Statistiques, Angers, France (2007) Google Scholar
  53. Vestrup, E.M.: The Theory of Measures and Integration. Wiley, New York (2003) zbMATHCrossRefGoogle Scholar
  54. Villemonteix, J.: Optimisation de fonctions coûteuses. Ph.D. thesis, Université Paris-Sud XI, Faculté des Sciences d’Orsay (2008) Google Scholar
  55. Villemonteix, J., Vazquez, E., Walter, E.: An informational approach to the global optimization of expensive-to-evaluate functions. J. Glob. Optim. 44(4), 509–534 (2009) MathSciNetzbMATHCrossRefGoogle Scholar
  56. Waarts, P.H.: Structural reliability using finite element methods: an appraisal of DARS. Ph.D. thesis, Delft University of Technology (2000) Google Scholar
  57. Welch, W.J., Buck, R.J., Sacks, J., Wynn, H.P., Mitchell, T.J., Morris, M.D.: Screening, predicting and computer experiments. Technometrics 34, 15–25 (1992) CrossRefGoogle Scholar
  58. Yaglom, A.M.: Correlation Theory of Stationary and Related Random Functions I: Basic Results. Springer Series in Statistics. Springer, New York (1986) Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2011

Authors and Affiliations

  1. 1.SUPELECGif-sur-YvetteFrance
  2. 2.Ecole Centrale ParisChatenay-MalabryFrance
  3. 3.Institute of Mathematical Statistics and Actuarial ScienceUniversity of BernBernSwitzerland

Personalised recommendations