Recent Results on Nonparametric Quantile Estimation in a Simulation Model

  • Adam KrzyżakEmail author
Part of the Studies in Computational Intelligence book series (SCI, volume 605)


We present recent results on nonparametric estimation of a quantile of distribution of Y given by a simulation model \(Y=m(X)\), where \(m: \mathbb {R}^d\rightarrow \mathbb {R}\) is a function which is costly to compute and X is a \(\mathbb {R}^d\)-valued random variable with given density. We argue that importance sampling quantile estimate of m(X), based on a suitable estimate \(m_n\) of m achieves better rate of convergence than the estimate based on order statistics alone. Similar results are given for Robbins-Monro type recursive importance sampling and for quantile estimation based on surrogate model.


Order Statistic Importance Sampling Surrogate Function Piecewise Polynomial Monte Carlo Estimate 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



The author would like to thank his co-authors and acknowledge the support from the Natural Sciences and Engineering Research Council of Canada.


  1. 1.
    Arnold BC, Balakrishnan N, Nagaraja HN (1992) A first course in order statistics. Wiley, New YorkzbMATHGoogle Scholar
  2. 2.
    Beirlant J, Györfi L (1998) On the asymptotic \({L}_2\)-error in partitioning regression estimation. J Stat Plan Inference 71:93–107CrossRefzbMATHGoogle Scholar
  3. 3.
    Benveniste A, Métivier M, Priouret P (1990) Adaptive algorithms and stochastic approximation. Springer, New YorkCrossRefGoogle Scholar
  4. 4.
    Bichon B, Eldred M, Swiler M, Mahadevan S, McFarland J (2008) Efficient global reliability analysis for nonlinear implicit performance functions. AIAA J 46:2459–2468CrossRefGoogle Scholar
  5. 5.
    Bourinet JM, Deheeger F, Lemaire M (2011) Assessing small failure probabilities by combined subset simulation and support vector machines. Struct Saf 33:343–353CrossRefGoogle Scholar
  6. 6.
    Cannamela C, Garnier J, Iooss B (2008) Controlled stratification for quantile estimation. Ann Appl Stat 2(4):1554–1580MathSciNetCrossRefzbMATHGoogle Scholar
  7. 7.
    Chen H-F (2002) Stochastic approximation and its applications. Kluwer Academic Publishers, BostonzbMATHGoogle Scholar
  8. 8.
    Das PK, Zheng Y (2000) Cumulative formation of response surface and its use in reliability analysis. Probab Eng Mech 15:309–315CrossRefzbMATHGoogle Scholar
  9. 9.
    de Boor C (1978) A practical guide to splines. Springer, New YorkCrossRefzbMATHGoogle Scholar
  10. 10.
    Deheeger F, Lemaire M (2010) Support vector machines for efficient subset simulations: \(^2\)SMART method. In: Proceedings of the 10th international conference on applications of statistics and probability in civil engineering (ICASP10), Tokyo, JapanGoogle Scholar
  11. 11.
    Devroye L, Wagner TJ (1980) Distribution-free consistency results in nonparametric discrimination and regression function estimation. Ann Stat 8:231–239MathSciNetCrossRefzbMATHGoogle Scholar
  12. 12.
    Devroye L (1982) Necessary and sufficient conditions for the almost everywhere convergence of nearest neighbor regression function estimates. Zeitschrift für Wahrscheinlichkeitstheorie und verwandte Gebiete 61:467–481MathSciNetCrossRefGoogle Scholar
  13. 13.
    Devroye L, Krzyżak A (1989) An equivalence theorem for \({L}_1\) convergence of the kernel regression estimate. J Stat Plan Inference 23:71–82CrossRefzbMATHGoogle Scholar
  14. 14.
    Devroye L, Györfi L, Krzyżak A, Lugosi G (1994) On the strong universal consistency of nearest neighbor regression function estimates. Ann Stat 22:1371–1385CrossRefGoogle Scholar
  15. 15.
    Dubourg V, Sudret B, Deheeger F (2013) Metamodel-based importance sampling for structural reliability analysis. Probab Eng Mech 33:47–57CrossRefGoogle Scholar
  16. 16.
    Egloff D, Leippold M (2010) Quantile estimation with adaptive importance sampling. Ann Stat 38(2):1244–1278MathSciNetCrossRefzbMATHGoogle Scholar
  17. 17.
    Enss GC, Kohler M, Krzyżak A, Platz R (2014) Nonparametric quantile estimation based on surrogate models. Submitted for publicationGoogle Scholar
  18. 18.
    Glasserman P (2004) Monte Carlo methods in financial engineering. Springer, New YorkzbMATHGoogle Scholar
  19. 19.
    Greblicki W, Pawlak M (1985) Fourier and Hermite series estimates of regression functions. Ann Inst Stat Math 37:443–454MathSciNetCrossRefzbMATHGoogle Scholar
  20. 20.
    Györfi L (1981) Recent results on nonparametric regression estimate and multiple classification. Probl Control Inf Theory 10:43–52zbMATHGoogle Scholar
  21. 21.
    Györfi L, Kohler M, Krzyżak A, Walk H (2002) A distribution-free theory of nonparametric regression. Springer series in statistics. Springer, New YorkGoogle Scholar
  22. 22.
    Holst U (1987) Recursive estimation of quantiles using recursive kernel density estimators. Seq Anal: Des Methods Appl 6(3):219–237MathSciNetCrossRefGoogle Scholar
  23. 23.
    Hurtado J (2004) Structural reliability—statistical learning perspectives, Lecture notes in applied and computational mechanics, vol 17. Springer, New YorkGoogle Scholar
  24. 24.
    Kaymaz I (2005) Application of Kriging method to structural reliability problems. Struct Saf 27:133–151CrossRefGoogle Scholar
  25. 25.
    Kim SH, Na SW (1997) Response surface method using vector projected sampling points. Struct Saf 19:3–19CrossRefGoogle Scholar
  26. 26.
    Koenker R (2005) Quantile regression. Cambridge University Press, New YorkCrossRefzbMATHGoogle Scholar
  27. 27.
    Kohler M (2000) Inequalities for uniform deviations of averages from expectations with applications to nonparametric regression. J Stat Plan Inference 89:1–23MathSciNetCrossRefGoogle Scholar
  28. 28.
    Kohler M, Krzyżak A (2001) Nonparametric regression estimation using penalized least squares. IEEE Trans Inf Theory 47:3054–3058CrossRefzbMATHGoogle Scholar
  29. 29.
    Kohler M (2014) Optimal global rates of convergence for noiseless regression estimation problems with adaptively chosen design. J Multivar Anal 132:197–208MathSciNetCrossRefGoogle Scholar
  30. 30.
    Kohler M, Krzyżak A, Walk H (2014) Nonparametric recursive quantile estimation. Stat Probab Lett 93:102–107CrossRefGoogle Scholar
  31. 31.
    Kohler M, Krzyżak A, Tent R, Walk H (2014) Nonparametric quantile estimation using importance sampling. Submitted for publicationGoogle Scholar
  32. 32.
    Kushner HJ, Yin G (2003) Stochastic approximation and recursive algorithms and applications, 2nd edn. Springer, New YorkzbMATHGoogle Scholar
  33. 33.
    Ljung L, Pflug G, Walk H (1992) Stochastic approximation and optimization of random systems. Birkhäuser Verlag, BaselCrossRefzbMATHGoogle Scholar
  34. 34.
    Lugosi G, Zeger K (1995) Nonparametric estimation via empirical risk minimization. IEEE Trans Inf Theory 41:677–687MathSciNetCrossRefzbMATHGoogle Scholar
  35. 35.
    Morio J (2012) Extreme quantile estimation with nonparametric adaptive importance sampling. Simul Model Pract Theory 27:76–89CrossRefGoogle Scholar
  36. 36.
    Nadaraya EA (1964) On estimating regression. Theory Probab Appl 9:141–142CrossRefGoogle Scholar
  37. 37.
    Nadaraya EA (1970) Remarks on nonparametric estimates for density functions and regression curves. Theory Probab Appl 15:134–137CrossRefzbMATHGoogle Scholar
  38. 38.
    Neddermeyer JC (2009) Computationally efficient nonparametric importance sampling. J Am Stat Assoc 104(486):788–802MathSciNetCrossRefGoogle Scholar
  39. 39.
    Oakley J (2004) Estimating percentiles of uncertain computer code outputs. J R Stat Soc: Ser C (Appl Stat) 53(1):83–93MathSciNetCrossRefzbMATHGoogle Scholar
  40. 40.
    Papadrakakis M, Lagaros N (2002) Reliability-based structural optimization using neural networks and Monte Carlo simulation. Comput Methods Appl Mech Eng 191:3491–3507CrossRefzbMATHGoogle Scholar
  41. 41.
    Polyak BT, Juditsky AB (2002) Acceleration of stochastic approximation by averaging. SIAM J Control Optim 30(4):838–855MathSciNetCrossRefGoogle Scholar
  42. 42.
    Rafajłowicz E (1987) Nonparametric orthogonal series estimators of regression: a class attaining the optimal convergence rate in L2. Stat Probab Lett 5:219–224CrossRefGoogle Scholar
  43. 43.
    Robbins H, Monro S (1951) A stochastic approximation method. Ann Math Stat 22(3):400–407MathSciNetCrossRefzbMATHGoogle Scholar
  44. 44.
    Ruppert D (1991) Stochastic approximation. In: Gosh BK, Sen PK (eds) Handbook of sequential analysis, Ch. 22. Marcel Dekker, New York, pp 503–529Google Scholar
  45. 45.
    Santner TJ, Williams BJ, Notz WI (2003) The design and analysis of computer experiments. Springer, New YorkCrossRefzbMATHGoogle Scholar
  46. 46.
    Stone CJ (1977) Consistent nonparametric regression. Ann Stat 5:595–645CrossRefGoogle Scholar
  47. 47.
    Stone CJ (1982) Optimal global rates of convergence for nonparametric regression. Ann Stat 10:1040–1053CrossRefzbMATHGoogle Scholar
  48. 48.
    Takeuchi I, Le QV, Sears TD, Smola AJ (2006) Nonparametric quantile estimation. J Mach Learn Res 7:1231–1264MathSciNetzbMATHGoogle Scholar
  49. 49.
    Tierney L (1983) A space-efficient recursive procedure for estimating a quantile of an unknown distribution. SIAM J Sci Stat Comput 4(4):706–711MathSciNetCrossRefGoogle Scholar
  50. 50.
    Watson GS (1964) Smooth regression analysis. Sankhya Ser A 26:359–372MathSciNetzbMATHGoogle Scholar
  51. 51.
    Wahba G (1990) Spline models for observational data. SIAM, PhiladelphiaCrossRefzbMATHGoogle Scholar
  52. 52.
    Yu K, Lu Z, Stander J (2003) Quantile regression: application and current research areas. J R Stat Soc, Ser D 52:331–350MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  1. 1.Department of Computer Science and Software EngineeringConcordia UniversityMontrealCanada

Personalised recommendations