Statistics and Computing

, Volume 28, Issue 4, pp 725–738 | Cite as

Reliable error estimation for Sobol’ indices

  • Lluís Antoni Jiménez Rugama
  • Laurent Gilquin


In the field of sensitivity analysis, Sobol’ indices are sensitivity measures widely used to assess the importance of inputs of a model to its output. The estimation of these indices is often performed through Monte Carlo or quasi-Monte Carlo methods. A notable method is the replication procedure that estimates first-order indices at a reduced cost in terms of number of model evaluations. An inherent practical problem of this estimation is how to quantify the number of model evaluations needed to ensure that estimates satisfy a desired error tolerance. This article addresses this challenge by proposing a reliable error bound for first-order and total effect Sobol’ indices. Starting from the integral formula of the indices, the error bound is defined in terms of the discrete Walsh coefficients of the different integrands. We propose a sequential estimation procedure of Sobol’ indices using the error bound as a stopping criterion. The sequential procedure combines Sobol’ sequences with either Saltelli’s strategy to estimate both first-order and total effect indices, or the replication procedure to estimate only first-order indices.


Sobol’ index Error bound Sequential method Quasi-Monte Carlo 

Mathematics Subject Classification

49Q12 62L12 65R10 



The authors thank Fred J. Hickernell and Clémentine Prieur for initiating this collaborative work, and Elise Arnaud for her proofreading. The authors are grateful to Stephen Joe, Frances Y. Kuo and Art B. Owen for their helpful answers and suggestions. The authors also thank the associate editor and the two anonymous reviewers for their helpful suggestions and comments which substantially improved the quality of this paper.


  1. Bratley, P., Fox, B.L., Niederreiter, H.: Implementation and tests of low-discrepancy sequences. ACM Trans. Model. Comput. Simul. 2(3), 195–213 (1992)CrossRefMATHGoogle Scholar
  2. Forrester, A., Sóbester, A., Keane, A.: Engineering Design Via Surrogate Modelling. Wiley, Chichester (2008)CrossRefMATHGoogle Scholar
  3. Gilquin, L., Arnaud, E., Monod, H., Prieur, C.: Recursive estimation procedure of Sobol’ indices based on replicated designs. Preprint available at (2016)
  4. Gilquin, L., Jiménez Rugama, L.A., Arnaud, E., Hickernell, F.J., Monod, H., Prieur, C.: Iterative construction of replicated designs based on Sobol’ sequences. C. R. Math. 355, 10–14 (2017)MathSciNetCrossRefMATHGoogle Scholar
  5. Hickernell, F.J., Jiménez Rugama, L.A.: Reliable adaptive cubature using digital sequences: Monte Carlo and Quasi-Monte Carlo. Methods 2014(163), 367–383 (2016)MATHGoogle Scholar
  6. Hickernell, F.J., Jiménez Rugama, L.A., Li, D.: Adaptive quasi-Monte Carlo methods for cubature. Preprint available at (2017)
  7. Hoeffding, W.F.: A class of statistics with asymptotically normal distribution. Ann. Math. Stat. 19(3), 293–325 (1948)MathSciNetCrossRefMATHGoogle Scholar
  8. Hong, H.S., Hickernell, F.J.: Algorithm 823: Implementing scrambled digital nets. ACM Trans. Math. Softw. 29, 95–109 (2003)CrossRefMATHGoogle Scholar
  9. Janon, A., Klein, T., Lagnoux, A., Nodet, M., Prieur, C.: Asymptotic normality and efficiency of two Sobol’ index estimators. ESAIM Probab. Stat. 18, 342–364 (2014)MathSciNetCrossRefMATHGoogle Scholar
  10. Jiménez Rugama, L.A., Hickernell, F.J.: Adaptive multidimensional integration based on rank-1 lattices: Monte Carlo and Quasi-Monte Carlo. Methods 2014(163), 407–422 (2016)MATHGoogle Scholar
  11. Joe, S., Kuo, F.Y.: Constructing Sobol sequences with better two-dimensional projections. SIAM J. Sci. Comput. 30(8), 2635–2654 (2008)MathSciNetCrossRefMATHGoogle Scholar
  12. Lemieux, C.: Monte Carlo and quasi-Monte Carlo Sampling. Springer, New York (2009)MATHGoogle Scholar
  13. Mara, T.A., Rakoto Joseph, O.: Comparison of some efficient methods to evaluate the main effect of computer model factors. J. Stat. Comput. Simul. 78(2), 167–178 (2008)MathSciNetCrossRefMATHGoogle Scholar
  14. McKay, M.D.: Evaluating prediction uncertainty, Los Alamos National Laboratory Report NUREG/CR- 6311, LA-12915-MS. (1995)Google Scholar
  15. Niederreiter, H.: Random Number Generation and Quasi-Monte Carlo Methods: CBMS-NSF Regional Conference Series in Applied Mathematics, vol. 63. SIAM, Philadelphia (1992)CrossRefGoogle Scholar
  16. Owen, A.B.: Randomly permuted \((t, m, s)\)-nets and \((t, s)\)-sequences: Monte Carlo and Quasi-Monte Carlo Methods in Scientific. Computing 106, 299–317 (1995)MATHGoogle Scholar
  17. Owen, A.B.: Better estimation of small Sobol’ sensitivity indices. ACM Trans. Model. Comput. Simul. 23(2), 11 (2013)MathSciNetCrossRefGoogle Scholar
  18. Saltelli, A.: Making best use of models evaluations to compute sensitivity indices. Comput. Phys. Commun. 145(2), 280–297 (2002)MathSciNetCrossRefMATHGoogle Scholar
  19. Sobol’, I.M.: On the distribution of points in a cube and the approximate evaluation of integrals. USSR Comput. Math. Math. Phys. 7(4), 86–112 (1967)MathSciNetCrossRefMATHGoogle Scholar
  20. Sobol’, I.M.: Sensitivity indices for nonlinear mathematical models. Math. Model. Comput. Exp. 1, 407–414 (1993)MATHGoogle Scholar
  21. Tissot, J.Y., Prieur, C.: A randomized orthogonal array-based procedure for the estimation of first- and second-order Sobol’ indices. J. Statist. Comput. Simul. 85(7), 1358–1381 (2015)MathSciNetCrossRefGoogle Scholar
  22. Tong, C.: Self-validated variance-based methods for sensitivity analysis of model outputs. Reliab. Eng. Syst. Saf. 95(3), 301–309 (2010)CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2017

Authors and Affiliations

  1. 1.Illinois Institute of TechnologyChicagoUSA
  2. 2.Inria Grenoble - Rhône-AlpesMontbonnotFrance

Personalised recommendations