Skip to main content
SpringerLink
Log in

Control of the false discovery rate under dependence using the bootstrap and subsampling

  • Invited Paper
  • Published:
TEST Aims and scope Submit manuscript

Abstract

This paper considers the problem of testing s null hypotheses simultaneously while controlling the false discovery rate (FDR). Benjamini and Hochberg (J. R. Stat. Soc. Ser. B 57(1):289–300, 1995) provide a method for controlling the FDR based on p-values for each of the null hypotheses under the assumption that the p-values are independent. Subsequent research has since shown that this procedure is valid under weaker assumptions on the joint distribution of the p-values. Related procedures that are valid under no assumptions on the joint distribution of the p-values have also been developed. None of these procedures, however, incorporate information about the dependence structure of the test statistics. This paper develops methods for control of the FDR under weak assumptions that incorporate such information and, by doing so, are better able to detect false null hypotheses. We illustrate this property via a simulation study and two empirical applications. In particular, the bootstrap method is competitive with methods that require independence if independence holds, but it outperforms these methods under dependence.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Abramovich F, Benjamini Y (1996) Adaptive thresholding of wavelet coefficients. Comput Stat Data Anal 22:351–361

    Article  MathSciNet  Google Scholar 

  • Abramovich F, Benjamini Y, Donoho DL, Johnstone IM (2006) Adapting to unknown sparsity by controlling the false discovery rate. Ann Stat 34(2):584–653

    Article  MATH  MathSciNet  Google Scholar 

  • Andrews DWK, Monahan JC (1992) An improved heteroskedasticity and autocorrelation consistent covariance matrix estimator. Econometrica 60:953–966

    Article  MATH  MathSciNet  Google Scholar 

  • Basford KE, Tukey JW (1997) Graphical profiles as an aid to understanding plant breeding experiments. J Stat Plann Inference 57:93–107

    Article  MATH  Google Scholar 

  • Benjamini Y, Hochberg Y (1995) Controlling the false discovery rate: A practical and powerful approach to multiple testing. J R Stat Soc Ser B 57(1):289–300

    MATH  MathSciNet  Google Scholar 

  • Benjamini Y, Hochberg Y (2000) On the adaptive control of the false discovery rate in multiple testing with independent statistics. J Educ Behav Stat 25(1):60–83

    Google Scholar 

  • Benjamini Y, Liu W (1999) A stepdown multiple hypotheses testing procedure that controls the false discovery rate under independence. J Stat Plann Inference 82:163–170

    Article  MATH  MathSciNet  Google Scholar 

  • Benjamini Y, Yekutieli D (2001) The control of the false discovery rate in multiple testing under dependency. Ann Stat 29(4):1165–1188

    Article  MATH  MathSciNet  Google Scholar 

  • Benjamini Y, Krieger AM, Yekutieli D (2006) Adaptive linear step-up procedures that control the false discovery rate. Biometrika 93(3):491–507

    Article  MATH  MathSciNet  Google Scholar 

  • Davison AC, Hinkley DV (1997) Bootstrap methods and their application. Cambridge University Press, Cambridge

    MATH  Google Scholar 

  • Drigalenko EI, Elston RC (1997) False discoveries in genome scanning. Genet Epidemiol 15:779–784

    Article  Google Scholar 

  • Efron B (1979) Bootstrap methods: Another look at the jackknife. Ann Stat 7:1–26

    Article  MATH  MathSciNet  Google Scholar 

  • Efron B, Tibshirani RJ (1993) An introduction to the bootstrap. Chapman & Hall, New York

    MATH  Google Scholar 

  • Genovese CR, Wasserman L (2004) A stochastic process approach to false discovery control. Ann Stat 32(3):1035–1061

    Article  MATH  MathSciNet  Google Scholar 

  • Götze F, Künsch HR (1996) Second order correctness of the blockwise bootstrap for stationary observations. Ann Stat 24:1914–1933

    Article  MATH  Google Scholar 

  • Hommel G, Hoffman T (1988) Controlled uncertainty. In: Bauer P, Hommel G, Sonnemann E (eds) Multiple hypothesis testing. Springer, Heidelberg, pp 154–161

    Google Scholar 

  • Joe H (2006) Generating random correlation matrices based on partial correlations. J Multivar Anal 97:2177–2189

    Article  MATH  MathSciNet  Google Scholar 

  • Kat HM (2003) 10 things investors should know about hedge funds. AIRC working paper 0015, Cass Business School, City University. Available at http://www.cass.city.ac.uk/airc/papers.html

  • Lahiri SN (1992) Edgeworth correction by ‘moving block’ bootstrap for stationary and nonstationary data. In: LePage R, Billard L (eds) Exploring the limits of bootstrap. Wiley, New York, pp 183–214

    Google Scholar 

  • Lahiri SN (2003) Resampling methods for dependent data. Springer, New York

    MATH  Google Scholar 

  • Lehmann EL, Romano JP (2005a) Generalizations of the familywise error rate. Ann Stat 33(3):1138–1154

    Article  MATH  MathSciNet  Google Scholar 

  • Lehmann EL, Romano JP (2005b) Testing statistical hypotheses, 3d edn. Springer, New York

    MATH  Google Scholar 

  • Lewandowski D, Kurowicka D, Joe H (2007) Generating random correlation matrices based on vines and extended Onion method. Preprint, Dept. of Mathematics, Delft University of Technology

  • Lo AW (2002) The statistics of Sharpe ratios. Financ Anal J 58(4):36–52

    Article  Google Scholar 

  • Mehrotra DV, Heyse JF (2004) Use of the false discovery rate for evaluating clinical safety data. Stat Methods Med Res 13:227–238

    MATH  MathSciNet  Google Scholar 

  • Politis DN, Romano JP (1992) A circular block-resampling procedure for stationary data. In: LePage R, Billard L (eds) Exploring the limits of bootstrap. Wiley, New York, pp 263–270

    Google Scholar 

  • Politis DN, Romano JP (1994) The stationary bootstrap. J Am Stat Assoc 89:1303–1313

    Article  MATH  MathSciNet  Google Scholar 

  • Politis DN, Romano JP, Wolf M (1999) Subsampling. Springer, New York

    MATH  Google Scholar 

  • Reiner A, Yekutieli D, Benjamini Y (2003) Identifying differentially expressed genes using false discovery rate controlling procedures. Bioinformatics 19:368–375

    Article  Google Scholar 

  • Romano JP, Shaikh AM (2006a) On stepdown control of the false discovery proportion. In: Rojo J (ed) Optimality: the second Erich L Lehmann symposium. IMS lecture notes—monograph series, vol 49, pp 33–50

  • Romano JP, Shaikh AM (2006b) Stepup procedures for control of generalizations of the familywise error rate. Ann Stat 34(4):1850–1873

    Article  MathSciNet  Google Scholar 

  • Romano JP, Wolf M (2006) Improved nonparametric confidence intervals in time series regressions. J Nonparametr Stat 18(2):199–214

    Article  MATH  MathSciNet  Google Scholar 

  • Romano JP, Wolf M (2007) Control of generalized error rates in multiple testing. Ann Stat 35(4):1378–1408

    Article  MATH  MathSciNet  Google Scholar 

  • Romano JP, Shaikh AM, Wolf M (2008) Formalized data snooping based on generalized error rates. Econom Theory 24(2):404–447

    Article  MATH  MathSciNet  Google Scholar 

  • Sarkar SK (2002) Some results on false discovery rate in stepwise multiple testing procedures. Ann Stat 30(1):239–257

    Article  MATH  Google Scholar 

  • Storey JD, Taylor JE, Siegmund D (2004) Strong control, conservative point estimation and simultaneous conservative consistency of false discovery rates: a unified approach. J R Stat Soc Ser B 66(1):187–205

    Article  MATH  MathSciNet  Google Scholar 

  • Troendle JF (2000) Stepwise normal theory test procedures controlling the false discovery rate. J Stat Plann Inference 84(1):139–158

    Article  MATH  MathSciNet  Google Scholar 

  • Van der Laan MJ, Dudoit S, Pollard KS (2004) Augmentation procedures for control of the generalized family-wise error rate and tail probabilities for the proportion of false positives. Stat Appl Genet Mol Biol 3(1):Article 15. Available at http://www.bepress.com/sagmb/vol3/iss1/art15/

  • Westfall PH, Young SS (1993) Resampling-based multiple testing: examples and methods for P-value adjustment. Wiley, New York

    Google Scholar 

  • Williams VSL, Jones LV, Tukey JW (1999) Controlling error in multiple comparisons, with examples from state-to-state differences in educational achievement. J Educ Behav Stat 24(1):42–69

    Google Scholar 

  • Yekutieli D, Benjamini Y (1999) Resampling-based false discovery rate controlling multiple test procedures for correlated test statistics. J Stat Plann Inference 82:171–196

    Article  MATH  MathSciNet  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Departments of Economics and Statistics, Stanford University, Stanford, USA

    Joseph P. Romano

  2. Department of Economics, University of Chicago, Chicago, USA

    Azeem M. Shaikh

  3. Institute for Empirical Research in Economics, University of Zurich, Bluemlisalpstrasse 10, 8006, Zurich, Switzerland

    Michael Wolf

Authors
  1. Joseph P. Romano
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Azeem M. Shaikh
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Michael Wolf
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Michael Wolf.

Additional information

This invited paper is discussed in the comments available at: http://dx.doi.org/10.1007/s11749-008-0127-5, http://dx.doi.org/10-1007/s11749-008-0128-4, http://dx.doi.org/10.1007/s11749-008-0129-3, http://dx.doi.org/10.1007/s11749-008-0130-x, http://dx.doi.org/10.1007/s11749-008-0131-9.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Romano, J.P., Shaikh, A.M. & Wolf, M. Control of the false discovery rate under dependence using the bootstrap and subsampling. TEST 17, 417–442 (2008). https://doi.org/10.1007/s11749-008-0126-6

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11749-008-0126-6

Keywords

Mathematics Subject Classification (2000)

Search

Navigation