Abstract
Many exploratory experiments such as DNA microarray or brain imaging require simultaneously comparisons of hundreds or thousands of hypotheses. Under such a setting, using the false discovery rate (FDR) as an overall Type I error is recommended (Benjamini and Hochberg in J. R. Stat. Soc. B 57:289–300, 1995). Many FDR controlling procedures have been proposed. However, when evaluating the performance of FDR-controlling procedures, researchers are often focused on the ability of procedures to control the FDR and to achieve high power. Meanwhile, under the multiple hypotheses, it may be also likely to commit a false non-discovery or fail to claim a true non-significance. In addition, various experimental parameters such as the number of hypotheses, the proportion of the number of true null hypotheses to the number of hypotheses, the samples size and the correlation structure may affect the performance of FDR controlling procedures. The purpose of this paper is to illustrate the performance of some existing FDR controlling procedures in terms of four indices, i.e., the FDR, the false non-discovery rate, the sensitivity and the specificity. Analytical results of these indices for the FDR controlling procedures are derived. Simulations are also performed to evaluate the performance of controlling procedures in terms of these indices under various experimental parameters. The result can be used to summarize as a guidance for practitioners to properly choose a FDR controlling procedure.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Benjamini, Y., Hochberg, Y.: Controlling the false discovery rate: a practical and powerful approach to multiple testing. J. R. Stat. Soc. B 57, 289–300 (1995)
Benjamini, Y., Hochberg, Y.: On the adaptive control of the false discovery rate in multiple testing with independent statistics. J. Educ. Behav. Stat. 25, 60–83 (2000)
Benjamini, Y., Liu, W.: A step-down multiple hypotheses testing procedure that controls the false discovery rate under independence. J. Stat. Plan. Inference 82, 163–170 (1999)
Benjamini, Y., Yekutieli, D.: The control of the false discovery rate in multiple testing under dependency. Ann. Stat. 29, 1165–1188 (2001)
Benjamini, Y., Krieger, A.M., Yekutieli, D.: Adaptive linear step-up procedures that control the false discovery rate. Biometrika 93, 491–507 (2006)
Black, M.A.: A note on the adaptive control of false discovery rates. J. R. Stat. Soc. B 66, 297–304 (2004)
Delongchamp, R.R., Bowyer, J.F., Chen, J.J., Kodell, R.L.: Multiple-testing strategy for analyzing cDNA array data on gene expression. Biometrics 60, 774–782 (2004)
Dudoit, S., Shaffer, J.P., Boldrick, J.C.: Multiple hypothesis testing in microarray experiments. Stat. Sci. 18, 71–103 (2003)
Farcomeni, A.: A review of modern multiple hypothesis testing, with particular attention to the false discovery proportion. Stat. Methods Med. Res. 17, 347–388 (2008)
Ferreira, J.A., Zwinderman, A.H.: On the Benjamini-Hochberg method. Ann. Stat. 34, 1827–1849 (2006)
Ge, Y., Dudoit, S., Speed, T.P.: Resampling-based multiple testing for microarray data hypothesis. Test 12, 1–44 (2003)
Genovese, C., Wasserman, L.: Operating characteristics and extensions of the false discovery rate procedure. J. R. Stat. Soc. B 64, 499–517 (2002)
Hsueh, H.M., Tsai, C.A., Chen, J.J.: Incorporating the number of true null hypotheses to improve power in multiple testing: application to gene microarray data. J. Stat. Comput. Simul. 77, 757–767 (2007)
Hwang, Y.T.: Comparisons of estimators of the number of true null hypotheses and adaptive FDR procedures in multiplicity testing, J. Stat. Comput. Simul. (2009, to appear)
Kim, K.I., van de Wiel, M.A.: Effects of dependence in high-dimensional multiple testing problems. BMC Bioinform. 9, 114–125 (2008)
Merriam, E.P., Genovese, C.R., Colby, C.L.: Spatial updating in human parietal cortex. Neuron 39, 361–393 (2003)
Pounds, S.B.: Estimation and control of multiple testing error rates for microarray studies. Brief. Bioinform. 7, 25–36 (2006)
Reiner, A., Yekutieli, D., Benjamini, Y.: Identifying differentially expressed genes using false discovery rate controlling procedures. Bioinformatics 19, 368–375 (2003)
Sarkar, S.K.: FDR-controlling stepwise procedures and their false negatives rates. J. Stat. Plan. Inference 125, 119–137 (2004)
Seeger, P.: A note on a method for the analysis of significance en masse. Technometrics 10, 583–593 (1968)
Storey, J.D.: A direct approach to false discovery rates. J. R. Stat. Soc. B 64, 479–498 (2002)
Storey, J.D., Taylor, J.E., Siemund, D.: Strong control, conservative point estimation, and simultaneous conservative consistency of false discovery rates: a unified approach. J. R. Stat. Soc. B 66, 187–205 (2004)
S̆idák, Z.: One multivariate normal probabilities of rectangles: their dependence on correlations. Ann. Math. Stat. 39, 1425–1434 (1968)
Yekutieli, D., Benjamini, Y.: Resampling-based false discovery rate controlling multiple test procedures for correlated test statistics. J. Stat. Plan. Inference 82, 171–196 (1999)
Yin, Y., Soteros, C.E., Bickis, M.G.: A clarifying comparison of methods for controlling the false discovery rate. J. Stat. Plan. Inference 139, 2126–2137 (2009)
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Hwang, YT., Chu, SK. & Ou, ST. Evaluations of FDR-controlling procedures in multiple hypothesis testing. Stat Comput 21, 569–583 (2011). https://doi.org/10.1007/s11222-010-9193-0
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11222-010-9193-0