Computational Statistics

, Volume 20, Issue 4, pp 643–654 | Cite as

A new nonparametric approach for multiplicity control:Optimal Subset procedures

  • Livio Finos
  • Luigi Salmaso


A new approach for multiplicity control (Optimal Subset) is presented. This is based on the selection of the best subset of partial (univariate) hypotheses producing the minimal p-value. In this work, we show how to perform this new procedure in the permutation framework, choosing suitable combining functions and permutation strategies. The optimal subset approach can be very useful in exploratory studies because it performs a weak control for multiplicity which can be a valid alternative to the False Discovery Rate (FDR). A comparative simulation study and an application to neuroimaging real data shows that it is particularly useful in presence of a high number of hypotheses. We also show how stepwise regression may be a special case of Optimal Subset procedures and how to adjust the p-value of the selected model taking into account for the multiplicity arising from the possible different models selected by a stepwise regression.


FDR FWE multiple tests neuroimaging permutation tests stepwise regression 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. [1]
    Benjamini, Y., Hochberg, Y. (1995). Controlling the false discovery rate: a new and powerful approach to multiple testing.Journal of the Royal Statistical Society, Series B, 57, 1289–1300.MathSciNetGoogle Scholar
  2. [2]
    Brook, R. J., Arnold, G. C. (1985).Applied regression analysis and experimental design. Marcel Dekker, New York.zbMATHGoogle Scholar
  3. [3]
    Copas, J. B. (1983). Regression, prediction and shrinkage (with discussion).Journal of the Royal Statistical Society, Series B, 45: 311–354.zbMATHMathSciNetGoogle Scholar
  4. [4]
    Finos, L., Pesarin, F., Salmaso, L. (2003). Combined tests for controlling multiplicity by closed testing procedures.Italian Journal of Applied Statistics, 15, 301–329.Google Scholar
  5. [5]
    Fisher, R.A. (1955).Statistical methods and scientific inference. Edinburgh: Oliver &: Boyd.Google Scholar
  6. [6]
    Freedman, L.S., Pee, D. and Midthune, D.N. (1992). The problem of underestimating the residual error variance in forward stepwise regression.The Statistician, 41, 405–412.CrossRefGoogle Scholar
  7. [7]
    Lubar, J. F., Congedo, M., Askew, J. H., 2003. Low-resolution electromagnetic tomography (LORETA) of cerebral activity in chronic depressive disorder.International Journal of Psychophysiology, 49(3), 175–185.CrossRefGoogle Scholar
  8. [8]
    Holm S. (1979). A simple sequentially rejective multiple test procedureScandinavian Journal of Statistics, 6, 65–70.MathSciNetGoogle Scholar
  9. [9]
    Marcus, R., Peritz, E. and Gabriel, K.R. (1976). On closed testing procedures with special reference to ordered analysis of variance.Biometrika, 63, 655–660.zbMATHCrossRefMathSciNetGoogle Scholar
  10. [10]
    Pascual-Marqui, R. D., 1999. Review of Methods for Solving the EEG Inverse Problem.International Journal of Bioelectromagnetism, 1 (1), 75–86.Google Scholar
  11. [11]
    Pesarin, F. (2001),Multivariate permutation tests with application to biostatistics. Wiley, Chichester.Google Scholar
  12. [12]
    Sherlin, L., Congedo, M. (2003). Obsessive Compulsive Disorder localized using Low-Resolution Electromagnetic Tomography, 34th annual meeting of the Association for Applied Psychophysiology & Biofeedback, March 27–30, Jacksonville, FL, USA.Google Scholar

Copyright information

© Physica-Verlag 2005

Authors and Affiliations

  • Livio Finos
    • 1
  • Luigi Salmaso
    • 2
  1. 1.Department of StatisticsUniversity of PadovaPadova
  2. 2.Department of Management and EngineeringUniversity of PadovaPadova

Personalised recommendations