Skip to main content

Heteroscedasticity: multiple degrees of freedom vs. sandwich estimation

Abstract

This article addresses the problem of multiple contrast tests in the presence of heteroscedasticity. A procedure applying multiple degrees of freedom and a procedure based on sandwich estimation are described and compared concerning the familywise error type I. The former procedure seems to be most robust in this regard, especially for situations where high variances meet small sample sizes.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

References

  1. Adler ID, Kliesch U (1990) Comparison of single and multiple treatment regiments in the mouse bone marrow micronucleus assay for hydroquinone and cyclophosphamide. Mutat Res 234:115–123

    Article  Google Scholar 

  2. Bretz F (2006) An extension of the Williams trend test to general unbalanced linear models. Comput Stat Data Anal 50(7):1735–1748

    Article  MathSciNet  MATH  Google Scholar 

  3. Bretz F, Genz A, Hothorn LA (2001) On the numerical availability of multiple comparison procedures. Biom J 43:645–656

    Article  MathSciNet  MATH  Google Scholar 

  4. Brown MB, Forsythe AB (1974) Anova and multiple comparisons for data with heterogeneous variances. Biometrics 30(4):719–724

    Article  MathSciNet  MATH  Google Scholar 

  5. Demirhan H, Dolgun NA, Demirhan YP, Dolgun MO (2010) Performance of some multiple comparison tests under heteroscedasticity and dependency. J Stat Comput Simul 80(10):1083–1100

    Article  MathSciNet  MATH  Google Scholar 

  6. Djira GD, Hasler M, Gerhard D, Schaarschmidt F (2012) mratios: Inferences for ratios of coefficients in the general linear model. http://CRAN.R-project.org/package=mratios, R package version 1.3.17. Accessed 2 Nov 2012

  7. Dunnett CW (1955) A multiple comparison procedure for comparing several treatments with a control. J Am Stat Assoc 50(272):1096–1121

    Article  MATH  Google Scholar 

  8. Dunnett CW (1980) Pairwise multiple comparisons in the unequal variance case. J Am Stat Assoc 75(372):796–800

    Article  Google Scholar 

  9. Freedman DA (2006) On the so-called “Huber Sandwich Estimator” and “Robust Standard Errors”. Am Stat 60(4):299–302

    Article  Google Scholar 

  10. Games PA, Howell JF (1976) Pairwise multiple comparison procedures with unequal n’s and/or variances: a monte carlo study. J Educ Stat 1(2):113–125

    Article  Google Scholar 

  11. Genz A, Bretz F (2002) Methods for the computation of multivariate \(t\)-probabilities. J Comput Graph Stat 11:950–971

    Article  MathSciNet  Google Scholar 

  12. Genz A, Bretz F (2009) Computation of multivariate normal and \(t\) probabilities, vol 195., Lecture notes in statisticsSpringer, Heidelberg

    Book  Google Scholar 

  13. Genz A, Bretz F, Miwa T, Mi X, Leisch F, Scheipl F, Hothorn T (2014) mvtnorm: Multivariate normal and \(t\) distributions. http://CRAN.R-project.org/package=mvtnorm, R package version 1.0-0. Accessed 8 July 2014

  14. Guilbaud O (2011) Note on simultaneous inferences about non-inferiority and superiority for a primary and a secondary endpoint. Biom J 53(6, SI):927–937

    Article  MathSciNet  MATH  Google Scholar 

  15. Hartung J, Argac D, Makambi KH (2002) Small sample properties of tests on homogeneity in one-way anova and meta-analysis. Stat Pap 43(2):197–235

    Article  MathSciNet  MATH  Google Scholar 

  16. Hasler M (2012) Multiple comparisons to both a negative and a positive control. Pharm Stat 11(1):74–81

    Article  MathSciNet  Google Scholar 

  17. Hasler M (2014) SimComp: simultaneous comparisons for multiple endpoints. http://CRAN.R-project.org/package=SimComp, R package version 2.2. Accessed 12 Sept 2014

  18. Hasler M, Hothorn LA (2008) Multiple contrast tests in the presence of heteroscedasticity. Biom J 50(5):793–800

    Article  MathSciNet  Google Scholar 

  19. Hasler M, Hothorn LA (2012) A multivariate Williams-type trend procedure. Stat Biopharm Res 4:57–65

    Article  Google Scholar 

  20. Hasler M, Vonk R, Hothorn LA (2008) Assessing non-inferiority of a new treatment in a three-arm trial in the presence of heteroscedasticity. Stat Med 27(4):490–503

    Article  MathSciNet  Google Scholar 

  21. Hayter AJ (1984) A proof of the conjecture that the Tukey–Kramer multiple comparisons procedure is conservative. Ann Stat 12(1):61–75

    Article  MathSciNet  MATH  Google Scholar 

  22. Herberich E, Sikorski J, Hothorn T (2010) A robust procedure for comparing multiple means under heteroscedasticity in unbalanced designs. PLoS One 5(3):e9788

    Article  Google Scholar 

  23. Holm S (1979) A simple sequentially rejective multiple test procedure. Scand J Stat 6:65–70

    MathSciNet  MATH  Google Scholar 

  24. Homma Y, Yamaguchi T, Yamaguchi O (2008) A randomized, double-blind, placebo-controlled phase ii dose-finding study of the novel anti-muscarinic agent imidafenacin in Japanese patients with overactive bladder. Int J Urol 15(9):809–815

    Article  Google Scholar 

  25. Hommel G (1988) A stagewise rejective multiple test procedure based on a modified Bonferroni test. Biometrika 75(2):383–386

    Article  MATH  Google Scholar 

  26. Hothorn T, Bretz F, Genz A (2001) On multivariate \(t\) and Gauss probabilities in R. R News 1(2):27–29

    Google Scholar 

  27. Hothorn T, Bretz F, Westfall PH, Heiberger RM, Schuetzenmeister A (2014) multcomp: Simultaneous inference in general parametric models. http://CRAN.R-project.org/package=multcomp, R package version 1.3-7. Accessed 2 Oct 2014

  28. Huber PJ (1967) The behavior of maximum likelihood estimates under nonstandard conditions. In: Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability, vol 1. pp 221–233

  29. Kromrey JD, LaRocca MA (2004) Power and type i error rates of new pairwise multiple comparison procedures under heterogeneous variances. J Exp Educ 63(4):343–362

    Article  Google Scholar 

  30. Li H, Ning W (2012) Multiple comparisons with a control under heteroscedasticity. J Appl Stat 39(10):2275–2283

    Article  MathSciNet  Google Scholar 

  31. Lumley T, Zeileis A (2014) Sandwich: robust covariance matrix estimators. http://CRAN.R-project.org/package=sandwich, R package version 2.3-2. Accessed 24 Aug 2014

  32. MacKinnon JG, White H (1985) Some heteroskedasticity-consistent covariance-matrix estimators with improved finite-sample properties. J Econom 29(3):305–325

    Article  Google Scholar 

  33. Mehrotra DV (1997) Improving the Brown–Forsythe solution to the generalized Behrens–Fisher problem. Commun Stat Simul Comput 26(3):1139–1145

    Article  MathSciNet  MATH  Google Scholar 

  34. R Core Team (2013) R: a language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. http://www.R-project.org/

  35. Ramsey PH, Ramsey PP (2009) Power and type I errors for pairwise comparisons of means in the unequal variances case. Br J Math Stat Psychol 62:263–281

    Article  MathSciNet  Google Scholar 

  36. Satterthwaite FE (1946) An approximate distribution of estimates of variance components. Biometrics 2:110–114

    Article  Google Scholar 

  37. Strassburger K, Bretz F (2008) Compatible simultaneous lower confidence bounds for the Holm procedure and other Bonferroni-based closed tests. Stat Med 27(24):4914–4927

    Article  MathSciNet  Google Scholar 

  38. Sun WG, McLain AC (2012) Multiple testing of composite null hypotheses in heteroscedastic models. J Am Stat Assoc 107(498):673–687

    Article  MathSciNet  MATH  Google Scholar 

  39. Tamhane AC, Logan BR (2004) Finding the maximum safe dose level for heteroscedastic data. J Biopharm Stat 14(4):843–856

    Article  MathSciNet  Google Scholar 

  40. Tukey JW (1953) The problem of multiple comparisons, dittoed manuscript of 396 pages. Department of Statistics, Princeton University, New Jersey

  41. Welch BL (1938) The significance of the difference between two means when the population variances are unequal. Biometrika 29:350–362

    Article  MATH  Google Scholar 

  42. Welch BL (1951) On the comparison of several mean values—an alternative approach. Biometrika 38(3–4):330–336

    Article  MathSciNet  MATH  Google Scholar 

  43. Westfall PH, Young SS (1993) Resampling-based multiple testing, Examples and methods for p-value adjustment. Wiley-Interscience, Berlin

    Google Scholar 

  44. Williams DA (1971) A test for differences between treatment means when several dose levels are compared with a zero dose control. Biometrics 27:103–117

    Article  Google Scholar 

  45. Zeileis A (2004) Econometric computing with HC and HAC covariance matrix estimators. J Stat Softw 11(10):1–17

  46. Zeileis A (2006) Object-oriented computation of sandwich estimators. J Stat Softw 16(9):1–16

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Mario Hasler.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Hasler, M. Heteroscedasticity: multiple degrees of freedom vs. sandwich estimation. Stat Papers 57, 55–68 (2016). https://doi.org/10.1007/s00362-014-0640-4

Download citation

Keywords

  • Family wise error type I
  • Heteroscedasticity
  • Multiple contrast tests
  • Multivariate \(t\)-distribution
  • Sandwich estimation