Computational Statistics

, Volume 15, Issue 3, pp 373–390 | Cite as

Type S error rates for classical and Bayesian single and multiple comparison procedures

  • Andrew Gelman
  • Francis Tuerlinckx


In classical statistics, the significance of comparisons (e. g., θ1–θ2) is calibrated using the Type 1 error rate, relying on the assuption that the true difference is zero, which makes no sense in many applications. We set up a more relevant framework in which a true comparison can be positive or negative, and, based on the data, you can state “θ1 > θ2 with confidence,”“θ2 > θ1 with confidence,” or “no claim with confidence.” We focus on the Type S (for sign) error, which occurs when you claim “θ1 > θ2 with confidence” when θ2 > θ1 (or vice-versa). We compute the Type S error rates for classical and Bayesian confidence statements and find that classical Type S error rates can be extremely high (up to 50%). Bayesian confidence statements are conservative in the sense that claims based on 95% posterior intervals have Type S error rates between 0 and 2.5%. For multiple comparison situations, the conclusions are similar.

Keywords: Bayesian Inference, Multiple Comparisons, Type 1 Error, Type M Error, Type S Error 

Copyright information

© Physica-Verlag, Heidelberg 2000

Authors and Affiliations

  • Andrew Gelman
    • 1
  • Francis Tuerlinckx
    • 2
  1. 1.Department of Statistics, University, New York, USAUS
  2. 2.Department of Psychology, University of Leuven, BelgiumBE

Personalised recommendations