The (mis)use of overlap of confidence intervals to assess effect modification
In randomized controlled trials as well as in observational studies, researchers are often interested in effects of treatment or exposure in different subgroups, i.e. effect modification [1, 2]. There are several methods to assess effect modification and the debate on which method is best is still ongoing [2–5]. In this article we focus on an invalid method to assess effect modification, which is often used in articles in health sciences journals , namely concluding that there is no effect modification if the confidence intervals of the subgroups are overlapping [7–9].
When assessing effect modification by looking at overlap of the 95% confidence intervals in subgroups, a type 1 error probability of 0.05 is often mistakenly assumed. In other words, if the confidence intervals are overlapping, the difference in effect estimates between the two subgroups is judged to be statistically insignificant. By using mathematical derivation, we calculated that the chance of finding non-overlappi ...
- Knol, MJ, Egger, M, Scott, P, Geerlings, MI, Vandenbroucke, JP (2009) When one depends on the other: reporting of interaction in case-control and cohort studies. Epidemiology 20: pp. 161-166 CrossRef
- Wang, R, Lagakos, SW, Ware, JH, Hunter, DJ, Drazen, JM (2007) Statistics in medicine—reporting of subgroup analyses in clinical trials. N Engl J Med 357: pp. 2189-2194 CrossRef
- Greenland, S (2009) Interactions in epidemiology: relevance, identification and estimation. Epidemiology 20: pp. 14-17 CrossRef
- Pocock, SJ, Collier, TJ, Dandreo, KJ (2004) Issues in the reporting of epidemiological studies: a survey of recent practice. BMJ 329: pp. 883 CrossRef
- Assmann, SF, Pocock, SJ, Enos, LE, Kasten, LE (2000) Subgroup analysis and other (mis)uses of baseline data in clinical trials. Lancet 355: pp. 1064-1069 CrossRef
- Schenker, N, Gentleman, JF (2001) On judging the significance of differences by examining the overlap between confidence intervals. The Am Stat 55: pp. 182-186 CrossRef
- Ryan, GW, Leadbetter, SD (2002) On the misuse of confidence intervals for two means in testing for the significance of the difference between the means. J Mod Appl Stat Methods 1: pp. 473-478
- Altman, DG, Bland, JM (2003) Interaction revisited: the difference between two estimates. BMJ 326: pp. 219 CrossRef
- Austin, PC, Hux, JE (2002) A brief note on overlapping confidence intervals. J Vasc Surg 36: pp. 194-195 CrossRef
- Goldstein, H, Healy, MJR (1995) The graphical presentation of a collection of means. J R Stat Soc 158: pp. 175-177 CrossRef
- The (mis)use of overlap of confidence intervals to assess effect modification
- Open Access
- Available under Open Access This content is freely available online to anyone, anywhere at any time.
European Journal of Epidemiology
Volume 26, Issue 4 , pp 253-254
- Cover Date
- Print ISSN
- Online ISSN
- Springer Netherlands
- Additional Links
- Industry Sectors