Drug Safety

, Volume 36, Supplement 1, pp 73–82 | Cite as

Empirical Performance of the Case–Control Method: Lessons for Developing a Risk Identification and Analysis System

  • David Madigan
  • Martijn J. Schuemie
  • Patrick B. Ryan
Original Research Article



Considerable attention now focuses on the use of large-scale observational healthcare data for understanding drug safety. In this context, analysts utilize a variety of statistical and epidemiological approaches such as case–control, cohort, and self-controlled methods. The operating characteristics of these methods are poorly understood.


Establish the operating characteristics of the case–control method for large scale observational analysis in drug safety.

Research Design

We empirically evaluated the case–control approach in 5 real observational healthcare databases and 6 simulated datasets. We retrospectively studied the predictive accuracy of the method when applied to a collection of 165 positive controls and 234 negative controls across 4 outcomes: acute liver injury, acute myocardial infarction, acute kidney injury, and upper gastrointestinal bleeding.


In our experiment, the case–control method provided weak discrimination between positive and negative controls. Furthermore, the method yielded positively biased estimates and confidence intervals that had poor coverage properties.


For the four outcomes we examined, the case–control method may not be the method of choice for estimating potentially harmful effects of drugs.


  1. 1.
    Food and Drug Administration Amendments Act of 2007. Pub. L. 110-85, 21 U.S., 2007.Google Scholar
  2. 2.
    Woodward M. Epidemiology study design and data analysis. London: Chapman & Hall/CRC; 1999.Google Scholar
  3. 3.
    Agresti A. Categorical data analysis. Hoboken: Wiley-Interscience; 2002.CrossRefGoogle Scholar
  4. 4.
    Breslow NE, Day NE. Statistical methods in cancer research. In: The analysis of case–control studies, vol. I. France: International Agency for Research on Cancer; 1993.Google Scholar
  5. 5.
    Rothman KJ, Greenland S, Lash TL. Modern epidemiology. 3rd ed. Philadelphia: Wolters Kluwer Health/Lippincott Williams & Wilkens; 2008.Google Scholar
  6. 6.
    Charlson ME, Pompei P, Ales KL, MacKenzie CR. A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. J Chronic Dis. 1987;40(5):373–83.PubMedCrossRefGoogle Scholar
  7. 7.
    Ryan PB, Schuemie M. Evaluating performance of risk identification methods through a large-scale simulation of observational data. Drug Saf (in this supplement issue). doi:10.1007/s40264-013-0110-2
  8. 8.
    Overhage JM, Ryan PB, Schuemie MJ, Stang PE. Desideratum for evidence based epidemiology. Drug Saf (in this supplement issue). doi:10.1007/s40264-013-0102-2
  9. 9.
    Ryan PB, Schuemie MJ, Welebob E, Duke J, Valentine S, Hartzema AG. Defining a reference set to support methodological research in drug safety. Drug Saf (in this supplement issue). doi:10.1007/s40264-013-0097-8
  10. 10.
    Wolfe MM, Lichtenstein DR, Singh G. Gastrointestinal toxicity of nonsteroidal antiinflammatory drugs. N Engl J Med. 1999;340(24):1888–99.PubMedCrossRefGoogle Scholar
  11. 11.
    Makarowski W, Zhao WW, Bevirt T, Recker DP. Efficacy and safety of the COX-2 specific inhibitor valdecoxib in the management of osteoarthritis of the hip: a randomized, double-blind, placebo-controlled comparison with naproxen. Osteoarthr cartil OARS Osteoarthr Res Soc. 2002;10(4):290–6.CrossRefGoogle Scholar
  12. 12.
    Maldonado G, Greenland S. Estimating causal effects. Int J Epidemiol. 2002;31(2):422–9.PubMedCrossRefGoogle Scholar
  13. 13.
    Hofler M. Causal inference based on counterfactuals. BMC Med Res Methodol. 2005;5:28.PubMedCrossRefGoogle Scholar
  14. 14.
    Schneeweiss S. A basic study design for expedited safety signal evaluation based on electronic healthcare data. Pharmacoepidemiol Drug Saf. 2010;19(8):858–68.PubMedCrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2013

Authors and Affiliations

  • David Madigan
    • 1
    • 4
  • Martijn J. Schuemie
    • 2
    • 4
  • Patrick B. Ryan
    • 3
    • 4
  1. 1.Department of StatisticsColumbia UniversityNew YorkUSA
  2. 2.Department of Medical InformaticsErasmus University Medical Center RotterdamRotterdamThe Netherlands
  3. 3.Janssen Research and Development LLCTitusvilleUSA
  4. 4.Observational Medical Outcomes Partnership, Foundation for the National Institutes of HealthBethesdaUSA

Personalised recommendations