Skip to main content

Empirical Performance of the Case–Control Method: Lessons for Developing a Risk Identification and Analysis System



Considerable attention now focuses on the use of large-scale observational healthcare data for understanding drug safety. In this context, analysts utilize a variety of statistical and epidemiological approaches such as case–control, cohort, and self-controlled methods. The operating characteristics of these methods are poorly understood.


Establish the operating characteristics of the case–control method for large scale observational analysis in drug safety.

Research Design

We empirically evaluated the case–control approach in 5 real observational healthcare databases and 6 simulated datasets. We retrospectively studied the predictive accuracy of the method when applied to a collection of 165 positive controls and 234 negative controls across 4 outcomes: acute liver injury, acute myocardial infarction, acute kidney injury, and upper gastrointestinal bleeding.


In our experiment, the case–control method provided weak discrimination between positive and negative controls. Furthermore, the method yielded positively biased estimates and confidence intervals that had poor coverage properties.


For the four outcomes we examined, the case–control method may not be the method of choice for estimating potentially harmful effects of drugs.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4


  1. Food and Drug Administration Amendments Act of 2007. Pub. L. 110-85, 21 U.S., 2007.

  2. Woodward M. Epidemiology study design and data analysis. London: Chapman & Hall/CRC; 1999.

  3. Agresti A. Categorical data analysis. Hoboken: Wiley-Interscience; 2002.

    Book  Google Scholar 

  4. Breslow NE, Day NE. Statistical methods in cancer research. In: The analysis of case–control studies, vol. I. France: International Agency for Research on Cancer; 1993.

  5. Rothman KJ, Greenland S, Lash TL. Modern epidemiology. 3rd ed. Philadelphia: Wolters Kluwer Health/Lippincott Williams & Wilkens; 2008.

  6. Charlson ME, Pompei P, Ales KL, MacKenzie CR. A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. J Chronic Dis. 1987;40(5):373–83.

    PubMed  Article  CAS  Google Scholar 

  7. Ryan PB, Schuemie M. Evaluating performance of risk identification methods through a large-scale simulation of observational data. Drug Saf (in this supplement issue). doi:10.1007/s40264-013-0110-2

  8. Overhage JM, Ryan PB, Schuemie MJ, Stang PE. Desideratum for evidence based epidemiology. Drug Saf (in this supplement issue). doi:10.1007/s40264-013-0102-2

  9. Ryan PB, Schuemie MJ, Welebob E, Duke J, Valentine S, Hartzema AG. Defining a reference set to support methodological research in drug safety. Drug Saf (in this supplement issue). doi:10.1007/s40264-013-0097-8

  10. Wolfe MM, Lichtenstein DR, Singh G. Gastrointestinal toxicity of nonsteroidal antiinflammatory drugs. N Engl J Med. 1999;340(24):1888–99.

    PubMed  Article  CAS  Google Scholar 

  11. Makarowski W, Zhao WW, Bevirt T, Recker DP. Efficacy and safety of the COX-2 specific inhibitor valdecoxib in the management of osteoarthritis of the hip: a randomized, double-blind, placebo-controlled comparison with naproxen. Osteoarthr cartil OARS Osteoarthr Res Soc. 2002;10(4):290–6.

    Article  CAS  Google Scholar 

  12. Maldonado G, Greenland S. Estimating causal effects. Int J Epidemiol. 2002;31(2):422–9.

    PubMed  Article  Google Scholar 

  13. Hofler M. Causal inference based on counterfactuals. BMC Med Res Methodol. 2005;5:28.

    PubMed  Article  CAS  Google Scholar 

  14. Schneeweiss S. A basic study design for expedited safety signal evaluation based on electronic healthcare data. Pharmacoepidemiol Drug Saf. 2010;19(8):858–68.

    PubMed  Article  Google Scholar 

Download references


The Observational Medical Outcomes Partnership is funded by the Foundation for the National Institutes of Health (FNIH) through generous contributions from the following: Abbott, Amgen Inc., AstraZeneca, Bayer Healthcare Pharmaceuticals, Inc., Biogen Idec, Bristol-Myers Squibb, Eli Lilly & Company, GlaxoSmithKline, Janssen Research and Development, Lundbeck, Inc., Merck & Co., Inc., Novartis Pharmaceuticals Corporation, Pfizer Inc, Pharmaceutical Research Manufacturers of America (PhRMA), Roche, Sanofi-aventis, Schering-Plough Corporation, and Takeda. Drs. Ryan and Schuemie are employees of Janssen Research and Development. Dr. Schuemie received a fellowship from the Office of Medical Policy, Center for Drug Evaluation and Research, Food and Drug Administration. Drs. Schuemie and Madigan have both received a grant previously from FNIH.

This article was published in a supplement sponsored by the Foundation for the National Institutes of Health (FNIH). The supplement was guest edited by Stephen J.W. Evans. It was peer reviewed by Olaf H. Klungel who received a small honorarium to cover out-of-pocket expenses. S.J.W.E has received travel funding from the FNIH to travel to the OMOP symposium and received a fee from FNIH for the review of a protocol for OMOP. O.H.K has received funding for the IMI-PROTECT project from the Innovative Medicines Initiative Joint Undertaking ( under Grant Agreement no 115004, resources of which are composed of financial contribution from the European Union’s Seventh Framework Programme (FP7/2007-2013) and EFPIA companies’ in kind contribution.

Author information

Authors and Affiliations


Corresponding author

Correspondence to David Madigan.

Additional information

The OMOP research used data from Truven Health Analytics (formerly the Health Business of Thomson Reuters), and includes MarketScan® Research Databases, represented with MarketScan Lab Supplemental (MSLR, 1.2 m persons), MarketScan Medicare Supplemental Beneficiaries (MDCR, 4.6 m persons), MarketScan Multi-State Medicaid (MDCD, 10.8 m persons), MarketScan Commercial Claims and Encounters (CCAE, 46.5 m persons). Data also provided by Quintiles® Practice Research Database (formerly General Electric’s Electronic Health Record, 11.2 m persons) database. GE is an electronic health record database while the other four databases contain administrative claims data.



figure a

The effect estimates for all test cases across the five databases using the optimal analysis choice setting (2000031). MSLR MarketScan Lab Supplemental, MDCD MarketScan Multi-state Medicaid, MDCR MarketScan Medicare Supplemental Beneficiaries, CCAE MarketScan Commercial Claims and Encounters, GE GE Centricity

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Madigan, D., Schuemie, M.J. & Ryan, P.B. Empirical Performance of the Case–Control Method: Lessons for Developing a Risk Identification and Analysis System. Drug Saf 36, 73–82 (2013).

Download citation

  • Published:

  • Issue Date:

  • DOI:


  • Acute Kidney Injury
  • Index Date
  • Salmeterol
  • Coverage Probability
  • Darbepoetin Alfa