Drug Safety

, Volume 36, Supplement 1, pp 27–32 | Cite as

How Well Do Various Health Outcome Definitions Identify Appropriate Cases in Observational Studies?

  • Richard A. Hansen
  • Michael D. Gray
  • Brent I. Fox
  • Joshua C. Hollingsworth
  • Juan Gao
  • Peng Zeng
Original Research Article



Observational data can be useful for drug safety research, but accurate measurement of adverse health outcomes is paramount. Best practices for identifying important health outcomes of interest (HOI) are needed.


To evaluate the extent to which health outcome definitions commonly used in observational database studies identify cases that are consistent with expert panel assessment of the underlying data.


Competing HOI definitions were used to identify potential cases of acute liver injury (ALI; n = 208), acute kidney injury (AKI; n = 200), and myocardial infarction (MI; n = 204) in the Truven MarketScan Lab Database (MSLR). Panelists reviewed patient-level data and answered questions about whether they believed the case actually reflected the HOI and their certainty of case classification on a 10-point scale (1 = unlikely to 10 = likely). Each patient was reviewed independently by two panelists. Case disagreements were resolved through consensus meetings. Positive predictive value (PPV) was calculated as the number of cases deemed to be true over the total number of sampled cases. Kappa statistics assessed inter-rater agreement.


PPV ranged from 0 to 52 % across ALI definitions, 12 to 82 % across AKI definitions, and 1 to 56 % across MI definitions. Certainty scores on the 10-point scale paralleled the PPV, with a range of mean values from 1.7 to 4.8 across ALI definitions, 3.1 to 6.0 across AKI definitions, and 2.8 to 5.7 across MI definitions. Inter-rater agreement was low to moderate (Kappa range 0.0–0.6).


Existing HOI definitions had relatively low PPV based on expert panel review. Experts commonly disagreed on case classification. Additional work is needed to refine HOI case definitions if observational data are to be reliably used for health outcome assessment.


Positive Predictive Value Acute Liver Injury Case Classification Common Data Model Observational Medical Outcome Partnership 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



This project was supported by the Foundation for the National Institutes of Health, grant HANSE11OMOP. We thank the following expert panelists: Jon Duke, MD, MS, Regenstrief Institute; Ben Powers, MD, MHS—St Luke’s Health System; Mei Sheng Duh, MPH, SCD, Analysis Group; T. Craig Cheetham, PharmD, MS, Kaiser Permanente; Scott Sanoff, MD, MPH, University of Virginia; Andrew Bomback, MD, MPH, Columbia University; Gregory Hess, MD, MBA, MSc, University of Pennsylvania; Themistocles Assimes, MD, PhD, Stanford University. A preliminary version of this work was presented as a poster at the OMOP Symposium, June 28, 2012, and at the Annual Meeting of the American Pharmacists Association, March 3, 2013. Dr. Hansen has received consulting support from Takeda Pharmaceuticals, Novartis, and Daiichi Sankyo for research unrelated to the content of this paper, and he has provided expert testimony for Allergan. Drs. Gray, Fox, Hollingsworth, and Zeng, as well as Ms. Gao declare no potential conflicts of interest.

This article was published in a supplement sponsored by the Foundation for the National Institutes of Health (FNIH). The supplement was guest edited by Stephen J.W. Evans. It was peer reviewed by Olaf H. Klungel who received a small honorarium to cover out-of-pocket expenses. S.J.W.E has received travel funding from the FNIH to travel to the OMOP symposium and received a fee from FNIH for the review of a protocol for OMOP. O.H.K has received funding for the IMI-PROTECT project from the Innovative Medicines Initiative Joint Undertaking ( under Grant Agreement no 115004, resources of which are composed of financial contribution from the European Union’s Seventh Framework Programme (FP7/2007-2013) and EFPIA companies’ in kind contribution.


  1. 1.
    Stang PE, Ryan PB, Racoosin JA, et al. Advancing the science for active surveillance: rationale and design for the Observational Medical Outcomes Partnership. Ann Intern Med. 2010;153(9):600–6.PubMedCrossRefGoogle Scholar
  2. 2.
    Observational Medical Outcomes Partnership (2010) [cited 2012 October 8].
  3. 3.
    Ryan PB, Madigan D. Observational Medical Outcomes Partnership (OMOP) methods evaluation. OMOP 2011 Symposium; 2011 January 11; Washington, DC.Google Scholar
  4. 4.
    Racoosin JA, Ryan PB. Implications of health outcomes of interest definitions: acute liver injury case study. OMOP 2011 Symposium; 2011 January 11; Washington, DC.Google Scholar
  5. 5.
    Jinjuvadia K, Kwan W, Fontana RJ. Searching for a needle in a haystack: use of ICD-9-CM codes in drug-induced liver injury. Am J Gastroenterol. 2007;102(11):2437–43.PubMedCrossRefGoogle Scholar
  6. 6.
    Vlasschaert ME, Bejaimal SA, Hackam DG, et al. Validity of administrative database coding for kidney disease: a systematic review. Am J Kidney Dis. 2011;57(1):29–43.PubMedCrossRefGoogle Scholar
  7. 7.
    Metcalfe A, Neudam A, Forde S, et al. Case definitions for acute myocardial infarction in administrative databases and their impact on in-hospital mortality rates. Health Serv Res. 2013;48(1):290–318.PubMedCrossRefGoogle Scholar
  8. 8.
    Fox BI, Hollingsworth JC, Gray MD, et al. Developing an expert panel process to refine health outcome definitions in observational data. J Biomed Inform 2013;46(5):795–804.PubMedCrossRefGoogle Scholar
  9. 9.
    Overhage JM, Ryan PB, Reich CG, et al. Validation of a common data model for active safety surveillance research. J Am Med Inform Assoc. 2012;19(1):54–60.PubMedCrossRefGoogle Scholar
  10. 10.
    Observational Medical Outcomes Partnership. OMOP Collaborator—Thonmson Reuters. [March 7, 2012].
  11. 11.
    Ryan PB. Lessons for building a risk identification and analysis system. Observational Medical Outcomes Partnership Symposium, June 28, 2012. Bethesda, MD.
  12. 12.
    Vessey MP, Doll R. Investigation of relation between use of oral contraceptives and thromboembolic disease. Br Med J. 1968;2(5599):199–205.PubMedCrossRefGoogle Scholar
  13. 13.
    Vessey MP. Learning how to control biases in studies to identify adverse effects of drugs. JLL Bulletin: Commentaries on the history of treatment evaluation. 2006. Accessed 26 June 2013.
  14. 14.
    Katz AJ, Ryan PB, Racoosin JA, et al. Assessment of case definitions for identifying acute liver injury in large observational databases. Drug Saf 2013;36(8):651–61.PubMedCrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2013

Authors and Affiliations

  • Richard A. Hansen
    • 1
  • Michael D. Gray
    • 2
  • Brent I. Fox
    • 1
  • Joshua C. Hollingsworth
    • 1
  • Juan Gao
    • 1
  • Peng Zeng
    • 3
  1. 1.Department of Pharmacy Care Systems, Harrison School of PharmacyAuburn UniversityAuburnUSA
  2. 2.HP LabsPalo AltoUSA
  3. 3.College of Science and MathematicsAuburn UniversityAuburnUSA

Personalised recommendations