Purpose of Review
An important component of the Food and Drug Administration’s Sentinel Initiative is the active post-market risk identification and analysis (ARIA) system, which utilizes semi-automated, parameterized computer programs to implement propensity-score adjusted and self-controlled risk interval designs to conduct targeted surveillance of medical products in the Sentinel Distributed Database. In this manuscript, we review literature relevant to the development of these programs and describe their application within the Sentinel Initiative.
These quality-checked and publicly available tools have been successfully used to conduct rapid, replicable, and targeted safety analyses of several medical products. In addition to speed and reproducibility, use of semi-automated tools allows investigators to focus on decisions regarding key methodological parameters. We also identified challenges associated with the use of these methods in distributed and prospective datasets like the Sentinel Distributed Database, namely uncertainty regarding the optimal approach to estimating propensity scores in dynamic data among data partners of heterogeneous size.
Future research should focus on the methodological challenges raised by these applications as well as developing new modular programs for targeted surveillance of medical products.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Price includes VAT (USA)
Tax calculation will be finalised during checkout.
Recently Published Papers of Particular Interest Have Been Highlighted as: • Of importance, •• Of major importance
Behrman RE, Benner JS, Brown JS, McClellan M, Woodcock J, Platt R. Developing the Sentinel System--a national resource for evidence development. N Engl J Med. 2011;364(6):498–9.
Platt R, Wilson M, Chan KA, Benner JS, Marchibroda J, McClellan M. The new Sentinel Network--improving the evidence of medical-product safety. N Engl J Med. 2009;361(7):645–7.
Coloma PM, Trifirò G, Patadia V, Sturkenboom M. Postmarketing safety surveillance : where does signal detection using electronic healthcare records fit into the big picture? Drug Saf. 2013;36(3):183–97.
Maro JC, Platt R, Holmes JH, Strom BL, Hennessy S, Lazarus R, et al. Design of a national distributed health data network. Ann Intern Med. 2009;151(5):341–4.
Brown JS, Holmes JH, Shah K, Hall K, Lazarus R, Platt R. Distributed health data networks: a practical and preferred approach to multi-institutional evaluations of comparative effectiveness, safety, and quality of care. Med Care. 2010;48(6 Suppl):S45–51.
Chen RT, Glasser JW, Rhodes PH, Davis RL, Barlow WE, Thompson RS, et al. Vaccine Safety Datalink project: a new tool for improving vaccine safety monitoring in the United States. The Vaccine Safety Datalink Team. Pediatrics. 1997;99(6):765–73.
Fahey KR. The Pioneering Role of the Vaccine Safety Datalink Project (VSD) to Advance Collaborative Research and Distributed Data Networks. EGEMS Wash DC. 2015;3(1):1195.
Baggs J, Gee J, Lewis E, Fowler G, Benson P, Lieu T, et al. The Vaccine Safety Datalink: a model for monitoring immunization safety. Pediatrics. 2011;127(Suppl 1):S45–53.
McNeil MM, Gee J, Weintraub ES, Belongia EA, Lee GM, Glanz JM, et al. The Vaccine Safety Datalink: successes and challenges monitoring vaccine safety. Vaccine. 2014;32(42):5390–8.
• Ball R, Robb M, Anderson SA, Dal Pan G. The FDA’s sentinel initiative--A comprehensive approach to medical product surveillance. Clin Pharmacol Ther. 2016;99(3):265–8. Provides a comprehensive overview of the Sentinel Initiative for medical product safety surveillance
Platt R, Carnahan R. The U.S. Food and Drug Administration’s Mini-Sentinel Program. Pharmacoepidemiol Drug Saf. 2012;21:1–303.
Curtis LH, Weiner MG, Boudreau DM, Cooper WO, Daniel GW, Nair VP, et al. Design considerations, architecture, and use of the Mini-Sentinel distributed data system. Pharmacoepidemiol Drug Saf. 2012;21:23–31.
Platt R. FDA’s Mini-Sentinel Program to Evaluate the Safety of Marketed Medical Products: Progress and Direction. Presentation to the Brookings Institution on January 31, 2013. [Internet]. [cited 2017 Jun 15]. Available from: https://www.brookings.edu/wp-content/uploads/2013/01/Richard-Platt-Presentation.pdf
Snapshot of Database Statistics | Sentinel System [Internet]. [cited 2017 Jun 29]. Available from: https://www.sentinelinitiative.org/sentinel/snapshot-database-statistics
Toh S, Reichman ME, Houstoun M, Ross Southworth M, Ding X, Hernandez AF, et al. Comparative risk for angioedema associated with the use of drugs that target the renin-angiotensin-aldosterone system. Arch Intern Med. 2012;172(20):1582–9.
Toh S, Hampp C, Reichman ME, Graham DJ, Balakrishnan S, Pucino F, et al. Risk for Hospitalized Heart Failure Among New Users of Saxagliptin, Sitagliptin, and Other Antihyperglycemic Drugs: A Retrospective Cohort Study. Ann Intern Med. 2016;164(11):705–14.
Implementation of a Randomized Controlled Trial to Improve Treatment with Oral Anticoagulants in Patients with Atrial Fibrillation (IMPACT-AFib) | Sentinel System [Internet]. [cited 2017 Jun 29]. Available from: https://www.sentinelinitiative.org/content/implementation-randomized-controlled-trial-improve-treatment-oral-anticoagulants-patients
Food and Drug Administration Amendments Act of 2007 [Internet]. Sect. 905, 110–85 Sep 27, 2007. Available from: https://www.gpo.gov/fdsys/pkg/PLAW-110publ85/html/PLAW-110publ85.htm
•• Gagne JJ, Wang SV, Rassen JA, Schneeweiss SA. modular, prospective, semi-automated drug safety monitoring system for use in a distributed data environment. Pharmacoepidemiol Drug Saf. 2014;23(6):619–27. Develops and tests a semi-automated, PS-based, modular program for safety monitoring in distributed databases, which is later developed into the PS adjustment tool
Stürmer T, Wyss R, Glynn RJ, Brookhart MA. Propensity scores for confounder adjustment when assessing the effects of medical interventions using nonexperimental study designs. J Intern Med. 2014;275(6):570–80.
Glynn RJ, Schneeweiss S, Stürmer T. Indications for propensity scores and review of their use in pharmacoepidemiology. Basic Clin Pharmacol Toxicol. 2006;98(3):253–9.
Glynn RJ, Gagne JJ, Schneeweiss S. Role of disease risk scores in comparative effectiveness research with emerging therapies. Pharmacoepidemiol Drug Saf. 2012;21(Suppl 2):138–47.
Arbogast PG, Ray WA. Use of disease risk scores in pharmacoepidemiologic studies. Stat Methods Med Res. 2009;18(1):67–80.
Wyss R, Glynn RJ, Gagne JJA. Review of Disease Risk Scores and Their Application in Pharmacoepidemiology. Curr Epidemiol Rep. 2016;3(4):277–84.
Rosenbaum PR, Rubin DB. The Central Role of the Propensity Score in Observational Studies for Causal Effects. Biometrika. 1983;70(1):41–55.
Hansen BB. The prognostic analogue of the propensity score. Biometrika. 2008;95(2):481–8.
Toh S, Gagne JJ, Rassen JA, Fireman BH, Kulldorff M, Brown JS. Confounding adjustment in comparative effectiveness research conducted within distributed research networks. Med Care. 2013;51(8 Suppl 3):S4–10.
Toh S, Reichman ME, Houstoun M, Ding X, Fireman BH, Gravel E, et al. Multivariable confounding adjustment in distributed data networks without sharing of patient-level data. Pharmacoepidemiol Drug Saf. 2013;22(11):1171–7.
Kumamaru H, Gagne JJ, Glynn RJ, Setoguchi S, Schneeweiss S. Comparison of high-dimensional confounder summary scores in comparative studies of newly marketed medications. J Clin Epidemiol. 2016;76:200–8.
Schneeweiss S, Rassen JA, Glynn RJ, Avorn J, Mogun H, Brookhart MA. High-dimensional propensity score adjustment in studies of treatment effects using health care claims data. Epidemiol Camb Mass. 2009;20(4):512–22.
Rassen JA, Glynn RJ, Brookhart MA, Schneeweiss S. Covariate selection in high-dimensional propensity score analyses of treatment effects in small samples. Am J Epidemiol. 2011;173(12):1404–13.
Rassen JA, Solomon DH, Curtis JR, Herrinton L, Schneeweiss S. Privacy-maintaining propensity score-based pooling of multiple databases applied to a study of biologics. Med Care. 2010;48(6 Suppl):S83–9.
Toh S, Shetterly S, Powers JD, Arterburn D. Privacy-preserving analytic methods for multisite comparative effectiveness and patient-centered outcomes research. Med Care. 2014;52(7):664–8.
Fireman B, Lee J, Lewis N, Bembom O, van der Laan M, Baxter R. Influenza vaccination and mortality: differentiating vaccine effects from bias. Am J Epidemiol. 2009;170(5):650–6.
Mini-Sentinel Methods Propensity Score Matching Tool Enhancements Report.pdf [Internet]. [cited 2017 Jun 29]. Available from: https://www.sentinelinitiative.org/sites/default/files/Methods/Mini-Sentinel_Methods_Propensity-Score-Matching-Tool-Enhancements-Report_0.pdf
Wahl PM, Gagne JJ, Wasser TE, Eisenberg DF, Rodgers JK, Daniel GW, et al. Early steps in the development of a claims-based targeted healthcare safety monitoring system and application to three empirical examples. Drug Saf. 2012;35(5):407–16.
Kulldorff M, Davis RL, Kolczak† M, Lewis E, Lieu T, Platt RA. Maximized Sequential Probability Ratio Test for Drug and Vaccine Safety Surveillance. Seq Anal. 2011;30(1):58–78.
Yih WK, Kulldorff M, Fireman BH, Shui IM, Lewis EM, Klein NP, et al. Active surveillance for adverse events: the experience of the Vaccine Safety Datalink project. Pediatrics. 2011;127(Suppl 1):S54–64.
Gagne JJ, Glynn RJ, Rassen JA, Walker AM, Daniel GW, Sridhar G, et al. Active safety monitoring of newly marketed medications in a distributed data network: application of a semi-automated monitoring system. Clin Pharmacol Ther. 2012;92(1):80–6.
• Gagne JJ, Rassen JA, Choudhry NK, Bohn RL, Patrick AR, Sridhar G, et al. Near-real-time monitoring of new drugs: an application comparing prasugrel versus clopidogrel. Drug Saf. 2014;37(3):151–61. Demonstrates the feasibility of conducting safety monitoring in distributed databases where data accumulate prospectively using the PS adjustment precursor tool
Level 2 Modular Program Queries | Sentinel System [Internet]. [cited 2017 Jun 29]. Available from: https://www.sentinelinitiative.org/sentinel/routine-querying-tools/level-2-modular-program-queries
Mini-Sentinel PROMPT Users Guide [Internet]. [cited 2017 Jun 29]. Available from: https://www.sentinelinitiative.org/sites/default/files/Methods/Mini-Sentinel_PROMPT_Users-Guide_0.pdf
Surveillance Tools | Sentinel System [Internet]. [cited 2017 Jun 16]. Available from: https://www.sentinelinitiative.org/sentinel/surveillance-tools
• Gagne JJ, Han X, Hennessy S, Leonard CE, Chrischilles EA, Carnahan RM, et al. Successful Comparison of US Food and Drug Administration Sentinel Analysis Tools to Traditional Approaches in Quantifying a Known Drug-Adverse Event Association. Clin Pharmacol Ther. 2016;100(5):558–64. Proof-of-principle study that targeted, semi-automated PS-adjusted analyses could reproduce results from full epidemiological analyses
Zhou M, Wang SV, Leonard CE, Gagne JJ, Fuller C, Hampp C, et al. Sentinel Modular Program for Propensity-Score Matched Cohort Analyses: Application to Glyburide, Glipizide, and Serious Hypoglycemia. Epidemiol Camb Mass. 2017 Jul 4;
•• Developments, Applications, And Methodological Challenges to The Use Of Propensity Score Matching Approaches In FDA’s Sentinel Program | Sentinel System [Internet]. [cited 2017 Jun 16]. Available from: https://www.sentinelinitiative.org/sentinel/methods/developments-applications-and-methodological-challenges-use-propensity-score Describes applications of the PS adjustment tool within Sentinel in detail, emphasizing associated methodological and technical challenges
Li L, Kulldorff M, Russek-Cohen E, Kawai AT, Hua W. Quantifying the impact of time-varying baseline risk adjustment in the self-controlled risk interval design. Pharmacoepidemiol Drug Saf. 2015;24(12):1304–12.
Gault N, Castañeda-Sanabria J, De Rycke Y, Guillo S, Foulon S, Tubach F. Self-controlled designs in pharmacoepidemiology involving electronic healthcare databases: a systematic review. BMC Med Res Methodol. 2017;17(1):25.
Petersen I, Douglas I, Whitaker H. Self controlled case series methods: an alternative to standard epidemiological study designs. BMJ. 2016;354:i4515.
Hallas J, Pottegård A. Use of self-controlled designs in pharmacoepidemiology. J Intern Med. 2014;275(6):581–9.
Baker MA, Lieu TA, Li L, Hua W, Qiang Y, Kawai AT, et al. A vaccine study design selection framework for the postlicensure rapid immunization safety monitoring program. Am J Epidemiol. 2015;181(8):608–18.
Nguyen M, Ball R, Midthun K, Lieu TA. The Food and Drug Administration’s Post-Licensure Rapid Immunization Safety Monitoring program: strengthening the federal vaccine safety enterprise. Pharmacoepidemiol Drug Saf. 2012;21(Suppl 1):291–7.
Tse A, Tseng HF, Greene SK, Vellozzi C, Lee GM, Rapid Cycle VSD. Analysis Influenza Working Group. Signal identification and evaluation for risk of febrile seizures in children following trivalent inactivated influenza vaccine in the Vaccine Safety Datalink Project, 2010-2011. Vaccine. 2012;30(11):2024–31.
Yih WK, Lee GM, Lieu TA, Ball R, Kulldorff M, Rett M, et al. Surveillance for adverse events following receipt of pandemic 2009 H1N1 vaccine in the Post-Licensure Rapid Immunization Safety Monitoring (PRISM) System, 2009-2010. Am J Epidemiol. 2012;175(11):1120–8.
Greene SK, Rett M, Weintraub ES, Li L, Yin R, Amato AA, et al. Risk of confirmed Guillain-Barre syndrome following receipt of monovalent inactivated influenza A (H1N1) and seasonal influenza vaccines in the Vaccine Safety Datalink Project, 2009-2010. Am J Epidemiol. 2012;175(11):1100–9.
McCarthy NL, Gee J, Lin ND, Thyagarajan V, Pan Y, Su S, et al. Evaluating the safety of influenza vaccine using a claims-based health system. Vaccine. 2013;31(50):5975–82.
Kawai AT, Li L, Kulldorff M, Vellozzi C, Weintraub E, Baxter R, et al. Absence of associations between influenza vaccines and increased risks of seizures, Guillain-Barré syndrome, encephalitis, or anaphylaxis in the 2012-2013 season. Pharmacoepidemiol Drug Saf. 2014;23(5):548–53.
Kawai AT, Martin D, Kulldorff M, Li L, Cole DV, McMahill-Walraven CN, et al. Febrile Seizures After 2010-2011 Trivalent Inactivated Influenza Vaccine. Pediatrics. 2015;136(4):e848–55.
Duffy J, Weintraub E, Hambidge SJ, Jackson LA, Kharbanda EO, Klein NP, et al. Febrile Seizure Risk After Vaccination in Children 6 to 23 Months. Pediatrics. 2016;138(1)
Li R, Stewart B, McNeil MM, Duffy J, Nelson J, Kawai AT, et al. Post licensure surveillance of influenza vaccines in the Vaccine Safety Datalink in the 2013-2014 and 2014-2015 seasons. Pharmacoepidemiol Drug Saf. 2016;25(8):928–34.
Nelson JC, Jackson ML, Weiss NS, Jackson LA. New strategies are needed to improve the accuracy of influenza vaccine effectiveness estimates among seniors. J Clin Epidemiol. 2009;62(7):687–94.
Jackson LA, Nelson JC, Benson P, Neuzil KM, Reid RJ, Psaty BM, et al. Functional status is a confounder of the association of influenza vaccine and risk of all cause mortality in seniors. Int J Epidemiol. 2006;35(2):345–52.
• Yih WK, Lieu TA, Kulldorff M, Martin D, McMahill-Walraven CN, Platt R, et al. Intussusception risk after rotavirus vaccination in U.S. infants. N Engl J Med. 2014;370(6):503–12. Influenza safety study that illustrates challenges of dynamic data in Sentinel Distributed Database
Yih WK, Greene SK, Zichittella L, Kulldorff M, Baker MA, de Jong JLO, et al. Evaluation of the risk of venous thromboembolism after quadrivalent human papillomavirus vaccination among US females. Vaccine. 2016;34(1):172–8.
Yih WK, Kulldorff M, Sandhu SK, Zichittella L, Maro JC, Cole DV, et al. Prospective influenza vaccine safety surveillance using fresh data in the Sentinel System. Pharmacoepidemiol Drug Saf. 2016;25(5):481–92.
Mini-Sentinel PROMPT Self Control Design Tool Technical Users Guide.pdf [Internet]. [cited 2017 Jun 29]. Available from: https://www.sentinelinitiative.org/sites/default/files/Methods/Mini-Sentinel_PROMPT_Self-Control-Design-Tool_Technical-Users-Guide_0.pdf
Sequential Analysis of Gardasil 9 Safety Surveillance Plan | Sentinel System [Internet]. [cited 2017 Jun 15]. Available from: https://www.sentinelinitiative.org/vaccines-blood-biologics/assessments/sequential-analysis-gardasil-9-safety-surveillance-plan
Ninth Annual Sentinel Initiative Public Workshop | Margolis Center for Health Policy. Slides 216–230. [Internet]. [cited 2017 Jun 16]. Available from: https://healthpolicy.duke.edu/events/ninth-annual-sentinel-initiative-public-workshop
Greene SK, Kulldorff M, Yin R, Yih WK, Lieu TA, Weintraub ES, et al. Near real-time vaccine safety surveillance with partially accrued data. Pharmacoepidemiol Drug Saf. 2011;20(6):583–90.
Wu Y, Jiang X, Kim J, Ohno-Machado L. Grid Binary LOgistic REgression (GLORE): building shared models without sharing data. J Am Med Inform Assoc JAMIA. 2012;19(5):758–64.
Analysis of Integrated Data without Data Integration. Chance. 2004;17(3):26–9.
Mini-Sentinel is a pilot project sponsored by the US Food and Drug Administration (FDA) to inform and facilitate development of a fully operational active surveillance system, the Sentinel System, for monitoring the safety of FDA-regulated medical products. Mini-Sentinel is one piece of the Sentinel Initiative, a multi-faceted effort by the FDA to develop a national electronic system that will complement existing methods of safety surveillance. Mini-Sentinel Collaborators include Data and Academic Partners that provide access to health care data and ongoing scientific, technical, methodological, and organizational expertise. The Mini-Sentinel Coordinating Center is funded by the FDA through the Department of Health and Human Services (HHS) Contract number HHSF223200910006I. This project was also supported in part by NIH U01 EB023683.
Conflict of Interest
John G. Connolly, Catherine A. Panozzo, Noelle Cocoros, Meijia Zhou, and Judith C. Maro each declare no potential conflicts of interest.
Sengwee Toh reports grants from US Food and Drug Administration during the conduct of the study.
Shirley V. Wang reports grants from Sentinel Initiative, during the conduct of the study; personal fees from Aetion, Inc., grants from Novartis, grants from Agency for Healthcare Research and Quality outside the submitted work.
Joshua J. Gagne reports grants from US FDA, during the conduct of the study; grants from Novartis Pharmaceuticals Corporation, grants from Eli Lilly and Company, personal fees from Aetion, Inc., personal fees from Optum, Inc., outside the submitted work.
Candace C. Fuller reports grants from US Food and Drug Administration during the conduct of the study.
Human and Animal Rights
All reported studies/experiments with human or animal subjects performed by the authors have been previously published and complied with all applicable ethical standards (including the Helsinki declaration and its amendments, institutional/national research committee standards, and international/national/institutional guidelines).
This article is part of the Topical Collection on Pharmacoepidemiology
About this article
Cite this article
Connolly, J.G., Wang, S.V., Fuller, C.C. et al. Development and Application of Two Semi-Automated Tools for Targeted Medical Product Surveillance in a Distributed Data Network. Curr Epidemiol Rep 4, 298–306 (2017). https://doi.org/10.1007/s40471-017-0121-0
- Drug safety