Skip to main content

Development and Application of Two Semi-Automated Tools for Targeted Medical Product Surveillance in a Distributed Data Network


Purpose of Review

An important component of the Food and Drug Administration’s Sentinel Initiative is the active post-market risk identification and analysis (ARIA) system, which utilizes semi-automated, parameterized computer programs to implement propensity-score adjusted and self-controlled risk interval designs to conduct targeted surveillance of medical products in the Sentinel Distributed Database. In this manuscript, we review literature relevant to the development of these programs and describe their application within the Sentinel Initiative.

Recent Findings

These quality-checked and publicly available tools have been successfully used to conduct rapid, replicable, and targeted safety analyses of several medical products. In addition to speed and reproducibility, use of semi-automated tools allows investigators to focus on decisions regarding key methodological parameters. We also identified challenges associated with the use of these methods in distributed and prospective datasets like the Sentinel Distributed Database, namely uncertainty regarding the optimal approach to estimating propensity scores in dynamic data among data partners of heterogeneous size.


Future research should focus on the methodological challenges raised by these applications as well as developing new modular programs for targeted surveillance of medical products.

This is a preview of subscription content, access via your institution.

Fig. 1


Recently Published Papers of Particular Interest Have Been Highlighted as: • Of importance, •• Of major importance

  1. 1.

    Behrman RE, Benner JS, Brown JS, McClellan M, Woodcock J, Platt R. Developing the Sentinel System--a national resource for evidence development. N Engl J Med. 2011;364(6):498–9.

    CAS  Article  PubMed  Google Scholar 

  2. 2.

    Platt R, Wilson M, Chan KA, Benner JS, Marchibroda J, McClellan M. The new Sentinel Network--improving the evidence of medical-product safety. N Engl J Med. 2009;361(7):645–7.

    CAS  Article  PubMed  Google Scholar 

  3. 3.

    Coloma PM, Trifirò G, Patadia V, Sturkenboom M. Postmarketing safety surveillance : where does signal detection using electronic healthcare records fit into the big picture? Drug Saf. 2013;36(3):183–97.

    Article  PubMed  Google Scholar 

  4. 4.

    Maro JC, Platt R, Holmes JH, Strom BL, Hennessy S, Lazarus R, et al. Design of a national distributed health data network. Ann Intern Med. 2009;151(5):341–4.

    Article  PubMed  Google Scholar 

  5. 5.

    Brown JS, Holmes JH, Shah K, Hall K, Lazarus R, Platt R. Distributed health data networks: a practical and preferred approach to multi-institutional evaluations of comparative effectiveness, safety, and quality of care. Med Care. 2010;48(6 Suppl):S45–51.

    Article  PubMed  Google Scholar 

  6. 6.

    Chen RT, Glasser JW, Rhodes PH, Davis RL, Barlow WE, Thompson RS, et al. Vaccine Safety Datalink project: a new tool for improving vaccine safety monitoring in the United States. The Vaccine Safety Datalink Team. Pediatrics. 1997;99(6):765–73.

    CAS  Article  PubMed  Google Scholar 

  7. 7.

    Fahey KR. The Pioneering Role of the Vaccine Safety Datalink Project (VSD) to Advance Collaborative Research and Distributed Data Networks. EGEMS Wash DC. 2015;3(1):1195.

    PubMed  PubMed Central  Google Scholar 

  8. 8.

    Baggs J, Gee J, Lewis E, Fowler G, Benson P, Lieu T, et al. The Vaccine Safety Datalink: a model for monitoring immunization safety. Pediatrics. 2011;127(Suppl 1):S45–53.

    Article  PubMed  Google Scholar 

  9. 9.

    McNeil MM, Gee J, Weintraub ES, Belongia EA, Lee GM, Glanz JM, et al. The Vaccine Safety Datalink: successes and challenges monitoring vaccine safety. Vaccine. 2014;32(42):5390–8.

    Article  PubMed  Google Scholar 

  10. 10.

    • Ball R, Robb M, Anderson SA, Dal Pan G. The FDA’s sentinel initiative--A comprehensive approach to medical product surveillance. Clin Pharmacol Ther. 2016;99(3):265–8. Provides a comprehensive overview of the Sentinel Initiative for medical product safety surveillance

    CAS  Article  PubMed  Google Scholar 

  11. 11.

    Platt R, Carnahan R. The U.S. Food and Drug Administration’s Mini-Sentinel Program. Pharmacoepidemiol Drug Saf. 2012;21:1–303.

    PubMed  Google Scholar 

  12. 12.

    Curtis LH, Weiner MG, Boudreau DM, Cooper WO, Daniel GW, Nair VP, et al. Design considerations, architecture, and use of the Mini-Sentinel distributed data system. Pharmacoepidemiol Drug Saf. 2012;21:23–31.

    Article  PubMed  Google Scholar 

  13. 13.

    Platt R. FDA’s Mini-Sentinel Program to Evaluate the Safety of Marketed Medical Products: Progress and Direction. Presentation to the Brookings Institution on January 31, 2013. [Internet]. [cited 2017 Jun 15]. Available from:

  14. 14.

    Snapshot of Database Statistics | Sentinel System [Internet]. [cited 2017 Jun 29]. Available from:

  15. 15.

    Toh S, Reichman ME, Houstoun M, Ross Southworth M, Ding X, Hernandez AF, et al. Comparative risk for angioedema associated with the use of drugs that target the renin-angiotensin-aldosterone system. Arch Intern Med. 2012;172(20):1582–9.

    CAS  Article  PubMed  Google Scholar 

  16. 16.

    Toh S, Hampp C, Reichman ME, Graham DJ, Balakrishnan S, Pucino F, et al. Risk for Hospitalized Heart Failure Among New Users of Saxagliptin, Sitagliptin, and Other Antihyperglycemic Drugs: A Retrospective Cohort Study. Ann Intern Med. 2016;164(11):705–14.

    Article  PubMed  PubMed Central  Google Scholar 

  17. 17.

    Implementation of a Randomized Controlled Trial to Improve Treatment with Oral Anticoagulants in Patients with Atrial Fibrillation (IMPACT-AFib) | Sentinel System [Internet]. [cited 2017 Jun 29]. Available from:

  18. 18.

    Food and Drug Administration Amendments Act of 2007 [Internet]. Sect. 905, 110–85 Sep 27, 2007. Available from:

  19. 19.

    •• Gagne JJ, Wang SV, Rassen JA, Schneeweiss SA. modular, prospective, semi-automated drug safety monitoring system for use in a distributed data environment. Pharmacoepidemiol Drug Saf. 2014;23(6):619–27. Develops and tests a semi-automated, PS-based, modular program for safety monitoring in distributed databases, which is later developed into the PS adjustment tool

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  20. 20.

    Stürmer T, Wyss R, Glynn RJ, Brookhart MA. Propensity scores for confounder adjustment when assessing the effects of medical interventions using nonexperimental study designs. J Intern Med. 2014;275(6):570–80.

    Article  PubMed  PubMed Central  Google Scholar 

  21. 21.

    Glynn RJ, Schneeweiss S, Stürmer T. Indications for propensity scores and review of their use in pharmacoepidemiology. Basic Clin Pharmacol Toxicol. 2006;98(3):253–9.

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  22. 22.

    Glynn RJ, Gagne JJ, Schneeweiss S. Role of disease risk scores in comparative effectiveness research with emerging therapies. Pharmacoepidemiol Drug Saf. 2012;21(Suppl 2):138–47.

    Article  PubMed  PubMed Central  Google Scholar 

  23. 23.

    Arbogast PG, Ray WA. Use of disease risk scores in pharmacoepidemiologic studies. Stat Methods Med Res. 2009;18(1):67–80.

    Article  PubMed  Google Scholar 

  24. 24.

    Wyss R, Glynn RJ, Gagne JJA. Review of Disease Risk Scores and Their Application in Pharmacoepidemiology. Curr Epidemiol Rep. 2016;3(4):277–84.

    Article  Google Scholar 

  25. 25.

    Rosenbaum PR, Rubin DB. The Central Role of the Propensity Score in Observational Studies for Causal Effects. Biometrika. 1983;70(1):41–55.

    Article  Google Scholar 

  26. 26.

    Hansen BB. The prognostic analogue of the propensity score. Biometrika. 2008;95(2):481–8.

    Article  Google Scholar 

  27. 27.

    Toh S, Gagne JJ, Rassen JA, Fireman BH, Kulldorff M, Brown JS. Confounding adjustment in comparative effectiveness research conducted within distributed research networks. Med Care. 2013;51(8 Suppl 3):S4–10.

    Article  PubMed  Google Scholar 

  28. 28.

    Toh S, Reichman ME, Houstoun M, Ding X, Fireman BH, Gravel E, et al. Multivariable confounding adjustment in distributed data networks without sharing of patient-level data. Pharmacoepidemiol Drug Saf. 2013;22(11):1171–7.

    CAS  Article  PubMed  Google Scholar 

  29. 29.

    Kumamaru H, Gagne JJ, Glynn RJ, Setoguchi S, Schneeweiss S. Comparison of high-dimensional confounder summary scores in comparative studies of newly marketed medications. J Clin Epidemiol. 2016;76:200–8.

    Article  PubMed  Google Scholar 

  30. 30.

    Schneeweiss S, Rassen JA, Glynn RJ, Avorn J, Mogun H, Brookhart MA. High-dimensional propensity score adjustment in studies of treatment effects using health care claims data. Epidemiol Camb Mass. 2009;20(4):512–22.

    Article  Google Scholar 

  31. 31.

    Rassen JA, Glynn RJ, Brookhart MA, Schneeweiss S. Covariate selection in high-dimensional propensity score analyses of treatment effects in small samples. Am J Epidemiol. 2011;173(12):1404–13.

    Article  PubMed  PubMed Central  Google Scholar 

  32. 32.

    Rassen JA, Solomon DH, Curtis JR, Herrinton L, Schneeweiss S. Privacy-maintaining propensity score-based pooling of multiple databases applied to a study of biologics. Med Care. 2010;48(6 Suppl):S83–9.

    Article  PubMed  PubMed Central  Google Scholar 

  33. 33.

    Toh S, Shetterly S, Powers JD, Arterburn D. Privacy-preserving analytic methods for multisite comparative effectiveness and patient-centered outcomes research. Med Care. 2014;52(7):664–8.

    Article  PubMed  Google Scholar 

  34. 34.

    Fireman B, Lee J, Lewis N, Bembom O, van der Laan M, Baxter R. Influenza vaccination and mortality: differentiating vaccine effects from bias. Am J Epidemiol. 2009;170(5):650–6.

    Article  PubMed  PubMed Central  Google Scholar 

  35. 35.

    Mini-Sentinel Methods Propensity Score Matching Tool Enhancements Report.pdf [Internet]. [cited 2017 Jun 29]. Available from:

  36. 36.

    Wahl PM, Gagne JJ, Wasser TE, Eisenberg DF, Rodgers JK, Daniel GW, et al. Early steps in the development of a claims-based targeted healthcare safety monitoring system and application to three empirical examples. Drug Saf. 2012;35(5):407–16.

    Article  PubMed  Google Scholar 

  37. 37.

    Kulldorff M, Davis RL, Kolczak† M, Lewis E, Lieu T, Platt RA. Maximized Sequential Probability Ratio Test for Drug and Vaccine Safety Surveillance. Seq Anal. 2011;30(1):58–78.

    Article  Google Scholar 

  38. 38.

    Yih WK, Kulldorff M, Fireman BH, Shui IM, Lewis EM, Klein NP, et al. Active surveillance for adverse events: the experience of the Vaccine Safety Datalink project. Pediatrics. 2011;127(Suppl 1):S54–64.

    Article  PubMed  Google Scholar 

  39. 39.

    Gagne JJ, Glynn RJ, Rassen JA, Walker AM, Daniel GW, Sridhar G, et al. Active safety monitoring of newly marketed medications in a distributed data network: application of a semi-automated monitoring system. Clin Pharmacol Ther. 2012;92(1):80–6.

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  40. 40.

    • Gagne JJ, Rassen JA, Choudhry NK, Bohn RL, Patrick AR, Sridhar G, et al. Near-real-time monitoring of new drugs: an application comparing prasugrel versus clopidogrel. Drug Saf. 2014;37(3):151–61. Demonstrates the feasibility of conducting safety monitoring in distributed databases where data accumulate prospectively using the PS adjustment precursor tool

    CAS  Article  PubMed  Google Scholar 

  41. 41.

    Level 2 Modular Program Queries | Sentinel System [Internet]. [cited 2017 Jun 29]. Available from:

  42. 42.

    Mini-Sentinel PROMPT Users Guide [Internet]. [cited 2017 Jun 29]. Available from:

  43. 43.

    Surveillance Tools | Sentinel System [Internet]. [cited 2017 Jun 16]. Available from:

  44. 44.

    • Gagne JJ, Han X, Hennessy S, Leonard CE, Chrischilles EA, Carnahan RM, et al. Successful Comparison of US Food and Drug Administration Sentinel Analysis Tools to Traditional Approaches in Quantifying a Known Drug-Adverse Event Association. Clin Pharmacol Ther. 2016;100(5):558–64. Proof-of-principle study that targeted, semi-automated PS-adjusted analyses could reproduce results from full epidemiological analyses

    CAS  Article  PubMed  Google Scholar 

  45. 45.

    Zhou M, Wang SV, Leonard CE, Gagne JJ, Fuller C, Hampp C, et al. Sentinel Modular Program for Propensity-Score Matched Cohort Analyses: Application to Glyburide, Glipizide, and Serious Hypoglycemia. Epidemiol Camb Mass. 2017 Jul 4;

  46. 46.

    •• Developments, Applications, And Methodological Challenges to The Use Of Propensity Score Matching Approaches In FDA’s Sentinel Program | Sentinel System [Internet]. [cited 2017 Jun 16]. Available from: Describes applications of the PS adjustment tool within Sentinel in detail, emphasizing associated methodological and technical challenges

  47. 47.

    Li L, Kulldorff M, Russek-Cohen E, Kawai AT, Hua W. Quantifying the impact of time-varying baseline risk adjustment in the self-controlled risk interval design. Pharmacoepidemiol Drug Saf. 2015;24(12):1304–12.

    Article  PubMed  Google Scholar 

  48. 48.

    Gault N, Castañeda-Sanabria J, De Rycke Y, Guillo S, Foulon S, Tubach F. Self-controlled designs in pharmacoepidemiology involving electronic healthcare databases: a systematic review. BMC Med Res Methodol. 2017;17(1):25.

    Article  PubMed  PubMed Central  Google Scholar 

  49. 49.

    Petersen I, Douglas I, Whitaker H. Self controlled case series methods: an alternative to standard epidemiological study designs. BMJ. 2016;354:i4515.

    Article  PubMed  Google Scholar 

  50. 50.

    Hallas J, Pottegård A. Use of self-controlled designs in pharmacoepidemiology. J Intern Med. 2014;275(6):581–9.

    CAS  Article  PubMed  Google Scholar 

  51. 51.

    Baker MA, Lieu TA, Li L, Hua W, Qiang Y, Kawai AT, et al. A vaccine study design selection framework for the postlicensure rapid immunization safety monitoring program. Am J Epidemiol. 2015;181(8):608–18.

    Article  PubMed  Google Scholar 

  52. 52.

    Nguyen M, Ball R, Midthun K, Lieu TA. The Food and Drug Administration’s Post-Licensure Rapid Immunization Safety Monitoring program: strengthening the federal vaccine safety enterprise. Pharmacoepidemiol Drug Saf. 2012;21(Suppl 1):291–7.

    Article  PubMed  Google Scholar 

  53. 53.

    Tse A, Tseng HF, Greene SK, Vellozzi C, Lee GM, Rapid Cycle VSD. Analysis Influenza Working Group. Signal identification and evaluation for risk of febrile seizures in children following trivalent inactivated influenza vaccine in the Vaccine Safety Datalink Project, 2010-2011. Vaccine. 2012;30(11):2024–31.

    Article  PubMed  Google Scholar 

  54. 54.

    Yih WK, Lee GM, Lieu TA, Ball R, Kulldorff M, Rett M, et al. Surveillance for adverse events following receipt of pandemic 2009 H1N1 vaccine in the Post-Licensure Rapid Immunization Safety Monitoring (PRISM) System, 2009-2010. Am J Epidemiol. 2012;175(11):1120–8.

    Article  PubMed  Google Scholar 

  55. 55.

    Greene SK, Rett M, Weintraub ES, Li L, Yin R, Amato AA, et al. Risk of confirmed Guillain-Barre syndrome following receipt of monovalent inactivated influenza A (H1N1) and seasonal influenza vaccines in the Vaccine Safety Datalink Project, 2009-2010. Am J Epidemiol. 2012;175(11):1100–9.

    Article  PubMed  Google Scholar 

  56. 56.

    McCarthy NL, Gee J, Lin ND, Thyagarajan V, Pan Y, Su S, et al. Evaluating the safety of influenza vaccine using a claims-based health system. Vaccine. 2013;31(50):5975–82.

    Article  PubMed  Google Scholar 

  57. 57.

    Kawai AT, Li L, Kulldorff M, Vellozzi C, Weintraub E, Baxter R, et al. Absence of associations between influenza vaccines and increased risks of seizures, Guillain-Barré syndrome, encephalitis, or anaphylaxis in the 2012-2013 season. Pharmacoepidemiol Drug Saf. 2014;23(5):548–53.

    Article  PubMed  Google Scholar 

  58. 58.

    Kawai AT, Martin D, Kulldorff M, Li L, Cole DV, McMahill-Walraven CN, et al. Febrile Seizures After 2010-2011 Trivalent Inactivated Influenza Vaccine. Pediatrics. 2015;136(4):e848–55.

    Article  PubMed  Google Scholar 

  59. 59.

    Duffy J, Weintraub E, Hambidge SJ, Jackson LA, Kharbanda EO, Klein NP, et al. Febrile Seizure Risk After Vaccination in Children 6 to 23 Months. Pediatrics. 2016;138(1)

  60. 60.

    Li R, Stewart B, McNeil MM, Duffy J, Nelson J, Kawai AT, et al. Post licensure surveillance of influenza vaccines in the Vaccine Safety Datalink in the 2013-2014 and 2014-2015 seasons. Pharmacoepidemiol Drug Saf. 2016;25(8):928–34.

    Article  PubMed  Google Scholar 

  61. 61.

    Nelson JC, Jackson ML, Weiss NS, Jackson LA. New strategies are needed to improve the accuracy of influenza vaccine effectiveness estimates among seniors. J Clin Epidemiol. 2009;62(7):687–94.

    Article  PubMed  Google Scholar 

  62. 62.

    Jackson LA, Nelson JC, Benson P, Neuzil KM, Reid RJ, Psaty BM, et al. Functional status is a confounder of the association of influenza vaccine and risk of all cause mortality in seniors. Int J Epidemiol. 2006;35(2):345–52.

    Article  PubMed  Google Scholar 

  63. 63.

    • Yih WK, Lieu TA, Kulldorff M, Martin D, McMahill-Walraven CN, Platt R, et al. Intussusception risk after rotavirus vaccination in U.S. infants. N Engl J Med. 2014;370(6):503–12. Influenza safety study that illustrates challenges of dynamic data in Sentinel Distributed Database

    CAS  Article  PubMed  Google Scholar 

  64. 64.

    Yih WK, Greene SK, Zichittella L, Kulldorff M, Baker MA, de Jong JLO, et al. Evaluation of the risk of venous thromboembolism after quadrivalent human papillomavirus vaccination among US females. Vaccine. 2016;34(1):172–8.

    Article  PubMed  Google Scholar 

  65. 65.

    Yih WK, Kulldorff M, Sandhu SK, Zichittella L, Maro JC, Cole DV, et al. Prospective influenza vaccine safety surveillance using fresh data in the Sentinel System. Pharmacoepidemiol Drug Saf. 2016;25(5):481–92.

    Article  PubMed  Google Scholar 

  66. 66.

    Mini-Sentinel PROMPT Self Control Design Tool Technical Users Guide.pdf [Internet]. [cited 2017 Jun 29]. Available from:

  67. 67.

    Sequential Analysis of Gardasil 9 Safety Surveillance Plan | Sentinel System [Internet]. [cited 2017 Jun 15]. Available from:

  68. 68.

    Ninth Annual Sentinel Initiative Public Workshop | Margolis Center for Health Policy. Slides 216–230. [Internet]. [cited 2017 Jun 16]. Available from:

  69. 69.

    Greene SK, Kulldorff M, Yin R, Yih WK, Lieu TA, Weintraub ES, et al. Near real-time vaccine safety surveillance with partially accrued data. Pharmacoepidemiol Drug Saf. 2011;20(6):583–90.

    Article  PubMed  Google Scholar 

  70. 70.

    Wu Y, Jiang X, Kim J, Ohno-Machado L. Grid Binary LOgistic REgression (GLORE): building shared models without sharing data. J Am Med Inform Assoc JAMIA. 2012;19(5):758–64.

    Article  PubMed  Google Scholar 

  71. 71.

    Analysis of Integrated Data without Data Integration. Chance. 2004;17(3):26–9.

    Article  Google Scholar 

Download references


Mini-Sentinel is a pilot project sponsored by the US Food and Drug Administration (FDA) to inform and facilitate development of a fully operational active surveillance system, the Sentinel System, for monitoring the safety of FDA-regulated medical products. Mini-Sentinel is one piece of the Sentinel Initiative, a multi-faceted effort by the FDA to develop a national electronic system that will complement existing methods of safety surveillance. Mini-Sentinel Collaborators include Data and Academic Partners that provide access to health care data and ongoing scientific, technical, methodological, and organizational expertise. The Mini-Sentinel Coordinating Center is funded by the FDA through the Department of Health and Human Services (HHS) Contract number HHSF223200910006I. This project was also supported in part by NIH U01 EB023683.

Author information



Corresponding author

Correspondence to John G. Connolly.

Ethics declarations

Conflict of Interest

John G. Connolly, Catherine A. Panozzo, Noelle Cocoros, Meijia Zhou, and Judith C. Maro each declare no potential conflicts of interest.

Sengwee Toh reports grants from US Food and Drug Administration during the conduct of the study.

Shirley V. Wang reports grants from Sentinel Initiative, during the conduct of the study; personal fees from Aetion, Inc., grants from Novartis, grants from Agency for Healthcare Research and Quality outside the submitted work.

Joshua J. Gagne reports grants from US FDA, during the conduct of the study; grants from Novartis Pharmaceuticals Corporation, grants from Eli Lilly and Company, personal fees from Aetion, Inc., personal fees from Optum, Inc., outside the submitted work.

Candace C. Fuller reports grants from US Food and Drug Administration during the conduct of the study.

Human and Animal Rights

All reported studies/experiments with human or animal subjects performed by the authors have been previously published and complied with all applicable ethical standards (including the Helsinki declaration and its amendments, institutional/national research committee standards, and international/national/institutional guidelines).

Additional information

This article is part of the Topical Collection on Pharmacoepidemiology

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Connolly, J.G., Wang, S.V., Fuller, C.C. et al. Development and Application of Two Semi-Automated Tools for Targeted Medical Product Surveillance in a Distributed Data Network. Curr Epidemiol Rep 4, 298–306 (2017).

Download citation


  • Surveillance
  • Pharmacoepidemiology
  • Sentinel
  • Aria
  • Targeted
  • Drug safety