Skip to main content
SpringerLink
Log in

Designing, Conducting, Monitoring, and Analyzing Data from Pragmatic Randomized Clinical Trials: Proceedings from a Multi-stakeholder Think Tank Meeting

  • Original Research
  • Published:
Therapeutic Innovation & Regulatory Science Aims and scope Submit manuscript

Abstract

In late 2018, the Food and Drug Administration (FDA) outlined a framework for evaluating the possible use of real-world evidence (RWE) to support regulatory decision-making. This framework was created to facilitate studies that would generate high-quality RWE, including pragmatic clinical trials (PCTs), which are randomized trials designed to inform clinical or policy decisions by assessing the real-world effectiveness of an intervention. There is general agreement among experts that the use of existing healthcare and patient-generated data holds promise for making randomized trials more efficient, less costly, and more generalizable. Yet the benefits of relying on real-world data sources must be weighed against difficulties with ensuring data integrity and completeness. Additionally, appropriately monitoring patient safety in randomized trials of new drugs using healthcare system data that might not be available in real time can be quite difficult. Recognizing that these and other concerns are critical to the development and acceptability of PCTs, a group of stakeholders from academia, industry, professional organizations, regulatory bodies, government agencies, and patient advocates discussed a path forward for PCT growth and sustainability at a think tank meeting entitled “Monitoring and Analyzing Data from Pragmatic Streamlined Randomized Clinical Trials,” which took place in January 2019 (Washington, DC). The goals of this meeting were to: (1) evaluate study design and methodological options specific to PCTs that have the potential to yield high-quality evidence; (2) discuss best practices to ensure data quality in PCTs; and (3) identify appropriate methods for study monitoring. Proceedings from the think tank meeting are summarized in this manuscript.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Figure 1.
Figure 2.
Figure 3.
Figure 4.

Similar content being viewed by others

References

  1. Reinsel D, Gantz J, Rydning J. International Data Corporation. Data Age 2025. The Digitization of the World: From Edge to Core. SEAGATE web site. https://www.seagate.com/files/www-content/our-story/trends/files/idc-seagate-dataage-whitepaper.pdf. Published Nov 2018. Accessed 31 Jul 2019.

  2. U.S. Food & Drug Administration (FDA). Framework for FDA’s Real-World Evidence Program. FDA web site. https://www.fda.gov/media/120060/download. Published Dec 2018. Accessed 31 Jul 2019.

  3. Sherman RE, Anderson SA, Dal Pan GJ, et al. Real-world evidence—what is it and what can it tell us? N Engl J Med. 2016;375:2293–7.

    Article  Google Scholar 

  4. First International Study of Infarct Survival Collaborative Group. Randomised trial of intravenous atenolol among 16 027 cases of suspected acute myocardial infarction: ISIS-1. Lancet. 1986;2:57–66.

    Google Scholar 

  5. ISIS-2 (Second International Study of Infarct Survival) Collaborative Group. Randomised trial of intravenous streptokinase, oral aspirin, both, or neither among 17,187 cases of suspected acute myocardial infarction: ISIS-2. Lancet. 1988;2:349–60.

    Google Scholar 

  6. ISIS-3 (Third International Study of Infarct Survival) Collaborative Group. ISIS-3: a randomised comparison of streptokinase vs tissue plasminogen activator vs anistreplase and of aspirin plus heparin vs aspirin alone among 41,299 cases of suspected acute myocardial infarction. Lancet. 1992;339:753–70.

    Article  Google Scholar 

  7. ISIS-4 (Fourth International Study of Infarct Survival) Collaborative Group. ISIS-4: a randomised factorial trial assessing early oral captopril, oral mononitrate, and intravenous magnesium sulphate in 58,050 patients with suspected acute myocardial infarction. Lancet. 1995;345:669–85.

    Article  Google Scholar 

  8. Gökbuget N, Kelsh M, Chia V, et al. Blinatumomab vs historical standard therapy of adult relapsed/refractory acute lymphoblastic leukemia. Blood Cancer J. 2016;6:e473.

    Article  Google Scholar 

  9. Daniel GW, Menis M, Sridhar G, et al. Immune globulins and thrombotic adverse events as recorded in a large administrative database in 2008 through 2010. Transfusion. 2012;52:2113–211.

    Article  CAS  Google Scholar 

  10. Jarow JP, LaVange L, Woodcock J. Multidimensional evidence generation and FDA regulatory decision making: defining and using “real-world” data. JAMA. 2017;318:703–4.

    Article  Google Scholar 

  11. Gabay M. 21st Century Cures Act. Hosp Pharm. 2017;52:264–5.

    Article  Google Scholar 

  12. Ford I, Norrie J. Pragmatic trials. N Engl J Med. 2016;375:454–63.

    Article  Google Scholar 

  13. Schwartz D, Lellouch J. Explanatory and pragmatic attitudes in therapeutical trials. J Clin Epidemiol. 2009;62:499–505.

    Article  Google Scholar 

  14. Stafford PB, Garrett A. Using real-time data to drive better decisions, fater. Drug Inform. J. 2011;45:495–502.

    Article  Google Scholar 

  15. Aroda VR, Saugstrup T, Buse JB, Donsmark M, Zacho J, Davies MJ. Incorporating and interpreting regulatory guidance on estimands in diabetes clinical trials: the PIONEER 1 randomized clinical trial as an example. Diabet Obes Metab. 2019;21(10):2203–10.

    Article  Google Scholar 

  16. Loudon K, Treweek S, Sullivan F, et al. The PRECIS-2 tool: designing trials that are fit for purpose. BMJ. 2015;350:h2147.

    Article  Google Scholar 

  17. Johnson KE, Neta G, Dember LM, et al. Use of PRECIS ratings in the National Institutes of Health (NIH) Health Care Systems Research Collaboratory. Trials. 2016;17:32.

    Article  Google Scholar 

  18. Loudon K, Zwarenstein M, Sullivan F, et al. Making clinical trials more relevant: improving and validating the PRECIS tool for matching trial design decisions to trial purpose. Trials. 2013;14:115.

    Article  Google Scholar 

  19. North American Symptomatic Carotid Endarterectomy Trial Collaborators, Barnett HJM, Taylor DW, et al. Beneficial effect of carotid endarterectomy in symptomatic patients with high-grade carotid stenosis. N Engl J Med. 1991;325:445–53.

    Article  Google Scholar 

  20. Jarvik JG, Comstock BA, James KT, et al. Lumbar imaging with reporting of epidemiology (LIRE): protocol for a pragmatic cluster randomized trial. Contemp Clin Trials. 2015;45:157–63.

    Article  Google Scholar 

  21. Thorpe KE, Zwarenstein M, Oxman AD, et al. A pragmatic-explanatory continuum indicator summary (PRECIS): a tool to help trial designers. CMAJ. 2009;180:E47–57.

    Article  Google Scholar 

  22. Welsing PM, Oude Rengerink K, Collier S, et al. Series: pragmatic trials and real world evidence: paper 6. Outcome measures in the real world. J Clin Epidemiol. 2017;90:99–107.

    Article  Google Scholar 

  23. Christian JB, Brouwer ES, Girman CJ, Dennett D, Davis KJ, Dreyer NA. Masking in pragmatic trials: who, what, and when to blind. Ther Innov Regul Sci. 2020;54:431–6.

    Article  Google Scholar 

  24. Turner EL, Li F, Gallis JA, et al. Review of recent methodological developments in group-randomized trials: part 1-design. Am J Public Health. 2017;107:907–15.

    Article  Google Scholar 

  25. Donner A, Klar N. Pitfalls of and controversies in cluster randomization trials. Am J Public Health. 2004;94:416–22.

    Article  Google Scholar 

  26. Turner EL, Prague M, Gallis JA, et al. Review of recent methodological developments in group-randomized trials: part 2-analysis. Am J Public Health. 2017;107:1078–86.

    Article  Google Scholar 

  27. Hughes JP, Granston TS, Heagerty PJ. Current issues in the design and analysis of stepped wedge trials. Contemp Clin Trials. 2015;45:55–60.

    Article  Google Scholar 

  28. Cook AJ, Delong E, Murray DM, et al. Statistical lessons learned for designing cluster randomized pragmatic clinical trials from the NIH Health Care Systems Collaboratory Biostatistics and Design Core. Clin Trials. 2016;13:504–12.

    Article  Google Scholar 

  29. Ellenberg SS. The stepped-wedge clinical trial: evaluation by rolling deployment. JAMA. 2018;319:607–8.

    Article  Google Scholar 

  30. McKinney RE, Beskow LM, Ford DE, et al. Use of altered informed consent in pragmatic clinical research. Clin Trials Lond Engl. 2015;12:494–502.

    Article  Google Scholar 

  31. U.S. Food & Drug Administration (FDA). Use of Electronic Informed Consent in Clinical Investigations—Questions and Answers | FDA. FDA web site. https://www.fda.gov/regulatory-information/search-fda-guidance-documents/use-electronic-informed-consent-clinical-investigations-questions-and-answers. Updated 24 Aug 2018. Accessed 21 May 2019.

  32. Aspirin Dosing: A Patient-Centric Trial Assessing Benefits and Long-term (ADAPTABLE). ClinicalTrials.gov web site. https://clinicaltrials.gov/ct2/show/NCT02697916. Updated 31 Jan 2019. Accessed 31 Jul 2019.

  33. Johnston A, Jones WS, Hernandez AF. The ADAPTABLE trial and aspirin dosing in secondary prevention for patients with coronary artery disease. Curr Cardiol Rep. 2016;18:81.

    Article  Google Scholar 

  34. ADAPTABLE prescreen video for patients. ADAPTABLE patient web site. https://adaptablepatient.com/en/prescreen/watch-video. Accessed July 31, 2019.

  35. Mahendraratnam N, Silcox C, Mercon K, et al. Determining real-world data’s fitness for use and the role of reliability. Duke-Margolis Center for Health Policy; September 26, 2019.

  36. Mayernik M, DiLauro T, Duerr R, et al. Data conservancy provenance, context, and lineage services: key components for data preservation and curation. Data Sci J. 2013;12:158–71.

    Article  Google Scholar 

  37. El Fadly A, Rance B, Lucas N, et al. Integrating clinical research with the Healthcare Enterprise: from the RE-USE project to the EHR4CR platform. J Biomed Inform. 2011;44:S94–S102.

    Article  Google Scholar 

  38. Nordo AH, Eisenstein EL, Hawley J, et al. A comparative effectiveness study of eSource used for data capture for a clinical research registry. Int J Med Inf. 2017;103:89–94.

    Article  Google Scholar 

  39. Nordo AH, Levaux HP, Becnel LB, et al. Use of EHRs data for clinical research: historical progress and current applications. Learn Health Syst. 2019;3:e10076.

    Article  Google Scholar 

  40. Guimarães PO, Krishnamoorthy A, Kaltenbach LA, et al. Accuracy of medical claims for identifying cardiovascular and bleeding events after myocardial infarction: a secondary analysis of the TRANSLATE-ACS study. JAMA Cardiol. 2017;2:750–7.

    Article  Google Scholar 

  41. Phelan M, Bhavsar NA, Goldstein BA. Illustrating informed presence bias in electronic health records data: how patient interactions with a health system can impact inference. EGEMS Wash DC. 2017;5:22.

    PubMed  PubMed Central  Google Scholar 

  42. Ahmad T, Lund LH, Rao P, et al. Machine learning methods improve prognostication, identify clinically distinct phenotypes, and detect heterogeneity in response to therapy in a large cohort of heart failure patients. J Am Heart Assoc. 2018;7:10.

    Google Scholar 

  43. Tudor Smith C, Stocken DD, Dunn J, et al. The value of source data verification in a cancer clinical trial. PLoS ONE. 2012;7:e51623.

    Article  Google Scholar 

  44. Anderson JR, Byrjalsen I, Bihlet A, et al. Impact of source data verification on data quality in clinical trials: an empirical post hoc analysis of three phase 3 randomized clinical trials. Br J Clin Pharmacol. 2015;79:660–8.

    Article  Google Scholar 

  45. Olsen R, Bihlet AR, Kalakou F, et al. The impact of clinical trial monitoring approaches on data integrity and cost: a review of current literature. Eur J Clin Pharmacol. 2016;72:399–412.

    Article  Google Scholar 

  46. Ellenberg SS, Culbertson R, Gillen DL, et al. Data monitoring committees for pragmatic clinical trials. Clin Trials Lond Engl. 2015;12:530–6.

    Article  Google Scholar 

Download references

Acknowledgements

The authors would like to thank Jennifer Gloc for her leadership and guidance in this manuscript effort, and Erin Campbell for her editorial contributions. Ms. Gloc and Ms. Campbell did not receive compensation for their contributions, apart from their employment at Duke Clinical Research Institute (Durham, NC USA).

Funding

This manuscript was funded internally by the Duke Clinical Research Institute (Durham, North Carolina). Funding support for the think tank meeting was provided through registration fees from AbbVie, Amgen, AstraZeneca, Bayer AG, Boehringer Ingelheim International GmbH, Eli Lilly and Company, Janssen Pharmaceutical Companies of Johnson & Johnson, Pfizer, and Sanofi.

Author information

Authors and Affiliations

  1. Duke Clinical Research Institute, 200 Morris Street, Durham, NC, 27701, USA

    Trevor A. Lentz PT, PhD, MPH, Lesley H. Curtis PhD, Frank W. Rockhold PhD, Carolyn Arias MPH, Adrian F. Hernandez MD, Matthew Roe MD & Myles Wolf MD, MMSc

  2. Food and Drug Administration, Center for Drug Evaluation and Research, Rockville, MD, USA

    David Martin MD, MPH

  3. AstraZeneca R&D, Gothenburg, Sweden

    Tomas L. G. Andersson MD, PhD

  4. Johnson & Johnson, Titusville, NJ, USA

    Jesse A. Berlin ScD & Joanne Waldstreicher MD

  5. Multiple Sclerosis Patient-Powered Research Network, Wakefield, RI, USA

    Cherie Binns RN, BS, MSCN

  6. Kaiser Permanente Washington Health Research Institute, Seattle, WA, USA

    Andrea Cook PhD

  7. HealthCore, Inc., Durham, NC, USA

    Mark Cziraky PharM

  8. Amgen Inc., Thousand Oaks, CA, USA

    Ricardo Dent MD, Thomas Hucko MD & Keri Monda PhD

  9. Stanford University, Stanford, CA, USA

    Manisha Desai MPH

  10. Pfizer Inc., New York, NY, USA

    Andrew Emmett & Jack Mardekian PhD

  11. Yale University, New Haven, CT, USA

    Denise Esserman PhD

  12. Boehringer Ingelheim International GmbH, Rhein, Germany

    Jyothis George PhD

  13. Boehringer Ingelheim Pharma GmbH & Co. KG, Biberach, Germany

    Stefan Hantel PhD

  14. University of Washington, Seattle, WA, USA

    Patrick Heagerty PhD

  15. AstraZeneca, Wilmington, DE, USA

    Naeem Khan MD

  16. Population Health Research Institute, Hamilton, ON, Canada

    Shun Fu Lee PhD

  17. Sanofi, Bridgewater, NJ, USA

    Robert LoCasale PhD & Lin Wang PhD

  18. RTP Consulting, Chapel Hill, NC, USA

    Debbe McCall MBA

  19. Harvard T. H. Chan School of Public Health, Medical School, Boston, MA, USA

    Sharon-Lise Normand PhD

  20. Eli Lilly and Company, Indianapolis, IN, USA

    Jeffrey Riesmeyer MD & Govinda Weerakkody PhD

  21. Bayer AG, Bergkamen, Germany

    Lothar Roessig MD & Harald Siedentop PhD

  22. Abbvie, Chicago, IL, USA

    Rob Scott MD

  23. University of Pennsylvania, Philadelphia, PA, USA

    Susan S. Ellenberg PhD

Authors
  1. Trevor A. Lentz PT, PhD, MPH
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lesley H. Curtis PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Frank W. Rockhold PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. David Martin MD, MPH
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Tomas L. G. Andersson MD, PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Carolyn Arias MPH
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Jesse A. Berlin ScD
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Cherie Binns RN, BS, MSCN
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Andrea Cook PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Mark Cziraky PharM
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Ricardo Dent MD
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Manisha Desai MPH
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Andrew Emmett
    View author publications

    You can also search for this author in PubMed Google Scholar

  14. Denise Esserman PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

  15. Jyothis George PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

  16. Stefan Hantel PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

  17. Patrick Heagerty PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

  18. Adrian F. Hernandez MD
    View author publications

    You can also search for this author in PubMed Google Scholar

  19. Thomas Hucko MD
    View author publications

    You can also search for this author in PubMed Google Scholar

  20. Naeem Khan MD
    View author publications

    You can also search for this author in PubMed Google Scholar

  21. Shun Fu Lee PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

  22. Robert LoCasale PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

  23. Jack Mardekian PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

  24. Debbe McCall MBA
    View author publications

    You can also search for this author in PubMed Google Scholar

  25. Keri Monda PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

  26. Sharon-Lise Normand PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

  27. Jeffrey Riesmeyer MD
    View author publications

    You can also search for this author in PubMed Google Scholar

  28. Matthew Roe MD
    View author publications

    You can also search for this author in PubMed Google Scholar

  29. Lothar Roessig MD
    View author publications

    You can also search for this author in PubMed Google Scholar

  30. Rob Scott MD
    View author publications

    You can also search for this author in PubMed Google Scholar

  31. Harald Siedentop PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

  32. Joanne Waldstreicher MD
    View author publications

    You can also search for this author in PubMed Google Scholar

  33. Lin Wang PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

  34. Govinda Weerakkody PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

  35. Myles Wolf MD, MMSc
    View author publications

    You can also search for this author in PubMed Google Scholar

  36. Susan S. Ellenberg PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Trevor A. Lentz PT, PhD, MPH.

Ethics declarations

Conflict of interest

Trevor A. Lentz reports no relevant disclosures. Lesley H. Curtis reports no relevant disclosures. Frank W. Rockhold reports grants from PCORI, the NIH, AstraZeneca, and Eidos Therapeutics; personal fees from Janssen, Amgen, Merck Healthcare KgaA, Merck Research Laboratories, NovoNordisk, and Eidos Therapeutics; other from GlaxoSmithKline and AthiraovoNordisk. David Martin reports no relevant disclosures. Tomas LG Andersson reports being an employee of AstraZeneca. Carolyn Arias reports no relevant disclosures. Jesse A. Berlin reports being an employee of Johnson & Johnson; holds equity in Johnson & Johnson. Cherie Binns reports no relevant disclosures. Andrea Cook reports no relevant disclosures. Mark Cziraky reports no relevant disclosures, but states even though he did not receive any direct payment, his company, HealthCore, conducts multiple research projects supported by various pharmaceutical companies. Ricardo Dent reports no relevant disclosures. Manisha Desai reports no relevant disclosures. Andrew Emmett reports no relevant disclosures. Denise Esserman reports no relevant disclosures. Jyothis George reports being an employee of Boehringer Ingelheim during the conduct of the study. Stefan Hantel reports being an employee of Boehringer Ingelheim Pharma GmbH & Co. KG, Germany. Patrick Heagerty reports no relevant disclosures. Adrian F. Hernandez reports no relevant disclosures. Thomas Hucko reports no relevant disclosures. Naeem Khan reports being a full-time employee of AstraZeneca. Shun Fu Lee reports no relevant disclosures. Robert LoCasale reports no relevant disclosures. Jack Mardekian reports being an employee of Pfizer Inc.; owning stock in Pfizer Inc. Debbe McCall reports no relevant disclosures. Keri Monda reports no relevant disclosures. Sharon-Lise Normand reports no relevant disclosures. Jeffrey Riesmeyer reports no relevant disclosures. Matthew Roe reports no relevant disclosures. Lothar Roessig reports being a full-time employee of Bayer AG. Rob Scott reports no relevant disclosures. Harald Siedentop has no relevant disclosures. Joanne Waldstreicher reports being an employee and shareholder for Johnson & Johnson; a former employee and former shareholder for Merck & Co. Lin Wang reports no relevant disclosures. Govinda Weerakkody reports no relevant disclosures. Myles Wolf served as a consultant to Akebia, AMAG, Amgen, Ardelyx, DiaDorin, Luitpold, and Pharmacosmos. Susan S. Ellenberg reports no relevant disclosures.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Disclaimer

The views expressed are those of the authors and do not necessarily represent the views of their organization.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 13 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lentz, T.A., Curtis, L.H., Rockhold, F.W. et al. Designing, Conducting, Monitoring, and Analyzing Data from Pragmatic Randomized Clinical Trials: Proceedings from a Multi-stakeholder Think Tank Meeting. Ther Innov Regul Sci 54, 1477–1488 (2020). https://doi.org/10.1007/s43441-020-00175-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s43441-020-00175-7

Keywords

Search

Navigation