Skip to main content

Advertisement

SpringerLink
Log in

A Simulation-Based Evaluation of Statistical Methods for Hybrid Real-World Control Arms in Clinical Trials

  • Published:
Statistics in Biosciences Aims and scope Submit manuscript

Abstract

Real-world (RW) data have been a source for creating external control arms to evaluate results from randomized controlled trials (RCTs) in rare diseases and scenarios where randomization to a control group is unethical or unfeasible. However, the validity of any decision making based on such comparative results depends heavily on the appropriateness and quality of the control arm data. FDA guidance lists multiple bias-generating concerns with the use of real-world controls arising from data quality and validity issues, which we frame as a data source ignorability assumption under the potential outcome framework. Hybrid control designs, RCTs with a full treatment group and a small underpowered control group supplemented with RW control data, have the potential to address some of these bias concerns. Statistical methods have been proposed for the analysis of hybrid designs and can adjust for potential violations of the data source ignorability assumption. A simulation study is presented to evaluate the operating characteristics of single and hybrid real-world control methods across the bias-generating scenarios mentioned in FDA guidance. Results suggest that certain methods can adjust for potential biases under these scenarios but may result in reduced efficiency through larger standard errors, or type I error inflation. Implications for the use of such methods and suggestions for additional work are discussed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Food and Drug Administration (2019) Demonstrating substantial evidence of effectiveness for human drug and biological products. https://www.fda.gov/media/133660/download. Accessed 17 Feb 2021

  2. Silverman B (2018) A baker’s dozen of US FDA efficacy approvals using real world evidence. Pharma Intelligence Pink Sheet

  3. Food and Drug Administration (2019) Rare diseases: Natural history studies for drug development. https://www.fda.gov/media/122425/download. Accessed 17 Feb 2021

  4. Food and Drug Administration (2014) Blinatumomab drug approval package. https://www.accessdata.fda.gov/drugsatfda_docs/nda/2014/125557Orig1s000MedRedt.pdf. Accessed 17 Feb 2021

  5. Food and Drug Administration (2016) Avelumab drug approval package. https://www.accessdata.fda.gov/drugsatfda_docs/nda/2017/761049orig1s000multidiscipliner.pdf. Accessed 17 Feb 2021

  6. Odogwu L, Mathieu L, Blumenthal G, Larkins E, Goldberg KB, Griffin N, Bijwaard K, Lee EY, Philip R, Jiang X, Rodriguez L, McKee AE, Keegan P, Pazdur R (2018) Fda approval summary: Dabrafenib and trametinib for the treatment of metastatic non-small cell lung cancers harboring braf v600e mutations. Oncologist 23(6):740–745. https://doi.org/10.1634/theoncologist.2017-0642

    Article  Google Scholar 

  7. Food and Drug Administration (2019) Capmatinib drug approval package. https://www.accessdata.fda.gov/drugsatfda_docs/nda/2020/213591Orig1s000MultidisciplineR.pdf. Accessed 17 Feb 2021

  8. Food and Drug Administration (2017) Axicabtagene ciloleuce drug approval package

  9. Food and Drug Administration (2001) E 10 choice of control group and related issues in clinical trials. https://www.fda.gov/media/71349/download. Accessed 17 Feb 2021

  10. Davies J, Martinec M, Delmar P, Coudert M, Bordogna W, Golding S, Martina R, Crane G (2018) Comparative effectiveness from a single-arm trial and real-world data: alectinib versus ceritinib. J Comp Effect Res 7(9):855–865. https://doi.org/10.2217/cer-2018-0032

    Article  Google Scholar 

  11. Bartlett CH, Mardekian J, Cotter MJ, Huang X, Zhang Z, Parrinello CM, Bourla AB (2020) Concordance of real-world versus conventional progression-free survival from a phase 3 trial of endocrine therapy as first-line treatment for metastatic breast cancer. PLoS ONE 15(4):1–16. https://doi.org/10.1371/journal.pone.0227256

    Article  Google Scholar 

  12. Carrigan G, Whipple S, Capra WB, Taylor MD, Brown JS, Lu M, Arnieri B, Copping R, Rothman KJ (2020) Using electronic health records to derive control arms for early phase single-arm lung cancer trials: proof-of-concept in randomized controlled trials. Clin Pharmacol Ther 107(2):369–377. https://doi.org/10.1002/cpt.1586

    Article  Google Scholar 

  13. Ford I, Norrie J (2016) Pragmatic trials. N Engl J Med 375(5):454–463. https://doi.org/10.1056/NEJMra1510059

    Article  Google Scholar 

  14. Zhu M, Sridhar S, Hollingsworth R, Chit A, Kimball T, Murmello K, Greenberg M, Gurunathan S, Chen J (2020) Hybrid clinical trials to generate real-world evidence: design considerations from a sponsor’s perspective. Contemp Clin Trials 94:105856. https://doi.org/10.1016/j.cct.2019.105856

    Article  Google Scholar 

  15. Viele K, Berry S, Neuenschwander B, Amzal B, Chen F, Enas N, Hobbs B, Ibrahim JG, Kinnersley N, Lindborg S, Micallef S, Roychoudhury S, Thompson L (2014) Use of historical control data for assessing treatment effects in clinical trials. Pharm Stat 13(1):41–54

    Article  Google Scholar 

  16. Wadsworth I, Hampson LV, Jaki T (2018) Extrapolation of efficacy and other data to support the development of new medicines for children: a systematic review of methods. Stat Methods Med Res 27(2):398–413. https://doi.org/10.1177/0962280216631359

    Article  MathSciNet  Google Scholar 

  17. Pocock SJ (1976) The combination of randomized and historical controls in clinical trials. J Chronic Dis 29(3):175–188

    Article  Google Scholar 

  18. Stuart EA, Rubin DB (2008) Matching with multiple control groups with adjustment for group differences. J Educ Behav Stat 33(3):279–306. https://doi.org/10.3102/1076998607306078

    Article  Google Scholar 

  19. Chen MH, Ibrahim JG (2000) Power prior distributions for regression models. Stat Sci 15(1):46–60. https://doi.org/10.1214/ss/1009212673

    Article  MathSciNet  Google Scholar 

  20. Duan Y, Ye K, Smith EP (2006) Evaluating water quality using power priors to incorporate historical information. Environmetrics 17(1):95–106. https://doi.org/10.1002/env.752

    Article  MathSciNet  Google Scholar 

  21. Neuenschwander B, Branson M, Spiegelhalter DJ (2009) A note on the power prior. Stat Med 28(28):3562–3566. https://doi.org/10.1002/sim.3722

    Article  MathSciNet  Google Scholar 

  22. Neuenschwander B, Capkun-Niggli G, Branson M, Spiegelhalter DJ (2010) Summarizing historical information on controls in clinical trials. Clin Trials 7(1):5–18. https://doi.org/10.1177/1740774509356002

    Article  Google Scholar 

  23. Hobbs BP, Carlin BP, Mandrekar SJ, Sargent DJ (2011) Hierarchical commensurate and power prior models for adaptive incorporation of historical information in clinical trials. Biometrics 67(3):1047–1056. https://doi.org/10.1111/j.1541-0420.2011.01564.x

    Article  MathSciNet  MATH  Google Scholar 

  24. Gamalo-Siebers M, Savic J, Basu C, Zhao X, Gopalakrishnan M, Gao A, Song G, Baygani S, Thompson L, Xia HA, Price K, Tiwari R, Carlin BP (2017) Statistical modeling for Bayesian extrapolation of adult clinical trial information in pediatric drug evaluation. Pharm Stat 16(4):232–249

    Article  Google Scholar 

  25. Dejardin D, Delmar P, Warne C, Patel K, van Rosmalen J, Lesaffre E (2018) Use of a historical control group in a noninferiority trial assessing a new antibacterial treatment: a case study and discussion of practical implementation aspects. Pharm Stat 17(2):169–181. https://doi.org/10.1002/pst.1843

    Article  Google Scholar 

  26. van Rosmalen J, Dejardin D, van Norden Y, Löwenberg B, Lesaffre E (2018) Including historical data in the analysis of clinical trials: Is it worth the effort? Stat Methods Med Res 27(10):3167–3182

    Article  MathSciNet  Google Scholar 

  27. Rubin D (1974) Estimating causal effects of treatments in randomized and nonrandomized studies. J Educ Psychol 66(5):688–701

    Article  Google Scholar 

  28. Rubin DB (1980) Randomization analysis of experimental data: the fisher randomization test comment. J Am Stat Assoc 75(371):591–593, http://www.jstor.org/stable/2287653

  29. Rubin DB (1986) Statistics and causal inference: comment: which ifs have causal answers. J Am Stat Assoc 81(396):961–962, http://www.jstor.org/stable/2289065

  30. Imbens GW, Rubin DB (2015) Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction. Cambridge University Press, Cambridge. https://doi.org/10.1017/CBO9781139025751

    Book  MATH  Google Scholar 

  31. Rosenbaum PR, Rubin DB (1983) The central role of the propensity score in observational studies for causal effects. Biometrika 70(1):41–55

    Article  MathSciNet  Google Scholar 

  32. Stuart EA (2010) Matching methods for causal inference: a review and a look forward. Stat Sci 25(1):1–21. https://doi.org/10.1214/09-STS313

    Article  MathSciNet  MATH  Google Scholar 

  33. Rubin DB (1973) Matching to remove bias in observational studies. Biometrics 29(1):159–183

    Article  Google Scholar 

  34. Rosenbaum PR (2002) Observational Studies, 2nd edn. Springer, New York

    Book  Google Scholar 

  35. Rubin DB (1987) Multiple Imputation for Nonresponse in Surveys. Wiley, New York

    Book  Google Scholar 

  36. Ibrahim JG, Chen MH (2000) Power prior distributions for regression models. Statist Sci 15(1):46–60. https://doi.org/10.1214/ss/1009212673

    Article  MathSciNet  Google Scholar 

  37. Franklin JM, Schneeweiss S, Polinski JM, Rassen JA (2014) Plasmode simulation for the evaluation of pharmacoepidemiologic methods in complex healthcare databases. Comput Stat Data Anal 72:219–226

    Article  MathSciNet  Google Scholar 

  38. Fialkowski AC (2018) SimMultiCorrData: simulation of correlated data with multiple variable types. https://CRAN.R-project.org/package=SimMultiCorrData, r package version 0.2.2

  39. Burcu M, Dreyer NA, Franklin JM, Blum MD, Critchlow CW, Perfetto EM, Zhou W (2020) Real-world evidence to support regulatory decision-making for medicines: considerations for external control arms. Pharmacoepidemiol Drug Saf 29(10):1228–1235. https://doi.org/10.1002/pds.4975

    Article  Google Scholar 

  40. Psioda MA, Ibrahim JG (2018) Bayesian clinical trial design using historical data that inform the treatment effect. Biostatistics 20(3):400–415

    Article  MathSciNet  Google Scholar 

  41. Kopp-Schneider A, Calderazzo S, Wiesenfarth M (2020) Power gains by using external information in clinical trials are typically not possible when requiring strict type i error control. Biom J 62(2):361–374

    Article  MathSciNet  Google Scholar 

  42. Normington J, Zhu J, Mattiello F, Sarkar S, Carlin B (2020) An efficient Bayesian platform trial design for borrowing adaptively from historical control data in lymphoma. Contemp Clin Trials 89:105890. https://doi.org/10.1016/j.cct.2019.105890

    Article  Google Scholar 

  43. Robins JM, Rotnitzky A, Zhao LP (1994) Estimation of regression coefficients when some regressors are not always observed. J Am Stat Assoc 89(427):846–866

    Article  MathSciNet  Google Scholar 

  44. Cole SR, Hernán MA (2008) Constructing inverse probability weights for marginal structural models. Am J Epidemiol 168(6):656–664

    Article  Google Scholar 

  45. Lee BK, Lessler J, Stuart EA (2011) Weight trimming and propensity score weighting. PLoS ONE 6:1–6. https://doi.org/10.1371/journal.pone.0018174

    Article  Google Scholar 

  46. Austin PC, Stuart EA (2015) Moving towards best practice when using inverse probability of treatment weighting (IPTW) using the propensity score to estimate causal treatment effects in observational studies. Stat Med 34(28):3661–3679. https://doi.org/10.1002/sim.6607

    Article  MathSciNet  Google Scholar 

  47. Lin J, Gamalo-Siebers M, Tiwari R (2019) Propensity-score-based priors for Bayesian augmented control design. Pharm Stat 18(2):223–238. https://doi.org/10.1002/pst.1918

    Article  Google Scholar 

  48. Wang C, Li H, Chen WC, Lu N, Tiwari R, Xu Y, Yue LQ (2019) Propensity score-integrated power prior approach for incorporating real-world evidence in single-arm clinical studies. J Biopharm Stat 29(5):731–748. https://doi.org/10.1080/10543406.2019.1657133

    Article  Google Scholar 

  49. Wang C, Lu N, Chen WC, Li H, Tiwari R, Xu Y, Yue LQ (2020) Propensity score-integrated composite likelihood approach for incorporating real-world evidence in single-arm clinical studies. J Biopharm Stat 30(3):495–507. https://doi.org/10.1080/10543406.2019.1684309

    Article  Google Scholar 

  50. Liu M, Bunn V, Hupf B, Lin J, Lin J (2021) Propensity-score-based meta-analytic predictive prior for incorporating real-world and historical data. Stat Med. https://doi.org/10.1002/sim.9095

    Article  MathSciNet  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Eli Lilly and Company, Real World Analytics, Indianapolis, IN, USA

    Mingyang Shan, Douglas Faries, Andy Dang & Zhanglin Cui

  2. CSL Behring, Quantitative Clinical Sciences and Reporting, King of Prussia, PA, USA

    Xiang Zhang

  3. Eli Lilly and Company, Global Patient Outcomes and Real World Evidence, Indianapolis, IN, USA

    Kristin M. Sheffield

Authors
  1. Mingyang Shan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Douglas Faries
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Andy Dang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xiang Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zhanglin Cui
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Kristin M. Sheffield
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mingyang Shan.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Shan, M., Faries, D., Dang, A. et al. A Simulation-Based Evaluation of Statistical Methods for Hybrid Real-World Control Arms in Clinical Trials. Stat Biosci 14, 259–284 (2022). https://doi.org/10.1007/s12561-022-09334-w

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12561-022-09334-w

Keywords

Search

Navigation