Skip to main content
SpringerLink
Log in

Evaluating the Generalisability of Trial Results: Introducing a Centre- and Trial-Level Generalisability Index

  • Original Research Article
  • Published:
PharmacoEconomics Aims and scope Submit manuscript

Abstract

Background

Few randomised controlled trials (RCTs) recruit centres representatively, which may limit the external validity of trial results.

Objective

The aim of this study was to propose a proof-of-concept method of assessing the generalisability of the clinical and cost-effectiveness findings of a given RCT.

Methods

We developed a generalisability index (Gix), informed by centre-level characteristics, as a measure of centre and trial representativeness. The centre-level Gix quantifies how representative a centre is in relation to its jurisdiction, e.g. a country or health authority. The trial-level Gix quantifies how representative trial recruitment is in relation to clinical practice in the jurisdiction. Taking a real-world RCT as a case study and assuming trial-wide results to represent ‘true jurisdiction values’, we used simulation methods to recreate 5000 RCTs and investigate the relationship between trial representativeness, reflected by the standardised trial-Gix, and the deviation of simulated trial results from the ‘true values’.

Results

The simulation study provides evidence that trial results (odds ratio for the primary outcome and incremental quality-adjusted life-years) were influenced by the representativeness of the sample of recruiting centres. Simulated RCTs with the closest results to the ‘true values’ were those whose recruitment closely mirrored the jurisdiction-wide context. Results appeared robust to six alternative specifications of the Gix.

Conclusions

Our findings suggest that an unrepresentative selection of centres limits the external validity of trial results. The Gix may be a valuable tool to help facilitate rational selection of trial centres and ensure the generalisability of results at the jurisdiction level.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Notes

  1. This estimate differs slightly from the published ROSSINI estimate (odds ratio 0.97, 95 % CI 0.69–1.36) because the complete-case dataset (n = 585 patients) was used in this analysis.

References

  1. Black N. Why we need observational studies to evaluate the effectiveness of health care. BMJ. 1996;312(7040):1215–8.

    Article  PubMed Central  CAS  PubMed  Google Scholar 

  2. Rothwell PM. External validity of randomised controlled trials: “to whom do the results of this trial apply?”. Lancet. 2005;365(9453):82–93.

    Article  PubMed  Google Scholar 

  3. Silverman SL. From randomized controlled trials to observational studies. Am J Med. 2009;122(2):114–20.

    Article  PubMed  Google Scholar 

  4. Kravitz RL, Duan N, Braslow J. Evidence-based medicine, heterogeneity of treatment effects, and the trouble with averages. Milbank Q. 2004;82(4):661–87.

    Article  PubMed Central  PubMed  Google Scholar 

  5. Kraemer H, Frank E, Kupfer D. Moderators of treatment outcomes: clinical, research, and policy importance. JAMA. 2006;296(10):1286–9.

    Article  CAS  PubMed  Google Scholar 

  6. O’Brien BJ. A tale of two (or more) cities: geographic transferability of pharmacoeconomic data. Am J Manage Care. 1997;3:S33–9.

    Google Scholar 

  7. Drummond MF, et al. Issues in the cross-national assessment of health technology. Int J Technol Assess Health Care. 1992;8(4):670–82.

    Article  Google Scholar 

  8. Moore DA, et al. How generalizable are the results of large randomized controlled trials of antiretroviral therapy? HIV Med. 2000;1(3):149–54.

    Article  CAS  PubMed  Google Scholar 

  9. Hoel AW, et al. Under-representation of women and ethnic minorities in vascular surgery randomized controlled trials. J Vasc Surg. 2009;50(2):349–54.

    Article  PubMed Central  PubMed  Google Scholar 

  10. Kalata P, et al. Differences between clinical trial participants and patients in a population-based registry: the German Rectal Cancer Study vs. the Rostock Cancer Registry. Dis Colon Rectum. 2009;52(3):425–37.

    Article  PubMed  Google Scholar 

  11. Maasland L, et al. Patients enrolled in large randomized clinical trials of antiplatelet treatment for prevention after transient ischemic attack or ischemic stroke are not representative of patients in clinical practice: the Netherlands Stroke Survey. Stroke. 2009;40(8):2662–8.

    Article  CAS  PubMed  Google Scholar 

  12. Costa DJ, et al. How representative are clinical study patients with allergic rhinitis in primary care? J Allergy Clin Immunol. 2011;127(4):920–6.

    Article  PubMed  Google Scholar 

  13. Falagas ME, et al. Patients included in randomised controlled trials do not represent those seen in clinical practice: focus on antimicrobial agents. Int J Antimicrob Agents. 2010;36(1):1–13.

    Article  CAS  PubMed  Google Scholar 

  14. Bartlett C, et al. The causes and effects of socio-demographic exclusions from clinical trials. Health Technol Assess. 2005;9(38):1–152.

    Article  Google Scholar 

  15. Halm EA, Lee C, Chassin MR. Is volume related to outcome in health care? a systematic review and methodologic critique of the literature. Ann Intern Med. 2002;137(6):511–20.

    Article  PubMed  Google Scholar 

  16. Devereaux PJ, et al. Need for expertise based randomised controlled trials. BMJ. 2005;330(7482):88.

    Article  PubMed Central  CAS  PubMed  Google Scholar 

  17. Gheorghe A, et al. Centre selection for clinical trials and the generalisability of results: a mixed methods study. PLoS One. 2013;8(2):e56560.

    Article  PubMed Central  CAS  PubMed  Google Scholar 

  18. Gheorghe A, et al. Rational centre selection for RCTs with a parallel economic evaluation: the next step towards increased generalisability? Health Economics. 2014;24(4):498–504.

    Article  PubMed  Google Scholar 

  19. Sculpher MJ, et al. Generalisability in economic evaluation studies in healthcare: a review and case studies. Health Technol Assess. 2004;8(49):iii–iv, 1–192.

  20. Goeree R, et al. Transferability of economic evaluations: approaches and factors to consider when using results from one geographic area for another. Curr Med Res Opin. 2007;23(4):671–82.

    Article  PubMed  Google Scholar 

  21. Wei JW, et al. Variations and determinants of hospital costs for acute stroke in China. PLoS One. 2010;5(9):e13041.

    Article  PubMed Central  PubMed  Google Scholar 

  22. Saleh SS, Racz M, Hannan E. The effect of preoperative and hospital characteristics on costs for coronary artery bypass graft. Ann Surg. 2009;249(2):335–41.

    Article  PubMed  Google Scholar 

  23. Becker ER. National trends and determinants of hospitalization costs and lengths-of-stay for uterine fibroids procedures. J Health Care Finance. 2007;33(3):1–16.

    PubMed  Google Scholar 

  24. Lee KH, Yang SB, Choi M. The association between hospital ownership and technical efficiency in a managed care environment. J Med Syst. 2009;33(4):307–15.

    Article  PubMed  Google Scholar 

  25. Chaikledkaew U, et al. Factors affecting health-care costs and hospitalizations among diabetic patients in Thai public hospitals. Value Health. 2008;11(Suppl 1):S69–74.

    Article  PubMed  Google Scholar 

  26. McCollam PL, Lage MJ, Bala M. A comparison of total hospital costs for percutaneous coronary intervention patients receiving abciximab versus tirofiban. Catheter Cardiovasc Interv. 2001;54(2):152–7.

    Article  CAS  PubMed  Google Scholar 

  27. Stock GN, McDermott C. Operational and contextual drivers of hospital costs. J Health Organ Manage. 2011;25(2):142–58.

    Article  Google Scholar 

  28. Kristensen T, et al. Cost variation in diabetes care delivered in English hospitals. Diabet Med. 2010;27(8):949–57.

    Article  CAS  PubMed  Google Scholar 

  29. Laudicella M, Olsen KR, Street A. Examining cost variation across hospital departments: a two-stage multi-level approach using patient-level data. Soc Sci Med. 2010;71(10):1872–81.

    Article  PubMed  Google Scholar 

  30. Daidone S, D’Amico F. Technical efficiency, specialization and ownership form: evidences from a pooling of Italian hospitals. J Product Anal. 2009;32(3):203–16.

    Article  Google Scholar 

  31. Gutacker N, et al. Truly inefficient or providing better quality of care? Analysing the relationship between risk-adjusted hospital costs and patients’ health outcomes. Health Econ. 2013;22(8):931–47.

    Article  PubMed  Google Scholar 

  32. Department of Health. PbR and the market forces factor (MFF) in 2013-14. 2013. Available at: https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/141395/PbR-and-the-MFF-in-2013-14.pdf. Accessed 11 Jul 2013.

  33. Department of Health. NHS reference costs 2011–2012. 2012. Available at: https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/127112/2011-12-reference-costs-publication.pdf. Accessed 11 Jul 2013.

  34. Pinkney TD, et al. Impact of wound edge protection devices on surgical site infection after laparotomy: multicentre randomised controlled trial (ROSSINI Trial). BMJ. 2013;347:f4305.

    Article  PubMed Central  PubMed  Google Scholar 

  35. Efron B, Tibshirani RJ. An introduction to the bootstrap. London: Chapman and Hall; 1993.

    Book  Google Scholar 

  36. National Institute for Health and Care Excellence. Guide to the methods of technology appraisal 2013. London: National Institute for Health and Care Excellence; 2013.

    Google Scholar 

  37. Efron B. Better bootstrap confidence intervals. J Am Stat Assoc. 1987;82(397):171–85.

    Article  Google Scholar 

  38. Briggs A, Wonderling DE, Mooney CZ. Pulling cost-effectiveness analysis up by its bootstraps: a non-parametric approach to confidence interval estimation. Health Econ. 1997;6(4):327–40.

    Article  CAS  PubMed  Google Scholar 

  39. R Development Core Team. R: a language and environment for statistical computing. Vienna: R Foundation for Statistical Computing; 2013.

  40. Gillam SJ, Siriwardena AN, Steel N. Pay-for-performance in the United Kingdom: impact of the quality and outcomes framework: a systematic review. Ann Fam Med. 2012;10(5):461–8.

    Article  PubMed Central  PubMed  Google Scholar 

  41. Majumdar SR, et al. Better outcomes for patients treated at hospitals that participate in clinical trials. Arch Intern Med. 2008;168(6):657–62.

    Article  PubMed  Google Scholar 

  42. Rochon J, du Bois A, Lange T. Mediation analysis of the relationship between institutional research activity and patient survival. BMC Med Res Methodol. 2014;14(1):9.

    Article  PubMed Central  PubMed  Google Scholar 

  43. Burton A, et al. The design of simulation studies in medical statistics. Stat Med. 2006;25(24):4279–92.

    Article  PubMed  Google Scholar 

  44. Deeks JJ, et al. Evaluating non-randomised intervention studies. Health Technol Assess. 2003;7(27):iii–x, 1–173.

  45. Gomes M, et al. Methods for covariate adjustment in cost-effectiveness analysis that use cluster randomised trials. Health Econ. 2012;21(9):1101–18.

    Article  PubMed  Google Scholar 

  46. McCarron CE, et al. Bayesian hierarchical models combining different study types and adjusting for covariate imbalances: a simulation study to assess model performance. PLoS One. 2011;6(10):e25635.

    Article  PubMed Central  CAS  PubMed  Google Scholar 

  47. Health & Social Care Information Centre. Hospital episode statistics, admitted patient care–England 2011–12. (2013). http://www.hscic.gov.uk/searchcatalogue?q=title%3A%22Hospital+Episode+Statistics%2C+Admitted+patient+care+-+England%22&area=&size=10&sort=Relevance. Accessed 29 May 2013.

Download references

Financial disclosure

Financial support for this study was provided in part by a National Institute for Health Research (NIHR) Research Support Facility PhD Studentship for Adrian Gheorghe, and in part by a grant from the MRC Midland Hub for Trials Methodology Research, University of Birmingham, Birmingham, UK (MRC Grant ID G0800808) for Melanie Calvert. The funding agreement ensured the authors’ independence in designing the study, interpreting the data, and writing and publishing the report.

Conflicts of interest

Adrian Gheorghe, Tracy Roberts, Karla Hemming and Melanie Calvert have no conflicts of interest to declare.

Acknowledgments

Adrian Gheorghe conceived the study with guidance and contributions from Melanie Calvert, Tracy Roberts and Karla Hemming. Adrian Gheorghe and Karla Hemming conducted the analysis, and Adrian Gheorghe wrote the first draft. All authors reviewed and contributed to the final version of the manuscript. Adrian Gheorghe is the study guarantor.

The authors are grateful to Thomas Pinkney, Dion Morton and the ROSSINI Trial Investigators for providing access to ROSSINI data. We would also like to thank participants at the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) 18th Annual Meeting, May 2013 (New Orleans, LA, USA), and the Health Economists’ Study Group (HESG) Summer Conference, June 2013 (University of Warwick, Coventry, UK), where earlier versions of this work were presented, as well as two anonymous reviewers for their constructive feedback.

Author information

Authors and Affiliations

  1. Primary Care Clinical Sciences and MRC Midlands Hub for Trials Methodology Research, University of Birmingham, Birmingham, UK

    Adrian Gheorghe & Melanie Calvert

  2. Health Economics Unit, University of Birmingham, Birmingham, UK

    Tracy Roberts

  3. Public Health, Epidemiology and Statistics, University of Birmingham, Birmingham, UK

    Karla Hemming

  4. Department of Global Health and Development, London School of Hygiene and Tropical Medicine, 15–17 Tavistock Place, London, WC1H 9SH, UK

    Adrian Gheorghe

Authors
  1. Adrian Gheorghe
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tracy Roberts
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Karla Hemming
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Melanie Calvert
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Adrian Gheorghe.

Appendices

Appendix 1

See Table 4.

Table 4 Centre-level Gix for ROSSINI centres
Full size table

Appendix 2

See Table 5.

Table 5 Results of the simulated RCTs across categories of standardised trial-Gix
Full size table

Appendix 3

See Table 6 and Figs. 5, 6, 7, 8, 9 and 10.

Table 6 Results of the simulated RCTs across various standardised trial-Gix formulations
Full size table
Fig. 5
figure 5

Clinical and cost-effectiveness estimates in simulated RCTs across categories of standardised trial-Gix3. Horizontal red lines represent point estimates of ROSSINI trial-wide results. Box plots depict first quartile, median, third quartile and extreme ranges from lower/upper quartile up to 1.5 IQR. Gix generalisability index, IQR interquartile range, OR odds ratio, SD standard deviation, QALYs quality-adjusted life-years, RCTs randomized controlled trials, ROSSINI Reduction of Surgical Site Infection using a Novel Intervention

Full size image
Fig. 6
figure 6

Clinical and cost-effectiveness estimates in simulated RCTs across categories of standardised trial-Gix4a. Horizontal red lines represent point estimates of ROSSINI trial-wide results. Box plots depict first quartile, median, third quartile and extreme ranges from lower/upper quartile up to 1.5 IQR. Gix generalisability index, IQR interquartile range, OR odds ratio, SD standard deviation, QALYs quality-adjusted life-years, RCTs randomized controlled trials, ROSSINI Reduction of Surgical Site Infection using a Novel Intervention

Full size image
Fig. 7
figure 7

Clinical and cost-effectiveness estimates in simulated RCTs across categories of standardised trial-Gix4b. Horizontal red lines represent point estimates of ROSSINI trial-wide results. Box plots depict first quartile, median, third quartile and extreme ranges from lower/upper quartile up to 1.5 IQR. Gix generalisability index, IQR interquartile range, OR odds ratio, SD standard deviation, QALYs quality-adjusted life-years, RCTs randomized controlled trials, ROSSINI Reduction of Surgical Site Infection using a Novel Intervention

Full size image
Fig. 8
figure 8

Clinical and cost-effectiveness estimates in simulated RCTs across categories of standardised trial-Gix4z. Horizontal red lines represent point estimates of ROSSINI trial-wide results. Box plots depict first quartile, median, third quartile and extreme ranges from lower/upper quartile up to 1.5 IQR. Gix generalisability index, IQR interquartile range, OR odds ratio, SD standard deviation, QALYs quality-adjusted life-years, RCTs randomized controlled trials, ROSSINI Reduction of Surgical Site Infection using a Novel Intervention

Full size image
Fig. 9
figure 9

Clinical and cost-effectiveness estimates in simulated RCTs across categories of standardised trial-Gix90. Horizontal red lines represent point estimates of ROSSINI trial-wide results. Box plots depict first quartile, median, third quartile and extreme ranges from lower/upper quartile up to 1.5 IQR. Gix generalisability index, IQR interquartile range, OR odds ratio, SD standard deviation, QALYs quality-adjusted life-years, RCTs randomized controlled trials, ROSSINI Reduction of Surgical Site Infection using a Novel Intervention

Full size image
Fig. 10
figure 10

Clinical and cost-effectiveness estimates in simulated RCTs across categories of standardised trial-Gix50. Horizontal red lines represent point estimates of ROSSINI trial-wide results. Box plots depict first quartile, median, third quartile and extreme ranges from lower/upper quartile up to 1.5 IQR. Gix generalisability index, IQR interquartile range, OR odds ratio, SD standard deviation, QALYs quality-adjusted life-years, RCTs randomized controlled trials, ROSSINI Reduction of Surgical Site Infection using a Novel Intervention

Full size image

Appendix 4

See Table 7.

Table 7 Correlation among the five centre-Gix dimensions in ROSSINI hospitals
Full size table

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gheorghe, A., Roberts, T., Hemming, K. et al. Evaluating the Generalisability of Trial Results: Introducing a Centre- and Trial-Level Generalisability Index. PharmacoEconomics 33, 1195–1214 (2015). https://doi.org/10.1007/s40273-015-0298-3

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40273-015-0298-3

Keywords

Search

Navigation