Skip to main content
SpringerLink
Log in

What Did Time Tell Us? A Comparison and Retrospective Validation of Different Survival Extrapolation Methods for Immuno-Oncologic Therapy in Advanced or Metastatic Renal Cell Carcinoma

  • Original Research Article
  • Published:
PharmacoEconomics Aims and scope Submit manuscript

Abstract

Background

The immuno-oncologic (IO) mechanism of action may lead to an overall survival (OS) hazard that changes over time, producing shapes that standard parametric extrapolation methods may struggle to reflect. Furthermore, selection of the most appropriate extrapolation method for health technology assessment is often based on trial data with limited follow-up.

Objective

To examine this problem, we fitted a range of extrapolation methods to patient-level survival data from CheckMate 025 (NCT01668784, CM-025), a phase III trial comparing nivolumab with everolimus for previously treated advanced renal cell carcinoma (aRCC), to assess their predictive accuracy over time.

Methods

Six extrapolation methods were examined: standard parametric models, natural cubic splines, piecewise models combining Kaplan–Meier data with an exponential or non-exponential distribution, response-based landmark models, and parametric mixture models. We produced three database locks (DBLs) at minimum follow-ups of 15, 27, and 39 months to align with previously published CM-025 data. A three-step evaluation process was adopted: (1) selection of the distribution family for each method in each of the three DBLs, (2) internal validation comparing extrapolation-based landmark and mean survival with the latest CM-025 dataset (minimum follow-up, 64 months), and (3) external validation of survival projections using clinical expert opinion and long-term follow-up data from other nivolumab studies in aRCC (CheckMate 003 and CheckMate 010).

Results

All extrapolation methods, with the exception of mixture models, underestimated landmark and mean OS for nivolumab compared with CM-025 long-term follow-up data. OS estimates for everolimus tended to be more accurate, with four of the six methods providing landmark OS estimates within the 95% confidence interval of observed OS as per the latest dataset. The predictive accuracy of survival extrapolation methods fitted to nivolumab also showed greater variation than for everolimus. The proportional hazards assumption held for all DBLs, and a dependent log-logistic model provided reliable estimates of longer-term survival for both nivolumab and everolimus across the DBLs. Although mixture models and response-based landmark models provided reasonable estimates of OS based on the 39-month DBL, this was not the case for the two earlier DBLs. The piecewise exponential models consistently underestimated OS for both nivolumab and everolimus at clinically meaningful pre-specified landmark time points.

Conclusions

This aRCC case study identified marked differences in the predictive accuracy of survival extrapolation methods for nivolumab but less so for everolimus. The dependent log-logistic model did not suffer from overfitting to early DBLs to the same extent as more complex methods. Methods that provide more degrees of freedom may accurately represent survival for IO therapy, particularly if data are more mature or external data are available to inform the long-term extrapolations.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Kearns B, Stevens J, Ren S, Brennan A. How uncertain is the survival extrapolation? A study of the impact of different parametric survival models on extrapolated uncertainty about hazard functions, lifetime mean survival and cost effectiveness. Pharmacoeconomics. 2020;38(2):193–204.

    Article  Google Scholar 

  2. Ferrara R, Pilotto S, Caccese M, Grizzi G, Sperduti I, Giannarelli D, et al. Do immune checkpoint inhibitors need new studies methodology? J Thorac Dis. 2018;10(Suppl 13):S1564–80.

    Article  Google Scholar 

  3. Kaufman HL, Atkins MB, Subedi P, Wu J, Chambers J, Joseph Mattingly T 2nd, et al. The promise of immuno-oncology: implications for defining the value of cancer treatment. J Immunother Cancer. 2019;7(1):129.

    Article  Google Scholar 

  4. Bullement A, Latimer NR, Bell GH. Survival extrapolation in cancer immunotherapy: a validation-based case study. Value Health. 2019;22(3):276–83.

    Article  Google Scholar 

  5. Ouwens M, Mukhopadhyay P, Zhang Y, Huang M, Latimer N, Briggs A. Estimating lifetime benefits associated with immuno-oncology therapies: challenges and approaches for overall survival extrapolations. Pharmacoeconomics. 2019;37(9):1129–38.

    Article  Google Scholar 

  6. Sculpher M, Palmer S. After 20 years of using economic evaluation, should NICE be considered a methods innovator? Pharmacoeconomics. 2020;38(3):247–57.

    Article  Google Scholar 

  7. Latimer N. NICE DSU technical support document 14: survival analysis for economic evaluations alongside clinical trials—extrapolation with patient-level data. 2013.

  8. Gallacher D, Auguste P, Connock M. How do pharmaceutical companies model survival of cancer patients? A review of NICE single technology appraisals in 2017. Int J Technol Assess Health Care. 2019;35(2):160–7.

    Article  Google Scholar 

  9. Kroep S, Kiff C, Kraan C, Bianco M, Johannesen K, Kurt M, et al. PCN451. Modeling the survival benefit of immuno-oncologic therapy: a review of methods used in NICE single technology appraisals. Value Health. 2019;22:S523–4.

    Article  Google Scholar 

  10. Latimer N. NICE DSU Technical support document 14: survival analysis for economic evaluations alongside clinical trials—extrapolation with patient-level data. 2011.

  11. Woods B, Sideris E, Palmer S, Latimer N, Soares M. NICE DSU Technical support document 19: partitioned survival analysis for decision modelling in health care: a critical review. 2017.

  12. Motzer RJ, Escudier B, McDermott DF, George S, Hammers HJ, Srinivas S, et al. Nivolumab versus everolimus in advanced renal-cell carcinoma. N Engl J Med. 2015;373(19):1803–13.

    Article  CAS  Google Scholar 

  13. National Institute for Health and Care Excellence. Final appraisal determination: nivolumab for previously treated advanced renal cell carcinoma. 2016.

  14. Statens legemiddelverk. Hurtig metodevurdering: Nivolumab (Opdivo) til andrelinjebehandling av avansert nyrecellekarsinom - Vurdering av innsendt dokumentasjon. 2016.

  15. Tandvårds- och Läkemedelsförmånsverket. Underlag foör beslut i landstingen: Opdivo som monoterapi är indicerat för behandling av vuxna med avancerad njurcellscancer efter tidigare behandling. 2016.

  16. Motzer RJ, Tykodi SS, Escudier B, Oudard S, Hammers HJ, McDermott DF, et al. Final analysis of the CheckMate 025 trial comparing nivolumab (NIVO) versus everolimus (EVE) with > 5 years of follow-up in patients with advanced renal cell carcinoma (aRCC). J Clin Oncol. 2020;38(6_suppl):617.

    Article  Google Scholar 

  17. Plimack ER, Motzer RJ, Escudier B, Sharma P, McDermott DF, George S, et al. Two-year efficacy and safety update: phase III CheckMate 025 study of nivolumab vs everolimus in patients with advanced renal cell carcinoma (aRCC). ESMO. 2016.

  18. Sharma P, Tykodi SS, Escudier B, Carducci M, Oudard S, Hammers HJ, et al. Three-year efficacy and safety update from the phase III CheckMate 025 study of nivolumab versus everolimus in patients with advanced renal cell carcinoma. In: 16th International kidney cancer symposium. 2017.

  19. Royston P, Parmar MK. Flexible parametric proportional-hazards and proportional-odds models for censored survival data, with application to prognostic modelling and estimation of treatment effects. Stat Med. 2002;21(15):2175–97.

    Article  Google Scholar 

  20. Grambsch PM, Therneau TM. Proportional hazards tests and diagnostics based on weighted residuals. Biometrika. 1994;81(3):515–26.

    Article  Google Scholar 

  21. Akaike H. A new look at the statistical model identification. IEEE Trans Automat Control. 1974;19(6):716–23.

    Article  Google Scholar 

  22. Schwarz G. Estimating the dimension of a model. Ann Stat. 1978;6(2):461–4.

    Article  Google Scholar 

  23. Mazza C, Escudier B, Albiges L. Nivolumab in renal cell carcinoma: latest evidence and clinical potential. Ther Adv Med Oncol. 2017;9(3):171–81.

    Article  CAS  Google Scholar 

  24. Topalian SL, Hodi FS, Brahmer JR, Gettinger SN, Smith DC, McDermott DF, et al. Safety, activity, and immune correlates of anti-PD-1 antibody in cancer. N Engl J Med. 2012;366(26):2443–54.

    Article  CAS  Google Scholar 

  25. Motzer RJ, Rini BI, McDermott DF, Redman BG, Kuzel TM, Harrison MR, et al. Nivolumab for metastatic renal cell carcinoma: results of a randomized phase II trial. J Clin Oncol. 2015;33(13):1430–7.

    Article  CAS  Google Scholar 

  26. McDermott DF, Motzer RJ, Atkins MB, Plimack ER, Sznol M, George S, et al. Long-term overall survival (OS) with nivolumab in previously treated patients with advanced renal cell carcinoma (aRCC) from phase I and II studies. J Clin Oncol. 2016;34(15_suppl):4507.

    Article  Google Scholar 

  27. O'Hagan A, Oakley JE. SHELF: the Sheffield Elicitation Framework; 2019. http://www.tonyohagan.co.uk/shelf/. Accessed Sept 2019.

  28. National Office for Statistics. National life tables: UK 2019.

  29. Bristol-Myers Squibb. Bristol-Myers Squibb receives FDA approval for Opdivo (nivolumab), the only treatment to deliver significant overall survival in advanced renal cell carcinoma vs. a standard of care, in patients who have received prior anti-angiogenic therapy. [2015 January 2020]. https://news.bms.com/press-release/bristol-myers-squibb-receives-fda-approval-opdivo-nivolumab-only-treatment-deliver-sig.

  30. European Medicines Agency. Opdivo EPAR2020.

  31. Motzer RJ, Tykodi SS, Escudier B, Carducci M, Oudard S, Hammers HJ, et al. Long-term follow-up of nivolumab versus everolimus in patients with advanced renal cell carcinoma: the phase 3 CheckMate 025 trial. In: 18th International Kidney Cancer Symposium. Miami, FL, USA2019. p. Poster.

  32. Liverpool Reviews and Implementation Group. Atezolizumab for treating locally advanced or metastatic non-small cell lung cancer after chemotherapy [ID970]. 2017.

  33. Bagust A, Beale S. Survival analysis and extrapolation modeling of time-to-event clinical trial data for economic evaluation: an alternative approach. Med Decis Mak. 2014;34(3):343–51.

    Article  Google Scholar 

  34. National Institute for Health and Care Excellence. TA531: Pembrolizumab for untreated PD-L1-positive metastatic non-small-cell lung cancer. 2018.

  35. National Institute for Health and Care Excellence. TA520: Atezolizumab for treating locally advanced or metastatic non-small-cell lung cancer after chemotherapy. 2018.

  36. Burnham KP, Anderson DR. Multimodel inference. Sociol Methods Res. 2016;33(2):261–304.

    Article  Google Scholar 

  37. Raftery AE. Bayesian model selection in social research. Sociol Methodol. 1995;25.

  38. Bullement A, Meng Y, Cooper M, Lee D, Harding TL, O’Regan C, et al. A review and validation of overall survival extrapolation in health technology assessments of cancer immunotherapy by the National Institute for Health and Care Excellence: how did the initial best estimate compare to trial data subsequently made available? J Med Econ. 2019;22(3):205–14.

    Article  Google Scholar 

  39. Cope S, Ayers D, Zhang J, Batt K, Jansen JP. Integrating expert opinion with clinical trial data to extrapolate long-term survival: a case study of CAR-T therapy for children and young adults with relapsed or refractory acute lymphoblastic leukemia. BMC Med Res Methodol. 2019;19(1):182.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Pharmerit - an OPEN Health Company, Marten Meesweg 107, 3068 AV, Rotterdam, The Netherlands

    Sven L. Klijn & Sonja Kroep

  2. Pharmerit - an OPEN Health Company, Oxford, UK

    Elisabeth Fenwick

  3. Bristol Myers Squibb, Stockholm, Sweden

    Kasper Johannesen

  4. Bristol Myers Squibb, Uxbridge, Greater London, UK

    Bill Malcolm, Christopher Kiff & John Borrill

  5. Bristol Myers Squibb, Princeton, NJ, USA

    Murat Kurt

Authors
  1. Sven L. Klijn
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Elisabeth Fenwick
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sonja Kroep
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Kasper Johannesen
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Bill Malcolm
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Murat Kurt
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Christopher Kiff
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. John Borrill
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sven L. Klijn.

Ethics declarations

Funding

This study was supported by Bristol-Myers Squibb Company.

Conflict of interest

Sven L. Klijn, Elisabeth Fenwick and Sonja Kroep are employees of Pharmerit - an OPEN Health Company, which received payment from Bristol Myers Squibb to conduct the analyses. John Borrill, Kasper Johannesen, Christopher Kiff, Murat Kurt, and Bill Malcolm are employees and shareholders of Bristol Myers Squibb.

Data availability

Bristol Myers Squibb Company policy on data sharing may be found at https://www.bms.com/researchers-and-partners/independent-research/data-sharing-request-process.html.

Author contributions

All authors were involved in the conceptualization and design of the work. SLK and SK performed data analysis. All authors were involved in data interpretation. All authors reviewed and revised the manuscript.

Acknowledgements

Jessica May and Kyna Gooden supported the data acquisition. Esra Cakar, Lisanne Verburg, and Poojee Sudhapalli contributed to the analyses of the study. Dr. Michael B. Atkins from Georgetown University provided clinical input for the external validation of the long-term survival extrapolations. Einar Torkilseng and Sébastien Branchoux contributed to the writing of this article.

Supplementary Information

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Klijn, S.L., Fenwick, E., Kroep, S. et al. What Did Time Tell Us? A Comparison and Retrospective Validation of Different Survival Extrapolation Methods for Immuno-Oncologic Therapy in Advanced or Metastatic Renal Cell Carcinoma. PharmacoEconomics 39, 345–356 (2021). https://doi.org/10.1007/s40273-020-00989-1

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40273-020-00989-1

Search

Navigation