Skip to main content
SpringerLink
Log in

R and Shiny for Cost-Effectiveness Analyses: Why and When? A Hypothetical Case Study

  • Original Research Article
  • Published:
PharmacoEconomics Aims and scope Submit manuscript

Abstract

Introduction

Health economics models are typically built in Microsoft Excel® owing to its wide familiarity, accessibility and perceived transparency. However, given the increasingly rapid and analytically complex decision-making needs of both the pharmaceutical industry and the field of health economics and outcomes research (HEOR), the demands of cost-effectiveness analyses may be better met by the programming language R.

Objective

This case study provides an explicit comparison between Excel and R for contemporary cost-effectiveness analysis.

Methods

We constructed duplicate cost-effectiveness models using Excel and R (with a user interface built using the Shiny package) to address a hypothetical case study typical of contemporary health technology assessment.

Results

We compared R and Excel versions of the same model design to determine the advantages and limitations of the modelling platforms in terms of (i) analytical capability, (ii) data safety, (iii) building considerations, (iv) usability for technical and non-technical users and (v) model adaptability.

Conclusions

The findings of this explicit comparison are used to produce recommendations for when R might be more suitable than Excel in contemporary cost-effectiveness analyses. We conclude that selection of appropriate modelling software needs to consider case-by-case modelling requirements, particularly (i) intended audience, (ii) complexity of analysis, (iii) nature and frequency of updates and (iv) anticipated model run time.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

Data Availability Statement

The datasets generated and analysed during the current study are available from the corresponding author on reasonable request. The Microsoft Excel® model presented in this publication is included in the supplementary material files. The R code for the intRface™ model produced for this study is not publicly available due to commercial interest, but is available from the corresponding author on reasonable request. A demonstration version of the front-end of the model can be found at https://bresmed-intrface-hypothetical-car-t-model.shinyapps.io/IntRface_Model-PharmacoEconomics/.

References

  1. Microsoft R Application Network. Introduction to R. 2019. https://mran.microsoft.com/documents/what-is-r. Accessed 6 Feb 2019.

  2. R Core Team. R: a language and environment for statistical computing. Vienna: R Foundation for Statistical Computing; 2018. https://www.R-project.org/. Accessed 10 May 2019.

  3. Woolacott N, Hawkins N, Mason A, Kainth A, Khadjesari Z. Etanercept and efalizumab for the treatment of psoriasis: a systematic review. Health Technol Assess. 2006;10(46):1–233.

    Article  CAS  Google Scholar 

  4. National Institute for Health and Care Excellence (NICE). Rituximab for the first-line treatment of stage III–IV follicular lymphoma (TA243). 2012. https://www.nice.org.uk/guidance/ta243. Accessed 11 June 2019.

  5. Institute for Clinical and Economic Review (ICER). Extended-release opioid agonists and antagonist medications for addiction treatment (MAT) in patients with opioid use disorder: effectiveness and value. 2018. https://icer-review.org/wp-content/uploads/2018/04/ICER_OUD_Final_Evidence_Report_120318.pdf. Accessed 11 June 2019.

  6. Baio G. R for trial and model-based cost-effectiveness analysis. 2019. http://www.statistica.it/gianluca/teaching/r-hta-workshop/. Accessed 30 Jan 2019.

  7. Incerti D, Thom H, Baio G, Jansen JP. R you still using excel? The advantages of modern software tools for health technology assessment. Value Health. 2019;22(5):575–9. https://doi.org/10.1016/j.jval.2019.01.003.

    Article  PubMed  Google Scholar 

  8. Krijkamp EM, Alarid-Escudero F, Enns EA, Jalal HJ, Hunink MGM, Pechlivanoglou P. Microsimulation modeling for health decision sciences using R: a tutorial. Med Decis Making. 2018;38(3):400–22. https://doi.org/10.1177/0272989X18754513.

    Article  PubMed  PubMed Central  Google Scholar 

  9. Hollman C, Paulden M, Pechlivanoglou P, McCabe C. A comparison of four software programs for implementing decision analytic cost-effectiveness models. Pharmacoeconomics. 2017;35(8):817–30. https://doi.org/10.1007/s40273-017-0510-8.

    Article  PubMed  Google Scholar 

  10. Williams C, Lewsey JD, Briggs AH, Mackay DF. Cost-effectiveness analysis in R using a multi-state modeling survival analysis framework: a tutorial. Med Decis Making. 2017;37(4):340–52. https://doi.org/10.1177/0272989X16651869.

    Article  PubMed  Google Scholar 

  11. Jalal H, Pechlivanoglou P, Krijkamp E, Alarid-Escudero F, Enns E, Hunink MGM. An overview of R in health decision sciences. Med Decis Making. 2017;37(7):735–46. https://doi.org/10.1177/0272989X16686559.

    Article  PubMed  Google Scholar 

  12. Baio G, Heath A. When simple becomes complicated: why excel should lose its place at the top table. Glob Reg Health Technol Assess. 2017;4(1):e3–6.

    Google Scholar 

  13. Sampson CJ, Arnold R, Bryan S, Clarke P, Ekins S, Hatswell A, et al. Transparency in decision modelling: what, why, who and how? Pharmacoeconomics. 2019;37(11):1355–69. https://doi.org/10.1007/s40273-019-00819-z.

    Article  PubMed  Google Scholar 

  14. Incerti D, Curtis JR, Shafrin J, Lakdawalla DN, Jansen JP. A flexible open-source decision model for value assessment of biologic treatment for rheumatoid arthritis. Pharmacoeconomics. 2019;37(6):829–43. https://doi.org/10.1007/s40273-018-00765-2.

    Article  PubMed  Google Scholar 

  15. Alarid-Escudero F, Krijkamp EM, Pechlivanoglou P, Jalal H, Kao SZ, Yang A, et al. A need for change! A coding framework for improving transparency in decision modeling. Pharmacoeconomics. 2019;37(11):1329–39. https://doi.org/10.1007/s40273-019-00837-x.

    Article  PubMed  Google Scholar 

  16. Eddy DM, Hollingworth W, Caro JJ, Tsevat J, McDonald KM, Wong JB, et al. Model transparency and validation: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force-7. Value Health. 2012;15(6):843–50. https://doi.org/10.1016/j.jval.2012.04.012.

    Article  PubMed  Google Scholar 

  17. Tappenden P, Caro JJ. Improving transparency in decision models: current issues and potential solutions. Pharmacoeconomics. 2019;37(11):1303–4. https://doi.org/10.1007/s40273-019-00850-0.

    Article  PubMed  Google Scholar 

  18. Allaire J, Xie Y, McPherson J, Luraschi J, Ushey K, Atkins A et al. rmarkdown: dynamic documents for R. R package version 1.13. 2019. https://rmarkdown.rstudio.com. Accessed 3 Dec 2019.

  19. RStudio. Version control with git and SVN. 2019. https://support.rstudio.com/hc/en-us/articles/200532077-Version-Control-with-Git-and-SVN. Accessed 3 Dec 2019.

  20. National Health Care Institute Zorginstituut Nederland (ZIN). Template Pharmacoeconomic dossier. 2016. https://www.zorginstituutnederland.nl/binaries/zinl/documenten/publicatie/2016/07/25/template-farmaco-economisch-dossier/Template+Pharmacoeconomic+dossier.doc. Accessed 12 Nov 2019.

  21. Pharmaceutical Management Agency (PHARMAC). Prescription for pharmacoeconomic analysis—methods for cost-utility analysis. 2015. https://www.pharmac.govt.nz/assets/pfpa-2-2.pdf. Accessed 12 Nov 2019.

  22. Canadian Agency for Drugs and Technologies in Health (CADTH). Procedure and submission guidelines for the CADTH common drug review. 2019. https://cadth.ca/sites/default/files/cdr/process/Procedure_and_Guidelines_for_CADTH_CDR.pdf. Accessed 12 Nov 2019.

  23. Healtheconomics.com. ICER using new economic modeling platform for value assessment. 2018. https://www.healtheconomics.com/industry-news/icer-using-new-economic-modeling-platform-for-value-assessment. Accessed 12 Nov 2019.

  24. RStudio. Shiny. 2019. https://shiny.rstudio.com/. Accessed 2 Dec 2019.

  25. Chang W, Cheng J, Allaire J, Xie Y, McPherson J. Shiny: web application framework for R. R package version 1.3.2. 2019. https://CRAN.R-project.org/package=shiny. Accessed 2 July 2019.

  26. Innovation and value initiative. open-source value platform. 2019. https://www.thevalueinitiative.org/open-source-value-project/. Accessed 6 Feb 2019.

  27. Food and Drug Administration (FDA). RR-Drug. 2019. https://openfda.shinyapps.io/RR_D/. Accessed 12 Nov 2019.

  28. (CHOICE). TUoW-TCHOPaEI. PriMER platform: diffusion of precision medicine technologies. 2019. https://uwchoice.shinyapps.io/primer/. Accessed 12 Nov 2019.

  29. Center for Evaluation of Value and Risk in Health (CEVR). Global health cost effectiveness analysis registry. 2019. http://healtheconomics.tuftsmedicalcenter.org/ghcearegistry/. Accessed 12 Nov 2019.

  30. Eichler HG, Oye K, Baird LG, Abadie E, Brown J, Drum CL, et al. Adaptive licensing: taking the next step in the evolution of drug approval. Clin Pharmacol Ther. 2012;91(3):426–37. https://doi.org/10.1038/clpt.2011.345.

    Article  PubMed  Google Scholar 

  31. Food and Drug Administration (FDA). Accelerated Approval Program. 2018. https://www.fda.gov/drugs/resourcesforyou/healthprofessionals/ucm313768.htm. Accessed 2 July 2019.

  32. European Medicines Agency. Conditional marketing authorisation: report on ten years of experience at the european medicines agency. 2016. https://www.ema.europa.eu/en/documents/report/conditional-marketing-authorisation-report-ten-years-experience-european-medicines-agency_en.pdf. Accessed 14 May 2019.

  33. Hatswell AJ, Baio G, Berlin JA, Irs A, Freemantle N. Regulatory approval of pharmaceuticals without a randomised controlled study: analysis of EMA and FDA approvals 1999–2014. BMJ Open. 2016;6(6):e011666. https://doi.org/10.1136/bmjopen-2016-011666.

    Article  PubMed  PubMed Central  Google Scholar 

  34. European Medicines Agency. Yescarta. 2018. https://www.ema.europa.eu/en/medicines/human/EPAR/yescarta. Accessed 9 Apr 2019.

  35. European Medicines Agency. Kymriah. 2018. https://www.ema.europa.eu/en/medicines/human/EPAR/kymriah. Accessed 9 Apr 2019.

  36. Food and Drug Administration (FDA). Yescarta (axicabtagene ciloleucel). 2018. https://www.fda.gov/vaccines-blood-biologics/cellular-gene-therapy-products/yescarta-axicabtagene-ciloleucel. Accessed 7 June 2019.

  37. Food and Drug Administration (FDA). Kymriah (tisagenlecleucel). 2019. https://www.fda.gov/vaccines-blood-biologics/cellular-gene-therapy-products/kymriah-tisagenlecleucel. Accessed 6 June 2019.

  38. Hettle R, Corbett M, Hinde S, Hodgson R, Jones-Diette J, Woolacott N et al. Exploring the assessment and appraisal of regenerative medicines and cell therapy products. 2015. https://www.nice.org.uk/Media/Default/About/what-we-do/Science%20policy%20and%20research/final-york-report-march-16.pdf. Accessed 2 July 2019.

  39. Picanco-Castro V, Goncalves Pereira C, Swiech K, Ribeiro Malmegrim KC, Tadeu Covas D, Silveira Porto G. Emerging CAR T cell therapies: clinical landscape and patent technological routes. Hum Vaccin Immunother. 2019. https://doi.org/10.1080/21645515.2019.1689744.

    Article  PubMed  Google Scholar 

  40. Abreu TR, Fonseca NA, Goncalves N, Moreira JN. Current challenges and emerging opportunities of CAR-T cell therapies. J Control Release. 2020;319:246–61. https://doi.org/10.1016/j.jconrel.2019.12.047.

    Article  CAS  PubMed  Google Scholar 

  41. National Institute for Health and Care Excellence (NICE). Tisagenlecleucel for treating relapsed or refractory B-cell acute lymphoblastic leukaemia in people aged up to 25 years [TA554]. 2018. https://www.nice.org.uk/guidance/ta554/documents/final-appraisal-determination-document. Accessed 17 Apr 2019.

  42. National Institute for Health and Care Excellence (NICE). Axicabtagene ciloleucel for treating diffuse large B-cell lymphoma and primary mediastinal large B-cell lymphoma after 2 or more systemic therapies [TA559]. 2019. https://www.nice.org.uk/guidance/ta559/documents/final-appraisal-determination-document. Accessed 23 Jan 2019.

  43. National Institute for Health and Care Excellence (NICE). Tisagenlecleucel for treating relapsed or refractory diffuse large B-cell lymphoma after 2 or more systemic therapies (TA567). 2019. https://www.nice.org.uk/guidance/ta567/documents/final-appraisal-determination-document. Accessed 05 July 2019.

  44. Woods B, Sideris E, Palmer S, Latimer N, Soares M. NICE DSU Technical Support Document 19. Partitioned survival analysis for decision modelling in health care: a critical review. 2017. http://scharr.dept.shef.ac.uk/nicedsu/wp-content/uploads/sites/7/2017/06/Partitioned-Survival-Analysis-final-report.pdf. Accessed 2 May 2019.

  45. National Institute for Health and Care Excellence (NICE). Guide to the methods of technology appraisal. 2013. https://www.nice.org.uk/guidance/pmg9/resources/guide-to-the-methods-of-technology-appraisal-2013-pdf-2007975843781. Accessed 2 May 2019.

  46. Sousa D, Ferreira F, Félix F, Lopesa M. Acute lymphoblastic leukemia in children and adolescents: prognostic factors and analysis of survival. Rev Bras Hematol Hemoter. 2015;37(4):223–9.

    Article  Google Scholar 

  47. Fuster JL. Current approach to relapsed acute lymphoblastic leukemia in children. World J Hematol. 2014;3(3):49–70. https://doi.org/10.5315/wjh.v3.i3.49.

    Article  Google Scholar 

  48. National Institute for Health and Care Excellence. Single technology appraisal axicabtagene ciloleucel for treating diffuse large B-cell lymphoma and primary mediastinal B-cell lymphoma after 2 or more systemic therapies (TA559): Committee Papers. 2019. https://www.nice.org.uk/guidance/ta559/evidence/appraisal-consultation-committee-papers-pdf-6661404973. Accessed 11 Apr 2019.

  49. Faria R, Hernandez Alava M, Manca A, Wailoo AJ. NICE DSU Technical Support Document 17: The use of observational data to inform estimates of treatment effectiveness for Technology Appraisal: methods for comparative individual patient data. 2015. http://scharr.dept.shef.ac.uk/nicedsu/wp-content/uploads/sites/7/2016/03/TSD17-DSU-Observational-data-FINAL.pdf. Accessed 10 May 2019.

  50. Rosenbaum PR, Rubin DB. The central role of the propensity score in observational studies for causal effects. Biometrika. 1983;70(1):41–55.

    Article  Google Scholar 

  51. Stuart EA. Matching methods for causal inference: a review and a look forward. Stat Sci. 2010;25(1):1–21. https://doi.org/10.1214/09-STS313.

    Article  PubMed  PubMed Central  Google Scholar 

  52. Latimer N. NICE DSU technical support document 14: undertaking survival analysis for economic evaluations alongside clinical trials—extrapolation with patient-level data. 2011. http://nicedsu.org.uk/wp-content/uploads/2016/03/NICE-DSU-TSD-Survival-analysis.updated-March-2013.v2.pdf. Accessed 2 May 2019.

  53. Lambert PC, Thompson JR, Weston CL, Dickman PW. Estimating and modeling the cure fraction in population-based cancer survival analysis. Biostatistics. 2007;8(3):576–94. https://doi.org/10.1093/biostatistics/kxl030.

    Article  PubMed  Google Scholar 

  54. Othus M, Barlogie B, Leblanc ML, Crowley JJ. Cure models as a useful statistical tool for analyzing survival. Clin Cancer Res. 2012;18(14):3731–6. https://doi.org/10.1158/1078-0432.CCR-11-2859.

    Article  PubMed  PubMed Central  Google Scholar 

  55. Lambert PC. Modeling of the cure fraction in survival studies. Stata J. 2007;7(3):351–75.

    Article  Google Scholar 

  56. Grant TS, Burns D, Kiff C, Lee D. A case study examining the usefulness of cure modelling for the prediction of survival based on data maturity. Pharmacoeconomics. 2020;38(4):385–95. https://doi.org/10.1007/s40273-019-00867-5.

    Article  PubMed  Google Scholar 

  57. RStudio. RStudio Connect. 2019. https://rstudio.com/products/connect/. Accessed 3 Dec 2019.

  58. Ghabri S, Stevenson M, Moller J, Caro JJ. Trusting the results of model-based economic analyses: is there a pragmatic validation solution? Pharmacoeconomics. 2019;37(1):1–6. https://doi.org/10.1007/s40273-018-0711-9.

    Article  PubMed  Google Scholar 

  59. National Institute for Health and Care Excellence (NICE). Guide to the processes of technology appraisal. 2014. https://www.nice.org.uk/process/pmg19. Accessed 11 June 2019.

  60. Canadian Agency for Drugs and Technologies in Health (CADTH). CADTH common drug review (CDR). 2019. https://www.cadth.ca/about-cadth/what-we-do/products-services/cdr. Accessed 11 June 2019.

  61. National Centre for Pharmacoeconomics (NCPE). HTA Guidelines. 2019. http://www.ncpe.ie/submission-process/hta-guidelines/. Accessed 11 June 2019.

  62. Pharmaceutical Benefits Advisory Committee (PBAC). The pharmaceutical benefits advisory committee guidlines. 2016. https://pbac.pbs.gov.au/information/printable-version-of-guidelines.html. Accessed 11 June 2019.

  63. Moller J, Davis S, Stevenson M, Caro JJ. Validation of a DICE simulation against a discrete event simulation implemented entirely in code. Pharmacoeconomics. 2017;35(10):1103–9. https://doi.org/10.1007/s40273-017-0534-0.

    Article  PubMed  Google Scholar 

  64. Philips Z, Ginnelly L, Sculpher M, Claxton K, Golder S, Riemsma R, et al. Review of guidelines for good practice in decision-analytic modelling in health technology assessment. Health Technol Assess. 2004;8(36):1–158. https://doi.org/10.3310/hta8360.

    Article  PubMed  Google Scholar 

Download references

Funding

No external funding was received for the development of the model or manuscript.

Author information

Authors and Affiliations

  1. BresMed Health Solutions, Sheffield, UK

    Rose Hart, Darren Burns, Daniel Gladwell, Will Sullivan, Indeg Sly, Theresa Cain & Dawn Lee

  2. Department of Clinical Epidemiology and Medical Technology Assessment, Maastricht University Medical Center, Maastricht, The Netherlands

    Bram Ramaekers

  3. School of Health and Related Research, University of Sheffield, Sheffield, UK

    Shijie Ren

  4. BresMed Health Solutions, Manchester, UK

    Niall Davison & Owain Saunders

Authors
  1. Rose Hart
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Darren Burns
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bram Ramaekers
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shijie Ren
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Daniel Gladwell
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Will Sullivan
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Niall Davison
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Owain Saunders
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Indeg Sly
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Theresa Cain
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Dawn Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

RH, WS, DG, ND and DL were involved in the conception, planning and writing of this manuscript. RH, DB and ND programmed the model demonstrated within this manuscript. OS, IS and TC were involved in programming methods and functions adapted for use within the model. SR, BR and DL were involved in quality control and validation of the methods used in the model demonstrated within this manuscript. All authors edited and commented on draft versions of the manuscript and approved the submitted versions.

Corresponding author

Correspondence to Rose Hart.

Ethics declarations

Conflict of interest

Rose Hart, Darren Burns, Bram Ramaekers, Shijie Ren, Daniel Gladwell, Will Sullivan, Niall Davison, Owain Saunders, Indeg Sly, Theresa Cain and Dawn Lee declare they have no conflicts of interest relevant to the content of this manuscript.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (XLSB 31099 kb)

Supplementary material 2 (PDF 567 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hart, R., Burns, D., Ramaekers, B. et al. R and Shiny for Cost-Effectiveness Analyses: Why and When? A Hypothetical Case Study. PharmacoEconomics 38, 765–776 (2020). https://doi.org/10.1007/s40273-020-00903-9

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40273-020-00903-9

Search

Navigation