Health economics models are typically built in Microsoft Excel® owing to its wide familiarity, accessibility and perceived transparency. However, given the increasingly rapid and analytically complex decision-making needs of both the pharmaceutical industry and the field of health economics and outcomes research (HEOR), the demands of cost-effectiveness analyses may be better met by the programming language R.
This case study provides an explicit comparison between Excel and R for contemporary cost-effectiveness analysis.
We constructed duplicate cost-effectiveness models using Excel and R (with a user interface built using the Shiny package) to address a hypothetical case study typical of contemporary health technology assessment.
We compared R and Excel versions of the same model design to determine the advantages and limitations of the modelling platforms in terms of (i) analytical capability, (ii) data safety, (iii) building considerations, (iv) usability for technical and non-technical users and (v) model adaptability.
The findings of this explicit comparison are used to produce recommendations for when R might be more suitable than Excel in contemporary cost-effectiveness analyses. We conclude that selection of appropriate modelling software needs to consider case-by-case modelling requirements, particularly (i) intended audience, (ii) complexity of analysis, (iii) nature and frequency of updates and (iv) anticipated model run time.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Price excludes VAT (USA)
Tax calculation will be finalised during checkout.
Data Availability Statement
The datasets generated and analysed during the current study are available from the corresponding author on reasonable request. The Microsoft Excel® model presented in this publication is included in the supplementary material files. The R code for the intRface™ model produced for this study is not publicly available due to commercial interest, but is available from the corresponding author on reasonable request. A demonstration version of the front-end of the model can be found at https://bresmed-intrface-hypothetical-car-t-model.shinyapps.io/IntRface_Model-PharmacoEconomics/.
Microsoft R Application Network. Introduction to R. 2019. https://mran.microsoft.com/documents/what-is-r. Accessed 6 Feb 2019.
R Core Team. R: a language and environment for statistical computing. Vienna: R Foundation for Statistical Computing; 2018. https://www.R-project.org/. Accessed 10 May 2019.
Woolacott N, Hawkins N, Mason A, Kainth A, Khadjesari Z. Etanercept and efalizumab for the treatment of psoriasis: a systematic review. Health Technol Assess. 2006;10(46):1–233.
National Institute for Health and Care Excellence (NICE). Rituximab for the first-line treatment of stage III–IV follicular lymphoma (TA243). 2012. https://www.nice.org.uk/guidance/ta243. Accessed 11 June 2019.
Institute for Clinical and Economic Review (ICER). Extended-release opioid agonists and antagonist medications for addiction treatment (MAT) in patients with opioid use disorder: effectiveness and value. 2018. https://icer-review.org/wp-content/uploads/2018/04/ICER_OUD_Final_Evidence_Report_120318.pdf. Accessed 11 June 2019.
Baio G. R for trial and model-based cost-effectiveness analysis. 2019. http://www.statistica.it/gianluca/teaching/r-hta-workshop/. Accessed 30 Jan 2019.
Incerti D, Thom H, Baio G, Jansen JP. R you still using excel? The advantages of modern software tools for health technology assessment. Value Health. 2019;22(5):575–9. https://doi.org/10.1016/j.jval.2019.01.003.
Krijkamp EM, Alarid-Escudero F, Enns EA, Jalal HJ, Hunink MGM, Pechlivanoglou P. Microsimulation modeling for health decision sciences using R: a tutorial. Med Decis Making. 2018;38(3):400–22. https://doi.org/10.1177/0272989X18754513.
Hollman C, Paulden M, Pechlivanoglou P, McCabe C. A comparison of four software programs for implementing decision analytic cost-effectiveness models. Pharmacoeconomics. 2017;35(8):817–30. https://doi.org/10.1007/s40273-017-0510-8.
Williams C, Lewsey JD, Briggs AH, Mackay DF. Cost-effectiveness analysis in R using a multi-state modeling survival analysis framework: a tutorial. Med Decis Making. 2017;37(4):340–52. https://doi.org/10.1177/0272989X16651869.
Jalal H, Pechlivanoglou P, Krijkamp E, Alarid-Escudero F, Enns E, Hunink MGM. An overview of R in health decision sciences. Med Decis Making. 2017;37(7):735–46. https://doi.org/10.1177/0272989X16686559.
Baio G, Heath A. When simple becomes complicated: why excel should lose its place at the top table. Glob Reg Health Technol Assess. 2017;4(1):e3–6.
Sampson CJ, Arnold R, Bryan S, Clarke P, Ekins S, Hatswell A, et al. Transparency in decision modelling: what, why, who and how? Pharmacoeconomics. 2019;37(11):1355–69. https://doi.org/10.1007/s40273-019-00819-z.
Incerti D, Curtis JR, Shafrin J, Lakdawalla DN, Jansen JP. A flexible open-source decision model for value assessment of biologic treatment for rheumatoid arthritis. Pharmacoeconomics. 2019;37(6):829–43. https://doi.org/10.1007/s40273-018-00765-2.
Alarid-Escudero F, Krijkamp EM, Pechlivanoglou P, Jalal H, Kao SZ, Yang A, et al. A need for change! A coding framework for improving transparency in decision modeling. Pharmacoeconomics. 2019;37(11):1329–39. https://doi.org/10.1007/s40273-019-00837-x.
Eddy DM, Hollingworth W, Caro JJ, Tsevat J, McDonald KM, Wong JB, et al. Model transparency and validation: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force-7. Value Health. 2012;15(6):843–50. https://doi.org/10.1016/j.jval.2012.04.012.
Tappenden P, Caro JJ. Improving transparency in decision models: current issues and potential solutions. Pharmacoeconomics. 2019;37(11):1303–4. https://doi.org/10.1007/s40273-019-00850-0.
Allaire J, Xie Y, McPherson J, Luraschi J, Ushey K, Atkins A et al. rmarkdown: dynamic documents for R. R package version 1.13. 2019. https://rmarkdown.rstudio.com. Accessed 3 Dec 2019.
RStudio. Version control with git and SVN. 2019. https://support.rstudio.com/hc/en-us/articles/200532077-Version-Control-with-Git-and-SVN. Accessed 3 Dec 2019.
National Health Care Institute Zorginstituut Nederland (ZIN). Template Pharmacoeconomic dossier. 2016. https://www.zorginstituutnederland.nl/binaries/zinl/documenten/publicatie/2016/07/25/template-farmaco-economisch-dossier/Template+Pharmacoeconomic+dossier.doc. Accessed 12 Nov 2019.
Pharmaceutical Management Agency (PHARMAC). Prescription for pharmacoeconomic analysis—methods for cost-utility analysis. 2015. https://www.pharmac.govt.nz/assets/pfpa-2-2.pdf. Accessed 12 Nov 2019.
Canadian Agency for Drugs and Technologies in Health (CADTH). Procedure and submission guidelines for the CADTH common drug review. 2019. https://cadth.ca/sites/default/files/cdr/process/Procedure_and_Guidelines_for_CADTH_CDR.pdf. Accessed 12 Nov 2019.
Healtheconomics.com. ICER using new economic modeling platform for value assessment. 2018. https://www.healtheconomics.com/industry-news/icer-using-new-economic-modeling-platform-for-value-assessment. Accessed 12 Nov 2019.
RStudio. Shiny. 2019. https://shiny.rstudio.com/. Accessed 2 Dec 2019.
Chang W, Cheng J, Allaire J, Xie Y, McPherson J. Shiny: web application framework for R. R package version 1.3.2. 2019. https://CRAN.R-project.org/package=shiny. Accessed 2 July 2019.
Innovation and value initiative. open-source value platform. 2019. https://www.thevalueinitiative.org/open-source-value-project/. Accessed 6 Feb 2019.
Food and Drug Administration (FDA). RR-Drug. 2019. https://openfda.shinyapps.io/RR_D/. Accessed 12 Nov 2019.
(CHOICE). TUoW-TCHOPaEI. PriMER platform: diffusion of precision medicine technologies. 2019. https://uwchoice.shinyapps.io/primer/. Accessed 12 Nov 2019.
Center for Evaluation of Value and Risk in Health (CEVR). Global health cost effectiveness analysis registry. 2019. http://healtheconomics.tuftsmedicalcenter.org/ghcearegistry/. Accessed 12 Nov 2019.
Eichler HG, Oye K, Baird LG, Abadie E, Brown J, Drum CL, et al. Adaptive licensing: taking the next step in the evolution of drug approval. Clin Pharmacol Ther. 2012;91(3):426–37. https://doi.org/10.1038/clpt.2011.345.
Food and Drug Administration (FDA). Accelerated Approval Program. 2018. https://www.fda.gov/drugs/resourcesforyou/healthprofessionals/ucm313768.htm. Accessed 2 July 2019.
European Medicines Agency. Conditional marketing authorisation: report on ten years of experience at the european medicines agency. 2016. https://www.ema.europa.eu/en/documents/report/conditional-marketing-authorisation-report-ten-years-experience-european-medicines-agency_en.pdf. Accessed 14 May 2019.
Hatswell AJ, Baio G, Berlin JA, Irs A, Freemantle N. Regulatory approval of pharmaceuticals without a randomised controlled study: analysis of EMA and FDA approvals 1999–2014. BMJ Open. 2016;6(6):e011666. https://doi.org/10.1136/bmjopen-2016-011666.
European Medicines Agency. Yescarta. 2018. https://www.ema.europa.eu/en/medicines/human/EPAR/yescarta. Accessed 9 Apr 2019.
European Medicines Agency. Kymriah. 2018. https://www.ema.europa.eu/en/medicines/human/EPAR/kymriah. Accessed 9 Apr 2019.
Food and Drug Administration (FDA). Yescarta (axicabtagene ciloleucel). 2018. https://www.fda.gov/vaccines-blood-biologics/cellular-gene-therapy-products/yescarta-axicabtagene-ciloleucel. Accessed 7 June 2019.
Food and Drug Administration (FDA). Kymriah (tisagenlecleucel). 2019. https://www.fda.gov/vaccines-blood-biologics/cellular-gene-therapy-products/kymriah-tisagenlecleucel. Accessed 6 June 2019.
Hettle R, Corbett M, Hinde S, Hodgson R, Jones-Diette J, Woolacott N et al. Exploring the assessment and appraisal of regenerative medicines and cell therapy products. 2015. https://www.nice.org.uk/Media/Default/About/what-we-do/Science%20policy%20and%20research/final-york-report-march-16.pdf. Accessed 2 July 2019.
Picanco-Castro V, Goncalves Pereira C, Swiech K, Ribeiro Malmegrim KC, Tadeu Covas D, Silveira Porto G. Emerging CAR T cell therapies: clinical landscape and patent technological routes. Hum Vaccin Immunother. 2019. https://doi.org/10.1080/21645515.2019.1689744.
Abreu TR, Fonseca NA, Goncalves N, Moreira JN. Current challenges and emerging opportunities of CAR-T cell therapies. J Control Release. 2020;319:246–61. https://doi.org/10.1016/j.jconrel.2019.12.047.
National Institute for Health and Care Excellence (NICE). Tisagenlecleucel for treating relapsed or refractory B-cell acute lymphoblastic leukaemia in people aged up to 25 years [TA554]. 2018. https://www.nice.org.uk/guidance/ta554/documents/final-appraisal-determination-document. Accessed 17 Apr 2019.
National Institute for Health and Care Excellence (NICE). Axicabtagene ciloleucel for treating diffuse large B-cell lymphoma and primary mediastinal large B-cell lymphoma after 2 or more systemic therapies [TA559]. 2019. https://www.nice.org.uk/guidance/ta559/documents/final-appraisal-determination-document. Accessed 23 Jan 2019.
National Institute for Health and Care Excellence (NICE). Tisagenlecleucel for treating relapsed or refractory diffuse large B-cell lymphoma after 2 or more systemic therapies (TA567). 2019. https://www.nice.org.uk/guidance/ta567/documents/final-appraisal-determination-document. Accessed 05 July 2019.
Woods B, Sideris E, Palmer S, Latimer N, Soares M. NICE DSU Technical Support Document 19. Partitioned survival analysis for decision modelling in health care: a critical review. 2017. http://scharr.dept.shef.ac.uk/nicedsu/wp-content/uploads/sites/7/2017/06/Partitioned-Survival-Analysis-final-report.pdf. Accessed 2 May 2019.
National Institute for Health and Care Excellence (NICE). Guide to the methods of technology appraisal. 2013. https://www.nice.org.uk/guidance/pmg9/resources/guide-to-the-methods-of-technology-appraisal-2013-pdf-2007975843781. Accessed 2 May 2019.
Sousa D, Ferreira F, Félix F, Lopesa M. Acute lymphoblastic leukemia in children and adolescents: prognostic factors and analysis of survival. Rev Bras Hematol Hemoter. 2015;37(4):223–9.
Fuster JL. Current approach to relapsed acute lymphoblastic leukemia in children. World J Hematol. 2014;3(3):49–70. https://doi.org/10.5315/wjh.v3.i3.49.
National Institute for Health and Care Excellence. Single technology appraisal axicabtagene ciloleucel for treating diffuse large B-cell lymphoma and primary mediastinal B-cell lymphoma after 2 or more systemic therapies (TA559): Committee Papers. 2019. https://www.nice.org.uk/guidance/ta559/evidence/appraisal-consultation-committee-papers-pdf-6661404973. Accessed 11 Apr 2019.
Faria R, Hernandez Alava M, Manca A, Wailoo AJ. NICE DSU Technical Support Document 17: The use of observational data to inform estimates of treatment effectiveness for Technology Appraisal: methods for comparative individual patient data. 2015. http://scharr.dept.shef.ac.uk/nicedsu/wp-content/uploads/sites/7/2016/03/TSD17-DSU-Observational-data-FINAL.pdf. Accessed 10 May 2019.
Rosenbaum PR, Rubin DB. The central role of the propensity score in observational studies for causal effects. Biometrika. 1983;70(1):41–55.
Stuart EA. Matching methods for causal inference: a review and a look forward. Stat Sci. 2010;25(1):1–21. https://doi.org/10.1214/09-STS313.
Latimer N. NICE DSU technical support document 14: undertaking survival analysis for economic evaluations alongside clinical trials—extrapolation with patient-level data. 2011. http://nicedsu.org.uk/wp-content/uploads/2016/03/NICE-DSU-TSD-Survival-analysis.updated-March-2013.v2.pdf. Accessed 2 May 2019.
Lambert PC, Thompson JR, Weston CL, Dickman PW. Estimating and modeling the cure fraction in population-based cancer survival analysis. Biostatistics. 2007;8(3):576–94. https://doi.org/10.1093/biostatistics/kxl030.
Othus M, Barlogie B, Leblanc ML, Crowley JJ. Cure models as a useful statistical tool for analyzing survival. Clin Cancer Res. 2012;18(14):3731–6. https://doi.org/10.1158/1078-0432.CCR-11-2859.
Lambert PC. Modeling of the cure fraction in survival studies. Stata J. 2007;7(3):351–75.
Grant TS, Burns D, Kiff C, Lee D. A case study examining the usefulness of cure modelling for the prediction of survival based on data maturity. Pharmacoeconomics. 2020;38(4):385–95. https://doi.org/10.1007/s40273-019-00867-5.
RStudio. RStudio Connect. 2019. https://rstudio.com/products/connect/. Accessed 3 Dec 2019.
Ghabri S, Stevenson M, Moller J, Caro JJ. Trusting the results of model-based economic analyses: is there a pragmatic validation solution? Pharmacoeconomics. 2019;37(1):1–6. https://doi.org/10.1007/s40273-018-0711-9.
National Institute for Health and Care Excellence (NICE). Guide to the processes of technology appraisal. 2014. https://www.nice.org.uk/process/pmg19. Accessed 11 June 2019.
Canadian Agency for Drugs and Technologies in Health (CADTH). CADTH common drug review (CDR). 2019. https://www.cadth.ca/about-cadth/what-we-do/products-services/cdr. Accessed 11 June 2019.
National Centre for Pharmacoeconomics (NCPE). HTA Guidelines. 2019. http://www.ncpe.ie/submission-process/hta-guidelines/. Accessed 11 June 2019.
Pharmaceutical Benefits Advisory Committee (PBAC). The pharmaceutical benefits advisory committee guidlines. 2016. https://pbac.pbs.gov.au/information/printable-version-of-guidelines.html. Accessed 11 June 2019.
Moller J, Davis S, Stevenson M, Caro JJ. Validation of a DICE simulation against a discrete event simulation implemented entirely in code. Pharmacoeconomics. 2017;35(10):1103–9. https://doi.org/10.1007/s40273-017-0534-0.
Philips Z, Ginnelly L, Sculpher M, Claxton K, Golder S, Riemsma R, et al. Review of guidelines for good practice in decision-analytic modelling in health technology assessment. Health Technol Assess. 2004;8(36):1–158. https://doi.org/10.3310/hta8360.
No external funding was received for the development of the model or manuscript.
Conflict of interest
Rose Hart, Darren Burns, Bram Ramaekers, Shijie Ren, Daniel Gladwell, Will Sullivan, Niall Davison, Owain Saunders, Indeg Sly, Theresa Cain and Dawn Lee declare they have no conflicts of interest relevant to the content of this manuscript.
About this article
Cite this article
Hart, R., Burns, D., Ramaekers, B. et al. R and Shiny for Cost-Effectiveness Analyses: Why and When? A Hypothetical Case Study. PharmacoEconomics 38, 765–776 (2020). https://doi.org/10.1007/s40273-020-00903-9