# A Comparison of Four Software Programs for Implementing Decision Analytic Cost-Effectiveness Models

## Abstract

The volume and technical complexity of both academic and commercial research using decision analytic modelling has increased rapidly over the last two decades. The range of software programs used for their implementation has also increased, but it remains true that a small number of programs account for the vast majority of cost-effectiveness modelling work. We report a comparison of four software programs: TreeAge Pro, Microsoft Excel, R and MATLAB. Our focus is on software commonly used for building Markov models and decision trees to conduct cohort simulations, given their predominance in the published literature around cost-effectiveness modelling. Our comparison uses three qualitative criteria as proposed by Eddy et al.: “transparency and validation”, “learning curve” and “capability”. In addition, we introduce the quantitative criterion of processing speed. We also consider the cost of each program to academic users and commercial users. We rank the programs based on each of these criteria. We find that, whilst Microsoft Excel and TreeAge Pro are good programs for educational purposes and for producing the types of analyses typically required by health technology assessment agencies, the efficiency and transparency advantages of programming languages such as MATLAB and R become increasingly valuable when more complex analyses are required.

## Keywords

Health Technology Assessment Integrate Development Environment Health Technology Assessment Agency Debug Tool Markov Tree## Notes

## Compliance with Ethical Standards

## Funding

This study was funded through grants from the Canadian Institutes for Health Research (CIHR) and Genome Canada. Christopher McCabe is supported through a Capital Health Research Chair in Emergency Medicine Research.

## Conflicts of interest

Mike Paulden and Christopher McCabe have taught introductory courses on decision modelling using Microsoft Excel, but have no relationships with the developer and have received no financial benefits for using this software to teach these courses. Petros Pechlivanoglou has taught introductory courses on decision modelling using R and has contributed to decision modelling courses that use TreeAge, but has no relationships with any of the developers and has received no financial benefits for using this software to teach these courses. Chase Hollman, Mike Paulden, Petros Pechlivanoglou and Christopher McCabe have no other potential conflicts of interest to report.

## Author contributions

Mike Paulden built the TreeAge model used for the benchmark comparisons and rebuilt this model using Microsoft Excel. Chase Hollman rebuilt this model in MATLAB and R, with support from Petros Pechlivanoglou, and conducted the benchmarking exercise. Christopher McCabe supervised the project. Chase Hollman wrote the first draft of the manuscript. All authors contributed to subsequent drafts of the manuscript, responses to peer review, and preparation of the manuscript for publication.

## Data availability statement

We have provided the models used in our benchmarking exercise as supplementary material.

## Supplementary material

## References

- 1.Tosh J, Wailoo A. Review of Software for Decision Modelling. Report by the NICE Decision Support Unit. 2008. http://www.nicedsu.org.uk/PDFs%20of%20reports/softwarereport-final.pdf. Accessed 1 May 2017.
- 2.Menn P, Holle R. Comparing three software tools for implementing markov models for health economic evaluations. Pharmacoeconomics. 2009;27:745–53.CrossRefPubMedGoogle Scholar
- 3.Davis S, Stevenson M, Tappenden P, Wailoo A. NICE DSU technical support document 15: cost-effectiveness modelling using patient-level simulation. Report by the NICE Decision Support Unit. 2014. http://www.nicedsu.org.uk/TSD15_Patient-level_simulation.pdf. Accessed 1 May 2017.
- 4.Jalal H, Pechlivanoglou P, Krijkamp E, Alarid-Escudero F, Enns E, Hunink MG. An overview of R in health decision sciences. Med Decis Making. 2017. doi: 10.1177/0272989X16686559.Google Scholar
- 5.Microsoft Corporation. Increase the productivity of Users’ with enhanced Office.js APIs in Office 2016. 2015. https://dev.office.com/blogs/Office-js-Public-Preview. Accessed 1 May 2017.
- 6.Edlin R, McCabe C, Hulme C, et al. Cost effectiveness modelling for health technology assessment: a practical course. Heidelberg: Springer; 2015.CrossRefGoogle Scholar
- 7.The Document Foundation. LibreOffice. 2017. https://www.libreoffice.org/. Accessed 1 May 2017.
- 8.The Apache Software Foundation. OpenOffice. 2017. https://www.openoffice.org/. Accessed 1 May 2017.
- 9.The Document Foundation. Frequently asked questions - Calc. 2016. https://wiki.documentfoundation.org/Faq/Calc. Accessed 1 May 2017.
- 10.Microsoft Corporation. Excel specifications and limits. 2017. https://support.office.com/en-us/article/Excel-specifications-and-limits-1672b34d-7043-467e-8e27-269d656771c3. Accessed 1 May 2017.
- 11.Botchkarev A. Assessing Excel VBA Suitability for Monte Carlo Simulation. Spreadsheets Educ (eJSiE). 2015;8(2):3.Google Scholar
- 12.Microsoft Corporation. Description of the RAND function in Excel. 2011. https://support.microsoft.com/en-us/help/828795/description-of-the-rand-function-in-excel. Accessed 1 May 2017.
- 13.Microsoft Corporation. Multithreaded recalculation in excel. 2012. https://msdn.microsoft.com/en-us/library/office/bb687899.aspx. Accessed 1 May 2017.
- 14.Microsoft Corporation. Excel 2010 performance: Tips for optimizing performance obstructions. 2011. https://msdn.microsoft.com/en-us/library/office/ff726673(v=office.14).aspx. Accessed 1 May 2017.
- 15.Microsoft Corporation. Microsoft Store—Microsoft Office. 2017. https://www.microsoftstore.com/store/msusa/en_US/list/Office/categoryID.71148700. Accessed 1 May 2017.
- 16.TreeAge Software Inc. Products—TreeAge Software. 2017. https://www.treeage.com/shop/. Accessed 1 May 2017.
- 17.TreeAge Software Inc. TreeAge Pro 2017 User’s Manual. 2017. http://files.treeage.com/treeagepro/17.1.0/20170109/TP-Manual-2017R1.pdf. Accessed 1 May 2017.
- 18.Hollenberg J. Markov cycle trees: a new representation for complex markov processes. Med Decis Making. 1984;4:529–30.Google Scholar
- 19.Claxton K, Eggington S, Ginnelly L, et al. A pilot study of value of information analysis to support research recommendations for NICE. CHE Research Paper 4. York: Centre for Health Economics, University of York; 2005. https://www.york.ac.uk/media/che/documents/papers/researchpapers/rp4_Pilot_study_of_value_of_information_analysis.pdf. Accessed 1 May 2017.
- 20.Ades AE, Lu G, Claxton K. Expected value of sample information calculations in medical decision modeling. Med Decis Making. 2004;24:207–27.CrossRefPubMedGoogle Scholar
- 21.Matsumoto M. Mersenne Twister Home Page. 2011. http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/emt.html. Accessed 1 May 2017.
- 22.GNU Octave. About GNU Octave. 2017. https://www.gnu.org/software/octave/about.html. Accessed 1 May 2017.
- 23.Mathworks Inc. File Exchange. 2017. https://www.mathworks.com/matlabcentral/fileexchange/. Accessed 1 May 2017.
- 24.Octave-Forge. Extra packages for GNU Octave. 2017. https://octave.sourceforge.io/packages.php. Accessed 1 May 2017.
- 25.Mathworks Inc. Products and Services. 2017. https://www.mathworks.com/products.html. Accessed 1 May 2017.
- 26.Mathworks Inc. SimEvents. 2017. https://www.mathworks.com/products/simevents.html. Accessed 1 May 2017.
- 27.Mathworks Inc. Data Types. 2017. https://www.mathworks.com/help/matlab/data-types_data-types.html. Accessed 1 May 2017.
- 28.Mathworks Inc. Mathworks Store. 2017. https://www.mathworks.com/store/. Accessed 1 May 2017.
- 29.R Foundation for Statistical Computing. What is R? 2017. https://www.r-project.org/about.html. Accessed 1 May 2017.
- 30.Revolution Analytics. Revolutions: popularity. 2017. http://blog.revolutionanalytics.com/popularity/. Accessed 1 May 2017.
- 31.Foundation for Open Access Statistics. Journal of Statistical Software. 2017. https://www.jstatsoft.org/index. Accessed 1 May 2017.
- 32.Microsoft Corporation. Microsoft R Open: The Enhanced R Distribution. 2017. https://mran.microsoft.com/open/. Accessed 1 May 2017.
- 33.Eddelbuettel D. CRAN Task View: High-Performance and Parallel Computing with R. 2017. https://cran.r-project.org/web/views/HighPerformanceComputing.html. Accessed 1 May 2017.
- 34.Eddy DM, Hollingworth W, Caro JJ, et al. Model transparency and validation: a report of the ISPOR-SMDM modeling good research practices task force-7. Med Decis Making. 2012;32:733–43.CrossRefPubMedGoogle Scholar
- 35.Sculpher M, Drummond M, Buxton M. Economic evaluation in health care research and development: undertake it early and often. Health Economics Research Group discussion paper no. 12. London: Brunel University, HERG; 1995.Google Scholar
- 36.Paulden M, Franek J, Pham B, et al. Cost-effectiveness of the 21-gene assay for guiding adjuvant chemotherapy decisions in early breast cancer. Value Health. 2013;16:729–39.CrossRefPubMedGoogle Scholar
- 37.Sadatsafavi M, Bansback N, Zafari Z, Najafzadeh M, Marra C. Need for speed: an efficient algorithm for calculation of single-parameter expected value of partial perfect information. Value Health. 2013;16(2):438–48.CrossRefPubMedGoogle Scholar
- 38.Strong M, Oakley JE, Brennan A. Estimating multiparameter partial expected value of perfect information from a probabilistic sensitivity analysis sample: a nonparametric regression approach. Med Decis Making. 2014;34(3):311–26.CrossRefPubMedGoogle Scholar
- 39.NHS England Cancer Drugs Fund Team. Appraisal and funding of cancer drugs from July 2016 (including the new Cancer Drugs Fund): a new deal for patients, taxpayers and industry. 2016. https://www.england.nhs.uk/wp-content/uploads/2013/04/cdf-sop.pdf. Accessed 1 May 2017.