A Comparison of Four Software Programs for Implementing Decision Analytic Cost-Effectiveness Models
The volume and technical complexity of both academic and commercial research using decision analytic modelling has increased rapidly over the last two decades. The range of software programs used for their implementation has also increased, but it remains true that a small number of programs account for the vast majority of cost-effectiveness modelling work. We report a comparison of four software programs: TreeAge Pro, Microsoft Excel, R and MATLAB. Our focus is on software commonly used for building Markov models and decision trees to conduct cohort simulations, given their predominance in the published literature around cost-effectiveness modelling. Our comparison uses three qualitative criteria as proposed by Eddy et al.: “transparency and validation”, “learning curve” and “capability”. In addition, we introduce the quantitative criterion of processing speed. We also consider the cost of each program to academic users and commercial users. We rank the programs based on each of these criteria. We find that, whilst Microsoft Excel and TreeAge Pro are good programs for educational purposes and for producing the types of analyses typically required by health technology assessment agencies, the efficiency and transparency advantages of programming languages such as MATLAB and R become increasingly valuable when more complex analyses are required.
KeywordsHealth Technology Assessment Integrate Development Environment Health Technology Assessment Agency Debug Tool Markov Tree
Compliance with Ethical Standards
This study was funded through grants from the Canadian Institutes for Health Research (CIHR) and Genome Canada. Christopher McCabe is supported through a Capital Health Research Chair in Emergency Medicine Research.
Conflicts of interest
Mike Paulden and Christopher McCabe have taught introductory courses on decision modelling using Microsoft Excel, but have no relationships with the developer and have received no financial benefits for using this software to teach these courses. Petros Pechlivanoglou has taught introductory courses on decision modelling using R and has contributed to decision modelling courses that use TreeAge, but has no relationships with any of the developers and has received no financial benefits for using this software to teach these courses. Chase Hollman, Mike Paulden, Petros Pechlivanoglou and Christopher McCabe have no other potential conflicts of interest to report.
Mike Paulden built the TreeAge model used for the benchmark comparisons and rebuilt this model using Microsoft Excel. Chase Hollman rebuilt this model in MATLAB and R, with support from Petros Pechlivanoglou, and conducted the benchmarking exercise. Christopher McCabe supervised the project. Chase Hollman wrote the first draft of the manuscript. All authors contributed to subsequent drafts of the manuscript, responses to peer review, and preparation of the manuscript for publication.
Data availability statement
We have provided the models used in our benchmarking exercise as supplementary material.
- 1.Tosh J, Wailoo A. Review of Software for Decision Modelling. Report by the NICE Decision Support Unit. 2008. http://www.nicedsu.org.uk/PDFs%20of%20reports/softwarereport-final.pdf. Accessed 1 May 2017.
- 3.Davis S, Stevenson M, Tappenden P, Wailoo A. NICE DSU technical support document 15: cost-effectiveness modelling using patient-level simulation. Report by the NICE Decision Support Unit. 2014. http://www.nicedsu.org.uk/TSD15_Patient-level_simulation.pdf. Accessed 1 May 2017.
- 5.Microsoft Corporation. Increase the productivity of Users’ with enhanced Office.js APIs in Office 2016. 2015. https://dev.office.com/blogs/Office-js-Public-Preview. Accessed 1 May 2017.
- 7.The Document Foundation. LibreOffice. 2017. https://www.libreoffice.org/. Accessed 1 May 2017.
- 8.The Apache Software Foundation. OpenOffice. 2017. https://www.openoffice.org/. Accessed 1 May 2017.
- 9.The Document Foundation. Frequently asked questions - Calc. 2016. https://wiki.documentfoundation.org/Faq/Calc. Accessed 1 May 2017.
- 10.Microsoft Corporation. Excel specifications and limits. 2017. https://support.office.com/en-us/article/Excel-specifications-and-limits-1672b34d-7043-467e-8e27-269d656771c3. Accessed 1 May 2017.
- 11.Botchkarev A. Assessing Excel VBA Suitability for Monte Carlo Simulation. Spreadsheets Educ (eJSiE). 2015;8(2):3.Google Scholar
- 12.Microsoft Corporation. Description of the RAND function in Excel. 2011. https://support.microsoft.com/en-us/help/828795/description-of-the-rand-function-in-excel. Accessed 1 May 2017.
- 13.Microsoft Corporation. Multithreaded recalculation in excel. 2012. https://msdn.microsoft.com/en-us/library/office/bb687899.aspx. Accessed 1 May 2017.
- 14.Microsoft Corporation. Excel 2010 performance: Tips for optimizing performance obstructions. 2011. https://msdn.microsoft.com/en-us/library/office/ff726673(v=office.14).aspx. Accessed 1 May 2017.
- 15.Microsoft Corporation. Microsoft Store—Microsoft Office. 2017. https://www.microsoftstore.com/store/msusa/en_US/list/Office/categoryID.71148700. Accessed 1 May 2017.
- 16.TreeAge Software Inc. Products—TreeAge Software. 2017. https://www.treeage.com/shop/. Accessed 1 May 2017.
- 17.TreeAge Software Inc. TreeAge Pro 2017 User’s Manual. 2017. http://files.treeage.com/treeagepro/17.1.0/20170109/TP-Manual-2017R1.pdf. Accessed 1 May 2017.
- 18.Hollenberg J. Markov cycle trees: a new representation for complex markov processes. Med Decis Making. 1984;4:529–30.Google Scholar
- 19.Claxton K, Eggington S, Ginnelly L, et al. A pilot study of value of information analysis to support research recommendations for NICE. CHE Research Paper 4. York: Centre for Health Economics, University of York; 2005. https://www.york.ac.uk/media/che/documents/papers/researchpapers/rp4_Pilot_study_of_value_of_information_analysis.pdf. Accessed 1 May 2017.
- 21.Matsumoto M. Mersenne Twister Home Page. 2011. http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/emt.html. Accessed 1 May 2017.
- 22.GNU Octave. About GNU Octave. 2017. https://www.gnu.org/software/octave/about.html. Accessed 1 May 2017.
- 23.Mathworks Inc. File Exchange. 2017. https://www.mathworks.com/matlabcentral/fileexchange/. Accessed 1 May 2017.
- 24.Octave-Forge. Extra packages for GNU Octave. 2017. https://octave.sourceforge.io/packages.php. Accessed 1 May 2017.
- 25.Mathworks Inc. Products and Services. 2017. https://www.mathworks.com/products.html. Accessed 1 May 2017.
- 26.Mathworks Inc. SimEvents. 2017. https://www.mathworks.com/products/simevents.html. Accessed 1 May 2017.
- 27.Mathworks Inc. Data Types. 2017. https://www.mathworks.com/help/matlab/data-types_data-types.html. Accessed 1 May 2017.
- 28.Mathworks Inc. Mathworks Store. 2017. https://www.mathworks.com/store/. Accessed 1 May 2017.
- 29.R Foundation for Statistical Computing. What is R? 2017. https://www.r-project.org/about.html. Accessed 1 May 2017.
- 30.Revolution Analytics. Revolutions: popularity. 2017. http://blog.revolutionanalytics.com/popularity/. Accessed 1 May 2017.
- 31.Foundation for Open Access Statistics. Journal of Statistical Software. 2017. https://www.jstatsoft.org/index. Accessed 1 May 2017.
- 32.Microsoft Corporation. Microsoft R Open: The Enhanced R Distribution. 2017. https://mran.microsoft.com/open/. Accessed 1 May 2017.
- 33.Eddelbuettel D. CRAN Task View: High-Performance and Parallel Computing with R. 2017. https://cran.r-project.org/web/views/HighPerformanceComputing.html. Accessed 1 May 2017.
- 35.Sculpher M, Drummond M, Buxton M. Economic evaluation in health care research and development: undertake it early and often. Health Economics Research Group discussion paper no. 12. London: Brunel University, HERG; 1995.Google Scholar
- 39.NHS England Cancer Drugs Fund Team. Appraisal and funding of cancer drugs from July 2016 (including the new Cancer Drugs Fund): a new deal for patients, taxpayers and industry. 2016. https://www.england.nhs.uk/wp-content/uploads/2013/04/cdf-sop.pdf. Accessed 1 May 2017.