A Need for Change! A Coding Framework for Improving Transparency in Decision Modeling
The use of open-source programming languages, such as R, in health decision sciences is growing and has the potential to facilitate model transparency, reproducibility, and shareability. However, realizing this potential can be challenging. Models are complex and primarily built to answer a research question, with model sharing and transparency relegated to being secondary goals. Consequently, code is often neither well documented nor systematically organized in a comprehensible and shareable approach. Moreover, many decision modelers are not formally trained in computer programming and may lack good coding practices, further compounding the problem of model transparency. To address these challenges, we propose a high-level framework for model-based decision and cost-effectiveness analyses (CEA) in R. The proposed framework consists of a conceptual, modular structure and coding recommendations for the implementation of model-based decision analyses in R. This framework defines a set of common decision model elements divided into five components: (1) model inputs, (2) decision model implementation, (3) model calibration, (4) model validation, and (5) analysis. The first four components form the model development phase. The analysis component is the application of the fully developed decision model to answer the policy or the research question of interest, assess decision uncertainty, and/or to determine the value of future research through value of information (VOI) analysis. In this framework, we also make recommendations for good coding practices specific to decision modeling, such as file organization and variable naming conventions. We showcase the framework through a fully functional, testbed decision model, which is hosted on GitHub for free download and easy adaptation to other applications. The use of this framework in decision modeling will improve code readability and model sharing, paving the way to an ideal, open-source world.
We thank Mr Caleb Easterly for his helpful comments and suggestions on the code developed for this framework and Dr Myriam Hunink for her overall contribution in the DARTH workgroup.
FAE, EK, PP, HJ SYK, AY, and EE: study design and analysis. All authors participated in the interpretation of the data, drafting of the manuscript, critical revision of the manuscript, and approval of the final manuscript.
Compliance with Ethical Standards
Data availability statement
Data and statistical code are provided in the GitHub repository (https://github.com/DARTH-git/darthpack) and the darthpack website (https://darth-git.github.io/darthpack). The version of darthpack released in this article is available at https://doi.org/10.5281/zenodo.3445451.
Dr Alarid-Escudero was supported by a Grant from the National Cancer Institute (U01-CA-199335) as part of the Cancer Intervention and Surveillance Modeling Network (CISNET). Dr Enns was supported by a Grant from the National Institute of Allergy and Infectious Disease of the National Institutes of Health under award no. K25AI118476. Dr Jalal was supported by a Grant from the National Institute of Health (KL2 TR0001856). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. The funding agencies had no role in the design of the study, interpretation of results, or writing of the manuscript. The funding agreement ensured the authors’ independence in designing the study, interpreting the data, writing, and publishing the report.
Conflict of interest
FAE reports no conflicts of interest. EK reports no conflicts of interest. PP reports no conflicts of interest. HJ reports no conflicts of interest. SYK reports no conflicts of interest. AY reports no conflicts of interest. EE reports no conflicts of interest.
- 1.Taichman DB. Data sharing statements for clinical trials: a requirement of the international committee of medical journal editors. Ann Intern Med. 2017;14:e1002315–e1002315.Google Scholar
- 2.Stanford. Data Availability Policies at Top Journals [Internet]. 2019. https://web.stanford.edu/~cy10/public/data/Data_Availability_Policies.pdf. Accessed 2 Aug 2019.
- 5.Baio G, Heath A. When simple becomes complicated: why Excel should lose its place at the top table. Glob Reg Heal Technol Assess. 2017;4(1):e3–6.Google Scholar
- 6.Canadian Agency for Drugs and Technologies in Health (CADTH). Procedure and Submission Guidelines for the CADTH Common Drug Review. 2019. p. 110. https://www.cadth.ca/sites/default/files/cdr/process/Procedure_and_Guidelines_for_CADTH_CDR.pdf. Accessed 17 Sept 2019.
- 7.Center for the Evaluation of Value and Risk in Health. Open-source model clearinghouse [Internet]. Tufts University Medical Center; 2019. http://healtheconomics.tuftsmedicalcenter.org/orchard/open-source-model-clearinghouse. Accessed 1 Feb 2019.
- 12.Decision Analysis in R for Technologies in Health (DARTH) workgroup. Decision analysis in R for technologies in health [Internet]. 2019. http://darthworkgroup.com. Accessed 1 Jan 2019.
- 13.R Core Team. R: A language and environment for statistical computing. Vienna: R Foundation for Statistical Computing; 2019.Google Scholar
- 23.Cancer Intervention and Surveillance Modelling Network (CISNET). About CISNET [Internet]. 2019. https://cisnet.cancer.gov/about/index.html. Accessed 16 July 2019.
- 24.Krijkamp EM, Alarid-Escudero F, Enns E, Pechlivanoglou P, Hunink MM, Jalal H. A multidimensional array representation of state-transition model dynamics. bioRxiv 670612. 2019.Google Scholar
- 25.Sculpher MJ, Basu A, Kuntz KM, Meltzer DO. Reflecting uncertainty in cost-effectiveness analysis. In: Neumann PJ, Sanders GD, Russell LB, Siegel JE, Ganiats TG, editors. Cost-effectiveness heal med. 2nd ed. New York: Oxford University Press; 2017. p. 289–318.Google Scholar
- 27.Raiffa H, Schlaifer RO. Applied statistical decision theory. Cambridge: Harvard Business School; 1961.Google Scholar
- 30.Wickham H. R packages: organize, test, document, and share your code. Spencer A, Marie Beaugureau, editors. Sebastopol: O’Reilly Media; 2015.Google Scholar
- 31.Cooper N, Hsing P-Y, editors. A guide to reproducible code in ecology and evolution. London: British Ecology Society; 2017.Google Scholar
- 33.Wickham H. The tidyverse style guide [Internet]. 2019. https://style.tidyverse.org. Accessed 19 July 2019.
- 34.Google. Google’s R Style Guide [Internet]. 2019. p. 1–6. https://google.github.io/styleguide/Rguide.xml. Accessed 24 July 2019.
- 35.Martin RC. Clean code: a handbook of agile software craftsmanship. Boston: Pearson Education; 2009.Google Scholar
- 37.Beeley C. Web application development with R using Shiny. Birmingham: Packt Publishing Ltd; 2013.Google Scholar
- 41.RStudio. Using projects [Internet]. 2019. https://support.rstudio.com/hc/en-us/articles/200526207-Using-Projects. Accessed 1 Feb 2019.