Skip to main content
SpringerLink
Log in

Barriers and Facilitators of Using R for Decision Analytic Modeling in Health Technology Assessment: Focus Group Results

  • Original Research Article
  • Published:
PharmacoEconomics Aims and scope Submit manuscript

Abstract

Background and Objective

Decision models for health technology assessment (HTA) are largely submitted to HTA agencies using commercial software, which has known limitations. The use of the open-source programming language R has been suggested because of its efficiency, transparency, reproducibility, and ability to consider complex analyses. However, its use in HTA remains limited. This qualitative study aimed to explore the main reasons for this slow uptake of R in HTA and identify tangible facilitators.

Methods

We undertook two semi-structured focus group discussions with 24 key stakeholders from government agencies, consultancy, pharmaceutical companies, and academia. Two 1.5-hour discussions reflected on barriers identified in a previous study and highlighted additional barriers. Discussions were recorded and semi-transcribed, and data were organized and summarized into key themes.

Results

Human resources constraints were identified as a key barrier, including a lack of training, prioritization and collaboration, and resistance to change. Another key barrier was the lack of acceptance, or clear guidance, around submissions in R by HTA agencies. Participants also highlighted a lack of communication around accepted packages and decision model structures, and between HTA agencies on standard decision modeling structures.

Conclusions

There is a need for standardization, which can facilitate decision model sharing, coding homogeneity, and improved country adaptations. The creation of training materials and tailored workshops was identified as a key short-term facilitator. Increased communication and engagement of stakeholders could also facilitate the use of R by identifying needs and opportunities, encouraging HTA agencies to address structural barriers, and increasing incentives to use R.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

References

  1. Henshall C, Mardhani-Bayne L, Frønsdal KB, Klemp M. Interactions between health technology assessment, coverage, and regulatory processes: emerging issues, goals, and opportunities. Int J Technol Assess Health Care. 2011;27(3):253–60. https://doi.org/10.1017/S0266462311000262.

    Article  PubMed  Google Scholar 

  2. International Network of Agencies for Health Technology Assessment (INAHTA), Health Technology Assessment International (HTAi); 2006. Health technology assessment (HTA) glossary. http://www.htaglossary.net [Accessed 27 Jun 2023].

  3. O’Donnell JC, Pham SV, Pashos CL, Miller DW, Smith MD. Health technology assessment: lessons learned from around the world: an overview. Value Health. 2009. https://doi.org/10.1111/j.1524-4733.2009.00550.x.

    Article  PubMed  Google Scholar 

  4. Philips Z, Bojke L, Sculpher M, Claxton K, Golder S. Good practice guidelines for decision-analytic modelling in health technology assessment: a review and consolidation of quality assessment. Pharmacoeconomics. 2006;24(4):355–71. https://doi.org/10.2165/00019053-200624040-00006.

    Article  PubMed  Google Scholar 

  5. Kunst N, Jackson C, Heath A. Chapter 1. Health economic modelling. In: Value of information for health economic evaluations. Boca Raton: CRC Press; 2023. p. 1–371.

    Google Scholar 

  6. Briggs A, Schulpher M, Claxton K. Decision modelling for health economic evaluation, vol. 1. New York: Oxford University Press Inc.; 2006. p. 1–256.

    Book  Google Scholar 

  7. Weinstein MC, O’Brien B, Hornberger J, Jackson J, Johannesson M, McCabe C, et al. Principles of good practice for decision analytic modeling in health-care evaluation: report of the ISPOR Task Force on Good Research Practices—modeling studies. Value Health. 2003;6(1):9–17. https://doi.org/10.1046/j.1524-4733.2003.00234.x.

    Article  PubMed  Google Scholar 

  8. Microsoft Corporation. Microsoft Excel. Redmond; 2019. https://office.microsoft.com/excel [Accessed 14 Aug 2023].

  9. Hollman C, Paulden M, Pechlivanoglou P, McCabe C. A comparison of four software programs for implementing decision analytic cost-effectiveness models. Pharmacoeconomics. 2017;35(8):817–30. https://doi.org/10.1007/s40273-017-0510-8.

    Article  PubMed  Google Scholar 

  10. TreeAge Pro. TreeAge Software, LLC. Williamstown; 2020. http://www.treeage.com [Accessed 14 Sep 2023].

  11. Incerti D, Thom H, Baio G, Jansen JP. R you still using Excel? The advantages of modern software tools for health technology assessment. Value Health. 2019;22(5):575–9. https://doi.org/10.1016/j.jval.2019.01.003.

    Article  PubMed  Google Scholar 

  12. Menn P, Holle R. Comparing three software tools for implementing Markov models for health economic evaluations. Pharmacoeconomics. 2009;27(9):745–53. https://doi.org/10.2165/11313760-000000000-00000.

    Article  PubMed  Google Scholar 

  13. Tosh J, Wailoo A. Review of software for decision modelling. National Institute for Health and Care Excellence (NICE); 2008. pp. 1–18. https://www.ncbi.nlm.nih.gov/books/NBK425836/ [Accessed 26 Jul 2023].

  14. Baio G, Heath A. When simple becomes complicated: why Excel should lose its place at the top table. Glob Reg Health Technol Assess. 2017;4(1): 5000247. https://doi.org/10.5301/grhta.5000247.

    Article  Google Scholar 

  15. Feenstra T, Corro-Ramos I, Hamerlijnck D, van Voorn G, Ghabri S. Four aspects affecting health economic decision models and their validation. Pharmacoeconomics. 2022;40:241–8. https://doi.org/10.1007/s40273-021-01110-w.

    Article  PubMed  Google Scholar 

  16. Sampson CJ, Arnold R, Bryan S, Clarke P, Ekins S, Hatswell A, et al. Transparency in decision modelling: what, why, who and how? Pharmacoeconomics. 2019;37:1355–69. https://doi.org/10.1007/s40273-019-00819-z.

    Article  PubMed  PubMed Central  Google Scholar 

  17. Hart R, Burns D, Ramaekers B, Ren S, Gladwell D, Sullivan W, et al. R and Shiny for cost-effectiveness analyses: why and when? A hypothetical case study. Pharmacoeconomics. 2020;38(7):765–76. https://doi.org/10.1007/s40273-020-00903-9.

    Article  PubMed  Google Scholar 

  18. Alarid-Escudero F, Krijkamp EM, Pechlivanoglou P, Jalal H, Kao SYZ, Yang A, et al. A need for change! A coding framework for improving transparency in decision modeling. Pharmacoeconomics. 2019;37(11):1329–39. https://doi.org/10.1007/s40273-019-00837-x.

    Article  PubMed  PubMed Central  Google Scholar 

  19. R Development Core Team. The R Project for statistical computing; 2021. https://www.r-project.org/ [Accessed 11 Aug 2023].

  20. Jalal H, Pechlivanoglou P, Krijkamp E, Alarid-Escudero F, Enns E, Hunink MGM. An overview of R in health decision sciences. Med Decis Making. 2017;37(7):735–46. https://doi.org/10.1177/0272989X16686559.

    Article  PubMed  Google Scholar 

  21. Pechlivanoglou P, Enns E, Alarid-Escudero F, Krijkamp E, Jalal H, Yang A, et al. Decision Analysis in R for Technologies in Health (DARTH) Group; 2023. Learn with us. Upcoming workshops and short courses. http://darthworkgroup.com/workshops/ [Accessed 17 Sep 2023].

  22. International Society for Pharmacoeconomics and Outcomes Research (ISPOR); 2023. HEOR education at ISPOR. https://www.ispor.org/education-training [Accessed 17 Sep 2023].

  23. Toronto Health Economics and Technology Assessment (THETA) Collaborative; 2023. THETA rounds. https://theta.utoronto.ca/theta-rounds [Accessed 17 Sep 2023].

  24. Canada’s Drug and Health Technology Agency (CADTH). Procedures for the CADTH common drug review and interim plasma protein product review; 2020. https://www.cadth.ca/sites/default/files/cdr/process/Procedure_and_Guidelines_for_CADTH_CDR.pdf [Accessed 28 Jul 2023].

  25. Dew K. A health researcher’s guide to qualitative methodologies. Aust N Z J Public Health. 2007;31(5):433–7. https://doi.org/10.1111/j.1753-6405.2007.00114.x.

    Article  PubMed  Google Scholar 

  26. Bouma GD, Ling R, Wilkinson L. The research process. 2nd ed. Oxford: Oxford University Press; 2012. p. 1–304.

    Google Scholar 

  27. Shaikh M, Cornford T. Navigating open source adoption in the public sector. In: 18th Americas Conference on Information Systems (AMCIS); 2012. pp. 2564–71. http://aisel.aisnet.org/amcis2012/proceedings/EGovernment/15. Accessed 29 Mar 2024.

  28. Coleman P. In-depth interviewing as a research method in healthcare practice and education: value, limitations and considerations. Int J Caring Sci. 2019;12(3):1879.

    Google Scholar 

  29. Sage Publications. Analyzing focus group results; 2015. https://us.sagepub.com/sites/default/files/upm-assets/65005_book_item_65005.pdf [Accessed 29 Mar 2024].

  30. Hennink MM, Leavy P. Writing focus group results. Understanding focus group discussions. Oxford: Oxford University Press; 2015. https://doi.org/10.1093/acprof:osobl/9780199856169.003.0004.

    Book  Google Scholar 

  31. Parker C, Scott S, Geddes A. Snowball sampling. Thousand Oaks: SAGE Publications Ltd, SAGE Research Methods Foundations; 2019.

    Google Scholar 

  32. Khusid A. MIRO. RealtimeBoard, Inc.; 2011. https://miro.com [Accessed 27 Jun 2023].

  33. Naylor NR, Williams J, Green N, Lamrock F, Briggs A. Extensions of health economic evaluations in R for Microsoft Excel users: a tutorial for incorporating heterogeneity and conducting value of information analyses. Pharmacoeconomics. 2023;41(1):21–32.

    Article  PubMed  Google Scholar 

  34. Green N, Lamrock F, Naylor N, Williams J, Briggs A. Health economic evaluation using Markov models in R for Microsoft Excel users: a tutorial. Pharmacoeconomics. 2023;41(1):5–19. https://doi.org/10.1007/s40273-022-01199-7.

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

We acknowledge all focus group participants.

Author information

Author notes

  1. Anna Heath and Petros Pechlivanoglou contributed equally.

Authors and Affiliations

  1. Child Health Evaluative Sciences, Peter Gilgan Centre for Research and Learning, The Hospital for Sick Children, Toronto, ON, Canada

    Yanara Marks, Anna Heath & Petros Pechlivanoglou

  2. University of California, Davis, CA, USA

    Jeffrey S. Hoch

  3. Division of Biostatistics, Dalla Lana School of Public Health, University of Toronto, Toronto, ON, Canada

    Anna Heath

  4. Department of Statistical Science, University College London, London, UK

    Anna Heath

  5. IHPME, Dalla Lana School of Public Health, University of Toronto, Toronto, ON, Canada

    Petros Pechlivanoglou

Authors
  1. Yanara Marks
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jeffrey S. Hoch
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Anna Heath
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Petros Pechlivanoglou
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yanara Marks.

Ethics declarations

Funding

This research activity was made possible with support from the Child Health Evaluative Sciences program at the Hospital for Sick Children through an unrestricted educational grant from PricewaterhouseCoopers. Anna Heath is funded by Canada Research Chair in Statistical Trial Design; Natural Sciences and Engineering Research Council of Canada (award No. RGPIN-2021-03366).

Conflict of Interest

Yanara Marks, Jeffrey S. Hoch, Anna Heath, and Petros Pechlivanoglou have no competing interests or other interests that might be perceived to influence the results and/or discussion reported in this paper.

Ethics Approval

The Hospital for Sick Children granted Research Ethics Board (REB#1000081313) Secondary Use Approval.

Consent to Participate

All participants were invited to attend the focus group discussions and attended voluntarily.

Consent for Publication

Participants who did not consent for publication were invited to leave the discussion.

Data Availability

De-identified focus group discussion data have been provided as supplementary material.

Code Availability

Not applicable.

Author Contributions

PP and AH conceived the research idea. YM, PP, and AH designed the focus group sessions. JH facilitated the focus groups. YM transcribed the data and drafted the manuscript. PP, JH, and AH reviewed and edited the manuscript. All authors approved the final version for publication.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (XLSX 33 KB)

Supplementary file2 (PDF 659 KB)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Marks, Y., Hoch, J.S., Heath, A. et al. Barriers and Facilitators of Using R for Decision Analytic Modeling in Health Technology Assessment: Focus Group Results. PharmacoEconomics (2024). https://doi.org/10.1007/s40273-024-01374-y

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s40273-024-01374-y

Search

Navigation