Skip to main content

Benefits, Challenges and Potential Strategies of Open Source Health Economic Models

Introduction

A common criticism of the pharmaceutical industry is that it fails to achieve transparency [1, 2]. To allay concerns, the industry needs to collaborate openly with stakeholders to make medicines widely available and affordable—this will mean companies need to address some of their practices [1, 3, 4]. We suggest that an important practice to address is the presentation of health economic models.

Formal health economic models are commonly used to support decision makers when deciding whether funding particular pharmaceuticals will improve the health of a population [5, 6]. The value of a formal model lies not only in its results but also in the revelation of the assumptions about both the data and the logical relationships it embodies [6]. Presented properly, the model provides clarity on key issues.

The value of adopting the correct modelling methodology has been recognised by national and international bodies who have issued guidance for developing models; for example, the reports generated from the recent US Panel on Cost-Effectiveness in Health and Medicine [7] and the collaboration between the International Society for Pharmacoeconomics and Outcomes Research and the Society of Medical-Decision Making (ISPOR-SMDM) [8, 9]. Similarly, the European Network for Health Technology Assessment (EUnetHTA) is also working towards international level collaboration, providing a core health technology assessment (HTA) model framework and a guide to best practices [10]. Multiple guidelines can produce some inconsistencies [5, 1014] but they have helped produce some clarity in dealing with uncertainty and validating and reporting models transparently [9]. Despite these welcome developments, individual models developed in the same therapeutic area may be significantly different from one another. Occasionally, efforts have been made to standardise them [15]. However, standardisation is not the norm and, in our experience, a model code is rarely shared.

It is not just pharmaceutical companies that have been challenged regarding the level of information shared. For example, the UK National Institute of Health and Care Excellence and the US Institute for Clinical and Economic Review are involved in debates around the extent their health economic models should be made publicly available [16, 17]. To facilitate this transparency, there is suggestion in the literature that open source modelling is desirable [8, 9]. The definition of open source can vary by discipline [1821]. An open source health economic model is defined here as one that is available to those who wish to access it. This means that the model, its underlying code and a written report describing its aim, methods, structure and results would all be accessible [8]. This could be in a fully public space (e.g. freely downloadable on the Internet) or only readily available to those who request it with conditions attached to access.

An open source culture allows existing models to be updated to answer new research questions and decision problems and creates a transparent public arena for model validation, education and collaboration across research, industry and healthcare communities [4, 8, 9]. There are few examples of published fully open source models; Sullivan et al. is a recent example in pain therapy where the code was made available in both Excel and R [22].

The objective of this letter is to highlight the paucity of examples of open source models in the literature and to report on a short piece of research undertaken to gain insight into the appetite of researchers to provide and use open source models.

Exploratory Expert Survey

An exploratory double-blind survey of health economic stakeholders was undertaken between 25 April, 2016 and 6 May, 2016 to assess whether there is any desire for open source health economic models. To gain access to a wide variety of people with experience and an active interest in health economics, the LinkedIn platform was used to access the relevant groups: ISPOR; HTA in Europe (maintained by EUnetHTA); and Institute for Medical Technology Assessment.

Eight out of the ten survey questions were quantitative, while two were qualitative, with the opportunity for the respondents to populate open text boxes. Within the qualitative responses, a framework analysis was carried out to identify themes based on the respondent’s answers.

Despite the short time the survey was open, 46 participants responded, and 35 completed the whole survey. Respondents worked predominantly in the pharmaceutical industry (35%), academia (33%) and government (20%), and the remaining worked in other fields such as independent research, non-government organisations, community pharmacy and hospitals. Among the respondents, 33/41 who answered (81%) had experience with one or more health economic models in their careers. Among the participants, 18/41 (44%) had both requested information and received a request for information about a model that was not publicly available. There were a number of reasons that the participants needed the information; Fig. 1 shows the most frequent (24/41; 59%) was learning about the technical aspects of the model for use in a different disease or decision problem. Across all stakeholders and countries, making health economic models available in an open format was considered beneficial; 34/35 (97%) of participants said that it would be occasionally or very beneficial to have access to the code when reviewing existing models.

Fig. 1
figure 1

Reasons for needing access to health economic models (41 responded, who could choose up to six answers that applied to them)

One of the survey questions focused on if respondents ever experienced challenges in accessing health economic models. Among the respondents, 26/41 (63%) had experienced challenges in accessing full details of a health economic model. Challenges included ownership and confidentiality issues, such as a lack of willingness to share and not knowing who to contact or being unable to contact someone to request information on a health economic model. Two qualitative questions were asked: what was their perceived value of improving access to health economic models and what were the best strategies to enable sharing. Figure 2 highlights the most common key themes mentioned by respondents relating to the perceived benefits.

Fig. 2
figure 2

Frequently mentioned benefits of open source health economic models

The best strategies to encourage the sharing of health economic models as suggested by the respondents included:

  • Collaboration: different stakeholders working together to develop open platforms or libraries to encourage sharing and to maximise the social value of their research; examples were journals, HTA agencies or consultants actively involved in publishing and sharing their health economic models and promoting a sense of sharing and value of open source.

  • Transparency: such as authors providing supplementary materials and highlighting relevant sources where information was gathered.

  • Confidentiality: finding ways to share models while keeping the necessary proprietary aspects confidential.

  • Changes to regulatory processes: potentially integrated open access processes across organisations and formal request processes by health authorities.

  • Consistent use of language: making models easier to understand across stakeholders and countries.

Discussion

The limited results of this exploratory analysis supported viewpoints found within the available guidelines and limited literature. Across all stakeholders and countries in the survey, making health economic models available in an open format is expected to be beneficial within industry, academia, government, non-government organisations or hospitals. This is consistent with arguments made by Fazio et al. that transparency with models are desired by various stakeholders [23]. The ISPOR-SMDM guidelines (specifically the ISPOR guidelines on model transparency and validation [8]) and the EUnetHTA joint action projects support the views of respondents on collaboration, transparency, confidentiality, processes and consistency, while the literature explores how best to improve this [4, 810]. It is also consistent with a proposal by Afzali et al. that there should be “reference models” that are built by an expert group for a particular therapy area and then made available to pharmaceutical companies for drug submissions [24]. However, we caution that any reference model approach would need to allow stakeholders the ability to debate, challenge and change the model as appropriate.

There were inconsistent qualitative responses from some participants who wanted tighter confidentiality yet more open access. This could be partially owing to the often high-profile and emotional nature of healthcare allocation decisions. This means policy makers and other stakeholders may not want to share all information in fear of it complicating the debate or hindering current or future decisions, as well as an organisation’s specific needs for not wanting to be fully transparent. The ISPOR-SMDM guidelines address some of these issues. They outline that there is potentially a conflict between the scientific desirability of making all methodological and technical details of a model available to peer reviewers and to other researchers and the need to protect intellectual property generated by substantial investments in the development of a model [8]. Rejecting the latter would significantly reduce the incentive to create models, and intellectual property rights cannot be ignored. One possible solution is to make a full technical documentation available within whatever agreements right holders feel are necessary to grant them adequate protection [8]. This should allow for detailed review of any model by other researchers, provided they accept the confidentiality restrictions [8]. We suggest that this is often the most practical solution in the short term. Despite challenges, there are already examples of open source models that are fully available in the public space [22].

Conclusion

The study reported here was small and lacks generalisability. However, it did demonstrate that among the respondents there is an appetite for greater use of open source models and the topic is deserving of further exploration. Future investigation could be particularly beneficial if it also focuses on the obstacles to transparency and how they may be overcome.

References

  1. 1.

    Dunlop W. Painting the picture: an open source health economics modelling initiative. Market Access, Pricing and Reimbursement Conference, London; 24 Feb 2016.

  2. 2.

    Eldessouki R, Dix Smith M. Health care system information sharing: a step toward better health globally. Value Health Reg Issues; 1(1):118–20.

  3. 3.

    Obama B. United States health care reform: progress to date and next steps. JAMA. 2016;316(5):525–32.

    Article  PubMed  PubMed Central  Google Scholar 

  4. 4.

    Meteos. Principles for collaborative, mutually acceptable drug pricing. 2016. http://www.meteos.co.uk/wp-content/uploads/PHARMADIPLOMACY-REPORT-low-res.pdf. Accessed 27 Nov 2016.

  5. 5.

    National Institute for Health and Care Excellence. Methods for the development of NICE public health guidance. 3rd ed. 2012. https://www.nice.org.uk/process/pmg4/chapter/introduction. Accessed Jul 2016.

  6. 6.

    Weinstein MC, O’Brien B, Hornberger J, et al. Principles of good practice for decision analytic modeling in health-care evaluation: report of the ISPOR task force on good research practices: modeling studies. Value Health. 2003;6(1):9–17.

    Article  PubMed  Google Scholar 

  7. 7.

    Sanders GD, Neumann PJ, Basu A, et al. Recommendations for conduct, methodological practices, and reporting of cost-effectiveness analyses: Second panel on cost-effectiveness in health and medicine. JAMA. 2016;316(10):1093–103.

    Article  PubMed  Google Scholar 

  8. 8.

    Eddy DM, Hollingworth W, Caro JJ, et al. Model transparency and validation: a report of the ISPOR-SMDM modeling good research practices task force-7. Med Decis Mak. 2012;32(5):733–43.

    Article  Google Scholar 

  9. 9.

    Caro JJ, Briggs AH, Siebert U, Kuntz KM. Modeling good research practices: overview. A report of the ISPOR-SMDM modeling good research practices task force-1. Value Health. 2012;15(6):796–803.

    Article  PubMed  Google Scholar 

  10. 10.

    EUnetHTA. HTA core model. 2016. http://www.eunethta.eu/hta-core-model. Accessed 9 Sep 2016.

  11. 11.

    International Society for Pharmacoeconomics and Outcomes Research. ISPOR health outcomes metrics index of open source code. 2016. http://www.ispor.org/opensourceindex/index.aspx. Accessed 25 Aug 2016.

  12. 12.

    Massetti M, Aballéa S, Videau Y, et al. A comparison of HAS & NICE guidelines for the economic evaluation of health technologies in the context of their respective national health care systems and cultural environments. J Mark Access Health Policy. 2015;12:3. doi:10.3402/jmahp.v3.24966.

    Google Scholar 

  13. 13.

    INAHTA. Guidelines for the economic evaluation of health technologies: Canada. 2006. http://www.inahta.org/wp-content/themes/inahta/img/AboutHTA_Guidelines_for_the_Economic_Evaluation_of_Health_Technologies.pdf. Accessed 23 Sep 2016.

  14. 14.

    Zechmeister-Koss I, Schnell-Inderst P, Zauner G. Appropriate evidence sources for populating decision analytic models within health technology assessment (HTA): a systematic review of HTA manuals and health economic guidelines. Med Decis Mak. 2014;34(3):288–99.

    Article  Google Scholar 

  15. 15.

    IMS. IMS CORE diabetes model user group forum. 2013. http://www.core-diabetes.com/. Accessed 6 Oct 2016.

  16. 16.

    ICER. Comments received on ICER’s value assessment framework. 2016. https://icer-review.org/methodology/icers-methods/icer-value-assessment-framework/ Accessed 12 Oct 2016.

  17. 17.

    Poole C, Agrawal S, Currie CJ. Let cost effectiveness models be open to scrutiny. BMJ. 2007;335(7623):735.

    Article  PubMed  PubMed Central  Google Scholar 

  18. 18.

    Perens B. The open source definition. 1999. http://www.oreilly.com/openbook/opensources/book/perens.html. Accessed 31 Aug 2016.

  19. 19.

    Initiative OS. The open source definition (annotated). https://opensource.org/osd.html. Accessed 31 Aug 2016.

  20. 20.

    PHACTS O. About open PHACTS. 2013. http://www.openphacts.org/about-open-phacts. Accessed 31 Aug 2016.

  21. 21.

    Harnad S. The green road to open access: a leveraged transition. 2007. http://eprints.soton.ac.uk/265753/1/greenroad.html. Accessed 31 Aug 2016.

  22. 22.

    Sullivan W, Hirst M, Beard S, et al. Economic evaluation in chronic pain: a systematic review and de novo flexible economic model. Eur J Health Econ. 2016;17(6):755–70.

    CAS  Article  PubMed  Google Scholar 

  23. 23.

    Fazio L, Rosner A, Drummond MF. How do U.S. payers use economic models submitted by life sciences organizations? Value Outcomes Spotlight. 2016;2(2):18–21.

  24. 24.

    Ali Afzali HH, Karnon J. Addressing the challenge for well informed and consistent reimbursement decisions. Pharmacoeconomics. 2011;29(10):823–5.

    Article  Google Scholar 

Download references

Acknowledgements

The authors thank all participants in the survey.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Nicola Mason.

Ethics declarations

Funding

Mundipharma International Limited provided funding to BresMed Health Solutions to conduct the research.

Conflict of interest

William C. N. Dunlop and James Kenworthy are employees of Mundipharma International. Ron Akehurst and Nicola Mason are employees of BresMed Health Solutions.

Author contributions

All of the authors were involved in the project design, the survey development, analysis and writing of this research letter.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution-NonCommercial 4.0 International License (http://creativecommons.org/licenses/by-nc/4.0/), which permits any noncommercial use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Dunlop, W.C.N., Mason, N., Kenworthy, J. et al. Benefits, Challenges and Potential Strategies of Open Source Health Economic Models. PharmacoEconomics 35, 125–128 (2017). https://doi.org/10.1007/s40273-016-0479-8

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40273-016-0479-8

Keywords

  • Open Source
  • Health Technology Assessment
  • Qualitative Response
  • Health Economic Model
  • Health Technology Assessment Agency