FormalPara Key Points for Decision Makers

The application of Open Science practices is gaining attention in all scientific disciples, including health economics and outcomes research (HEOR).

Current HEOR professionals are not often trained to apply these principles, which emphasises the need to train future generations of HEOR professionals to achieve sustainable changes in the practice of HEOR.

The current paper describes the design, implementation, and evaluation of a teaching innovation aiming at introducing Open Science practices to Master’s degree students.

1 Introduction

Health economics and outcomes research (HEOR) is an applied research field focusing on informing health policy decisions through qualitative and quantitative value assessments of (new) healthcare technologies and interventions. The results of HEOR assessments may therefore have important consequences for society, for example HEOR assessments inform decisions concerning the reimbursement of new promising pharmaceuticals for patients with unmet clinical need. However, HEOR assessments are not easily accessible for all members of society. This may lead to incomprehension and reluctance to accept decisions taken based on the results of these assessments. Applying the Open Science philosophy when performing HEOR assessments may improve their transparency and thereby improve the acceptance of their results among society. Open Science aims at making research processes and results more transparent, reusable, reproducible and accessible for all members of society [1].

The practice of Open Science is gaining increasing attention in all research fields, including HEOR, in which Open Science practices are implemented or championed through initiatives such as the ISPOR’s Open Source Models (OSMs) Special Interest Group [2], the Innovation and Value Initiative [3], the Decision Analysis in R for Technology in Health [4] and the PeerModels Network [5]. These initiatives mainly focus on the development, use, and dissemination of OSMs. They do not however address one of the core issues in implementing Open Science and OSMs practices in HEOR, which is the lack of formal education in the Open Science philosophy and practices within academic HEOR-related curricula [6, 7]. While individual HEOR professionals may work according to the Open Science philosophy out of their own interest, systemic changes in the practice of Open Science within HEOR will not be achieved without (a) reforming how HEOR professionals are educated [8] and (b) rewarding adherence to the Open Science philosophy in the context of career development.

Since education plays a central role in shaping HEOR research and practice, it is crucial to embed the Open Science philosophy within HEOR-related curricula. Therefore, the aim of this manuscript is to present the design, implementation, and evaluation of a teaching innovation focusing on introducing Open Science and Open Source Modelling theory and practices during a health economic modelling Masters’ course.

2 The Context

The teaching innovation was implemented in 2023 during the 10-weeks Masters’ course ‘Advanced Simulation for Health Economic Analysis’ (ASHEA), which is taught at the University of Twente, Enschede, the Netherlands, as part of the Healthcare Technology and Management (HCTM) specialisation of the Industrial Engineering and Management (IEM) Master’s programme. During the IEM Master programme, students learn to use programming languages to perform statistical analysis and develop simulation models to assess the efficiency of production processes and improve these. The HCTM specialisation of this programme focuses on improving healthcare processes and delivery. In this specialization, students follow courses related to statistical learning, advanced simulation (ASHEA), and healthcare decision making. None of the courses on this programme provided an introduction to Open Science or OSMs before the implementation of this teaching innovation.

During ASHEA, students learn to develop a discrete event simulation (DES) model using the statistical software R [9]. Following lectures and tutorials, they work in pairs on a project assignment focusing on developing a DES model to assess the health economic impact of a new biomarker combination for monitoring response during cancer treatment. The main product that is assessed in this course is a project report which should contain a description and justification of the choices they made during the development of the DES model and reflections on lessons learned during the course.

3 The Teaching Innovation

The research proposal for this project has been published elsewhere [10]. Changes made to this proposal during the project are described in Supplementary Material 1. Ethical approval for this research has been obtained from the Ethics Committee from the Faculty of Behavioural, Management and Social Sciences of the University of Twente (request number 221418).

3.1 Design

To formally include Open Science within ASHEA, the learning objective ‘Apply open-source modelling practices in the context of health economic modelling’ was added to the already-existing learning objectives of the course.

To design the teaching innovation, an electronic survey was sent to HEOR professionals and previous ASHEA course participants (N = 22) to determine which aspects of Open Science and OSM this teaching innovation should focus on. The survey was designed by XP and was tested for comprehensiveness and completeness by a PhD student in HEOR (Supplementary Material 2A). The survey was spread among HEOR professionals through a LinkedIn post and through a HEOR professional newsletter in the Netherlands. Previous ASHEA course participants were sent the survey via email. The first part of the survey asked respondents to mention the length of their professional career, their work environment and whether they develop and/or review health economic (HE) models as part of their daily work activities. The second part of the survey asked the respondents which Open Science-related skills they believe should be taught to future HEOR professionals. Responses to this second part of the survey were used to decide on which topics the teaching innovation should focus. Finally, respondents could add general feedback and comments on the survey and on teaching Open Science in general.

Four previous ASHEA course participants (18%) and 19 HEOR professionals returned the survey. Due to the limited number of respondents, survey results from both groups were combined and are discussed together. Most respondents worked within a consultancy (N = 9, 39%) or academic environment (N = 8, 35%). Thirteen (56%) had professional career of 5 years or shorter. Most respondents neither developed (N = 15, 65%), nor reviewed (N = 14, 61%) HE models during their daily work activities (Table 1). According to the 23 respondents, the most important OS-related skills were (1) being able to use a script-based statistical software, (2) script annotation and documentation, (3) sharing software code, (4) communicating your work to a lay audience, and (5) publishing open access (Table 2). Based on these results, the teaching activities described in Table 3 were developed. The slides of these teaching activities are available at the following link https://doi.org/10.5281/zenodo.8214973 for interested (HEOR) teachers who may want to include these activities in their own teaching. Table 3 also contains references to selected relevant resources that may be used to design teaching activities on these Open Science topics.

Table 1 Demographic characteristics of the respondents to the survey on important (HEOR) Open Science-related skills
Full size table
Table 2 Most important Open Science-related skills for future HEOR professionals according to survey respondents
Full size table
Table 3 Overview of the developed teaching activities
Full size table

3.2 Implementation

The teaching innovation started with a 2 h lecture introducing the rationale behind and the concept of Open Science. Additionally, this lecture introduced course participants to Open Science and OSM-related initiatives within HEOR. All other topics were introduced during shorter lectures of 15–30 min at the beginning of the hands-on tutorial sessions that were scheduled during the course. At the end of the course, the student pairs had to present how they would design and evaluate a public outreach activity to disseminate the process of developing their HE model and their results to a non-scientific audience. Each pair could choose which public outreach activity to design from the following list: developing a R shiny interactive application, a game, an infographic, a podcast, a TED-talk-like presentation, social media posts, a blog, or a video. The best presentation was elected by the students, according to criteria that they had formulated themselves. These criteria were gathered by the teaching staff before the presentations and communicated to the students before the start of the presentations. The criteria that students formulated to elect the best presentation related to the content and the form of the presentation, and to the impact on the societal behaviour of the public outreach activity. The content-related criteria were that the content should be (a) concrete and directly applicable in practice and (b) complete, thus containing all relevant aspects. The form-related criteria were the clarity of content, the attractiveness of the presentation, the duration, the engagement with the audience and whether there was a demonstration of the public outreach activity.

To formally assess whether participants achieved the learning objective related to Open Science, pairs of students had to write a reflection on Open Science in relation to HE modelling and their project in the final project report. This reflection had to include a description of how participants had included Open Science practices during their project and describe its importance in the HE context. Besides, participants had to select one Open Science practice that they thought was less valuable for HE modelling and motivate their choice. The actual implementation of the Open Science practices in their project was not graded separately.

3.3 Evaluation

The effectiveness of the teaching innovation was assessed using multiple methods. Firstly, a survey was distributed to participants (N = 10) at the beginning and end of the course to assess their knowledge of Open Science (Supplementary Material 2B). Participants were asked to define Open Science. We assessed whether the participants’ definition contained the following Open Science concepts mentioned in the United Nations Educational, Scientific and Cultural Organization (UNESCO) recommendation on Open Science [1] (Box 1):

  • Openly available,

  • Openly accessible,

  • Reusable,

  • Increase collaboration,

  • Sharing information (with society),

  • Open the processes of scientific knowledge creation (transparency),

  • Open the processes of scientific knowledge evaluation,

  • Communication to societal actors beyond the traditional scientific community,

  • Open scientific knowledge,

  • Open science infrastructures,

  • Science communication,

  • Open engagement of societal actors.

figure a

Box 1: UNESCO’s definition of Open Science

In the same survey, attitudes of participants towards scientific knowledge and the participants’ opinions concerning researchers’ compliance to Open Science philosophy and practices were also elicited before and after the teaching innovation using five-point Likert scales (Supplementary Material 2B; Figs. 2, 3). Participants were asked to rate their proficiency level concerning Open Science-related skills before and after the teaching innovation using the following scale: ‘No knowledge at all’, ‘Basic knowledge’, ‘Advanced knowledge’, ‘Expert’. The included Open Science-related skills were the same as those included in the survey distributed among HEOR professionals and previous ASHEA course participants (Table 2). The survey was concluded with an open field where participants could write general feedback and comments. Frequency plots of the participants’ answers to the Likert scales were drawn to assess the participants’ attitudes towards Science, their opinions concerning researchers’ compliance with Open Science principles and their self-reported proficiency level in the different Open Science-related skills.

Secondly, informal group discussions to evaluate the relevance and quality of the Open Science-related teaching activities were organised after half of the Open Science-related teaching activities had taken place and at the end of the course. XP asked the following questions to the group during these evaluation sessions: ‘I find the selected Open Science-related topics relevant for my professional development’, ‘The content of the Open Science-related activities is too complicated | just right | or too easy’ (multiple-choice question), ‘Are there any Open Science-related topics you are curious about and would like to discuss in a following session?’, ‘Would you like to have more practical assignment to apply these Open Science principles to your own health economic model? (Yes/No)’ and ‘Do you have any further feedback?’. During these evaluations XP took notes which he summarised.

Seven (70%) participants filled in the survey before and after the teaching innovation. When comparing the participants’ definition of Open Science to the UNESCO definition, the following concepts were more often mentioned after the teaching innovation: openly available, reusability, sharing information with society, transparency, communication beyond scientific communities and open scientific knowledge (Fig. 1). The concepts increase collaboration, science communication and open engagement of societal actors were not mentioned in any participants’ definitions.

Fig. 1
figure 1

Comparison of the proportion of participants’ definitions of Open Science (before and after the teaching innovation) mentioning concepts from the UNESCO definition of Open Science. OS, Open Science; after, after the teaching innovation; before, before the teaching innovation

Full size image

Concerning the participants’ attitudes towards scientific knowledge, less participants agreed after the teaching innovation compared with before the teaching innovation with the statement that ‘all scientific knowledge is produced according to open and transparent processes’ and that ‘open-source publishing of HE models is common practice’ (Fig. 2). After the teaching innovation, less participants indicated that (HEOR) researchers complied with Open Science practices, such as pre-registering their study, sharing software code and public outreach to a non-scientific audience (Fig. 3).

Fig. 2
figure 2

Participants’ opinions towards scientific knowledge before and after the teaching innovation. A I trust the results described in scientific journal articles. B All scientific knowledge is produced according to an open and transparent process. C Studies contesting the research hypothesis (negative results) are published as often as studies confirming the research hypothesis. D When a health economic evaluation complies to reporting guidelines, it means that it is highly reproducible. E Researchers are rewarded for performing transparent and reproducible research. F Open-source publishing of health economic evaluations is common practice. G Industry-sponsorship influences the results of health economic evaluations

Full size image
Fig. 3
figure 3

Participants’ opinions concerning (HEOR) researchers’ compliance with Open Science principles and practices before and after the teaching innovation. A Researchers pre-register research hypothesis and analysis plan before performing the research in question. B Researchers share data openly. C Researchers share software code used for data analysis. D Researchers take measures to ensure the reproducibility of their research. E Researchers involve stakeholders in scientific knowledge production (i.e. citizen science). F Researchers communicate scientific knowledge to lay audience or perform any type of public outreach

Full size image

Finally, participants reported less often that they had ‘No knowledge at all’ of Open Science-related skills after the teaching innovation compared with before (except for the skill ‘identifying fake news’). The skills for which at least 50% of participants rated their proficiency level as ‘Advanced’ after the teaching innovation were (1) using a script-based programming language, (2) script annotation and documentation and (3) collaborative writing (Fig. 4).

Fig. 4
figure 4

Self-reported participants’ level in Open Science-related skills before and after the teaching innovation. A Being able to use a script-based statistical software, e.g. R, Python, … B Pre-registering your analysis plan. C Writing a data and/or software management plan. D Version controlling (script, software, …). E Code/script annotation and documentation. F Licensing software code. G Collaborative (software) script development. H Identifying and reusing open data. I Creating reproducible reports, e.g. using (R) Markdown or Jupiter notebook. J Creating interactive applications, e.g. R-Shiny. K Collaborative writing. L Sharing software code. M Sharing data. N Communicating your work to a lay audience. O Communicating your work through social media. P Critically review someone else’s work and share the review openly. Q Archiving your work in an open online repository. R Adhering to the findable, accessible, interoperable, reusable (FAIR) principles. S Publishing open access. T Identifying fake news

Full size image

During the group discussions aiming at evaluating the Open Science-related teaching activities, the participants mentioned that the topics addressed during these activities were interesting and relevant for their development as researchers. However, participants mentioned that Open Science encompasses many different concepts and that some of these concepts require knowledge of different digital tools. The practical use of all these digital tools was not yet clear for all participants, most likely because these digital tools were introduced to the participants but they did not need to use them all during the course. Additionally, participants mentioned that they enrolled in this specific course to learn about developing a DES model, rather than for the Open Science-related teaching. Finally, they indicated that the Open Science topics would fit better within the context of a more general course of the programme, such as ‘Research Orientation’ where students learn generic research skills and get acquainted with the different specialisations of the Master’s programme, instead of within a specific simulation course.

4 Discussion

The current teaching innovation aimed at introducing Master’s degree students to the Open Science philosophy and practices during an existing course focusing on learning how to develop a DES model. The teaching innovation has led to an increased awareness among participants of what Open Science entails, and of their Open Science-related skills, such as being able to use a script-based statistical software, annotating and documenting one’s script, and publishing open access.

To the best of our knowledge, this is one of the first reported teaching innovations focusing on teaching Open Science and OSMs within HEOR, even though the amount of open resources for teaching open and transparent HE analyses is increasing. For instance, recent publications describe tutorials to develop HE models in R [4] and R Shiny applications linked to HE models [11], packaging HE model developed in R to improve their transparency [12], a framework to increase the transparency of HE models developed in R [13], and the development of an OSM dedicated to teaching the concepts of HE evaluations [14].

During the teaching innovation, we noticed limited engagement of students in implementing Open Science practices. This may be explained by the lack of grading of the implementation of Open Science practices in the participants’ project since we only asked them to reflect on Open Science in their final project report. Our results are in concordance with a previous teaching innovation on Open Science introduced in a course from a sociology bachelor programme [15]. During this teaching innovation, students engaged moderately with the Open Science component of a course because it was not compulsory [15].

Originally, we wanted to convey information on how Open Science could be implemented in all phases of the research cycle, and let students apply each of the practices through small practical assignments. However, we realised during the development of the teaching activities and during teaching that this plan was too ambitious to achieve within a time frame of 10 weeks (the duration of the course) and within an existing five European Credit Transfer and Accumulation System (ECTS)-points course. Hence, the teaching activities we designed were focused more on introducing the Open Science philosophy and practices than on applying these practices. A broader and structural implementation of Open Science within academic curricula is therefore needed if we want students to become open (HEOR) scientists and practitioners. This was also indicated by the participants during the evaluation of this teaching evaluation. They mentioned that Open Science education did not entirely fit the focus of the ASHEA course but would fit better within a course focusing on general research skills.

For the ASHEA course, we are currently considering shifting the focus from a general introduction to Open Science to a more comprehensive introduction and application of OSM only. This change of focus would permit to dedicate more time on teaching students to develop skills such as version control, code documentation, reproducible computational research, open peer review and using digital tools such as Github and R packaging functionalities. Narrowing the Open Science focus to developing an OSM does not mean other Open Science principles are less important, but would better align the Open Science teaching with the learning goals of ASHEA.

4.1 Limitations

The number of respondents to all surveys and the number of participants in the course were limited, which prohibited formal statistical analyses of the survey responses. The reported results should therefore be interpreted with caution. Additionally, most HEOR professionals who responded to the survey did not develop or review HE models during their daily work activities and may not be aware of necessary skills to develop OSMs. Due to time constraints, we could not provide extensive explanations of all Open Science-related topics and had to make choices concerning which topics (not) to address.

4.2 Conclusions

This practical application describes the design, implementation and evaluation of a teaching innovation focusing on teaching the Open Science philosophy and practices within an existing HE modelling course. During the teaching innovation, students had to reflect on how they adhered to Open Science practices when performing the project related to this course (developing a DES model) and had to design a public outreach activity to disseminate the results of HE models to a broader non-scientific audience.

Properly introducing Open Science to students requires a more structural embedding of Open Science philosophy and practices within academic curricula. This would allow students to develop (a) a critical mindset concerning science and (b) the required skills to become open researchers and practitioners. During next editions of ASHEA, we aim to shift the focus on developing and using OSMs instead of Open Science in general. Narrowing the Open Science focus during ASHEA may result in a better integration of Open Science practices within our HE modelling course.

We hope that through open reporting of this teaching innovation, the materials used and the results obtained, others can build on our experience and more teaching innovations focusing on the Open Science philosophy and practices will find their way into academic curricula.