Introduction

To structurally promote a healthy lifestyle among students, an increasing number of Dutch primary, secondary, and secondary vocational schools have obtained the Healthy School (HS) program certificate [1]. To date, however, limited scientific knowledge is available on the effectiveness of the HS program. Previous studies mostly focused on the short-term effects of initiatives belonging to one of the pillars (health education, school environment, referral, and school policy) within a specific health topic (nutrition, physical activity, well-being, smoking, alcohol and drug prevention, relationships and sexuality, physical safety, environment and nature, and media literacy) [2,3,4]. Additionally, for these national and international studies, it is difficult to draw unambiguous conclusions about the effectiveness of (initiatives similar to) the HS program [5, 6]. This can be partly explained by the limited understanding of the factors, characteristics, and processes, i.e., the conditions, that strengthen the implementation and intended outcomes of the HS program [5,6,7].

The Netherlands Organization for Health Research and Development (ZonMw) is therefore subsidizing a national evaluation study on the conditions for effectiveness of the HS program. The study aims not only to investigate whether the HS program matters, but, above all, to find starting points for improving the national and regional approaches as well. A broad consortium, consisting of three academic living labs (Academic Collaborative Center for Public Health Limburg, Academic Collaborative Center AGORA, and Academic Collaborative Center AMPHI), the Netherlands Organization for Applied Scientific Research (TNO) and nine regional Public Health Services (PHSs), will be in charge of conducting the four-year study (start date: April 1, 2019). This article describes the research design, which focuses on three levels: the student, the school, and the region. Although the focus is on the HS program, the concept of “healthy school” will be considered more broadly. After all, a school without an HS program certificate might still implement aspects of school health promotion.

The main research question is: Under what conditions does the HS program matter? The conceptual framework in Fig. 1 forms the basis of the study and has guided the formulation of three research questions:

  1. 1.

    What is the degree of development of students regarding lifestyle, health, and personal development and educational performance and school absenteeism, and how and to what extent can differences between schools be explained by factors in the school context?

  2. 2.

    What is the degree of implementation of the HS program in schools, and how and to what extent can differences between schools be explained by (interaction between) factors in the school context and regional professional support?

  3. 3.

    What is the degree of regional professional support in the nine PHS regions, and how and to what extent can differences between regions be explained by (interaction between) factors in the school and regional contexts?

Fig. 1
figure 1

Preliminary version of the conceptual framework for the evaluation of the Healthy School Program in the Netherlands

Full size image

Conceptual framework

The conceptual framework distinguishes three levels, namely that of the student, the school, and the region. The school and the region are considered to be so-called complex adaptive systems [7,8,9,10,11,12,13,14,15]. This means they are systems made up of many components, which are constantly interacting with each other. These components include the people in the system (the actors) and the factors and characteristics of the system. Changes in one part (e.g., implementation of an HS initiative) lead to changes in other parts or are resisted, in an effort to stay balanced as a whole [8, 12, 14]. For this purpose, there is constant feedback between the components [12]. This process is unpredictable since small efforts may have large effects and vice versa [8, 14]. Therefore, in each system, i.e., in every school or region, there is a unique context [7,8,9,10,11,12,13,14]. Based on this theoretical vision, (the preliminary version of) the conceptual framework provides a visualization of the expected process of professional support at the regional level, its implementation at the school level, and outcomes at the student level (Fig. 1).

The school is central to the framework and is visualized by the circle in the middle. The lower part represents the implementation of the HS program, as part of the context of the complex adaptive school system. The upper part shows the actors, factors, and characteristics of the school context that are important for implementation of the program. Circular arrows indicate continuous feedback between the two parts to keep the school system in balance [7, 16]. The wavy line in the school system shows that there is no strict boundary between the parts, but that they flow into each other (for example, when an HS activity is organized by a group of teachers). Actors in the school context include teachers, students, parents, management, and care coordinators. Relevant factors may be directly related to these actors (e.g., students’ work ethic or teachers’ pedagogical competencies for the HS program), but also to the school as an organization (e.g., the number of hours available for the HS program and coordination by management) and the sociopolitical environment (laws and regulations regarding HS) [16]. In addition, general school characteristics (e.g., school size) and, if applicable, HS program characteristics (e.g., which topic certificates have been obtained) determine the school context [17].

The PHS region is shown at the top of the framework. It is assumed that the implementation process of the HS program in schools does not only depend on factors in the school context, but also on the professional support schools receive [18,19,20,21]. Top-down input from the national HS program, consisting of financial and practical support, enters the complex adaptive PHS regional system. This reaches the school through an HS adviser from the PHS or directly via the website [22]. The lower part of the regional system shows the level of support that has been offered to schools. The upper part shows the actors, factors, and characteristics of the context. Circular arrows again indicate constant feedback between these two parts to maintain balance in the system. The wavy line shows that the parts overlap (for example, when a PHS professional supports a school). Furthermore, the overlap between the regional and school systems assumes reciprocal influence between the regional support and the school context, represented by circular arrows. Actors in the regional context include the PHS, municipalities, Youth Health Care Services, local sports companies, environmental organizations, and addiction prevention. Four categories of contextual factors were selected based on existing literature [18, 23,24,25]. These are factors related to the professional (e.g., attitude toward HS program support), the organization (including available hours for support), the broader context (e.g., shared vision on regional prevention policies), and the collaboration (e.g., method of communication). The regional context is also determined by general characteristics (e.g., the number of collaborative partners) and HS-related characteristics (e.g., the year of adoption of HS program support).

The framework further assumes a relationship between the HS program and student-level outcomes. Indicators of these outcomes are based on the objectives of the program: 1) healthy lifestyle, 2) health and personal development, and 3) educational performance and school absenteeism [26]. This is shown in the bottom section of the framework. It is hypothesized that the HS program is associated with lifestyles related to the topics of the program, including fruit and vegetable consumption, physical activity, substance use, and sexual risk behavior [6, 27]. This might in turn improve general and psychosocial health [28,29,30]. Previous research further demonstrates that unhealthy lifestyles and deteriorated health may be associated with poorer educational performance, such as lower grade point averages, increased likelihood of grade repetition, and school absenteeism [2, 31,32,33]. Therefore, it is hypothesized that the HS program also leads to improvements of the other indicators via improvements in lifestyle [6, 26, 29, 33]. This is a complex process of reciprocal influence on lifestyle, health, and educational performance that is shaped by the home, the school, and the broader environment [34,35,36]. This reciprocal influence is represented by circular arrows between the three indicators. In addition, the impact of the HS program at the student level may vary by school, in part due to differences in HS program implementation [37]. However, even schools that implement the program in similar ways may vary in outcomes, as these may also be influenced by the factors in the school context described earlier [38, 39]. The impact of these possible moderators is visualized by the arrow on the left side of the framework.

Method

Conditions for student outcomes

To answer the first research question, two substudies will be conducted (Fig. 1 and Tab. 1). For substudy 1.1, retrospective analyses of the development of student outcomes and characteristics of schools and students that contribute to explaining the differences found between schools will be conducted. For this purpose, existing datasets from, among others, participating PHSs and Statistics Netherlands (CBS) that contain indicators regarding lifestyle, health and personal development, and/or educational performance/school absenteeism of students will be used. Usable datasets consist of student-level data from both schools with and without HS program certification and include an indicator to determine which students were in the same school during that period. For primary schools, in particular, digitalized data from the PHSs of ~5- and 10-year-olds from multiple birth cohorts will be used. This mainly involves two types of data: 1) growth data (height and weight), which are used to calculate BMI z‑scores, and 2) outcomes of the Strengths and Difficulties Questionnaire (SDQ), which is used to measure psychosocial problems in children and adolescents [40]. For secondary schools, data from the national Youth Health Monitor for 2015 and 2019, filled out by second and fourth graders, will be used primarily [41]. These include data on health and lifestyle (including dietary intake, physical activity, substance use, and sexual risk behavior), personal development (including SDQ), and school absenteeism. Data on educational performance for primary and secondary schools will be obtained through the Education Executive Agency (DUO) and Statistics Netherlands, among others. For secondary vocational schools, the possibilities are still being investigated.

Table 1 Overview of data collection per substudy
Full size table

For substudy 1.1, methods will be applied that allow reasonable statements to be made about associations between the HS program and the intended outcomes. These could include multilevel regression analyses or quasi-experimental methods. The final choice of quantitative analysis strategies will depend on the characteristics of the available data on students and schools. We will first identify the variation across schools for the intended outcomes and then examine to what extent differences can be explained by school characteristics (e.g., school type), school population (e.g., students’ educational performance), and HS program characteristics (e.g., which topic certificates were obtained). To analyze data from different sources as a whole, data harmonization techniques will be used. These are techniques that allow different data sources to be combined prior to statistical analyses [42]. To deal with missing data, both at the individual and school level, multiple imputation will be used [43]. To account for the nested structure of the data, we will use multilevel imputation [44].

Next, for substudy 1.2, school conditions for student outcomes will be examined in more detail. Based on data from substudy 1.1, a total of ~50 schools for primary, secondary, and secondary vocational education that perform relatively better or worse on student outcomes will be selected. Within this selection, matching will be applied based on school population characteristics, such as the educational track in secondary schools, where possible. The follow-up analysis will consist of predominantly qualitative research. Data on school characteristics (e.g., facilities), school population (e.g., teacher attitudes), and HS program characteristics (e.g., the degree of implementation) will be collected through interviews with school staff, parents, and students, document analysis, observations, and—if not yet filled out by the school—the developed HS implementation questionnaire (see substudy 2.1). All data will be analyzed with Qualitative Comparative Analysis (QCA) to explain differences between schools. QCA is a method by which complex relationships between many factors of different groups (in this case, schools) can be examined [45].

Conditions for implementation at schools

To answer the second research question, two substudies will be conducted (Fig. 1 and Tab. 1). Substudy 2.1 will map the degree of HS program implementation in schools. For this purpose, a questionnaire will be developed for school staff. During the development process, experts in school health promotion (n = 10–15) from academia, PHSs, schools, and the HS program will be asked to define the degree of HS implementation in semi-structured interviews. This will be done using seven concepts: 1) adherence to the principles (by HS topic), 2) the dosage of HS program implementation at the school, 3) the quality of delivery, 4) the degree to which actors are involved, 5) the degree to which the HS program contains unique elements to make a difference, 6) the degree of HS program adaptation to fit the school context, and 7) the degree to which the HS program is part of their DNA (the routines, norms and identity) [46,47,48,49,50,51]. Using the interview data, a questionnaire is drafted, which is then further developed during an online expert consultation (n = ~40 experts in school health promotion) and a pretest among school staff (n = ~10). The final questionnaire will be distributed to all primary, secondary, and secondary vocational schools in the nine participating PHS regions (n = ~3000) and filled out by the person who is most knowledgeable about the school’s health promotion practices. This results in quantitative data, which can be used to differentiate schools for each of the seven dimensions (concepts).

Substudy 2.2 aims to understand factors in the school context that may influence the degree of HS program implementation. Based on the results of substudy 2.1, ~50 schools will be selected, stratified by the degree of implementation and education sector. Although these schools will be selected based on different criteria than those used in substudy 1.2, the selections may overlap. At these schools, data on factors related to the actors, school organization, and sociopolitical environment will be collected, as well as general school and HS program characteristics (also see “Conceptual Framework” and Fig. 1). Factor selection will be guided by the Measurement Instrument for Determinants of Innovations (MIDI) [52]. Furthermore, the degree of HS program implementation, as found in substudy 2.1, will be verified. Data stemming from a combination of semi-structured (group) interviews, observations, registration systems, and document analysis will be collected. The interviews will be conducted with teachers, principals, the care team, parents, and students. Using the predominantly qualitative data, a descriptive analysis of the school context will be performed.

Next, differences in the degree of HS program implementation (substudy 2.1) will be explained by the school context (substudy 2.2) and regional support (see substudy 3.1 and 3.2). Data consisting of information on explanatory factors will be analyzed using QCA [45]. Data concerning actors’ perceptions of the association between explanatory factors and the degree of HS program implementation will be analyzed descriptively.

Conditions for regional support

To answer the third research question, three substudies will be conducted (Fig. 1 and Tab. 1). Substudy 3.1 provides insight into the regional professional support received by schools. The study sample will be the selection of schools (n = ~50) from substudy 2.2. At the schools, insight into regional support will be obtained through staff interviews by inquiring, for example, about the type and intensity of HS program support received. In substudy 3.2, the offer from the participating PHS regions (n = 9) will be mapped. These PHS regions were selected for participation based on variation in location and willingness to participate. For each region, data from semi-structured (group) interviews, observations, document analysis, and registration systems will be collected. This includes, for example, the hours and financial resources available to provide HS program support. Initial participants are coordinators of HS advisers from the respective regions, after which other relevant stakeholders are identified by the snowball method [53]. In substudy 3.3, information on factors in the regional context related to the stakeholders and their organizations, the broader context and collaboration between the organizations, and general and HS program characteristics will be collected in each region (also see “Conceptual Framework” and Fig. 1). These data will be collected simultaneously with those for substudy 3.2 and entail information from the semi-structured (group) interviews with stakeholders, observations, document analysis, and registration systems.

The predominantly qualitative data from substudies 3.1, 3.2 and 3.3 will first be analyzed descriptively. Next, differences in the level of regional professional support (substudies 3.1 and 3.2) will be explained by the school context (see substudy 2.2) and regional context (substudy 3.3). Data consisting of information about the explanatory factors will be analyzed with QCA [45], whereas data regarding actors’ perceptions of the association between the explanatory factors and regional support will be analyzed descriptively.

Community of practice

In semi-annual meetings of the so-called Community of Practice, researchers and practice professionals exchange information and knowledge. This improves the connection between research and daily practice and increases the knowledge and skills of school health promotion professionals. Participants are (coordinators of) HS advisers and epidemiologists from PHSs in the Netherlands, employees of the national professional support system for the HS program, employees of other relevant organizations in the health and education sectors, and scientists.

Discussion

This article describes the research design for an evaluation of the conditions for effectiveness of the HS program. For this purpose, a combination of qualitative and quantitative methods will be used. This will provide insight into the outcomes of the HS program at the level of the student, school implementation, and regional professional support. By combining methods, more complete and practically relevant insights will be gained, compared with using the same methods separately [5]. Furthermore, a context-driven approach will be applied at each of the three levels. As one size fits all is no longer the default for intervention development and evaluation research, the context must take a more central place [5, 54]. Examining the role of the context is important to understand its impact and increase generalizability to different settings [39]. This will enable us to further investigate differences within and between levels, which contributes to explaining the effectiveness of the HS program.

A preliminary study by the Netherlands Organization for Applied Scientific Research describes in detail the research possibilities for investigating the conditions for effectiveness of the HS program with regard to student outcomes [5]. This study concludes that the classical randomized experiment is not feasible at the current state of knowledge. Therefore, for the first research question, a quantitative retrospective study will be used to gain insight into (the conditions for) student outcomes. By using existing datasets, the time investment for students and schools will remain limited, but data of a large number of students spread across the country can still be included [5]. Furthermore, this research assumes natural variation between schools, which increases external validity [5, 55]. However, using data collected for other purposes also presents challenges. HS program effectiveness can only be examined for indicators that are included in existing data sources [5]. Moreover, data are limited for secondary vocational schools, and if they are available, they are mainly cross-sectional [5]. This deficiency could be partially overcome by collecting data prospectively using the “Test Your Lifestyle” questionnaire [56].

In addition, several studies will use descriptive analyses to map school implementation, regional professional support offered and received, and school and regional contexts. Descriptive research is highly suitable to examine such complex systems in the real world. It provides detailed insight into the school and regional systems and allows for comparative analysis of factors influencing the processes. This results in more complete and rich data, creating a real understanding of school health promotion practice, and contributing to external validity [5, 57, 58]. However, a major challenge is the dependence on the willingness of schools to participate in the study, which may lead to selection bias [57]. This could apply to a greater extent to schools with lower degrees of implementation or worse student outcomes. Therefore, it is important to organize recruitment via existing connections (e.g., HS advisers, educational boards, and local organizations) whenever possible, as well as emphasize that our study is not an assessment. In the PHS regions, identifying the right stakeholders using the snowball method depends on referrals from participants [53]. Therefore, in each region we will start with the (coordinators of) HS advisers. These professionals are familiar with HS program support and the regional context and are expected to be able to identify other stakeholders. A final challenge related to the above is the generalizability of the results, which may allow conclusions to be drawn mainly about the participating schools and regions in the study [57, 58].

Moreover, for the entire study, complexity is a challenge. In order to fully understand the conditions for effectiveness, many explanatory factors must be included, but in order to limit the study size to some extent, selection is necessary. To interpret the results, it is also important to acknowledge that the three-level contextual approach assumes the presence of dynamic processes in which continuous change occurs, whereas this evaluation only focusses on a specific moment in time.

We consider the described research design to be the best possible method to investigate under what conditions the HS program matters. The results will provide insight into the HS program outcomes at the level of the student, school implementation, and regional professional support. Furthermore, insight will be provided into factors and characteristics that may facilitate or impede these outcomes. This allows for providing specific advice to optimize conditions at the school and regional level. Moreover, a proven association between the HS program and healthy lifestyles and better educational performance of students can increase support among both school staff and policy makers and further strengthen the position of school health promotion [5, 19]. Finally, we intend to continue the Community of Practice after completion of the study to create a permanent learning network on school health promotion in the Netherlands. This will increase support among professionals and the likelihood that recommendations will be implemented in practice.