Trial Design
Full details of the trial design and participants are reported elsewhere [29]. In brief, a two-arm cluster RCT design was utilised, with schools used as the unit of randomisation. Schools were randomly allocated on a 1:1 basis to implement PATHS or continue with their usual practice. Data were collected at baseline (May–July 2012) and at 12- and 24-month follow-ups. Adaptive stratification was employed to ensure that the trial arms were balanced on the proportions of children eligible for free school meals and speaking English as an additional language (EAL). Randomisation was performed independently by the Manchester Academic Health Science Centre Clinical Trials Unit. The study received ethical approval from the University of Manchester Research Ethics Committee (Ref: 11470).
Participants
All mainstream, state-maintained institutions providing education for children aged 4–11 years in the ten local authorities in Greater Manchester were eligible for the study. School participation required consent from the head teacher. Participants within each school were children aged 7–9 years at baseline (year groups 3–5). Child participation required a lack of parental and child opt-out.
Sample Size
Initially, 58 schools were recruited, 45 of which met the eligibility criteria for randomisation. In total, 23 schools were randomised to receive PATHS; 140 parents (3%) exercised their right to opt their children out of the trial. A total of 5218 children participated in the study (n = 2676 in the intervention arm).
Characteristics of participating schools mirrored those of primary schools in England in respect of attendance, attainment and proportion of pupils speaking EAL but were significantly larger, with higher proportions of pupils eligible for free school meals and lower proportions identified as having special education needs. Characterisics of participating pupils (aged 7–9 at baseline) aligned closely with those of English primary school-aged children [30, 31], although the prevalence of children eligible for free school meals and speaking EAL was higher [32,33,34,35].
Intervention and Comparator
Intervention
Each participating class received curriculum packs containing lessons (and some supplementary send-home activities) built around four conceptual units (emotional understanding, self-control, social problem-solving, peer relations and self-esteem), plus associated material resources (e.g. posters, feelings dictionaries) [36]. Teachers also received a guidance manual developed by the research team that emphasized the programme theory and importance of effective implementation.
PATHS was implemented and delivered by class teachers as part of the general classroom timetable for year groups 3–5 in the first year of the trial (4–6 in the second year). All were qualified teachers and had an average of 8 years’ teaching experience; 81% were female. PATHS lessons lasted 30–40 min and were designed to be delivered twice weekly throughout the school year. Curriculum packs contained an average of 40 lessons.
In the first year of the trial, teachers of year groups 3–5 in the treatment arm received a full day of initial group training prior to the school year, with a half-day follow-up 4 months later. This training was led by certified trainers from Pennsylvania State University (PSU). They were aided by three PATHS coaches from the research team, who received training on supporting PATHS implementation during a visit to PSU. Where teachers could not attend the group training sessions, one-on-one training was carried out by a PATHS coach. Where teachers left part-way through a school year, training for replacement teachers was also carried out on a one-on-one basis by a PATHS coach.
In the second year of the trial, teachers of year group 6 and newly starting year group 4–5 teachers also received the above training. PATHS coaches conducted all second-year training and provided teachers with ongoing technical support and assistance, including modelling of and feedback on lessons.
Usual Practice
School-level surveys ascertaining the level of engagement with non-PATHS SEL-related activities were completed by the member of staff with lead responsibility for personal, social and health education in both PATHS and control schools before and during the intervention period. Findings from these surveys, presented elsewhere [29], revealed that, both pre and post-implementation, schools in both the PATHS and the control arms engaged in a range of other SEL-related activities. The volumes of these activities were relatively balanced across arms pre-PATHS implementation. Post-PATHS implementation, most control schools were implementing the universal elements of the Social and Emotional Aspects of Learning (SEAL) programme [37] (82% using the whole school resources, such as assemblies; 72% delivering SEAL lessons) and were delivering the National Healthy Schools Programme [38] (75%) and Circle Time [39] (60%). In total, 32% of control schools were using nurture groups [40], 28% were implementing the small group work component of SEAL, 25% were involved in the Targeted Mental Health in Schools initiative [41] and 26% were using restorative justice techniques [42]. All of these activities were conducted in classroom time and delivered by classroom teachers, and all aimed to improve children’s emotional wellbeing. Although PATHS was designed to be implemented in addition to these other activities, a comparison of pre and post-implementation surveys suggested that PATHS did lead to some displacement of other SEL-related activities, with reductions primarily seen for elements of the SEAL programme [29].
Perspective and Time Horizon
A 2-year time horizon (consistent with the length of the trial) was used. Analysis was conducted from a UK health service perspective, assuming that the NHS would fund incremental costs of PATHS compared with costs incurred in usual practice (e.g. additional training requirements) but that schools would continue to be responsible for costs involved in the delivery of the curriculum (e.g. teachers’ salary costs).
Outcome Measure
The Child Health Utility Nine-Dimension (CHU-9D) is a generic, multi-attribute, preference-based measure of HRQoL constructed specifically for use in children aged 7–11 years [43, 44]. It measures HRQoL based on nine attributes (worried, sad, pain, tired, annoyed, schoolwork/homework, sleep, daily routine and activities), each scored on five levels. Preference weights derived using the standard gamble method in a sample of UK adults were used to transform responses to the CHU-9D to utility values ranging from 0.33 to 1 (perfect health), with a value of 0 equivalent to death [45].
Participants self-completed the CHU-9D questionnaire in a classroom setting at baseline and at the 12-month and 24-month follow-ups. CHU-9D utility values at each time point were used to calculate quality-adjusted life-years (QALYs) using the area under the curve method, assuming utilities followed a linear path over time [46]. QALYs were discounted at a rate of 3.5%.
Intervention Costs
Cost analysis was based on only the costs of the PATHS intervention itself, as resource use data (e.g. use of educational psychologists) were not collected alongside the trial. Micro-costing was used to calculate the incremental costs of the intervention in the trial setting, compared with usual practice. Incremental costs comprised costs of materials, training for teachers and PATHS coaches and ongoing support. Specific cost components, assumptions and sources of unit costs are provided in Table 1, with further detail in Appendix 1 in the Electronic Supplementary Material (ESM).
Table 1 Assumptions and sources of unit costs for costing in the trial setting and roll-out scenario Costs were assigned as recurrent or non-recurrent (full justifications are provided in Appendix 1 in ESM). As non-recurrent costs (classroom materials, PATHS coach training and initial full-day classroom teacher training) were likely to provide benefits beyond the trial period, costs were annuitized over their expected life at a discount rate of 3.5% [47]. The expected life of classroom materials and PATHS coach training was set equal to the expected life of the intervention (5 years) [20]. For initial-full day classroom teacher training, the expected life was set equal to the minimum of either the 5-year expected intervention life or the expected length of service for a teacher. The latter was estimated as the reciprocal of the annual rate of teacher attrition across both years of the trial, assuming a constant attrition rate.
Costs were reported in £ and inflated to year 2018/19 values. Costs incurred beyond the first year of the trial were discounted at a rate of 3.5%.
The total incremental cost of the intervention was derived by summing across all cost elements. Per-child incremental costs were calculated by dividing total cost by the number of children receiving the intervention (n = 2745), comprising 2676 study participants and 69 opt-outs from the PATHS arm.
Missing Data
The proportion of children with any missing data on CHU-9D utility values over all time points was 51%. We followed recommendations outlined in a recent guide to handling missing data in within-trial CEAs [48].
Descriptive analysis found large differentials in missing data across trial arms, with a smaller proportion of individuals in the control arm providing CHU-9D utility values at all time points (Appendix 2 in the ESM). This reflected increased missingness at the 12- and 24-month follow-ups, primarily because five control schools failed to provide follow-up data. Logistic regression models also identified a range of baseline covariates measured at both the child level (ethnicity, free school meals eligibility, special education needs) and the school level (school size) as predictors of CHU-9D missingness. The existence of any predictors of missingness provided evidence against data being missing completely at random. Logistic regressions also confirmed that the probability of CHU-9D missingness in any given time period significantly decreased with higher CHU-9D utility values in all other waves, providing evidence against covariate-dependent missingness. Data were therefore treated as missing at random, assuming the probability of missingness was independent of unobserved factors, i.e. factors other than baseline covariates and CHU-9D utility values in other waves.
Consistent with the missing at random assumption, multiple imputation (MI) methods using chained equations were employed to handle missing CHU-9D data. Given low levels of domain-specific missingness for the CHU-9D, utility scores were imputed. In total, 50 imputations were chosen—approximately equal to the percentage of cases with missing data [49].
Imputation was carried out separately by trial arm, as the posterior conditional distribution of the missing data given the observed data may differ across treatment arms [50]. Predictors included in the imputation model included the CHU-9D utility values at all other time points and a range of child-level (e.g. sex, year group) and school-level (e.g. school size) covariates, consistent with the main trial [29]. Single-level MI was conducted, assuming all between-school variation in missingness was accounted for by school-level covariates.
To account for the skewness and multi-modality in the distribution of CHU-9D utility values, predictive mean matching was used to impute CHU-9D utility values [49]. Incremental QALYs were calculated as the mean of incremental QALY estimates generated in each imputed dataset, following Rubin’s rule. The validity of the imputation procedure was confirmed by establishing similarity in the distribution of non-missing and imputed QALYs [51].
Data Analysis
Incremental costs and QALYs were modelled separately. The incremental cost of PATHS was calculated as the simple difference between costs in the treatment and control arms and was equal to the average child-level incremental intervention costs. To estimate incremental QALYs, child-level random intercept models were used to account for clustering of outcomes at school level. Explanatory variables comprised intervention status (PATHS vs. control), baseline CHU-9D utility values [46] and the child-level and school-level covariates used in the imputation model.
Presentation of Results
The main trial found PATHS led to very small improvements in children’s psychological wellbeing and no improvements in social skills and mental health problems relating to internalising and externalising behaviour [29]. Negative incremental QALY estimates were therefore possible. Consequently, incremental net benefits (INBs) were chosen over incremental cost-effectiveness ratios (ICERs) to present results [52] and were calculated as follows:
$${\text{INB}}\left( \lambda \right) = \lambda \left( {{\text{QALY}}_{p} - {\text{QALY}}_{u} } \right) - \left( {{\text{cost}}_{p} - {\text{cost}}_{u} } \right)$$
\(p\) and \(u\) refer to intervention and control arms, respectively, and \(\lambda\) denotes the cost-effectiveness threshold, representing willingness to pay (WTP) for an additional QALY. PATHS was deemed cost effective at a given threshold if the INB was positive. INBs were estimated for thresholds ranging from £0 to 150,000, increasing in £1000 increments. In the UK, NICE sets this threshold at £20,000–30,000 [28].
Identifying Sample Uncertainty
Uncertainty was examined via non-parametric bootstrapping techniques using 10,000 iterations. This was combined with the MI methods as follows. First, for each bootstrap replication, a sample size of 5218 was randomly drawn with replacement from each of the 50 imputed datasets. Incremental QALYs and costs were re-estimated within each sample and Rubin’s rules used to combine incremental statistics across samples in each imputed dataset to form average statistics for the bootstrap replication. INBs were then calculated for each value of the threshold within each bootstrap replication, and standard bootstrap methods were used to construct confidence intervals [53]. These estimates were then used to construct cost-effectiveness acceptability curves (CEACs).
One-Way Sensitivity Analyses
We undertook five one-way sensitivitiy analyses. First, we examined the robustness of results to using alternative preference weights for constructing CHU-9D utility values derived using a best–worst scaling discrete-choice experiment in an Australian adolescent sample, which assign greater weights to the CHU-9D mental health domains [54]. Second, we examined the effect of performing a complete case analysis where all observations with missing data were removed. Third, we explored the impact of changing the expected life of the intervention from 5 to 10 years when annuitizing non-recurrent costs.
Fourth, we considered the effects of including costs relating to classroom teachers’ time spent training, preparing for and delivering the intervention as previous studies have treated these costs as incremental [20, 55]. These costs were not considered as incremental in the base-case analysis as the fixed-length school day means PATHS necessarily displaced other components of the school curriculum that required equal time resources and therefore consumed identical time resources as in usual practice [56]. Teacher’s salary costs are currently paid by the UK Department for Education. These costs would therefore only be considered incremental under a UK health sector perspective if the NHS were to agree to fund the costs of the proportion of the school day required for PATHS implementation. We examined the effects of assuming 40 min [56] or 45 min [20] of preparation time per teacher per week.
Finally, we costed the intervention in a hypothetical scenario similar to that in the event of nationwide roll-out, given that these costs would likely differ from those incurred in the trial. The assumptions in the roll-out scenario are presented in Table 1 and in Appendix 3 in the ESM. Costs were based on those charged by Birmingham City Council, as they were the only council providing a traded PATHS service with English schools during the implementation period [56]. This provided a total bundled cost of PATHS for a single-form entry school with seven year groups (reception to year 6). This cost included (1) a curriculum pack and supplementary materials for each class, (2) initial training for seven teachers, (3) training of a PATHS coach/co-ordinator (usually a head teacher) and (4) 2 days of PATHS coaching support (assumed to occur in the first year and be non-recurrent). Unbundled costs for curriculum packs and supplementary material and full-day and half-day teacher training for seven teachers were also provided. Total costs were divided by seven to get a total cost per class and multiplied by the number of classes in the PATHS arm of the trial to get a cost equivalent to the size of the PATHS school sample. This cost was then divided by the number of children receiving the PATHS intervention to get an equivalent cost per pupil.