Introduction

Improving the capacity of schools to enhance students’ academic achievement and wellbeing is a central focus of governments in Australia (e.g. Gillard, 2010; Jensen & Sonnemann, 2014) and internationally (e.g. Ainscow and West, 2006; Fullan, 2007; Resnick, 2010; Sammons, 2007). Schools play a key role in developing the social and emotional skills that underpin students’ mental health and wellbeing alongside academic learning (Brackett et al., 2009; Feigenberg, Watts, & Buckner, 2010; Mellin et al., 2016). They are recognised as critical sites for the prevention and early identification of mental health problems, as well as the promotion of wellbeing (Green et al., 2013; Payton et al., 2008; Sklad, Diekstra, Ritter, Ben, & Gravesteijn, 2012; Durlak et al., 2011; Taylor et al., 2017). Importantly, a whole-school approach to promotion of emotional and social health stands to benefit all students, not just those at risk for poor mental health (WHO, 2004). Research indicates that “schools will be most successful in their educational mission when they integrate efforts to promote children’s academic, social, and emotional learning” (Zins et al., 2004) and that “strong bonds between student behaviour, attainment and learning and their social and emotional development” are central (Sammons, 2007). This evidence has been actioned in nation-wide school-based initiatives that address students’ wellbeing and bolster each child’s capacity for effective academic learning (e.g. CASEL, 2008; DCSF, 2009; ENSEC, 2009; KidsMatter, 2009; Benningfield et al., 2015).

The extent to which such initiatives are implemented is critical in effecting change (Durlak and DuPre, 2008; Basch, 2011). Given the complexity of implementing mental health promotion school-wide, some schools do this more readily than others. For example, in Australia, significant positive relationships were found between the extent of implementation of a whole-school mental health initiative and students’ differential mental health outcomes (Slee et al., 2009) and academic performance (Dix, Slee, Lawson, & Keeves, 2012). In order to assess implementation quality in the 96 schools studied, Dix, Slee, Lawson, and Keeves (2012) used latent class analysis to develop an ‘implementation index’. The index categorised schools as high, moderate, or low implementers of the mental health initiative based on a set of criteria (fidelity, dosage, quality of delivery) and found that high implementing schools showed greater academic gains than low implementing schools. Similar results have been reported elsewhere (e.g. Simonsen et al., 2012; Payton et al., 2008; Sklad, Diekstra, Ritter, Ben, & Gravesteijn, 2012; Durlak et al. 2011; Kidger, Araya, Donovan, & Gunnell, 2012; Wells, Barlow, & Stewart‐Brown, 2003; Basch, 2011; Novins, Green, Legha, & Aarons, 2013; Weissberg, Goren, Domitrovich, & Dusenbury, 2013). However, despite the large number of school-based programs that foster positive mental health, there continues to be little evidence available concerning how schools measure the implementation of such programs and the impact that they may have (Adelman & Taylor, 2000; Domitrovich et al., 2008; Durlak and DuPre, 2008). Feigenberg, Watts, and Buckner (2010) suggest that it has been challenging to measure the impact of formal whole-school mental health interventions at the school-level due in part to the lack of assessment tools that measure school-level characteristics (Tseng and Seidman, 2007). In addition to the plethora of formal programs, schools may also be engaged in mental health-promoting activities outside the remit of a formal program, suggesting the need for assessment tools that are independent of the programs.

A number of instruments assessing implementation have focused on the whole-school setting. These include assessments that examined cultural climate and school safety (e.g. Brand, Felner, Shim, Seitsinger, & Dumas, 2003), school-wide positive behaviour (e.g. Horner et al., 2004), school mental health capacity and quality (e.g. Feigenberg, Watts, & Buckner, 2010; Weist et al., 2006; Wyn, Cahill, Holdsworth, Rowling, & Carson, 2000; Graetz et al., 2008), and school health (e.g. Staten et al., 2005). However, while there are toolkits and self-assessment planning guides that schools can use to improve their wellbeing policies and programs (e.g. KidsMatter, 2013; Staten et al., 2005; E4L, 2017), their emphasis is not quantitative and they do not readily enable psychometric assessment, nor allow comparisons between schools, and do not allow school wellbeing promotion to be readily examined as an outcome variable. Accordingly, research and practice stand to benefit from the development of a quantitative assessment tool that describes the extent to which whole-school mental health promotion policies and practices have been implemented. For this reason, we developed the Survey of School Promotion of Emotional and Social Health (SSPESH).

In the development of the SSPESH, we adopted a conceptual approach reflecting protective factors known to influence student wellbeing. This was based on the World Health Organization’s (WHO) Health Promoting Schools (HPS) framework that incorporates: (a) ethos and environment of the school; (b) formal health curriculum; and (c) engagement with families, or communities, or both (Langford et al., 2014). In Australia, the major health-promoting school initiatives, such as KidsMatter and MindMatters, operationalised the HPS framework by separating (c) into two parts to create four components that foster the mental health of every student: (a) positive school community; (b) social and emotional learning for students; (c) parenting support and education; and (d) early intervention for students experiencing mental health difficulties (Graetz et al., 2008; Beyondblue, 2017). This conceptual approach guided our design of the SSPESH measurement tool.

Research Context

This research was conducted as part of the New South Wales Child Development Study (NSW-CDS) in Australia (Carr et al., 2016; Green et al., in press). The NSW-CDS is a multi-agency, intergenerational data linkage study that brings together diverse health, education, welfare, and justice records in a population cohort of approximately 90,000 children and their parents. It aims to identify childhood factors that promote positive mental and physical health, and social and educational outcomes in adolescence and adulthood. It also aims to identify new opportunities for detecting and mitigating early vulnerability for ill-health and other adverse outcomes. In 2015, the investigator team administered online a self-report survey of mental health and wellbeing to a subsample of children in the NSW-CDS cohort (the Middle Childhood Survey; MCS), via 829 primary (elementary) schools in NSW (Laurens et al., 2017). MCS data were obtained from 27,792 children aged approximately 11 years of age during their final year of primary schooling (Year 6 in NSW). In addition, the NSW-CDS surveyed the primary school Principals (or their delegate from the school leadership group, such as a Deputy Principal) regarding the extent to which whole-school mental health promotion activities were implemented in the participating school. In Australia, the Principal is the head of the school and oversees its management and administration. Promoting school improvement is an important part of a Principal’s role, which can be assisted by self-assessment tools that help monitor and identify particular areas for improvement.

The aim of the present study was to develop a brief, school-level instrument with the potential to profile schools and identify characteristics that differentiate high, moderate, and low implementation of whole-school promotion of emotional and social health. This approach has been successfully used in national evaluations of whole-school mental health promotion initiatives in Australia (Dix, Slee, Lawson, & Keeves, 2010; Slee et al., 2012; 2009). It acknowledges that implementing whole-school policy is complex and that schools in diverse contexts accomplish this to varying extents. The diversity of school contexts in Australia arise from systemic and geographical factors (e.g. government and non-government systems, metropolitan and remote location, socio-economic disparity, multiculturalism). A key feature of this approach was to develop a measure of implementation (an ‘implementation index’) that could account for the likelihood that not all schools implement whole-school wellbeing promotion to the same degree. We sought to strengthen capacity to attribute significant differences in implementation and the impact this may have on students. Rather than arbitrarily assigning implementation thresholds (cut-points) to a total survey score, latent class analysis (LCA) was used to profile schools. LCA is a statistical method for identifying subtypes of related cases (latent classes) from multivariate categorical data. The main benefit of using LCA was its non-dependence on assumptions of normality and the establishment of non-arbitrary scoring thresholds (cut-points) for the instrument (Dix, 2009). Accordingly, this paper discusses the development and preliminary validation of the SSPESH and its derived ‘implementation index’, and considers potential uses of the SSPESH in research and practice.

Method

Instrument Development

The SSPESH was based on two earlier measures developed in a broader context to meet the needs of schools participating in the Australian KidsMatter Primary initiative (2013). These precursor tools were designed for and completed only by schools participating in the KidsMatter initiative; though no psychometric analyses have been published on those survey tools, they closely aligned with our requirements for the current study and provided a useful pool of items for consideration. The precursor surveys were developed over a number of years by integrating stakeholder opinion from a team of mental health and education experts with evidence from existing literature on whole-school mental health promotion (KidsMatter, 2017). The 52-item ‘Staff Survey’ and the 31-item ‘School Check-up’ were designed around the four-component KidsMatter framework to assess a school’s capacity to promote mental health and wellbeing across the school community (Graetz et al., 2008). While the original items were written for generalisability across school settings, many of the items were considered to be too conceptually complex and poorly designed.

The goal in developing the SSPESH was to simplify the previous measures and create a brief, generic school-level instrument (not aligned to a specific program or initiative such as KidsMatter) that could assess a school’s capacity to promote social and emotional wellbeing. Importantly, to minimise respondent burden and to maximise response rates by NSW-CDS schools, the instrument had to be shorter than its predecessors and easily completed by school Principals. As the leader of the school, Principals have overall responsibility for monitoring and promoting social and emotional health in their school. Predominantly, these types of school climate and wellbeing surveys have targeted teachers, parents, and students. This can render the information collection process more complex and often ‘too hard’ to undertake within the context of the busy school environment. We sought to design a brief tool that could be completed quickly by school leadership to support an efficient reflective process that informed school improvement policy.

As part of the instrument development process, the lead author (with an academic background in educational measurement and survey design) consulted multiple experts in school-based mental health, school leadership, whole-school improvement, and questionnaire design from three federal agencies (Beyondblue, Australian Psychological Society, and Principals Australia Institute). These consultations guided the selection of items that were thought to align with the four conceptual domains identified in the literature as important agents of change in terms of mental health promotion, prevention and early intervention (e.g. Graetz et al., 2008) and school improvement (e.g. Jensen & Sonnemann, 2014); namely: (a) positive school community, (b) student social emotional learning skills, (d) working with families, and (d) supporting students experiencing mental health difficulties.

While the conceptual elements in the original items, sourced from the ‘Staff Survey’ and ‘School Check-up’, were retained, in consultation with the experts, the wording of each item was modified to gather information about whole-school practices rather than those of individual staff, and to ensure that the questions were not specific to a particular initiative (e.g. KidsMatter). Care was taken to ensure that the survey items used plain English and were readable by educators. This was reflected in a Flesch reading-ease score of 42, equivalent to College level (Flesch, 1980). Nevertheless, the items were complex in nature in order to convey a breadth of related content. For example, the item, Our school has specific policies and practices that promote inclusion and a safe environment, comprised two components (inclusion and a safe environment). While a high implementing school could be expected to have both in place, a school with only one of these aspects in place would be expected to report a lower rating on this item. This approach enabled us to keep the survey brief while maximising the breadth of content, and also lowered the risk of a measurement ceiling effect (i.e. it was more difficult for schools to select the top response option).

In the final stages of measurement design, the item-redevelopment process involved cognitive interviews and a focus group discussion conducted with a convenience sample of six school Principals and school wellbeing coordinators. A cognitive laboratory approach was taken (Leighton, 2017). Participants were guided through a short-list of possible items, to gauge their understanding around the intent of each item, and identify items that were confusing, not relevant, unclear, or too complex. The final items were selected because they were perceived to be readily identifiable and rateable by school leadership. The resultant measure, the SSPESH, included 14 statements that characterised favourable practices within a school, rated on a four-point Likert-type scale of: Not yet in place (scored 0), Introducing (1), Taking hold (2), and Completely in place (3). To comprehensively characterise school performance in wellbeing promotion, the SSPESH was designed to provide an overall summary score as well as domain scores for each of the four components.

The instrument also contained additional questions that were not intended to be used in the calculation of the total SSPESH score, and as such, are not reported in detail in this paper. These questions asked about specific social and emotional learning (SEL) programs and wellbeing frameworks being implemented by schools. School leadership could identify up to five SEL programs from a list of over 100 formally verified programs (Trinder, Roberts, & Cavanagh, 2009). They were asked to indicate when the program commenced, what year levels were targeted, and their perception of how effective the program was. An option to provide other SEL programs not listed was also available.

The SSPESH also asked schools to indicate if they were implementing any whole-school mental health frameworks. This selection included KidsMatter Primary, which provides mental health promotion to primary school students according to the four-component framework that guided the development of the SSPESH (Graetz et al., 2008; Beyondblue, 2017).

Sampling and Procedure

A target sample of 689 NSW primary schools with a Year 6 enrolment were invited to participate in the research; these were a subset (83%) of the 829 schools that had participated in the MCS and indicated willingness to be contacted with an invitation to participate in a Principal/school leadership survey of school-based mental health promotion.

Principals (or their delegate from the school leadership group, such as a Deputy Principal) were invited via email to complete the SSPESH following the conclusion of the MCS administration in October 2015. Participants completed the SSPESH using an online survey (software supported by PeoplePulse; www.peoplepulse.com). A unique survey link was provided to each school. Informed consent was indicated via survey completion. To maximise response rates, participants were given a 2-month period in which to complete the survey. Responses were monitored during the administration period, and system-generated reminder emails were sent to those schools yet to respond, followed by a telephone reminder to the Principal, yielding the participation of 598 schools.

The SSPESH online survey presented the original 14-item instrument as a set of statements and radio-button choices, in matrix format. This was followed by the section capturing school implementation of SEL programs and whole-school wellbeing frameworks, which used drop-downs, check-boxes and open-text response formats. The brief survey was presented on a single page, and no items enforced mandatory responding.

All procedures performed were in accordance with the ethical approvals issued by the University of New South Wales Human Research Ethics Advisory Panel D: Biomedical (Reference number: HC14348) and the New South Wales Department of Education’s State Education Research Approvals Process (SERAP reference number: 2015083).

Statistical Analyses

Analyses were conducted using SPSS Version 22 (IBM Corp, 2013) and Mplus version 5.3 (Muthén & Muthén, 2007).

Structural Analysis

We first examined the distribution of the 14 individual items, and determined inter-item correlations using Spearman’s rank-order correlation to account for any non-normal distribution of items. Reliability was assessed using Cronbach’s alpha coefficient, and the item-total correlations were examined to identify items that appeared not to contribute to the overall construct being measured by the SSPESH.

Exploratory factor analyses (EFA) were then conducted, initially on the full 14-item scale, to examine the instrument’s dimensionality and to identify for removal any item that did not contribute to the overall scale (based on the item-total correlation coefficients and on non-loading of any item with other items). A split-half methodology was adopted for the structural analysis in Mplus, such that the sample was split randomly into two groups of 299 and 298 schools, respectively, with the EFA conducted on one sample, and confirmatory factor analysis (CFA) on the other. EFA was used to explore solutions with one to five factors using Oblimin rotation (Muthén & Muthén 2007). Goodness-of-fit indices used to evaluate model fits included the root mean square error of approximation (RMSEA), the standardised root mean square residual (SRMR), the comparative fit index (CFI), the Tucker Lewis Index (TLI), and the normed Chi-squared statistic (χ2/df). These indices perform better than other indices under non-normal distribution conditions (Fan, Thompson, & Wang, 1999; Hu & Bentler, 1999). Factors were evaluated for non-loading and cross-loading items, using a threshold for loading of 0.3. Next, we examined the factor structure of the resultant instrument using CFA. Like the EFA, model fit was examined using maximum likelihood parameter estimates with standard errors that are robust to non-normality (Hooper, Coughlan, & Mullen, 2008).

Finally, in order to provide a check of the criterion validity of the resulting SSPESH structure, we used the information collected from the schools regarding their engagement with the KidsMatter Primary whole-school initiative. Independent t-tests were conducted on the SSPESH total scale (summed) score, and on the subscale summed scores based on the factors derived from the CFA, comparing scores for schools that indicated that they had engaged, verses those that had not engaged, with KidsMatter Primary.

Latent Class Analysis

LCA was conducted in Mplus to identify profiles (classes) of schools that could be distinguished with respect to the quality of implementation reported (i.e. to identify classes of schools with similar patterns of responses to the SSPESH items). Using the four response options (Not yet in place, Introducing, Taking hold, and Completely in place) for each of the items, LCA models were conducted to determine the optimal number of latent classes signifying varying degrees of implementation. Robust maximum likelihood (MLR) was used to estimate solutions for one to four classes. The number of initial stage random starts was set at 50 with the 15 best solutions retained for final stage of the optimisations. To obtain the best solution, and considering both model parsimony and interpretability, we were informed by the Akaike Information Criterion (AIC), the Bayesian Information Criterion (BIC), the sample-size adjusted BIC (SSA-BIC), Entropy, the adjusted Lo-Mendell-Rubin test (LMR), and the bootstrapped parametric likelihood ratio tests (BLRT) to compare the model with K classes to a model with K-1 classes (Lo, Mendell, & Rubin, 2001; Asparouhov & Muthén, 2012; Morgan, 2015; UCLA, 2009).

Results

Participating Schools

From the pool of 689 schools invited to participate in the SSPESH, 598 (87%) Principals participated on behalf of their school. The survey took, on average, 10.7 minutes to complete based on the start and finish time-stamps. The representativeness of participating schools relative to all NSW schools was estimated using national school-level data on socio-demographic indices acquired from the Australian Curriculum, Assessment and Reporting Authority. Table 1 summarises the demographic characteristics of all NSW schools with a Year 6 enrolment in 2015 compared with SSPESH participating schools, firstly as distributions of unweighted data, and secondly as distributions after weighting by Year 6 enrolment and number of MCS participants per school. This shows that the 598 schools participating in the SSPESH were comparable on a range of socio-demographic indices to the total population of NSW schools with a Year 6 enrolment; all prevalences reported for the SSPESH participating schools (both unweighted and weighted estimates) lie within ~ 2% of NSW rates. The participating schools comprised two-thirds (66.9%) government and one-third (33.1%) non-government schools, mainly located in metropolitan (63.4%) and provincial (35.3%) communities. Schools were near-evenly distributed across socio-educational quartiles, with a small over-representation of socio-educational disadvantage (29.3% in lowest quartile). School communities were diverse, but on average comprised 23.7% of students with Language Background Other Than English (LBOTE) and 9.2% of students from Aboriginal or Torres Strait Islander background.

Table 1 Demographic characteristics of SSPESH participating schools relative to all NSW schools with a Year 6 student enrolment (unweighted, and weighted by enrolment)
Full size table

Missing Data Imputation

Of the 598 participating schools, 556 (93.1%) had no missing data across the 14 statements. One school did not respond to the 14 statements and was excluded. The small amount (1%) of missing data from the 597 schools were well within acceptable ranges, and Little’s (1988) MCAR test suggested that missing values were completely at random across the 14 items (χ 2(274)  = 299.5, p > 0.10). Given limited missing values (in 41 schools), and a desire to retain as many schools as possible in the analysis, the missing data were imputed by replacing the missing value(s) with the rounded median score of the school’s responses to the non-missing items. This method preserved the categorical structure of the data and minimised possible bias relative to replacement with the series mean (Brick & Kalton, 1996). Statistical analyses of the 14 items were performed using the full database of n = 597 schools.

Structural Analyses

Item Distributions and Inter-Item Correlations

Across the 14 items, shown in Table 2, mean scores ranged from 1.27 to 2.54, within a possible range of 0–3, and the standard deviations ranged between 0.63 and 1.05. Most items exhibited non-normal distributions (skewness and kurtosis z-scores of over 3). All Spearman rank item correlation coefficients (see Supplementary Table 1) were significant (p < 0.01) and ranged from very weak (rs = 0.13) to strong (rs = 0.67). In accordance with our theoretical expectation, this suggested that the underlying factors would be moderately correlated, supporting our interest in deriving both a total scale and subscale scores across the four hypothesised conceptual domains.

Table 2 Descriptive statistics for the 14 SSPESH items (N = 597)
Full size table

An initial Cronbach’s alpha reliability coefficient for the 14 items was α = 0.85. The item-total correlations for all but one item ranged between 0.4 and 0.6 (see Supplementary Table 1); the item-total correlation for item 4 was low (r = 0.31), and unlike the other items, deletion of Item 4 did not reduce Cronbach’s alpha, suggesting a 13-item scale (α = 0.85) for structural analysis. Item 4, ‘Students, parents, and staff in our school community regularly participate in providing structured feedback that informs school improvement’, was aimed at assessing schools’ data-driven decision making and, while important, conceptually stands alone from the other items.

Exploratory Factor Analyses

EFA was conducted initially on the 14-item scale, using data from the split-half subsample of 298 randomly selected schools, to examine the instrument’s dimensionality and to assess the need for removal of the item (Item 4) that was not contributing to the overall scale based on the item-total correlations. These analyses confirmed that Item 4 loaded independently of all other items. EFA was thus subsequently conducted on 13-items, fitting one to five factors using Oblimin rotation. The extraction of eigenvalues greater than 1.00 suggested a potential three-factor solution; however, this was not supported by the cross-loading of items onto multiple factors. On the basis of the fit statistics presented in Supplementary Table 2 (CFI, TLI, RMSEA, SRMR and χ2/df), the optimal solution was achieved with a 13-item scale loading onto four factors. This solution aligned with the theoretical model. The first factor explained 42.0% of the variance, with the four-factor model explaining 67.9% of the variance.

Supplementary Table 3 presents the four-factor EFA of the 13-item SSPESH scale, with the four domains conceptually interpreted as 1: Positive school community (PSC), 2: Student social emotional learning (SEL), 3: Engaging families (FAM), and 4: Supporting students experiencing difficulties (SUP). All items loaded above 0.4, with the exception of Items 2 and 5, where cross-loading was apparent; these items were assigned to the factor with the highest loading (for both items, this loading was 0.34). Correlations between the four factors ranged from r = 0.10 to 0.51, p > 0.05 (r1,2 = 0.26; r1,3 = 0.10; r1,4 = 0.21; r2,3 = 0.51; r2,4 = 0.48; r3,4 = 0.44). These results suggest that there was underlying unidimensionality, and the use of the instrument as a total scale was acceptable, alongside the derivation of total (summed) subscale scores reflecting each of the four factors. Acceptable internal consistency (alpha coefficients) for each factor was demonstrated, ranging from 0.67 to 0.72 (Gliem and Gliem 2003).

Confirmatory Factor Analysis

A four-factor CFA was specified on the 13 items using data from the other subsample of 299 randomly selected schools. This model fit the data well, with fit indices of CFI = 0.975, TLI = 0.987, RMSEA = 0.048, χ2/df = 1.675 (Hu & Bentler, 1999). Thus, the CFA yielded support for the four-domain solution and our conceptual model, as illustrated in Fig. 1.

Fig. 1
figure 1

Four-factor model derived from confirmatory factor analysis (CFA) of the 13 SSPESH items, indicating standardised factor loadings for each item and inter-construct correlations (N = 299)

Full size image

Criterion Validity

Independent t-tests conducted on the SSPESH total scale and the four subscale scores (sum of items loading on the factor) indicated significant differences between schools which had implemented KidsMatter Primary and those which had not, on the SSPESH total scale: t(595) = 3.38, p < 0.01; and the subscale scores of: Student social emotional learning: t(595) = 2.70, p = 0.01; Engaging families: t(595) = 3.57, p < 0.01; and Supporting students experiencing mental health difficulties: t(595) = 3.25, p < 0.01. There was no significant difference between schools on the Positive school community subscale: t(595) = 0.46, p = 0.65.

Latent Class Analysis

From the LCA testing of one through four class solutions (see Supplementary Table 4 for the fit statistics for each model), the preferred solution was a 3-class model with acceptable fit indices (AIC = 16,392.58; BIC = 16,915.22; SSA–BIC = 16,537.43; Entropy = 0.86); model fit was supported also by the adjusted LMR test (p = 0.019) and the bootstrapped parametric likelihood ratio test (p < 0.001). Average posterior probabilities were high (Class 1 = 0.93; Class 2 = 0.94, Class 3 = 0.94) and the average probability of being assigned to an incorrect class was low (0.07 or less). Based on the LCA of response patterns across the 13 items, 19.4% of schools were classified as Class 1, 53.7% of schools were Class 2, and 26.9% of schools were Class 3.

Figure 2 graphs the resulting three-class probability estimates for schools selecting each of the four response categories (i.e. the likelihood of scoring 0, 1, 2 or 3 against each item, depicted as lines). The mean scores, expressed as a percentage of the maximum possible score (3) for each item, are also shown (depicted as bars; described on the right vertical axes). The left vertical axes can be interpreted as the probability of selecting a particular response category and the horizontal axes present each of the 13 items.

Fig. 2
figure 2

Three-class probability estimates, from latent class analysis, of being a high (class 1), moderate (class 2), or low (class 3) implementing school of emotional and social health promotion

Full size image

Class 1 (19.4% of schools) showed a high probability of selecting Category 3 (Completely in place) and low probability of selecting Category 0 (Not yet in place) or Category 1 (Introducing) across most items, and was interpreted as high implementing schools. The first graph in Fig. 2 suggests that these schools were more likely to have Completely in place a positive school community, student social and emotional learning, and support for students experiencing difficulties, while engaging parents was still Taking hold.

The profile of Class 2 (53.7% of schools) showed a higher probability across many items of selecting Category 2 (Taking hold) and was interpreted as moderate implementing schools. The second graph in Fig. 2 suggests that these schools were more likely to have Completely in place a positive school community and some elements of support for students experiencing difficulties, while student social and emotional learning was Taking hold, and engaging families was being Introduced.

Class 3 (26.9% of schools) showed higher probability of selecting Category 0 or 1 across most items, and were interpreted as low implementing schools. The third graph in Fig. 2 suggests that in these schools, a positive school community was still Taking hold, while elements of supporting students experiencing difficulties were being Introduced or were Taking hold. Moreover, these schools were Introducing social and emotional learning in the curriculum, while engaging families was Not yet in place or was starting to be Introduced.

Identifying Scoring Thresholds

As a final step in the process, we applied the LCA grouping of schools to the summed scores to establish scoring thresholds (cut-points) for total scale and subscale scores. Using the response category scores (minimum = 0 to maximum = 3) for each of the 13 items, a total index score was calculated for each school to differentiate high and low performances (minimum possible score = 0, maximum = 39). The schools’ total scores ranged from a low score of 2 to a high score 39. The distribution of total scores within each class identified by the LCA is shown in Supplementary Figure 1. Inspection of the scores and class boundaries suggested thresholds set at the total scores of 21 for moderate implementing schools, and 33 for high implementing schools. Schools scoring less than 21 were interpreted as low implementing schools.

Scoring thresholds for SSPESH subscales were also calculated and are presented in Table 3 along with the thresholds for the total score. Threshold scores were adjusted to ensure that the sum of the subscale scores in each class fell within the total score ranges, in order to simplify interpretation for the end-user (i.e. researchers, schools). The variation across subscale scoring threshold indicated that the subscales of Positive school community and Student social emotional learning were relatively easier to implement, and thus had higher thresholds, compared to Engaging families and Supporting students experiencing mental health difficulties. Table 3 compares the resulting scoring bands and the distribution of Class 1 (high implementation), Class 2 (moderate implementation) and Class 3 (low implementation) schools in each scoring band. The SSPESH total score evidenced the most accurate classification, with 92% of schools on average, correctly allocated. The subscale Positive school community was less accurate, with 65% of schools, on average, being correctly allocated. The subscales of Student social emotional learning and Engaging families correctly allocated 68% of schools on average, and Supporting students experiencing difficulties correctly allocated 72% of schools on average.

Table 3 Scoring thresholds for the SSPESH total score and the four subscale scores at low, moderate or high implementation of mental health promotion, and their distribution across the high (class 1), moderate (class 2), and low (class 3) implementing classes derived from latent class analysis
Full size table

Discussion

In this paper we describe the development of a new instrument, the Survey of School Promotion of Emotional and Social Health (SSPESH), which is designed to measure the school-level construct of whole-school emotional and social health promotion. The SSPESH offers the potential to profile schools and identify characteristics that differentiate states of high, moderate, and low implementation. The measure is also intended to serve as a guide for schools in identifying target areas for whole-school improvement in mental health and wellbeing in their community. Towards this end, an administrable version of the SSPESH, along with a scoring key to support interpretation, is provided in the Supplementary material.

Exploratory and confirmatory factor analysis supported the theoretical four-component structure of the 13-item version of the scale, with three items each related to Positive school community, Student social and emotional learning, and Engaging families, and four items related to Supporting students experiencing mental health difficulties. The preliminary psychometric properties obtained, provided support for the criterion-related validity of the scale based on its sensitivity to schools implementing KidsMatter Primary. Schools that were implementing this initiative were more likely to have in place policies and practices that promoted student wellbeing overall, and in three of the four components. However, there was no significant difference between schools implementing and not implementing KidsMatter on the Positive school community subscale, with most schools rating their capacity strongly within this domain. It is possible that the items comprising this domain may not have had sufficient discriminating power to avoid a ceiling effect.

The resulting 13-item SSPESH survey offers a brief instrument to assess a school’s current social and emotional health promotion activities, and to determine target areas for whole-school improvement, that will be easy for school leaders to use. A school’s overall social and emotional health promotion score is calculated by summing the scores on the 13 items, resulting in a theoretical range for the total scale score of 0–39. Latent class analysis identified three types of schools based on the characteristics of their response profiles to each of the 13 items. From these profiles, scoring thresholds were established to facilitate the interpretation of scores as an indication of a school’s current activities in whole-school mental (social and emotional) health promotion. The three ‘classes’ of school were described as ‘high implementing’ (19.4% of schools), ‘moderate implementing’ (53.7%), and ‘low implementing’ schools (26.9%). Scoring thresholds for the total score at 21 and 33 demarcated the moderate and high implementing schools, respectively.

Although less refined, a summed score was also calculated for each of the subscales to provide an indication of current capacity in the four SSPESH domains. The theoretical range for the three-item subscales of Positive school community, Student social and emotional learning and Engaging families is 0–9, and for the four-item subscale Supporting students experiencing difficulties is 0–12. Scoring thresholds vary across the subscales in response to the differential challenges schools have implementing each domain. These differences indicate that Principals submitted a range of ratings across the domains rather than appraise their school positively on every domain. Inspection of the probability profiles indicated that schools did not implement all four areas to the same extent, and showed that Engaging families was challenging, even for the high implementing schools. The domains of Positive school community and Student social and emotional learning are relatively easier for schools to implement and thus have higher scoring thresholds to distinguish moderate and high implementing schools (6 and 9, and 5 and 9, respectively). The lack of significant differentiation of Positive school community between schools that had and had not implemented KidsMatter Primary, and the likelihood of a ceiling effect, are reflected in the high scoring thresholds. The domains Engaging families and Supporting students experiencing difficulties have lower thresholds (3 and 7, and 7 and 11, respectively) to indicate moderate and high implementing schools. Accordingly, school leaders can use the SSPESH instrument to perform a brief assessment of their school, and using the threshold scores as a guide, have some understanding of their current activities in promoting emotional and social health, benchmarked against other schools.

Implications for School Mental Health Promotion

The development of a means to assess and quantify implementation of school promotion of emotional and social health offers the potential for fruitful new avenues of research and provides a user-friendly tool to help schools address the mental health needs of their community. The SSPESH can help briefly describe a school’s current functioning around emotional and social health promotion, aligned to contemporary health-promoting frameworks and initiatives in Australia and internationally. Such information could help identify where a school might benefit from a capacity-building intervention. A SSPESH assessment could help establish a school’s readiness to work with external services and health professionals on its path to establishing community partnerships. Moreover, schools could use the SSPESH survey as an evaluative guide to assess the comprehensiveness of whole-school programs in terms of the extent to which they address all of the domains. It might further be used over time to monitor a school’s progress on implementation of whole-school mental health promotion initiatives in relation to student wellbeing and academic outcomes, though further research is needed to determine the sensitivity of the instrument to measuring changes in implementation within schools. Finally, the present research has identified that, while schools find it relatively easy to build a positive school community, schools are most challenged by engaging and working effectively with parents. This signifies a shortcoming of current whole-school initiatives and calls for the development of specific resources and programs to better meet the needs of schools regarding parent engagement, as has been noted also by others (e.g. Fox & Olsen, 2014).

Limitations and Future Research

A limitation of this current study is that participating schools were only representative of one state in Australia and included only primary schools. Accordingly, we recommend caution in generalising findings to schools in other states and territories in Australia (and internationally) and to secondary schools. More representative data would allow generation of robust scoring thresholds and norms for the instrument and enable comparisons across schools in different settings (and could potentially be extended to include early childhood centres).

Another limitation was the approach used to establish criterion validity. While schools indicated that they were involved in implementing specific whole-school initiatives such as KidsMatter Primary, collection of information about the fidelity or duration of the initiatives’ implementation was beyond the scope of the study. Moreover, our sole reliance on the Principal’s report may have introduced response bias. Principals might have been inclined to over-estimate the extent to which their school was performing well. Principals might be encouraged to consult with other members of their school leadership team to form a collective view on the performance of the school. Nonetheless, schools that have engaged in a whole-school mental health program, such as KidsMatter Primary, are often more self-critical than other schools (Dix & Van Velsen, 2014). Such schools have a more comprehensive understanding of what is involved in achieving effective school-based mental health promotion (the Dunning-Kruger effect). Further study using the SSPESH alongside a more comprehensive community-level assessment, drawing on staff, parent, and student school satisfaction surveys, would further establish the validity of the instrument in primary and secondary schools. However, we acknowledge that the complexity of several items in the instrument, those which assess multiple components, may limit its utility to members of the school leadership only, with adapted versions to be developed for other informants. Likewise, the complexity of the item ‘Students, parents, and staff in our school community regularly participate in providing structured feedback that informs school improvement’ (Item 4 in the original, 14-item version administered) may have contributed to the lack of loading by this item on any of the four factors within the model.

While the current paper is limited to establishing the measurement properties of the SSPESH, there is opportunity to link this data with other school datasets. Data linkage would permit examination of the extent to which a school’s promotion of emotional and social health influences various outcomes, or is influenced by various factors. It offers the opportunity to address the question: Does whole-school mental health promotion make a difference to the social, emotional, behavioural and educational outcomes of children? Such work is envisaged to occur within the NSW-CDS cohort (Carr et al., 2016; Green et al., in press), via linkage to the self-reported Middle Childhood Survey mental health and wellbeing outcomes gathered from children at approximately 11 years of age, in their final year of primary schooling (Laurens et al., 2017), as well as multi-agency administrative records (Carr et al., 2016; Green et al., in press). Accordingly, examining the relationship between the SSPESH and student social emotional health over time is beyond the scope of the present paper, but will be explored in subsequent manuscripts from the study.

It is likely that schools in Australia and internationally will continue to be an important setting for addressing the mental health and wellbeing needs of children and adolescents. Schools that are highly engaged in whole-school emotional and social health promotion are likely to be more effective in providing for the psychosocial and educational needs of their whole student community. The SSPESH survey provides researchers and school leadership teams with an efficient and effective way to briefly assess and quantify the construct of whole-school promotion of emotional and social health.