Introduction

Contemporary community-based mental health residential rehabilitation services combine medium-to-long term accommodation with intensive rehabilitation and psychosocial support (Parker et al., 2019a, 2019b, 2019c). These services provide transitional residential rehabilitation (TRR) to people experiencing severe and persistent mental illness to enable them to live more independently in the community. Most people accessing these services are diagnosed with schizophrenia and have complex care needs (Dalton-Locke et al., 2020; Parker et al., 2019a, 2019b, 2019c). Providing intensive support over an extended duration in a residential setting incurs high costs per episode of care (Parker et al., 2020). Despite the costs associated with these services, there have been limited comparative studies to establish their effectiveness (Dalton-Locke et al., 2020; Parker et al., 2019a, 2019b, 2019c), and TRR service capacity has expanded considerably in Australia over the last decade (Karan et al., 2022).

Australian TRR-type services have been adapted based on changing policy agendas and local service priorities. For example, services have generally shifted from a focus on providing a permanent residence to transitional support (Gerrand et al., 2007), increasingly emphasize recovery-oriented practice (McKenna et al., 2016), and are exploring novel staffing configurations with reduced emphasis on clinical roles (Karan et al., 2022; Parker et al., 2016; Saraf & Newton, 2017). Including staff with a lived experience of mental illness (Peer Support Workers, PSWs) in traditional clinical mental health services is increasingly encouraged in Australia (Saraf & Newton, 2017; State of Victoria, 2021). One such approach is the 'integrated staffing model' (Karan et al., 2022). Under this model, PSWs, rather than mental health nurses, represent the majority staffing component and draw on their personal recovery experiences to support consumers. The integrated staffing model was not intended to alter the principles and objectives of TRR care (Parker et al., 2016).

While there has been strong advocacy for the benefits of incorporating PSW roles in mental health services, the supporting evidence is mainly qualitative, with largely equivocal findings emerging from quantitative studies (Lloyd-Evans et al., 2014; O'Connor et al., 2017). Furthermore, there are concerns about how efforts to integrate paid PSWs within routine mental health care might undermine the value derived from 'real-world interactions between people supporting each other with their emotional distress' (p342) (Gillard, 2019).

Qualitative research conducted at TRR units trialing the integrated staffing model found that consumers and staff held positive expectations of this approach (Meurk, Parker, Newman, & Dark, 2018; Parker et al., 2018, 2019a, 2019b, 2019c). Furthermore, 12–18 months following service entry, consumers supported under this staffing approach emphasized the value of PSW availability (Parker et al., 2021). A recent cross-sectional study in Queensland found that the integrated staffing model was associated with lower levels of restrictive practice, pharmacological treatment, and greater staff-rated consumer engagement than the clinical staffing model (Karan et al., 2022). The authors of this study emphasized the need for findings to be interpreted with caution as no inferences could be drawn as to whether the differential treatment was appropriate or preferable. No available research compares care outcomes between the integrated and clinical staffing models. Despite the limited quantitative evidence, the integrated staffing model was identified as 'show[ing] promise in supporting recovery-oriented practice and maximizing consumer choice and control' (p37) in a report commissioned by the 2020 Victorian Royal Commission into Mental Health (Harvey & Brophy, 2020). Before the broader dissemination of such an approach, it is critical to demonstrate that a substantial reduction in clinical staff within TRR services to accommodate PSW availability does not degrade the clinical and functional outcomes achieved.

Aims

This study considers whether clinical and functional outcomes differ between consumers admitted to community rehabilitation units operating the integrated and clinical staffing models. Given that the integrated staffing model was not intended to alter the core function of the service, we hypothesized that significant differences in outcomes between the staffing approaches would not emerge (Parker et al., 2016). However, if differences are present, this would affect the optimal staffing approach for future services.

Methods

Data were collected as part of a mixed-methods evaluation of the comparative effectiveness of integrated and clinical staffing models for Community Care Units (CCUs; ethics approval HREC/14/QPAH/62) (Parker et al., 2016). A prospective observational design was used due to the ethical and clinical inappropriateness of randomized site allocation or waitlist control. Focusing on 'comparative effectiveness' followed the assumption of clinical equipoise at the policy level. Individualized change rather than group-level comparisons were chosen based on advocacy for this approach (Trauer, 2010) and its increasing use in similar contexts (Barbato et al., 2007; Gonda et al., 2012; Maxwell, Tsoutsoulis, Menon Tarur Padinjareveettil, Zivkovic, & Rogers, 2019; Murugesan et al., 2007). The ISPOR Task Force Report for comparative effectiveness research (Berger et al., 2012) and STROBE statement (von Elm et al., 2007) guided study reporting. Publications based on related data are available, including cohort description (Parker et al., 2019a, 2019b, 2019c), modelling predictors of unplanned discharge (Arnautovska et al., 2021), and qualitative research of stakeholder perspectives (Meurk et al., 2018; Parker et al., 2018, 2019, 2021).

Study Context

CCUs are the dominant community-based TRR service-type operated by public mental health services in Australia. These units provide living skills development and community integration support to consumers residing in independent living units (in a cluster housing configuration) over 6-to-24-months. Staff support is available 24-h a day.

This study considered data from consumers admitted across three CCUs in Queensland over 3-years (12/2014-to-12/2017). One site operated the clinical staffing model, and two operated the integrated staffing model. Under the clinical staffing model, nursing staff reflect most staff roles; there are also senior allied health practitioners and medical staff. Under the integrated staffing model, PSWs reflect the majority team component (> 50%), with a reduction in the number of nursing roles. PSW staff have their own leadership structure and draw on their lived recovery experience to support consumers and guide clinical staff toward recovery-oriented practices. There is no specified interventional framework for PSWs at the CCUs. However, a qualitative study provides a rich description of what PSWs perceived their roles to be, emphasizing self-disclosure and connection through ‘shared engagement in everyday activities… providing authentic opportunities to support residents deal with their experiences and fears… [building] relationships and trust… [and] reducing shame and isolation' (p5) (Wyder et al., 2020).

Participants

Consumers were admitted to the CCU closest to their most-recent principal residence and were included in the cohort if they provided consent and stayed beyond the assessment period (6-weeks, n = 145/161). Ninety-one percent and 89% of eligible clinical and integrated staffing model site consumers consented. Recruitment exceeded the target to achieve > 80% power to detect a 15% difference in the Health of the Nation Outcome Scales (HoNOS) (Wing et al., 1998) set for the parent evaluation (n ≥ 100) that was based on the treatment change observed in an Australian community residential step-up/down service (Siskind et al., 2013).

Data Collection and Measures

Unblinded trained multi-disciplinary team members completed an assessment battery on admission and discharge. Diagnostic and demographic information was collected at admission, and treatment-related variables were collected at admission and discharge (see Table 1). Based on the literature, a range of known confounders relevant to understanding rehabilitation outcomes were available in our data set (see Supplementary Materials 1).

Table 1 Features of the study sites, including characteristics of the clinical and integrated staffing models.
Full size table

The assessment battery included measures relevant to the real-world planning of rehabilitation care (Parker et al., 2016). These covered: functioning and disability (HoNOS, Social Functioning Scale (SFS) (Birchwood et al., 1990), Allen’s Cognitive Levels (ACL) (Velligan et al., 1998), and Life Skills Profile-16 (LSP-16) (Rosen et al., 2001); clinical symptoms [Brief Psychiatric Rating Scale (BPRS-18) (Flemenbaum & Zimmermann, 1973), Scale for the Assessment of Negative Symptoms (SANS) (Andreasen, 1982), and Alcohol Use Disorders Identification Test (AUDIT) (Babor, Higgins-Biddle, Saunders, & Monteiro, 2001)]; and, wellbeing/recovery (Mental Health Inventory (MHI-38) (Veit & Ware, 1983) and the Stages of Recovery Instrument (STORI-30) (Andresen et al., 2013). Measures were selected based on the availability of data to support their reliability and validity, as well as pragmatic considerations relevant to the service context (Parker et al., 2016).

Defining Reliable and Clinically Significant Change

Difference scores (admission-discharge) and subsequent analyses were conducted on measures with > 50% of paired data. Where relevant, scores were transformed so that positive differences reflected improvement on all measures.

The reliable change index (RCI) was calculated using the Christensen and Mendoza formula (Christensen & Mendoza, 1986). The clinical significance of an individual discharge score was operationalized based on three cut-off methods (N. S. Jacobson & Truax, 1991): [Cut-off 1] More than 2 SDs from the dysfunctional population mean (i.e., cohort mean at admission); [Cut-off 2] Within 2 SDs of the functional population mean (i.e., normative data); and [Cut-off 3] Closer to the functional population mean than the dysfunctional population mean.

Normative data for Cut-offs 2 and 3 for HoNOS and LSP-16 came from a study of Australian individuals with a psychotic disorder accessing mental health services who had experienced at least one inpatient or emergency department admission within five years but none within six months (n = 114) (Maxwell et al., 2019). For BPRS-18 and SANS, normative data came from a study of community-dwelling individuals with clinically stable 'chronic schizophrenia' without admissions in the previous six months (n = 120) (Baynes et al., 2000). Relevant functional population data was not identifiable for the other measures; thus, only Cut-off 1 could be applied. Where skewed data limited the ability to interpret RCS based on Cut-off 1 meaningfully, RCS improvement was not considered (N. Jacobson et al., 1988).

Reliable and clinically significant (RCS) change was assumed where the change between admission and discharge score exceeded the RCI and crossed a clinical significance threshold (i.e., RCS improvement or RCS deterioration).

Analysis

Analyses were completed in IBM SPSS Statistics Version 27 (SPSS, 2017). Comparability of the three sites was supported by examining measures at admission using the Kruskal–Wallis test (Supplementary Materials 2). Data from integrated staffing model sites were merged for subsequent analyses.

Individual difference scores were categorized as 'reliable improvement ‘or’ no reliable improvement' (i.e., stable/deterioration) based on the RCI. The RCS improvement cut-off producing the largest proportion of improved consumers (Gonda et al., 2012) was used to categorize scores into ‘improvement’ and ‘no improvement’ for RCS change. Outcomes for the staffing model groups were compared using Chi-Square/Fisher’s Exact tests, with effect size estimated using Cramer’s V (Kim, 2017).

The impact of known confounders on the relationship between staffing model and reliable improvement was explored using binomial logistic regression modelling (see Supplementary Materials 1). Modelling was not undertaken for RCS improvement due to the low event rates. Independent variables (IVs) additional to the ‘Integrated staffing model’ entered in the final models were rationalized based on a threshold of p < 0.200 (Mickey & Greenland, 1989). All IVs were entered simultaneously. Where time-related covariates were included, interactions with staffing model were explored. Events-to-IVs ratios in the final models exceeded the acceptable minimum threshold (5:1) in assessing confounders (Vittinghoff & McCulloch, 2007).

Results

The sample included 145 consumers aged 18–59 years (M = 31.4, SD = 9.0, see Table 2). The median duration of CCU care was 303-days. Although there were no differences in the frequency of involuntary treatment on admission between the two staffing models, consumers under the integrated staffing model were more frequently voluntary at the time of discharge (χ2(1) = 4.061, p = 0.044). Paired admission and discharge data were available for > 50% of consumers on all measures except ACL and STORI-30 (Table 2).

Table 2 Characteristics of consumers by staffing model
Full size table

Most consumers showed reliable improvement on SANS (72.5%), MHI Index (64.0%), SFS (55.6%), LSP-16 (51.1%), and HoNOS (50.4%, see Table 3 and Fig. 1). Approximately half the consumers showed reliable improvement on BPRS-18 (47.8%). Using the RCS improvement criterion, gains occurred most frequently on the symptomatic measures (SANS, 37.4%; BPRS-18, 27.3%), and almost a quarter of participants improved on HoNOS (24.8%). Few consumers met the RCS improvement criterion on SFS (9.6%), MHI Index (2.2%), and LSP-16 (2.1%).

Table 3 Admission and discharge total scores on measures within the assessment battery where paired data was available for > 50% of participants (N = 145)
Full size table
Figure. 1
figure 1

Plot of admission and discharge assessment scores for measures where reliable and clinically Significant change was calculable (deterioration / no change / improvement)

Full size image

In unadjusted analyses, only BPRS-18 scores differed between the two staffing models. A higher proportion of consumers admitted under the integrated staffing approach were reliably improved compared to the clinical staffing approach (60.4% vs. 30.8%, OR 3.43, CI 1.43–8.22, Cramer’s V 0.293). No group differences were statistically significant when the stricter RCS improvement criteria were applied.

Covariate analyses are detailed in Table 4. The increased likelihood of reliable improvement in BPRS-18 scores among consumers in the integrated staffing approach (Exp(B) = 2.726, CI = 0.994–7.480, p = 0.051) was attributable in part to baseline differences in HoNOS Total (Exp(B) = 1.090, CI = 1.009–1.177, p = 0.029) and education level (Exp(B) = 0.454, CI = 0.248–0.830, p = 0.010). After adjusting for confounders, consumers in the integrated staffing model also had higher odds of reliable improvement in SFS (Exp(B) = 3.218, CI = 1.122–9.229, p = 0.030). Additionally, analyses indicated the likelihood of RCI improvement was lower for consumers admitted later in the cohort for HoNOS (Exp(B) = 0.635, CI = 0.407–0.989, p = 0.045) and SANS (Exp(B) = 0.440, CI = 0.232–0.835, p = 0.012) Table 5 Table 4.

Table 4 Comparison of reliable and clinically significant improvement in outcome variables where > 50% of paired admission-to-discharge data was available
Full size table
Table 5 Results of covariate analysis using logistic regression to identify predictors of reliable improvement
Full size table

Discussion

This study considered whether consumers receiving TRR support under integrated and clinical staffing models achieve equivalent functional and clinical outcomes at discharge. Regardless of staffing model, most consumers (50.4%-72.5%) showed reliable improvements in negative psychotic symptoms (SANS), psychological wellbeing and distress (MHI-38), social functioning (SFS), disability (LSP-16), and mental health and social functioning (HoNOS). The unadjusted odds of reliable improvement were equivalent between the staffing model groups on all measures, except in general psychiatric symptoms (BPRS-18, favoring those in the integrated staffing model). No significant differences emerged in the likelihood of RCS improvement between the staffing model groups. Covariate analyses suggested that consumers admitted under the integrated staffing model were more likely to experience reliable improvement than in the clinical staffing model on two outcomes (BPRS-18 and SFS). Additional predictors of reliable improvement on the BPRS-18 were having a primary diagnosis of schizophrenia, higher HoNOS total score on admission, and lower levels of education. Additional predictors of reliable improvement in SFS emerging through the covariate analyses were the presence of comorbid substance use, involuntary mental health act status, and lower LSP-16 total scores on admission.

The gains in clinical and functional outcomes are consistent with the literature supporting the positive impact of mental health rehabilitation (Chan et al., 2021; Dalton-Locke et al., 2020). Unlike a recent retrospective cohort study that included only clinical staffing model sites (Parker et al., 2020), reliable improvements in disability occurred for most consumers. However, our study focused on admission-discharge outcomes rather than pre-admission and post-discharge. Our results also compare favorably to a recent Australian inpatient rehabilitation cohort study (Maxwell et al., 2019), with higher frequencies of reliable improvement in HoNOS and LSP. However, the comparability of these outcomes is limited by our mean length of admission being over three times longer.

The absence of marked differences between the integrated and clinical staffing configurations is consistent with the quantitative literature considering PSWs as care providers in Australian (O'Donnell et al., 1999) and international (Pitt et al., 2013) clinical services. Our findings indicate that the integrated staffing model achieved at least equivalent outcomes and that consumers under this model were more likely to have their involuntary treatment revoked prior to discharge. These findings provide reassurance that reduced restrictive and pharmacologically focused treatment at CCUs under the integrated staffing model (Karan et al., 2022) is not associated with inferior clinical and functional outcomes. Additionally, our findings add weight to the relevance of considering consumer preferences in terms of their emphasis on the value of the availability of PSWs under an integrated staffing model (Parker et al., 2021).

Limitations

A key limitation is the absence of process evaluation to identify treatment differences between the approaches. Additionally, inter-rater reliability data was unavailable, and a later admission date was associated with a lower likelihood of reliable improvement on HoNOS and SANS, suggesting possible impacts of processes and staff changes. Unmeasured and unknown confounders may have also impacted the results. For example, service-level factors that impact organizational performance were not considered, such as staffing turnover, shortages, and burnout (Coates & Howe, 2015).

Outcomes were considered without correcting for multiple comparisons, increasing the risk of Type 1 error. The nature of the planned analyses meant that applying such corrections would have limited the ability to draw meaningful conclusions due to inflation of the Type 2 error risk (Armstrong, 2014). Another important consideration is limitations in statistical power to detect small differences between the staffing models. This is a particularly relevant consideration for the outcomes with higher levels of missing data (AUDIT, BPRS, SANS, and SFS). Given this, our findings should be interpreted cautiously.

Unavailability of paired admission data occurred more frequently for consumers who had experienced unplanned discharge. This means the findings may be biased toward consumers who are more likely to have favorable outcomes. Additionally, high rates of missing data prevented comparing the personal recovery measure (STORI-30), an outcome highly relevant to the service focus.

Conclusions

Reliable improvements in symptoms and functioning generally occurred between admission to and discharge from community-based residential rehabilitation. Furthermore, most consumers demonstrated clinically significant improvements in negative psychotic symptoms and disability. Under the integrated and clinical staffing models, consumers had at least equivalent clinical and functional outcomes. In the context of other emerging research, our findings further emphasise the promising nature of the integrated approach as an alternative to traditional clinical staffing models. More research in other contexts will enhance the ability for future decisions about mental health rehabilitation services staffing to be evidence-informed.