Background

It has been well established that people with a diagnosis of schizophrenia or bipolar have poorer physical health and social outcomes. The factors contributing to this are multiple, reciprocally interacting and include difficulties with engagement between healthcare providers and service users, along with health related behaviours. [1, 2]. There is evidence that primary care is centrally involved with the care of people with a diagnosis of schizophrenia, bipolar or other psychoses in the UK: Reilly et al. [3] found that nearly a third were seen only in primary care and those seen in secondary care received only minimal support. These authors identified a number of obstacles to primary care practitioners supporting this group to achieve improved outcomes, including a lack of continuity of care within primary care teams and poor continuity and information exchange at the interface with secondary care [3]. Primary care practitioners also often lack the necessary time and training to effectively support people with these diagnoses in addressing their mental health needs [4,5,6]. Collaborative care has been identified as a potential strategy for overcoming these obstacles [5] and has been found to be effective in improving mental, physical and social functioning across a range of mental health conditions [7, 8]. However, few trials have included people with a diagnosis of schizophrenia or similar forms of psychosis and so it is not possible to draw conclusions about its effectiveness for this group [5, 8]. Further, trials that have included people with a diagnosis of bipolar, schizophrenia or other psychoses have principally taken place in the USA, with none in the UK, where the organisation and funding of healthcare provision is substantially different.

The PARTNERS2 (develoPing integrAted primaRy care for paTieNts with sERiouS mental illness) programme intends to address this gap through the development of a model of collaborative care for people with a diagnosis of bipolar, schizophrenia or other psychoses, which will be tested through a randomised controlled trial in three sites in England. The process of the development of the model and the intervention theory are described in more detail in Gwernan-Jones et al. [9]. A Cochrane review [8] of collaborative care approaches for people with severe mental illness, defined as schizophrenia or other types of schizophrenia like psychosis (e.g. schizophreniform and schizoaffective disorders), bipolar affective disorder or other psychosis, has identified 14 components of collaborative care interventions. These are listed in Table 1, alongside a description of how they are operationalised in the PARTNERS intervention.

Table 1 Description of collaborative care components included in the PARTNERS model
Full size table

As with other collaborative care approaches [5], the PARTNERS model is based on Wagner’s Chronic Care Model [10]. Additionally, and as a subtle departure from other collaborative care interventions, which tend to be focussed on specific disorders, it incorporates personal recovery processes [11] as a specific orientation to supporting service users to become more informed and active in managing their mental health, as well as ensuring care is focussed to support their priorities. Coaching was selected as the psycho-social intervention to support this recovery orientation and an approach supporting goals identified by individuals.

The PARTNERS programme theory was developed using realist principles [12] aiming to identify underlying causal mechanisms by which the intervention had its effects and the influence of context on the operation of these mechanisms. Within the realist approach, mechanisms are defined as the reasoning and reactions of human agents in response to the resources provided by an intervention. Mechanisms thus reflect the internal mental processes which take place within the individuals involved in an intervention that lead to them choosing to make the desired changes in practices or behaviours necessary to implement the intervention or that form the outcomes of the intervention [13]. Mechanisms include a diverse range of processes, such as increasing acceptance, motivation, knowledge and skills, trust, and self-confidence [14,15,16]. The initial programme theory development resulted in a large number of ‘explanatory statements’ [9], which are summarised in Fig. 1 and Table 2.

Fig. 1
figure 1

The PARTNERS2 initial programme theory

Full size image
Table 2 Initial PARTNERS programme theory
Full size table

MRC guidelines for the evaluation of complex interventions [17, 18] recommend a feasibility and piloting phase, during which formative evaluation can take place, in order to test and refine the underlying programme theory and to establish what support would be needed to ensure delivery of the intervention as intended, and thus allow effective outcome evaluation during the main trial. As implementation of collaborative care for mental health in general has been limited, despite evidence of its efficacy [19], factors affecting delivery were a particularly important focus for the formative evaluation of the PARTNERS model.

Two recent systematic reviews, which included 9 UK studies, have identified barriers and facilitators to the implementation of collaborative care for depression and anxiety [20, 21]. These included practitioners’ understanding of collaborative care and their attitudes towards change, the presence or absence of regular communication between practitioners and structures which supported this and the qualities, knowledge and skills of the case manager, in engaging both with other practitioners and service users. A potential barrier was that some service users preferred not to have mental and physical health integrated but generally the experience of service users was not investigated. These findings are consistent with other systematic reviews of factors affecting implementation of collaborative care across all conditions [22, 23] suggesting they were also likely to operate during our feasibility study. There is further some evidence that facilitation can lead to improved implementation of collaborative care models for mental health conditions [24, 25].

Aims

In the context of the above evidence, the aims of our formative evaluation were therefore to:

  1. 1.

    Assess how well the intervention as delivered matched the model

  2. 2.

    Identify the barriers and facilitators to delivering the model as intended

  3. 3.

    Identify additional support for implementation likely to be required in the main trial

  4. 4.

    Test and refine our initial theory, through exploring any perceived effects of the intervention and comparing these to the programme theory

The use of qualitative data is recommended for addressing such aims [26] and so this methodology was adopted.

Methods

Ethical approval was obtained from the West Midlands – Edgbaston Ethics Committee (14/WM/0052).

Design

The formative evaluation consisted of the small scale delivery of the PARTNERS intervention in three geographically diverse regions and qualitative data collection about the experiences of the intervention from all stakeholders. In order to maximise rigour, we used multiple methods of data collection: semi-structured interviews, recordings of consultations between care partners and service users and tape-assisted recall interviews about these recordings with service users and care partners. This enabled us to triangulate multiple perspectives, including our own observations of the intervention in action, and participants’ reflections both on the intervention overall and specific interactions, allowing us to build a more detailed and accurate understanding of how the intervention had been delivered and experienced [27].

Intervention

Through negotiations with the NHS Trusts providing secondary care mental health services in each site, agreements were reached whereby a qualified mental health worker and a senior clinician would have a portion of their time re-allocated to delivering the PARTNERS service as a care partner and supervisor respectively. The portion of time allocated varied across the sites with one Trust agreeing to a full time equivalent and another agreeing to half a day a week. The individual practitioners were nominated by the Trusts as being available for and interested in the roles. The care partners were two mental health nurses and a social worker and the supervisors were a mental health nurse, a social worker and an occupational therapist currently working as team leaders.

A number of GP (general practitioner) practices in each site were approached to establish interest in receiving the intervention. The number recruited in each site varied according to the time allocated to the care partner, with a total of six practices receiving the PARTNERS service.

To deliver the components of the intervention involving direct patient contact, care partners met individually with service users, usually at their GP practice although this was negotiated individually and some meetings took place at service users’ homes or community venues. Consultations, known as intervention sessions, varied in length according to need, preference and stage of the intervention, lasting between 20 and 90 min. Initial sessions were used to build a therapeutic relationship and develop a ‘shared understanding’ of the service user’s life experiences and priorities for change. This fed into the identification of the service user’s personal goals, the exploration of resources which could help them meet those goals and the agreement of action plans, which would be reviewed in subsequent sessions, along with routine monitoring of service users’ mental health and wellbeing using standardised measures: the CORE-10 (Clinical Outcomes in Routine Evaluation 10-item) [28] and the WEMWBS (Warwick Edinburgh Mental Wellbeing Scale) [29]. Care partners would liaise with other people and services who could provide support for the service user to work towards their goals on an individually agreed basis.

Care partners were based at GP surgeries for the allocated time with the expectation that they would become active members of that primary care team, engaging in two way consultation with other team members in whatever way was appropriate and recording information about service users in computerised record systems.

Care partners were expected to meet with supervisors at least once a fortnight for guidance on goal setting, level of intensity of care and decision-making in relation to working collaboratively with other services and agencies. A protocol was provided to ensure that care partners and supervisors maintained an overview of all service users on the caseload.

The inclusion criteria for receiving the intervention were: adults with a diagnosis of schizophrenia or bipolar, including those only receiving mental health care from general practice and those with stable but ongoing mental health needs under specialist services; individuals receiving acute crisis care were excluded. Service users were identified by screening primary and secondary care records and their suitability was confirmed by a practitioner who was known to them. They were then approached by the practitioner to discuss receiving the new service. The number of service users varied across sites, according to the time allocated to the care partner, with a total of thirty-eight service users receiving the intervention.

The intervention was delivered over a period of eight to 10 months; expectations of the minimum number of contacts a service user would receive were adjusted so that they would have at least two contacts during this period. The intervention was delivered concurrently in sites A and C but the start was delayed by 6 months in site B. No changes were made to the intervention in Site B of the basis of the evaluation of the other sites but this did influence the data collection (see below).

Participants

Participants in the formative evaluation were service users receiving the intervention, their family carers, and practitioners involved in delivering and supporting the service. A required sample size for each category of participant in each site was agreed prior to data collection, with an aim to achieve maximum diversity in the sample rather than data saturation. This was a pragmatic decision made in relation to funding constraints and we judged that these sample sizes would provide sufficient data about the range of experiences. Data was collected from sixteen service users, five family carers, three care partners, three supervisors, four GPs and six other primary or secondary care practitioners.

Procedure

Topic guides for semi-structured interviews were developed collaboratively by the research team, with input from members of the PARTNERS2 Lived Experience Advisory Panels (LEAPs). The LEAPs consisted of people with personal experience of living with schizophrenia or bipolar, as a service user or a family carer, and an interest in research. The study has one LEAP per site, three in total.

Topic guides focussed on the key elements of the PARTNERS service, as specified in the manual and associated service user guide, whether and how the participant had experienced these in practice and what difference this made in terms of the quality of care delivered. Time pressure on sampling prevented pilot testing of topic guides. Copies of the topic guides are available on request from the authors.

We adopted the tape-assisted recall method outlined by Cape et al., [30] whereby consultations were audio-recorded and reviewed by researchers, to identify five or six excerpts from the session where there seemed to be divergence from the model or where the care partner and service user may have perceived the care partner’s action differently. Service users and care partners were then invited to participate in separate tape-assisted recall interviews, in which these excerpts were played back and the participant was asked for their views of what had been happening at that point and whether this was helpful or unhelpful.

After service users had been receiving the intervention for a few weeks, researchers contacted them by phone to attempt to recruit them to the evaluation. Service users were asked to consent separately to the different forms of data collection. In site A, service users were sampled purposively for interviews to achieve a range of gender, diagnosis and experiences of mental health services but were identified for session recordings by the care partner. In site B, service users were identified for interviews and session recordings by the care partner on the basis of their availability and willingness to participate. In Site C, all service users were approached by researchers for interviews, although not all were available. One service user declined to have a session recorded and one was not approached for a session recording as the care partner judged it would be too distressing. Participating service users were asked to identify family carers who could be approached. Family carers were then approached by phone; all family carers who were approached agreed to be interviewed. Practitioners who had some form of contact with the PARTNERS service were contacted by phone and asked to consent to data collection relevant to their role; all those approached agreed to participate.

Data was collected over a 6 month period from sites A and C, where the intervention was delivered concurrently. Preliminary analysis of this data, which identified which components of the intervention had not been delivered as intended in these sites and possible barriers and facilitators informed the approach to data collection in site B, such that the care partner and supervisors were re-interviewed towards the end of the intervention period to see what learning through experience might have taken place.

Data was collected by a number of researchers (MC, LGa, LGi, RGJ, CMc, SR, AR), including two service user researchers. All received training in qualitative methods as appropriate, and the use of the specific interview techniques, from members of the research team experienced in qualitative research (RGJ, NB, VP, LGa).

Semi-structured interviews lasted one to 2 h and tape-assisted recall interviews lasted about 1 h. All service user and family carer interviews were conducted face to face, at the participant’s GP practice or home, according to their preference. Two service users chose to have family members present during the interview but otherwise participants were alone. Practitioner participants were interviewed face to face at their regular place of work or by telephone, according to their preference. Service user participants, care partners and supervisors had met the researcher previously as part of the process of setting up the PARTNERS service and so the interviewer would have been familiar to them. Other categories of participant had not met the interviewer previously. Service user researchers were encouraged to disclose their experience of using mental health services to service user participants. Otherwise participants were informed on Participant Information Sheets that the interviewers belonged to a group of ‘very experienced mental health researchers’. All interviews were audio-recorded and interviewers also made field notes after the interview, using an agreed template. Service users and carers received a £10 shopping voucher for each interview and recorded intervention session.

Several participants contributed to more than one source of data and the total number of data sources by type is shown in Table 3.

Table 3 Number and type of data sources by site
Full size table

Data analysis

All data sources were transcribed and imported into NVivo 11. Transcripts were checked against the recordings for accuracy by researchers but not returned to participants due to time pressures created by preparing for the full trial. Preliminary deductive thematic analysis, was carried out by a number of the interviewers (MC, LGi, RGJ, CMc, AR). Data from sites A and C was coded against the detailed sub-components of the model and intended outcomes as articulated in the manual and wider programme theory. A descriptive analysis was then conducted (RGJ) that was reported and discussed at a researcher meeting, where areas for further exploration were prioritised as needing to be understood to support planning for the main trial. Written summaries of key issues relating to each component were then produced for discussion at a stakeholder meeting.

The stakeholder meeting was attended by the whole research team, care partners and supervisors and members of the LEAPs, enabling data interpretation to be grounded in lived experience. This meeting focussed on understanding why some components had not been delivered or experienced as intended and identifying strategies to address these issues; the outcome of these discussions formed the basis of the implementation support strategy and informed revisions to programme materials in advance of the full trial.

Subsequent to this meeting, further data was collected; this consisted of two tape-assisted recall interviews with the care partner and one interview with a service user in site C and all the data collection in site B. There was insufficient time to revise topic guides to take account of the emerging themes but the preliminary analysis informed the development of new topic guides for the follow up interviews with care partners and supervisors in site B. Once all the data had been collected, the entire data set was analysed by one researcher (EB) with more detailed themes, and the relationships between them, being identified and defined. Coding of data from the diverse range of sources required a degree of interpretation, especially where observational data was inconsistent with interview data or two informants gave conflicting accounts of the same event. These interpretations were supported by reference to field notes and discussion with a researcher who had been involved in collecting data and the initial coding (RGJ) and an experienced qualitative researcher (NB), taking into account the main coder’s professional background as a clinical psychologist. A framework analysis [31] was then used to systematically explore differences between data sources and reach conclusions about how best to understand the data. Interpretations were shared and discussed with the wider researcher team.

In order to address the aims of the study, the analysis initially focussed on barriers and facilitators to delivery of the intervention components. The themes identified were compared against the ongoing actions agreed at the stakeholder meeting to ensure that the emerging implementation support strategy still reflected the data. Subsequent analysis focussed on the intended outcomes of the intervention. The themes identified for each outcome domain were compared to the mechanisms hypothesised in the programme theory, as articulated in the intervention logic model and ‘explanatory statements’ [9] (see Fig. 1 and Table 2).

Results

Sixteen service users, five family carers and sixteen practitioners participated in the formative evaluation; distribution of participants across the three sites is shown in Table 4, along with the demographic data.

Table 4 Distribution of participants and service user demographic data
Full size table

Aim 1. Evidence of whether the intervention as delivered matched the model

The preliminary analysis identified that a number of components of the model had not been delivered as intended across the sites and one component required clearer specification as it was not clear whether what had been delivered was consistent with the intervention. The extent to which each component of the model was found to have been delivered is summarised in Table 5.

Table 5 Model components not consistently delivered as intended
Full size table

Aim 2. Barriers and facilitators to delivering the model as intended

Further analysis initially focussed on identifying barriers and facilitators in relation to the components that had been identified as not being delivered. The most informative data was available in relation to three components: systematic communication, psychological intervention (in the form of coaching and goal setting) and multi-disciplinary working (in the form of supervision by senior mental health practitioners); we will therefore present detailed findings in relation to these.

Limited information was available about the barriers to provider integration as our topic guides did not focus on systemic issues and few of our participants worked at an organisational level where they would have been able to comment on this. Interpretation of the data in relation to the other components was complicated by inconsistencies between data sources. Data from session recordings, service user interviews and primary care practitioner interviews indicated that care partners were not fully delivering the model; however this was not necessarily recognised either by the care partner or their supervisor. Consequently care partners and supervisors did not offer any account of barriers and descriptions of facilitators were considered only partial accounts. In relation to systematic monitoring and shared decision making this prevented any firm conclusions being reached about barriers and facilitators. The themes that were identified as barriers and facilitators to systematic communication, coaching and goal setting, and supervision are shown in Table 6.

Table 6 Barriers and facilitators to delivery of systematic communication, coaching and supervision
Full size table

Systematic communication

The extent to which care partners communicated with primary care teams appeared to be influenced both by the atmosphere of the primary care service and the communication style of the care partner. Communication was most likely to take place when the primary care service was hospitable and the care partner had an interpersonal route in to the care team, who could raise awareness of the care partner’s role and provide practical support:

the manager…was very welcoming and introduced me to as many people as possible and… assigned a – I think she’s an admin worker or something, to me, so if I have any problems I just go to M. and M. does everything, and it’s great. [care partner 3]

Conversely if the care partner’s presence went ‘under the radar’ [GP4] this acted as a barrier.

Care partners were still able to successfully communicate with primary care teams, if they were pro-active in creating and capitalising on opportunities to do so, either routinely, through using records or attending meetings, or for a specific clinical purpose. However, care partners’ lack of knowledge about primary care systems and processes could reduce their confidence to approach primary care team members, such that they focussed on more familiar clinical work rather than liaison:

in [name of surgery] I feel that it’s kind of in process now, it’s working, people are turning up and it’s almost like, well, do I need to do something? People are aware that I’m there, but I don’t feel that they have the quality understanding of why I’m there [care partner 3]

The support that care partners sought, and were offered, to overcome barriers to communication was a further influence, with some supervisors not being aware that this was an area of difficulty and so the issue was not addressed:

I can’t remember him saying that there was ever a particular issue? Not one, maybe, that he brought to supervision. [supervisor 2]

In contrast, another care partner sought support from the research team, which seemed to lead to greater engagement with primary care.

If service users wanted the care partner’s support in accessing physical health care this also created opportunities for the care partner to liaise. However some service users did not see the value in this, which acted as a further barrier to liaison:

If I need to see me GP about health problems, I just go to me GP, I don’t involve [name of care partner]. [service user 1]

Coaching and goal setting

A goal-setting approach was most likely to be delivered when both care partners and service users saw the value of it, and some service users identified it as a key active ingredient of the intervention:

the goal-oriented approach for me is the crucial factor [service user 11]

However, there were times when both service users and care partners questioned the value of goal setting, suggesting that the emotional support provided by the sessions was as important:

I was just thinking about sometimes allowing the sessions to just be, because whilst there is an agenda, … of the coaching… sometimes I’ve found people don’t want to necessarily be coached, but they want to come along to the sessions [care partner 3]

This appeared to be linked to an interpretation of ‘goals’ as relating to practical achievements rather than psychological needs, which service user 11 described as ‘airy fairy, nebulous goals’.

The care partner’s pre-existing personal style, and whether this was congruent with a coaching approach was also influential. Where coaching was too great a contrast to the care partner’s usual practice, this tended to result in the prioritisation of their perception of the service user’s needs, especially in relation to managing risk:

To me, me getting [name of service user] and taking her physically and saying ‘Come on, [name of service user]’ and if I have to ‘There, there’ and wrap her up in cotton wool for a little bit, I’ll know the job was done and dusted then… I’ll know if she’s safe. [care partner 1]

The availability, or lack, of supervision in the use of a coaching approach was also linked to delivery. One supervisor was comfortable and familiar with coaching, which supported the care partner, but another was not able to provide such guidance, leaving the care partner struggling:

I was getting meself bogged down …and I’d go to me manager and I’d say …‘ I need help here’, and it was like ‘Read your manual’ and I felt like saying [shouts] ‘You read the manual!’ [care partner 1]

Service user motivation also impacted on the delivery of coaching and goal setting, with some not engaging with the approach as they did not wish to make changes and others taking it up readily:

some people have already set their own goals, they’ve been very good… very insightful into their own difficulties … just thinking of somebody that I’m working with at [name of surgery], she’s been very good at planning what she needs to do to improve her quality of life [care partner 3]

Supervision

The delivery of systematic communication and coaching was therefore underpinned by the delivery of supervision. While some supervisors were able to prioritise providing PARTNERS supervision, in one site there were too many competing demands:

we took on the single point of access team, recovery team as well as the crisis team, so my role sort of really expanded. So I feel really bad because I didn’t apply myself the way I probably should have done [supervisor 1]

Consequently, the care partner felt inhibited about seeking supervision:

I know he’s busy [care partner 1]

The supervisor’s availability was associated with the extent to which the supervisor understood the model. When the supervisor found it hard to understand, it was more likely to be experienced as an additional demand. Conversely, regularly engaging with the model increased the supervisor’s familiarity with it and commitment to the intervention:

the manuals are on the table but they’re closed because – I know how we’ve been working over time and I understand the model [supervisor 3]

Whether the content of supervision was consistent with the model appeared to be influenced by the extent to which both supervisors and care partners recognised the need to challenge the care partner to engage in detailed discussions about their work. Where supervisors assumed the care partner was competent and the care partner did not recognise a need to change their practice, supervision did not focus on the intervention elements, as intended. This focus was provided when the supervisor engaged the care partner in ‘extended case discussions’ [supervisor 3] and the care partner valued the opportunity to reflect on their practice:

what I like is supervisions where people are challenging me and… giving me ideas of how I can improve upon things… rather than saying ‘Oh yeah, it’s really great’ [care partner 3]

Aim 3. Additional support for implementation likely to be required in the main trial

Drawing on these findings and approaches to facilitation that have been found to be effective in supporting the implementation of collaborative care for mental health [24, 25], we developed an implementation support strategy for the main trial. This included actions in relation to each of the three components where barriers and facilitators were identified.

Actions to facilitate systematic communication between care partners and primary care practitioners

  • A link person to be identified within the primary care team, who will contribute to a local needs assessment, clarifying the most effective strategies for the care partner to communicate with the team about their work.

  • Regular contact between researchers and care partners to ensure these strategies are being used and identify any concerns, and regular contact with the practice link person for problem-solving.

  • The supervision protocol to be amended to include reviewing communication with primary care practitioners and providing support for developing communication strategies.

Actions to facilitate delivery of coaching and goal setting

  • Revisions to care partner and supervisor manual and training, clarifying that goals can be psychological as well as practical and may include addressing the need for ongoing emotional support.

  • Revisions to care partner and supervisor manual and training to provide clearer examples of coaching approaches and more opportunities for care partners and supervisors to practice through role play.

  • Additional follow up training, in which care partners reflect on audio-recordings of their own work, together with supervisors to support learning from practice.

Actions to facilitate delivery of supervision

  • Clear, negotiated agreements with secondary care service leadership about supervisors’ time commitments and regular contact between researchers and supervisors and researchers and service leadership for progress checking and trouble shooting.

  • Follow up training focussed on audio recordings of care partners’ practice, to encourage critical self-refection and support supervisors engaging care partners in detailed discussion of their work.

Aim 4. Comparison of perceived effects to the programme theory

Our preliminary analysis also identified a number of components that had been delivered but had not been experienced as intended. This mostly related to the experience of a few service users and were not universally reported. These were: not relating to the language of one of the measures used for ‘systematic monitoring’ and experiencing its use as burdensome rather than supportive; concerns about the use of coaching as the ‘psychological intervention’ because of its focus on identifying goals, which was felt to be unrealistic in the context of complex difficulties, and finding the service user guide, intended as a support for ‘shared decision making’ to be too lengthy, complex or paternalistic. Additionally, despite the provision of ‘education for mental health providers’ we found that care partners and supervisors sometimes did not have the knowledge or skills required to deliver their role in key areas following training, including physical health care and use of IT systems.

In relation to the programme theory, these findings indicate that practitioners did not react as expected to the resources provided the intervention training, in that they did not develop the knowledge required to deliver the intervention. Additionally, service users did not react as expected to some of the resources provided by the care partner, in that rather than becoming activated in managing their mental health they felt directed to attend to it in particular ways. Discussion of these findings within the team concluded that the failure of these mechanisms to operate as intended was a consequence of inadequate operationalisation of the programme theory, through the care partner manual and training and the materials used with service users, such that these did not adequately convey what was intended. It was agreed that, rather than revisions being required to the programme theory itself, revisions should be made to these practical expressions of it, in consultation with members of the LEAPs, in order to find forms of expression that would be experienced as more empowering for service users.

Further analysis focussed on the intended outcome domains shown in Fig. 1: physical health, stability of mental health and quality of life, including the personal recovery processes of connectedness, hope, identity, meaning and empowerment. The themes identified in each of these domains were compared to the programme theory, to see whether it reflected the operation of the hypothesised mechanisms.

Physical health

We found evidence for three mechanisms operating as predicted by the programme theory. Firstly, care partner liaison with GPs could increase GP’s understanding of service users’ needs, resulting in GPs providing appropriate support, including feeling able to attend to the service user’s physical health concerns rather than mental health:

I can see that one patient that did come to see… my colleague, he was able to focus on his shoulder and his other thing that he’d come about and not try to do everything all at once. ‘Cos the danger is, you see a patient who’s on those sorts of drugs, you think ‘Oh god, I really need to think about their mental health’ and ‘Oh my god I need to check they’re not seeing things or they’re not in danger’, and then… there’s no time to look at stuff like stopping smoking or what their blood pressure might be [GP 4]

Secondly, closer working with primary care could increase the care partner’s awareness of physical health issues and the support available, resulting in them introducing this as a possible goal with service users:

it’s also having that wider perspective of kind of interest… looking at… physical history, which I wouldn’t normally have access to those kind of records [care partner 3]

Thirdly, when the care partner shared knowledge about physical health, this could increase service user knowledge, empowering them to seek appropriate support from primary care:

the other week she [care partner] was explaining that… I should have been having more… heart checks and stuff with the medication I’m on…and that’s the only thing that is a little bit weak, ‘cos I only had my first ever ECG [electrocardiogram]… a couple of weeks ago, and that’s ‘cos I asked for it. [service user 12]

Quality of life and CHIME framework outcomes

Also consistent with the programme theory, we found evidence that service users responded to the resources provided by the care partner, particularly the coaching approach and consistency of contact, by experiencing increased hope and confidence in their ability to change as well as trust in the support from the care partner. This resulted in them taking small steps towards goals such as starting voluntary work or increasing their social contact:

it’s giving me the confidence to get back out on the tightrope knowing that there’s safety net underneath me [service user 11]

There was also evidence that this set up a virtuous cycle, whereby increased activity reinforced service users’ confidence, as predicted by the programme theory:

we take the dog over there and we walk around the park a couple of times and I come home and I feel, yeah, I've got out and that's an achievement. Where beforehand I would have thought, ooh, no, no, I just can't do that. [service user 7]

Stability of mental health

We also found evidence that service user responses to coaching and consistency of contact with the care partner resulted in service users changing their behaviours in ways which would help maintain the stability of their mental health, in the ways predicted. This included trust in the care partner leading to a service user becoming more open and gaining reassurance:

there was a sense that [name of service user] felt that we were in a comfortable space, a comfortable environment for him to be able to talk about… he said that he could have bizarre, dark thoughts at times in the past. And I guess it’s a place that we can talk through those and then kind of – we can explore whether it’s something that’s out of the ordinary or whether it’s just kind of eccentric feelings, views, thoughts or whatever, to talk those things through [care partner 3]

Additionally there was some evidence that monitoring of service users’ mental health could increase service user hope and confidence about their ability to maintain stability and access to support if necessary:

even though I find putting a number on it is quite hard, but at the same time it gives you that general score of where you are… you might be depressed, but you're not having bad thoughts, you're not kind of not confident but you don’t need to worry [service user 12]

There was also evidence that care partner liaison with secondary care practitioners led to increases in their understanding of individual service user needs and preferences, resulting in appropriate support being provided, in the way expected:

links to the psychiatrist, that’s been pretty good, ‘cos there was somebody …whose medication had been changed around and as a result she’d become a lot less motivated … so I spoke with a psychiatrist here, ‘cos I knew that she was going to be meeting as an outpatient, um, to see what their thoughts were about medication, and they’ve changed it back and she’s very pleased about that [care partner 3]

Discussion

This formative evaluation of a novel collaborative care intervention for people with a diagnosis of bipolar, schizophrenia or other psychoses in England had four aims: we will discuss our findings in relation to each in turn.

How well the intervention as delivered matched the model

Our data indicated that while many of the intervention components were delivered as intended, a number were only partially or inconsistently delivered, with one component requiring clearer specification as it was not clear whether it had been delivered as intended. Key components that were not delivered in at least two, if not all three, sites were: increased interaction between care partners and primary care practitioners, either through becoming integrated into primary care teams or through routinely sharing information about service users, supervision of the intervention, use of a coaching and goal-setting approach, and a collaborative style of interaction with service users.

These findings are consistent with those of other qualitative evaluations of the delivery of collaborative care for mental health in England, with limited integration of specialist mental health workers into primary care teams being most commonly identified [32,33,34]. Other evaluations have also identified inconsistent delivery of supervision [35] and practitioners having difficulty adapting to a new therapeutic style [36]. This indicates that these components may need additional implementation support in order to enable evaluation of the intervention in a full trial and that such support may also need to be considered if more widespread implementation is indicated.

Barriers and facilitators to delivering the model

We identified a number of factors that influenced the delivery of increased communication, the use of coaching, and supervision: service leadership that supported organisational change through raising awareness of new roles and making resources available, the use of strategies to support regular communication between practitioners, the communication skills of care partners, both with other practitioners and service users, practitioners’ understanding of the intervention, effective supervision and service user motivation. Our results indicate that recovery-focussed collaborative care for people with a diagnosis of bipolar, schizophrenia or other psychoses is most likely to be implemented when the primary care team are receptive and the secondary care practitioner is prepared to adopt new working practices, including a pro-active approach to liaison and a more positive approach to risk. Further, that such changes to practice may be most effectively supported by supervisors with sufficient time and knowledge and when practitioners are sufficiently self-aware to engage in focussed reflection on practice. The intervention is most likely to be accepted by service users who are ready and willing to make changes, especially in addressing their physical health.

The barriers and facilitators that we identified are largely similar to those described in two recent systematic reviews of the implementation of collaborative care for depression [20, 21]. However, there has been limited previous investigation of service user views and experiences of collaborative care for mental health [21]; our findings indicate that service user motivation may also be a contributing factor. These reviews conclude that evidence-based approaches to implementation should be used to overcome such barriers, especially adequate training and supervision and setting up robust systems for communication; we have attempted to achieve this through our implementation strategy.

Implementation support required for the main trial

Our implementation strategy aims to support increased interaction between care partners and primary care practitioners and the delivery of supervision through researcher facilitation, which has previously been found to be effective in supporting the implementation of collaborative care for mental health [24, 25]. This will identify communication strategies tailored to the local context and monitor their use, and monitor the delivery of supervision, engaging with service leadership to support the delivery of the new roles and systems where necessary.

We aim to improve care partner understanding and skills in relation to coaching, goal setting and collaborative interaction styles through additional training and clearer written information in the manual. The extent to which care partners adopted a coaching approach appeared to reflect their preferred style of practice. Clinicians’ personal style is a well-established barrier to implementation in primary care settings [37]and was also identified as influential during a randomised controlled trial of a team level pro-recovery intervention, which had a coaching component [38, 39]. The process evaluation of this trial identified that training was most effective when it involved opportunities to use the skills in practice, through role play with colleagues or initial experiences with service users, and receive direct personalised feedback. We have revised our training process to provide increased opportunities for skills practice and feedback, through role play and the use of audio recordings of sessions. Whether these strategies will be successful in influencing practitioners’ personal style is a remaining ‘key uncertainty’ [18] and will therefore be included as a specific research question in the process evaluation of the full trial.

Our data also indicated that when service users were ready and motivated for a collaborative care style intervention that they responded well. As, through its use of a coaching approach, the PARTNERS intervention focusses on building readiness for change, increased support for care partner skills development should also address barriers arising from service user motivation. Whether the reliable application of such an intervention can enable service users who are less receptive to become more engaged is an additional question that can be explored in the process evaluation of the full trial.

Comparison to the programme theory

The partial delivery of key intervention components limited the extent to which we were able to test the programme theory. Our preliminary analysis identified instances where intervention components were delivered but not experienced as intended; these appeared to reflect difficulties with operationalising the intervention and learning from these contributed to the process of revising the written materials and training that supported intervention delivery.

Where there was evidence of the intervention being delivered as intended, the themes identified in relation to each of the intended outcome domains reflected the mechanisms proposed within the programme theory. This indicates that the programme theory itself does not require revision in advance of the full trial and provides some validation of the comprehensive theory development process outlined in Gwernan-Jones et al. [9]. However, the model requires more robust testing during the full trial, when we anticipate that increased implementation support will lead to greater fidelity and there will be thus be more opportunity to explore in depth how the intervention may have its effects.

Strengths and limitations

In contrast to existing studies of the implementation of collaborative care for mental health, which did not collect data from all professional groups [20] or service users [21] we collected data from a wide range of sources,. We also made use of session recordings and linked tape assisted recall interviews, which deepened our understanding of interactions between care partners and service users [27]. However, there were inconsistencies between some data sources, and we did not make direct observations of how care partners related to primary care teams or supervisors. Consequently, interpretation of this data required a greater degree of inference about potential influences on implementation and uncertainty remains. Care partners and supervisors were possibly motivated to present a positive impression of the intervention rather than engage in critical reflection. This indicates a need for a different approach to understanding care partner and supervisor experience in the process evaluation of the full trial. Increased researcher facilitation may result in better rapport between researchers and practitioners and create opportunities for honest feedback. We have also revised our interview topic guides, to attempt to convey a spirit of mutual enquiry rather than evaluation.

Sampling of service users was to some extent reliant on care partners and is therefore likely to have been influenced by their perceptions of service users. This may have resulted in a positively skewed response from service users. A consistently purposive approach to sampling will be important in the main trial to enable exploration of the full range of service user experiences.

This study provides an example of the contribution of formative evaluation in the development of a complex intervention, of which few are available [40]. However as delivery of the intervention was variable, interpretations about how the intervention worked in practice in comparison to the realist programme theory and facilitators to delivery are based on limited data. Further, the initial data analysis was driven by a pragmatic need to address key issues in planning for the main trial which was imminently due to start and therefore did not involve detailed comparison between the mechanisms specified in the programme theory and the data. However, as the themes derived from our subsequent analysis were consistent with the logic model and detailed theory articulation, this provides some validation of our programme theory. Data analysis in the full trial will adopt a systematic approach to whether the mechanisms specified by the programme theory are operating.

Conclusions

This study found that enhanced implementation support is likely to be required for the PARTNERS model to be delivered as intended; this has implications both for the full trial and for healthcare services attempting to establish collaborative care for mental health as routine practice. In particular, additional support may be required to improve understanding of, and increase practitioner confidence to implement, new working practices. Practitioner training may need to focus on practical skills development for supporting service user activation. It may be that the impact of introducing a recovery-focussed collaborative care intervention is limited by both practitioner and service user readiness to change and there is a need for a greater understanding of the contextual influences which contribute to this. This question will be addressed through the process evaluation in our main trial and has shaped our approach to data collection. Although we found promising indications that the PARTNERS model may operate as predicted by the programme theory, there is a need to test this more fully, an opportunity which the full trial will also provide.