Background

Improving the quality of care that patients receive is a key objective for healthcare providers across the world. As well as safety and clinical effectiveness, patients experience of care is a key component of most definitions of quality of healthcare [1].

In recent years evidence has begun to emerge that patient-reported experience can offer a reliable indicator of the quality of care that services provide [2,3,4]. A systematic review summarising evidence from 55 studies, showed consistent positive associations between patient-reported experience and clinical effectiveness [2]. However, most studies included in this systematic review focused on patients using primary care and general hospital services. Only two studies included data from patients using mental health services and one study specifically excluded patients with a psychiatric diagnosis. It has been argued that assessment of patient experience of mental health services is challenging because people may not have sufficient information about what such services are able to provide [5], and because severe mental illness may impair a person’s insight [6].

Small scale studies have examined the relationship between patient-reported experience and other measures of quality of care. For instance, Shipley and colleagues [7] investigated the relationship between the quality of a community mental health service and patient-rated satisfaction with the care they received. Based on data from 113 randomly selected patients referred to this service over a two-month period, the authors found that patient-reported experience of care provided a more accurate indicator of quality of care than standard quality indicators such as waiting times. Although the results of this study support the use of patient experience as a measure of service quality in mental health settings, the generalisability of the results of study are limited by the small sample size and single site.

The second round of the National Audit of Schizophrenia conducted between 2013 and 2014 assessed the quality of care provided to patients with schizophrenia across all the Mental Health Trusts in England and Health Boards in Wales. As the audit provided data on a wide range of outcomes of care for patients with schizophrenia around the county, we set out to use data from this audit to examine whether patient-reported experience provides an indication of the services’ overall quality of care in a nationwide cohort of patients with severe mental health illness.

Methods

The National Audit of Schizophrenia provided an in-depth examination of service performance against 16 standards of care derived from national guidelines for the treatment of people with schizophrenia in England [8]. These standards concern the quality of physical health monitoring, prescribing practice, use of evidence-based psychological treatments and patient involvement in care planning. Our study comprised a secondary analysis of cross-section data from an audit of clinical records of the quality of care provided to people with psychosis and the results of a patient survey that was conducted in parallel with the audit.

Data collection

A total of 64 NHS Mental Health Trusts in England and Health Boards in Wales (referred to collectively as ‘Trusts’ in the remainder of this report) participated in the audit. Data were collected via an audit of practice tool and a contemporaneous patient survey. The audit of practice tool was developed specifically to measure standards of care and the relevant data were obtained from the patients’ case records (see Appendix for a list of the standards). The tool was completed by consultants, junior doctors and other allied health professionals within the services. The patient survey was developed to assess patient-reported experience of care. An expert patient reference group helped refine draft versions of both the audit of practice tool and the patient survey prior to the start of the audit.

Trusts were asked to complete the audit of practice tool for a random sample of 100 patients. In addition, Trusts were required to distribute the patient survey to 200 randomly selected patients including the 100 patients whose notes were examined using the audit of practice tool. The data collection was conducted between August 2013 and November 2013.

Patient sample

Patients were eligible for inclusion in the audit if they were aged above 18, with a current ICD-10 diagnosis of schizophrenia or schizoaffective disorders. To be eligible for the audit patients had to have received treatment from community mental healthcare setting within the Trust for at least 1 year.

A composite measure of global trust performance

For each Trust, we calculated the proportion of patients for whom each individual standard was met and the mean percentage of compliance for each standard relative to the total national sample. We then calculated a global trust rating score based on the mean percentage of compliance of each Trust across all standards assessed with the audit of practice tool. We also calculated five sub-scores from standards relating to:

  1. 1).

    The quality of physical health monitoring the patient received (e.g. assessments and interventions delivered).

  2. 2).

    The quality of patient involvement (e.g. whether there was shared decision making process in the selection of medication).

  3. 3).

    Compliance with recommended prescribing practice (e.g. extent of polypharmacy and prescribing above BNF recommended maxima).

  4. 4).

    Access and uptake of evidence-based psychological treatments.

  5. 5).

    Provision of a care plan jointly developed with the patients to meet their needs.

Patient-reported experience measures

The patient-reported experience measures we used included a measure of patient satisfaction with care and a patient-rated outcome measure. The question on patient satisfaction with care asked: “Taking everything into consideration, are you pleased with the care you have received from the service so far?” Responders had to indicate their answer by using a four-points Likert scale from 1 (not satisfied at all) to 4 (very satisfied). The question on patient-rated outcome asked: “To what extent have services helped you to achieve good mental health in the last year?” and patients responded using a four-point Likert scale ranging from 1 (made me worse) to 4 (helped a lot). We calculated the mean trust scores for these two questions and converted them into a 100-point scale.

Statistical analysis

We carried out Pearson product-moment correlations to investigate associations between patient satisfaction and patient-rated outcome scores with the global trust performance and subgroups of care scores [9].

Ethics approval and consent to participate

The National Audit of Schizophrenia (NAS) was commissioned by the Healthcare Quality Improvement Partnership (HQIP) as part of the National Clinical Audit Programme (NCA). As part of NCA, all Mental Health Trusts in England and Wales were expected to take part in NAS. Consent was not required from the patients to publish the audit data in the audit’s main report or for research purposes and the data were collected by the audit team in such a manner that the patients could not be directly identified.

Results

The audit team received 5733 returns from the audit of practice, of which 5608 were used in the analysis after data cleaning with a mean of 88 (SD = 17) returns per Trust. In total, 3379 patients submitted a survey form with a mean of 53 (SD = 19) per Trust and a response rate of 26.4%. Table 1 shows a breakdown of the demographic characteristics of the patients included in the audit of practice according to whether they did or did not respond to the survey. The demographic data on the responders and non-responders to the patient survey refer to the only two Trusts that were able to collect this information. The responders and not-responders group did not differ with regard to gender, ethnicity, and time since diagnosis.

Table 1 Demographic characteristics of the patients included in the audit and of those who responded and did not respond to the survey
Full size table

Association between patient-reported experience measures and overall trust rating

There was a positive significant correlation between the patient satisfaction and the patient-rated outcome measure (rs = 0.582, p < 0.01). In Table 2, we present (i) correlations between patient-rated outcome and quality of care data from the audit, and (ii) correlations between patient-rated satisfaction and quality of care data from the audit. While patient-rated outcome scores were significantly associated with the global trust rating, patient satisfaction scores were not. Figure 1 shows the scatterplot of the relationship between global trust rating and patient-rated outcome about their mental health.

Table 2 Correlations between the global trust rating and its subgroups with patient satisfaction and patient experience
Full size table
Fig. 1
figure 1

Scatterplot showing a positive linear correlation between global trust rating scores and patient-rated outcome scores among

Full size image

The results relating to the five subgroups of patient care we examined are also presented in Table 2. Only the provision of a care plan was significantly correlated with high patient satisfaction scores. In contrast, patient-rated outcome scores were positively correlated with patient involvement and with the services’ compliance with recommended prescribing practice. No significant associations were observed between the patient-reported experience measures and the other subgroups of patient care.

Discussion

Secondary analysis of data from this national audit of care for people with schizophrenia found that patient-rated outcome was positively associated with an independent rating of the global quality of care provided by these services. In contrast, we did not find an association between global quality of care and the patients’ satisfaction with the service they received. High scores on patient-rated outcome were also correlated with high patient involvement in the decisions about their care and with high compliance of the services with the recommended prescribing practice. In contrast, high levels of patient satisfaction were associated only with the provision of a care plan tailored to the patients’ needs. Nonetheless the strength of these associations are not strong and caution needs to be exercised in interpreting the results of this analysis.

While the previous work of Doyle and colleagues [2] demonstrated consistent positive associations between patient-reported experience and clinical effectiveness, our study demonstrates that these associations are also true of users of mental health services. The idea that patient-reported experience of care could be a valuable indicator of quality of care in mental health services has been previously tested only in small studies with patients from a single service provider [7]. Our study builds up on this work and counters previously reported concerns about the ability of people with mental disorder to provide valid feedback about the quality of care they receive [10].

The main strength of this study is that we were able to examine the relationship between patient experience and quality of care across a large number of providers of mental health services throughout England and Wales. However, the study also has a number of limitations which need to be considered when interpreting the results. Firstly, the audit used two different samples for collecting case note data and for the patient survey, which did not allow us to determine whether individual patient ratings related to the quality of care provided to them personally by their Trust. The use of aggregate data from different samples of patients within each Trust may have underestimated the strength of the associations we might have seen, had both sets of data come from the same patient sample.

A second limitation of the study was that the measures of patient-rate experience and the global trust rating we used have not been validated. The global rating of trust quality of care was developed from standards for the treatment and management of people with schizophrenia published by the National Institute for Health and Clinical Excellence following a systematic review of published literature and extensive consultation with patients and providers [8]. However, the construct validity of the patient-rate experience questions and the global trust rating we used have not been tested. A further limitation of the patient-rate experience questions was that they were based on a single item.

While we found a moderate degree of association between patient satisfaction and patient-reported outcome in our analysis, only the latter was correlated with the independent rating of global quality of care provided Trusts. This suggests that patient-rated outcomes might be a better indicator of the quality of care that a service provides than patient-rated satisfaction with care. We also found that the two measures of patient-rated experience were associated with different elements of care delivered by participating Trusts. The more the services promoted the patients’ involvement in decisions about care and higher levels of compliance with recommended prescribing practice were associated with patients being more likely to state that services were improving their mental health. On the other hand, services that had higher compliance with recommendations that patients should have a care plan that was jointly developed to meet their needs had higher ratings on patient satisfaction with care.

These results support previous research in other healthcare settings which indicate that patients rate their experience and outcomes of care according to different criteria and highlight the importance of both in assessing the overall quality of care that healthcare services provide [11, 12].

In recent years concerns have been expressed about the burden that providers of healthcare services face in assuring the quality of care they deliver [13, 14]. The results of this study, together with other reports linking patient experience with service quality and safety [15, 16] suggest that greater use of patient surveys could be made in assuring the quality of healthcare and reducing the burden of data collection on service providers and assisting efforts to improve the quality of care that patients receive [17].

Conclusion

In summary our findings highlight that, feedback from patients with severe mental disorders can provide important information about the quality of care they receive. Therefore, greater use of patient-reported experience measures should be made when assessing the quality of care provided to people with psychosis. Future work should focus on exploring the best way to combine patient views with audit data to obtain a comprehensive picture of the quality of care that services provide.