Background

Information provided in the referral process constitutes the main communication from primary care to specialist health care [1, 2]. Existing literature reveals, however, that the information relayed in the referral process is seen as insufficient by the receivers [1–4]. The transit from primary care to specialist health care constitutes a major clinical handover situation, implying a large risk of adverse events [5, 6]. Further, coordination across services is one of the major challenges to health care [7]. Improving the information transference is the principal means of reducing the risk of adverse events in clinical handovers and ensuring continuity and coordination of care [5, 6, 8–10]. Mental health care is often provided by various primary health and social services, in combination with periods of specialised mental health care [11]. Patients with mental illnesses are therefore particularly vulnerable to the effects of insufficient referral information. Nevertheless, there is a striking lack of research on whether and how the quality of referral information affects the quality of care [1, 12].

To explore the impact of referral information on quality of care, as well as the underlying mechanisms through which this effect may be realised, it is necessary to establish valid measurements for the output of the referral process [13]. The quality of the referral process can be assessed on three dimensions: necessity (whether the patient should be referred), destination (where the patient should be referred) and quality [14]. The ‘quality’ dimension concerns the process of referral, in which the quality of the referral letter is essential [14, 15]. Sufficient information is the most essential criterion for assessing the quality of the referral letter; most of the existing literature on referral letters’ quality and interventions to improve this is on the completeness of information relayed in the letter [1, 2, 15]. The construct of ‘high quality referral letter to specialised mental health care’ is therefore often defined by the completeness of information in the letter, as was done by Hartveit et al. [16]. The Quality of Referral information-Mental Health (QRef-MH), a recently developed and tested instrument, provides a valid operationalisation of the construct [17]. The instrument includes 19 items regarding identification of the patient, essential introductory information (included as check-off points), case history and social situation, present state and results, past and on-going treatment efforts and involved professional network, the patient’s assessment, and reason for the referral [17].

Existing literature reveals a large set of outcome indicators relevant for exploring the quality of health care, including readmission rate, mortality and patient experiences measured through surveys [18]. Indicators can be defined as ‘measures that assess a particular health care process or outcome’ [19]. They should be valid and reliable, sensitive to change, acceptable, feasible and easy to communicate [13, 19]. Indicators are used to assess structures, processes and outcomes in health care [19]. Existing outcome measures do not enable us to understand how and why referral information may affect the quality of care. It has therefore been recommended to develop indicators for sub-processes in health care, such as the referral process [12, 19, 20].

Exploring the underlying mechanisms through which referral information may influence quality of care is recommended for several reasons. First, an understanding of the underlying processes linking referral information to quality of care (e.g., mediating factors) will enable us to develop interventions tailored to support these mechanisms [13]. Second, mediating factors can affect a wide range of important outcome measures [13]. Consequently, the detection of such key mediating factors will facilitate the effective improvement of outcomes. Third, the use of indicators measuring mediating factors will make possible the identification of improvement potential and evaluation of improvement efforts, because these indicators are more sensitive to change than are outcome measures [20]. In the complex intervention of a care pathway (a systematic method to improve care across different patient groups), which is found to be effective in improving coordination and communication in health care processes, indicators serve an essential role in the improvement process [21]. For research purposes, revealing mediating factors is essential for developing theories of causality and exploring to what degree changes in these factors predict improved patient outcomes [22, 23]. The thorough development of valid process and outcome indicators is supported by the guidelines of the United Kingdom’s Medical Research Council ((UK) MRC) for exploring the causality and predictive value of a complex intervention on relevant outcomes [24]. Theory and evidence derived through research exploring components in complex processes and interventions will enable the informed use of theory in improvement programmes, as recommended by Davidoff and colleagues [25]. For mental and substance use health care, the development of indicators is particularly recommended, because few measures have been developed and the improvement infrastructure within these services suffers from limitations [11]. This is also true for the referral process, where the evidence for valid indicators to detect the mechanisms and effects of improved referral is clearly limited [1].

The present study’s aim was to develop quality indicators measuring the impact of referral information from primary care to specialised mental health care to explore how the quality of this information can affect the quality of mental health care for adults. The construct of ‘referral information’ was defined in accordance with the guidelines established by Hartveit et al. [16] and operationalised using QRef-MH [17].

Methods

The study was conducted in the region of the Western Norway Regional Health Authority, which is responsible for public specialised health care for a population of approximately one million. In response to the research question ‘What indicators are relevant and valid in the assessment of the potential impact of improved referral information on specialised mental health care for adults?’, we adapted the RAND/UCLA appropriateness method [26, 27] and used a stepwise process as described in Table 1. First, we organised focus group interviews with participants representing the most central stakeholders. Second, we conducted a systematic literature review. Finally, indicators identified in the focus group interviews and the literature review were assessed using criteria for indicators (see Table 2) by expert panels [13, 27]. The RAND/UCLA appropriateness method was chosen for its strengths in combining the best available evidence and collective judgement by experts to assess and select indicators in areas with limited existing knowledge, as is the case for the referral process [26]. To enrich the material and gain a deeper insight into areas of mental health care potentially affected by referral information, the method was supplemented by focus groups interviews.

Table 1 The steps in the RAND/UCLA appropriateness method and the present study
Full size table
Table 2 Criteria for indicators used by the expert panels
Full size table

Focus group interviews

Four focus group interviews [28] were conducted to define quality indicators or areas expected to be affected by improved referral information. To stimulate discussion and gain insight into the subject from different perspectives [28], each focus group was composited by health professionals, patient representatives and managers. Nine focus group participants worked in primary or specialised health care, six were managers and four were patient representatives. Of the 15 participants representing the professional and management perspective, nine were medical doctors (two general practitioners), four were psychologists and two were nurses. Twelve of these were specialists. Three of the four patient representatives had more than 15 years of experience with mental health care. The participants were selected by their organisations in the region because of their interest in and knowledge of the topic.

At the beginning of the group interviews, the participants discussed what type of information they recommended including in a specialised mental health care referral request. (These findings have been published separately [16]) After the discussion, they were asked, ‘If the referral letters were improved in the way you suggest, how do you think this would affect the process of care?’ The interviews were structured using the ‘affinity diagram’ [29], which included steps for written brainstorming using post-it notes. This method ensures a common understanding of ideas among the group members and excludes overlapping ideas [29]. The brainstorming was conducted in two sessions, with an oral discussion in between. The interviews were moderated by a researcher (EB) and observed by a second researcher (MH). All interviews, where the participants explain their ideas, were audio recorded to provide additional information for the analyses.

The suggested ideas (written by the participants on post-it notes) were analysed by two researchers (MH and OT) individually, guided by the steps of systematic text condensation by Giorgi, as described by Malterud [30]. Both researchers read all of the notes and listened to the audiotape to clarify the content of the notes to gain an overview, and the notes were then categorised by similarities in content and a code was defined for each category. For each category, the emerging indicators were defined. Finally, the results of the individual analyses were discussed by the two researchers, and a consensus about categories and preliminary indicators was reached.

Literature review

The literature search was conducted using PsycINFO, Embase and PubMed over a period of 10 years (2002–week 26 in 2012). The scarcity of existing literature necessitated wide inclusion criteria: All papers revealing, suggesting or discussing a potential causal chain between contents of referral information and aspects of quality of care were included. However, articles suggesting indicators clearly relevant for only one mental health diagnosis were excluded as ‘diagnosis-specific’. The search was conducted in the three databases for articles where the phrase ‘referral letter(s)’ occurred in the title or in the abstract, and was limited to adult patients. Based on the abstracts, articles were selected for full text reading, and relevant preliminary indicators were identified. Two authors (MH and OT) discussed and reached a consensus on the combined results from the interviews and the systematic literature review.

Expert panels

Three expert panels were set up, with three, three and two participants. The participants were all experienced psychiatrists or trained psychologists, and four were also experienced researchers. They were asked to assess each indicator using criteria for indicators regarding validity, reliability, sensitivity to change, acceptability, feasibility, simplicity and communicability [13, 19, 27]. The criteria, as introduced to the panels, are described in Table 2.

The aim of the study was described to the panels before they were presented with the indicators and their evidence basis, which was derived from the focus groups and literature review. Indicators were first evaluated by the individual members of the panel. The panel then discussed to what degree the indicators met the criteria for good indicators (Table 2). The expert panels were requested to place the indicators in one of three groups: bad/unacceptable, acceptable/needs adjustment or good/recommended. Further, they were encouraged to suggest improvements to the indicators. One researcher (MH) presented information to the panels and moderated the discussion, and two of the three groups also included an observer. At the end of the discussion, the moderator introduced relevant arguments made by the other expert panels and gave the panellists an opportunity to assess the suggested indicator once more to maximise the benefits of conducting multiple panels.

Results

The results of each step in the study are shown in Fig. 1.

Fig. 1
figure 1

Illustration of the study

Full size image

Focus group interviews

After excluding intergroup duplicates, the four groups suggested 128 indicators or areas (potential mediating factors) expected to be affected by improved referral information. During the analyses, three categories of suggestions emerged: co-operation, timely access and organisation/logistics. ‘Co-operation’ included suggestions such as a common understanding of and respect for the distribution of responsibility between primary care and specialised health care, avoiding duplication of interventions and improved co-ordination between the involved services. ‘Timely access’ comprised performance measures on improved decision making to ensure that the patients assessed as (medically) most in need receive specialised mental health care first. Most suggestions within ‘organisation/logistics’ concerned delays and waste in the process of care and focused on the optimal use of scarce specialised health care resources, such as the specialists’ time. Ten preliminary indicators emerged from the three categories. Of these 10 indicators, four where in the category of ‘co-operation’, three were in ‘timely access’ and three were in ‘organisation/logistics’.

Literature review

The literature search resulted in a total of 253 hits (PubMed: 88, PsycINFO: 24 and Embase: 141). Applying the inclusion criteria, 30 articles were included, whereas only three were from the database for mental health, PsycINFO. During the analyses, five categories evolved defining potential areas expected to be affected by the quality of referral information (with reference to the included papers in square brackets): timeliness and delay [31–33], attendance/drop-out [34–37], unnecessary consultations and investigations [32, 38–42], appropriateness of the referral [32, 43–53] and correctness of prioritisation of patients [36, 40, 44, 54–60]. Fifteen preliminary indicators were derived from the abstract of these five categories.

The 15 preliminary indicators suggested by the literature review were fully supported by the areas suggested by the focus group interviews. In addition to these, the focus group participants suggested measuring the degree of common understanding of the treatment plan among the involved services and health professionals. For further specification of the 16 indicators, the research team used their experience in mental health service provision and indicator development and consulted colleagues in the clinic on an ad hoc basis.

Expert panels

The expert panels’ assessment of the appropriateness of the indicators resulted in the recommendation of four of the 16 suggested indicators (Described in Table 3). The indicator ‘timely access’ measures whether the specialist’s assessment of urgency (maximum acceptable waiting time) based on information given in the referral letter correlates with a corresponding assessment based on a clinical evaluation. Two indicators measuring delay in the process were also among the recommended indicators. The first of these measures was whether the receiver of the referral was immediately able to determine the priority of the patient, or whether he/she had to request further information to prioritise the patient correctly. The second delay in process indicator concerned waiting time to start specialised health care treatment for patients with a severe condition and for patients with a less severe condition. Severity is defined by ‘severity factors’ [16] regarding risk of harming oneself or others, substance abuse, psychosis and caring for children. The fourth recommended indicator is appropriateness of referral. It measures whether the hospital specialist perceives the referral to be timely and to describe a situation where referral is recommended.

Table 3 Description of the four recommended indicators
Full size table

In all expert panels, participants spontaneously expressed that they saw the quality of referral information as a factor important for the quality of health care. However, they were also explicit about the difficulties they saw with defining good indicators according to the defined criteria [13, 27]. Seven of the 16 indicators presented were assessed as unacceptable by all three panels or as unacceptable by two and ‘acceptable/in need of adjustments’ by the third panel. The panellists saw the suggested causal chain as clearly weak or questionable because of a large expected risk of confounding factors affecting these seven indicators. Further, limited feasibility was given as a counterargument for some of the indicators. Five indicators were seen as acceptable or in need of improvements by all panels or by two and as unacceptable by the third. The participants expressed that they expected these indicators to represent existing causal chains but were in doubt as to the strength of the causal chains, strength of confounding factors and/or reliability. The 12 indicators that were not recommended, i.e., found to be in need of adjustments or to be unacceptable, are described in Table 4.

Table 4 Description of the 12 indicators that were not recommended
Full size table

The focus group interviews and expert panels revealed local factors that may affect the indicators’ validity and reliability for benchmarking, such as how the assessment of referral letters is organised and the capacity of the various specialised mental health units. Further, it was emphasised that diagnosis is not seen as an appropriate way to define the degree of patients’ needs or severity of condition and should be replaced by ‘severity factors’, as suggested in a previous study [16]. For all indicators, including those recommended, the expert panels emphasised the need for further development by exploring which factors should be controlled for and testing these factors.

Discussion

Using a modified version of the RAND/UCLA appropriateness method, the present study explored underlying mechanisms for the potential impact of referral information on the quality of care by responding to the research question, ‘What indicators are relevant and valid in the assessment of the potential impact of improved referral information on specialised mental health care for adults?’ The construct of ‘referral information’ was defined by the inclusion of recommended content in referral letters to specialised mental health care, as described by Hartveit and colleagues [16, 17]. The present study revealed a set of 16 indicators measuring the potential impact of the quality of primary care referral letters on quality of care. Of the identified indicators, four were recommended for use, and five were seen as having potential but in need of further adjustments.

Results discussed in light of existing literature

Guevara and colleagues have developed a model for the specialty referral process that suggests that the impact of the referral process can be measured within the areas of coordination, resource use, quality and outcomes [12]. The indicators suggested by the present study are in accordance with the model by Guevara and colleagues: Indicators regarding delay and waste of time in the process of handling the referral request translate as ‘resource use’ and ‘coordination’. Indicators of co-operation and timely access regard elements of ‘co-operation’ and ‘quality’ in the model of Guevara and colleagues (i.e., equity, timeliness, appropriateness and integration of care). Further, the results are supported by the Institute of Medicine (IOM), which defines being ‘timely’ as one of the six aims for high-quality health care [7]. Co-operation between services is also highlighted as a main challenge to health care by the IOM, as it was in the present study. Also supporting our results is the research on clinical handover and patient safety, which reveals that co-operation and coordination between involved services are essential for the quality of health care [5].

The indicators designated as recommended or acceptable in the present study are all process measures (i.e., measuring expected mediating factors for health care outcomes). The reservations expressed by the participants in both the focus groups and the expert panels regarding expected confounding factors in the complex referral and care process underline the importance of measuring mediating factors [13, 22, 23]. This finding is in accordance with previous literature on indicators, which asserts that outcome measures are preferred only when it is likely that improvement in the care will lead to significant change in health status or patient evaluation of care [20]. Further, process measures are more sensitive to change and easier to interpret, which is of great importance for facilitating both research and quality improvement efforts [20].

Strengths and limitations

The existing knowledge about indicators that measure the impact of improved referral information is clearly weak. The RAND/UCLA appropriateness method has become an acknowledged method to define indicators on areas with limited or diverging knowledge by utilising existing knowledge in combination with collective judgments [26, 27]. Further, this method is in line with the thorough preparation of process and outcome measures recommended by the (UK) MRC [24]. However, the method has been criticised for not conveying the patient perspective [61]. In the present study, focus groups representing patients, health professionals and managers were conducted to supplement the limited existing literature and to ensure representation of all stakeholders, as recommended by the framework for developing and assessing the quality of clinical practice guidelines, AGREE II (Appraisal of Guidelines for Research & Evaluation, second version) [62].

A systematic literature review was conducted and presented to the expert panels. However, because there is limited existing literature and the referral and care process is complex, gathering existing knowledge was found to be challenging. Although the search strategy used was assessed to be the most appropriate alternative, there are limitations to the literature review in the present study. The research team found additional relevant literature later in the research process, but this new literature did not introduce new areas or indicators. The lack of more evidence-based studies in the literature review means there are some limitations within the third domain of AGREE II: ‘rigour of development’ [62]. Further, the main body of existing literature found was not within mental health care. However, the combination of a systematic literature review and expert opinion with an agreed standard for the quality of referral information within mental health care, as used in the present study, provides a broader basis for further development of quality indicators and increases content validity in situations with clearly weak evidence bases [61]. The present study included only indicators that were non-specific with regard to condition or diagnosis within the spectrum of mental diseases. For specific conditions, there may be other valid indicators than the ones identified by this study.

Generalisability

The recommended indicators for measuring the impact of the quality of referral information are based on international literature and focus group interviews representing the relevant perspectives [63]. The results are therefore expected to be valid for mental health services that employ a similar system for the referral process and access to specialised care, as described by Guevara and colleagues [12]. However, participants emphasised that local context can have implications for the interpretation of the data from the indicators. Contextual factors may require adjustments to the definition of the numerator and denominator, and this negatively affects the generalisability of these definitions. The suggested indicators are therefore of interest primarily for improvement efforts-and less for benchmarking-until further exploration of the impact of contextual factors has been conducted. Although the literature review included studies from somatic health care, the participants selected to enrich the evidence base (focus group members) and to assess the qualifications of the indicators (expert panels) were selected for their mental health care expertise. The validity of the findings for services other than mental health is therefore unknown.

Conclusions

The study revealed a convincing agreement among experts on the expected importance of high quality referral information from primary care to specialised mental health care. The suggested indicators are expected to represent mediating factors affecting a wide range of main goals for health care as defined by the IOM (safe, effective, patient-centred, timely, efficient and equitable) [7]. The risk to health care of not being ‘timely’ is highlighted as particularly relevant when exploring the potential impact of low quality referral information.

We argue that, to enable effective quality improvement, it is necessary to explore underlying mechanisms to understand how outcome indicators can be affected. The present study calls for further inquiry into whether the quality of referral information affects the expected mediating factors for the quality of care and on the predictive value of various types of referral information for the quality of care. This will enable improvements to the suggested definition of high quality referral information by establishing an evidence-based minimum set of information to include in referral requests. Further, it will enable the development of quality improvement interventions tailored to support the underlying mechanisms for achieving high quality mental health care. We recommend that future research emphasise further exploration of mediating factors in the complex referral process, as well as their relevance for patient outcomes, and investigate whether and how contextual factors affect the validity of the suggested indicators.