Introduction

The call for clinically relevant outcome criteria in palliative care has been raised, as assessment of adequate quality of service providers is essential with increasing momentum in the development of palliative care in most European countries. In Germany, since the implementation of the first palliative ward in 1983 and the first Hospice in 1985, the number of palliative service providers has increased significantly, from 32 palliative wards (297 beds) in 1993 [1] to 127 palliative care units and 142 inpatient hospices (4,381 beds) in 2007 [2]. This expansion necessitates an evaluation of the care being currently delivered, which should be supported by health authorities [3]. A programme on documentation and quality assurance has been performed in Germany since 1996 with yearly census periods and a large number of palliative and hospice services participating [4–6]. However, in palliative care, certifications of services or mandatory quality assurance programmes have not yet been implemented.

Germany has a rather differentiated system of inpatient and outpatient hospice and palliative care. Hospices focus on nursing needs, are freestanding and, in general, patients remain there until they die. Reimbursement is regulated since 1997 through a combination of long-term care and health care insurance. For outpatient hospice services, which are mainly volunteer-based, funding for the coordination of the volunteers is ensured by law since 2001 [7]. In 2007, the German Parliament passed a bill on the right to receive specialised palliative care at home, paid for by health insurance. Inpatient palliative care units can be reimbursed beyond the DRG-System on the basis of fixed prices per day [8].

Policymakers, managers of palliative care programmes, clinicians, and others committed to improvement of end-of-life care need methods and criteria to measure and evaluate the care delivered [9]. These should assess the effectiveness, value, and efficiency of health-related services, in order to provide the basis for quality improvement [10]. In addition, performance measurement allows comparison of results and assessment of progress toward goals [11].

Concepts about quality of care and health outcome as well as measurement tools have been developed in the last decades [12]. In the specific field of end-of-life care, quality indicators should be able to demonstrate adequate and appropriate management of the patient. Quick and efficient assessment of service performance is a challenge for quality improvement, as it facilitates modification of services and practises to improve the quality of care at an individual but also at the institutional level. Important scientific work has been undertaken in order to define appropriate assessment and evaluation methods in palliative care [3, 9, 13, 14].

However, the criteria and scales that have been suggested have failed to prove their effectiveness in the differentiation of different settings or in quality management [15–19]. In order to understand these problems, it is necessary to build a comprehensive theoretical model of what the criteria of quality in palliative care are.

The aim of this study is to investigate important dimensions and indicators for assessment and evaluation of palliative care from the perspective of multi-disciplinary German experts working for years in the field of palliative care. This study is part of a structured research process on quality indicators and dimensions in palliative care from the research network of the German Association for Palliative Care.

Dimensions are defined as a specifiable aspect or fact of a concept [20], while quality indicators are measurable and explicitly defined items referring to the structures, processes, or outcomes of care [21] that we choose to consider as reflection of a variable we wish to study, in our case, “quality of palliative care” [20].

Methods

The focus group used the structured consensus method of the improved nominal group technique (INGT) [22, 23]. The INGT was conducted on 23rd August 2007 in Aachen, Germany. Nine experts from different regions in Germany and different disciplines (sociology: TP, medicine: LR, FN & CO, theology: GH, psychology: MF & MP, social work: JR, nursing: NK) participated in the study, who also co-authored the paper. The participants were selected because of their expertise in the field, representing academic and community practise and different specialities (‘expert for themselves’ [24]) and their experience with assessment tools and methodologies, respectively. Intended was a balanced representation of expertise from various sectors of professionals capable of working in a collaborative [25]. The sampling was realised among members of the research working group of the German Association of Palliative Medicine (DGP).

The nominal group technique (NGT) was developed by Delbecq and Van de Ven in 1971 [22] and has been defined as “a special purpose group process appropriate for identifying elements of a problem situation, identifying elements of a solution programme and establishing priorities” [26]. The NGT’s characteristics of a formal consensus development involve individual, private decision-making, formal feedback of groups’ choices, highly structured interaction and an explicit aggregation method [27]. Consensus methods have many advantages such as the wider range of experience and knowledge as well the interaction between the participants, facilitation of new ideas and mutual stimulation [28]. This method ensures that all relevant issues are considered, provides a democratic approach (avoiding conformity, domination, social pressure, or not having enough time for all individuals to develop their ideas) and attempts to relate outcomes of the research to practise [28, 29]. Fox [23] improved this technique (calling it INGT) by using 3 × 5 cards, allowing more-ideas-per-person-per-turn without reducing the opportunity of equal participation.

However, agreement in INGT groups is not always high [30]. The decision-making process in the group is subject to cognitive biases and motivational biases [28]. Other factors influencing consensus are the composition of the group or attributes of the facilitator [27, 28]. Nevertheless, multi-speciality seems to be of benefit [28, 30], as well as a group size between six and 12 people [28].

The final rating of indicators is the result of a multi-stage process. In the first stage, a round of thinking about and becoming aware of currently applied assessments methods in the everyday practise and their advantages and disadvantages in independent contributions was performed and subsequently discussed (‘opening statement’). In the second stage, participants were asked to write down any idea about indicators of quality in palliative care in answer to the question: “What is important to assess in order to evaluate quality in palliative care?” (‘silent generation of ideas’). Then, all participants equally read aloud each idea (‘round robin’) and the card was put in the flip chart, an open discussion followed (‘clarification of ideas’). Finally, the participants were asked to rank important issues in assessment and evaluation of palliative care (‘voting and ranking’). Each participant could allocate ten points to the indicators, resulting in a maximum ranking score of 90. In addition, the essential properties of outcome indicators were discussed.

The session was recorded with a digital recorder and then was transcribed and analysed with a qualitative data analysis software (MAX-QDA), intending to delve into the discussion by using an inductive thematic analysis.

Results

Thinking about the problem

The German experts reported the use of a broad spectrum of instruments to assess the quality of their work (see Table 1). Assessment is especially focussed on symptom control (BSI [31], MIDOS [32, 33], or VAS/NRS), quality of life (FACIT [34], POS [35], QLQ-30 EORTC [36], or SEIQoL [37]), psychological (HADS [38], PoMS [39], or SCL-90 [40]) and spiritual (SMILE [41], or SPIR [42]) aspects, mostly relying on patient self-report for information. Trust in physician (TPS) [43], as well as assessment of social support, was also mentioned. In addition to patients’ self-report relatives or staff members are used as sources of information. Another informal criterion used was asking relatives and patients about satisfaction with specific support opportunities such as bereavement counselling. Information from the team is provided in an unstructured way in team meetings, but also with documentation systems such as Optiplan®.

Table 1 Instruments used in clinical practise
Full size table

Assessment was rated as very helpful, as the brief instruments used in current clinical practise are easily applicable in daily routine, are reliable, and allow for comparison in different settings. Information from these instruments offers a basis for discussion and generating questions. Establishing routine in the assessment provides a training effect and raises awareness for uncommon cases. Sometimes, the assessment itself has an intervention effect for the patient. Palliative care services have to prove their efficacy in the delivery of care, and documentation forms the basis for evaluation. Furthermore, outcome assessment helps to visualise the disease trajectories of the patients (see Table 2).

Table 2 Advantages and challenges of the used instruments
Full size table

However, with the documentation and assessment instruments used in palliative care, in general, several problems were identified:

According to the experts, some outcomes may be difficult to capture directly after an intervention, as palliative care interventions often take time to produce an effect, for instance in psychological treatment. In addition, assessment often is deficit-oriented, looking for reduced function and symptom burden, rather than resource-oriented. Important outcomes such as ‘autonomy’ or ‘dignity’ are not operationalised in a satisfactory and acknowledged way. Many of the concepts are closed systems with their own semantics and do not communicate easily with other systems.

Assessment instruments in palliative care often provide unsatisfactory instructions of use, timing and frequency. Sometimes, these instruments are complex (e.g. Schedule for the Evaluation of Individual Quality of Life (SEIQoL)) and may lead to an increased burden for the patients by of directing their awareness to the loss of functionality. Acknowledged parameters for assessment systems such as validity, reliability and sensitivity may be challenged in palliative care. Existing instruments are not sensitive to measure change in time. Furthermore, although palliative care is multidimensional by definition, many instruments are only one-dimensional, for instance, measuring just the somatic dimension.

Application of instruments by the team implies that staff members have to be trained in the outcome assessment in order to support common concepts for the parameters used in the assessment and to be aware of an unexpected or untypical outcome. Standardised instruments bear the risk of being selective according to their construction and ignoring important areas and symptoms which are not included in the checklist. Furthermore, assessment might be reduced to an automatic act (‘hack down’), without consequences.

The performance status of patients in palliative care varies significantly according to the stage of the disease. Patients usually are not able to complete long and complex questionnaires and checklists. Self-assessment is often not possible because of the poor performance status of the patients; and thus, other assessment methods are required. For approximately one third of the patients, assessment instruments are not feasible. Furthermore, the continuous assessment could be an unwelcome reminder of declining function and thus place an extra burden on the patients.

Using other sources to get relevant information is subject to validity problems. Staff assessment often underrates the subjective experience of symptoms of the patients. However, the ambiguous interpretation of signals at end-of life may also reduce validity, as it is difficult to differentiate pain, anxiety and distress.

Timing

Where do we measure, what, how, when and how frequently? Retrospective data collection is used, for instance, during the bereavement café (Trauerkaffee). Some instruments are inappropriate if used only for a single time point in the illness trajectory. For example, when counselling bereaved relatives, little change is reported during the counselling process. However, some time after counselling has been done, relatives often report good effect of this intervention retrospectively.

The complexity by the quality assessment in the everyday praxis is shown in this quotation:

We attained to have a unique questionnaire [HOPE—Hospice and Palliative Care Evaluation] that should be filled at the initiation and at the end of treatment in order to evaluate treatment in a palliative care unit. This was not possible, because a number of patients die where the deterioration of physical function is linked to the underlying disease, (...) and we don’t see the expected effect. So, we thought about follow-up questionnaires to be completed, for instance, every three days. This doesn’t work, because completing an extended questionnaire is evidently not practicable for patient, relatives and staff. Then we thought that the follow-up questionnaire could be completed weekly. This may be a good time span to assess the effect of the initial therapeutic regimen, but without the deterioration of the performance status. However, many patients in our unit stay only 10 days, others die even sooner. (...). In consequence we do not know which time span to recommend for evaluation. This would be similar for other checklists and questionnaires. (Focus group)

Dimensions of outcome indicators

An abundance of topics (16) emerged from the nominal group technique, pointing at the complexity of the issue. These themes and their definitions are shown in Table 3.

Table 3 Themes
Full size table

The categorisation identified six main topics as crucial priorities: ‘quality of life’, ‘needs assessments of patients and relatives’, ‘resource assessment’, ‘surveillance of decision making processes’, ‘symptom control’ as well as ‘spiritual well-being’ (Table 4). However, scores for these topics were not bigger than one eighth of the possible maximum score, showing significant dispersion of priorities among the experts. ‘Quality of life’ and ‘needs assessments of patients and relatives’ were ranked highest by the participants, followed by ‘resource assessment’ and ‘surveillance of decision making processes’. Of the remaining themes, ‘symptom management’ was rated with 7/90.

Table 4 Top themes from ranking
Full size table

‘Quality of life’ refers to subjective quality of life, involving psychological, spiritual and existential dimensions. Even though there was a tacit agreement among the experts that quality of life is a central aspect in the definition palliative care, or even the underlying goal, it was of interest that the construct of quality of life was not discussed, despite the lack of agreement about its measurement and its assessment or well-known problems such as response shift.

The ‘needs assessment of patients and relatives’ and their completion reflects the individual approach of palliative care, where the focus of interest lies in the exploration of each case in particular.

Also the ‘resource assessment’, especially the social support and network and the improvement of its utilisation, confirms the individual character of palliative care and the adjustment to particular situation.

An important aspect in the evaluation of palliative care is the ‘surveillance of decision making processes’, such as in case of treatment withdrawal, change of treatment goals with progression of the disease or dealing with limited resources. Hereby is appealed the ethic aspect.

‘Spiritual wellbeing’ was used by the experts as a ‘well-know’ term, with a general agreement—similar to quality of life—on its importance within palliative care. However, no attempts were made to clarify its definition and operationalisation despite the wide range of professions in the focus group.

Given that palliative care deals with somatic, psychological and spiritual aspects of the treatment and care, an ideal outcome system would have to consider these aspects.

Based on ratings and rankings during the meeting, five themes were identified as being the most important: These are: ‘quality of life’, ‘needs assessments of patients and relatives’, ‘resource assessment’, ‘surveillance of decision making processes’ and ‘spiritual wellbeing’. Interestingly, often-discussed issues such as ‘dignity’ were not ranked primarily as important themes.

The concept of ‘dignity’ was broadly discussed especially because of the difficulties to operationalise dignity and its practical use as a concept for operational procedures. Furthermore, dignity should not be a normative sentence, but rather a guideline to understand what is dignified in the patient’s view. The questions “To what degree do you perceive the situation to be dignified?” or “How does the situation impact on your dignity?” can clarify whether we are in contact with the patient’s concept and experience of dignity. This has to be differentiated from what the team perceives as dignity. Associations emerging from this concept were the relationship with suffering, ethical and legal issues and meaning of life.

Essential properties of outcome criteria

From the background of the assessment instruments currently used in clinical practise, the experts claim the following properties as essential for outcome criteria: multidimensional, sensitive, without additional burden on patients, easily applicable, scientifically valid, related to clinical practise, and beneficial for the communication within the team, and externally, for ethical discussions as well as for quality management. In addition, they should evaluate the success of a therapy and allow comparison with other settings.

Concerns of different interest groups

Other problems in the outcome assessment in palliative care are the divergent interests and requirements of different stakeholders other than the directly involved health care professionals, patients and relatives. Stakeholder groups that were identified in the focus group included health insurance company, health policy makers, academic and educational organisations, professional associations, volunteers, church, rival networks (such as oncologist), expectations of the public and international networks. The vast range of divergent interests in these groups puts up additional challenges for the selection of uniform outcome criteria. New emerging interests may also introduce new outcome criteria. For example, in 2007, a new legislation introduced the right for specialist palliative home care (SPHC) in Germany. This resulted in a new primary outcome indicator for services offering SPHC focussing on the percentage of patients that actually die at home.

Discussion

In general, the discussion revealed a lack of agreement among the experts regarding predominant concepts in theoretical and in empirical areas of palliative care. This was evident not only in the wide range of instruments used, but also in the different priorities on predominant dimensions of outcome indicators. Within the time frame of the focus group, it was not possible to agree on a common concept for outcome assessment, even though the experts agreed on the scope to be covered. Important questions such as what is palliative care, what are its actual tasks or who is a palliative care patient are still under discussion. This complicates the identification of accepted outcomes since these should be based on the answers to those questions.

The discussion showed clearly that it is difficult (and maybe even impossible) to assess the whole construct “palliative care” with only one outcome indicator or one instrument, but that there should be specific outcomes for different patient groups and specific disease stages, situations, settings and areas of care. However, some essential criteria have been identified which should be included in each instrument or that should be assessed in each area.

The question to which extent the assessment by staff or care givers agrees with self-assessment remains controversial [9, 44–46], and has practical implication [47]. However, considering the high percentage of patients in palliative care not able to self-assess, assessment by others seems to be inevitable in palliative care.

Sixteen dimensions for palliative care were identified using INGT and the expertise of a multi-professional team. These dimensions were ranked according to their relative importance.

The five most important themes in our study (quality of life, needs assessments of patients and relatives, resource assessment on social support, surveillance of decision-making processes, spiritual well-being) are well supported in the existing literature: interestingly, symptom management was only rated in the sixth place. The identified quality dimensions cover a great area of the definition of palliative care [48].

The fact that the group was constituted predominately by clinical experts could explain the tendency to choose individual outcome indicators in contrast to outcome indicators from macro-level, such as organisational or structural aspects (costs, personal). However, without consensus on meaningful individual outcome indicators evaluation of cost–benefit ratios will not be possible and thus would have to be discussed subsequently.

The focus group did not discuss the definition of quality of life but rather emphasised the subjective attribute of this construct. This ties in with the ongoing discussion in the literature on divergent definitions of quality of life. Numerous studies in palliative care have addressed important components of this construct [49, 50]. In contrast to popular expectations, quality of life may be perceived as high by patients receiving palliative care, and may even improve despite ongoing disease progression. This difference between the expectations and the actual performance as the major determinant of subjective quality of life has been described as the Calman Gap [51]. Accordingly, this gap can be reduced by improving performance or by reducing expectations. In consequence, recent developments show that individual priorities are important in quality of life assessment which has led to the development of the Schedule for the Evaluation of the Individual Quality of Life (SEIQoL) [52].

Constructing quality indicators based on needs assessment of patients and relatives and resource assessment shows an orientation towards patients/relatives, as well as a resource orientation. This could be interpreted as a more comprehensive aspect of care as it considers the subjective reality as it is constructed by the patient and his relatives, in their specific context

Surveillance of decision-making processes is essential in palliative care. Patient autonomy and informed consent are well acknowledged aspects on the end of life [53–55]. However, again, no clear definition of these concepts is available, and this reduces the use as outcome parameters.

The high ranking of spiritual well-being in this study substantiates the crucial role of spirituality in end-of-life care, which has also been highlighted in previous research [41, 56].

The focus group and the INGT produced rich information on core-questions in palliative care, and allowed for the discovery of new ideas and the identification of challenges. Using a modified form of NGT, Aspinal et al. [57] found that for professionals, symptom management was the most important dimension of outcome. Other groups such as bereaved relatives and patients answer the question “what things are important to measure in the last months and weeks of life”; the former with quality of life or relationship, the latter with preparation on death and dying and coordination and continuity [57].

There are a few limitations of this study. First, the composition of the group is a central determinant in the results of this study. The study participants represented the scope of professions involved in palliative care in Germany. However, a generalisation to the whole country could be difficult, even though the participants represented different regions and interests. Nevertheless, even in this small sample, we found a wide range of opinions and a lack of consensus and it is most probably that these findings would be even more prominent in a larger sample. The feasibility of the assessment of the selected dimensions is not established by clinical measurement. Bias may also have been introduced by the group composition. Giving top priority to the spiritual dimension might have been related to the high percentage of non-physicians in the group. In addition, the definition of a concept of quality should lead to the description of outcome parameters. However, as the definition of quality was not included in the focus group, and no consensus on a uniform definition was sought, this could have influenced the selection of outcome criteria. Furthermore, the discussion on different interests of stakeholder groups in outcome identified a number of priorities not related to quality control, which would therefore not have been influenced by the underlying concept of quality. On the other hand, patients and carers were not represented in the group, and their inclusion might have led to a different focus in the discussion. Results from the focus group should be correlated with patient representatives and other professionals to confirm the validity of our results.

The dimensions identified in the INGT should be operationalised into quality indicators in order to measure and confirm their feasibility, reliability, and validity in different German settings. In addition, further research is needed to consider the perspective of patients, bereaved relatives and other professionals involved in palliative care in order to develop valid quality indicators. However, it is necessary to consider the implications of the quality measurement for the allocation of health resources, for the organisation of medical practises, for the professionalism of health care personal and for their concept of quality [58]. Results from this study also will be used to evaluate the findings from a systematic literature review on outcome parameters used in palliative care.

The quality indicators should be suitable to identify and measure things that are part of usual clinical care. At the same time, this means that in the assessment of quality of end-of-life care, it is necessary to go beyond the highly individual dying process in order to establish a normative judgment. This is in contrast with the concept of palliative care with flexible and individual goals which form the basis of the treatment plan. This may imply a paradox that might be impossible to solve.

Conclusions

The study identified topics considered important by experts in clinical practise. ‘Quality of life’ and ‘needs assessments of patients and relatives’ were outstanding issues for the German experts. The discussion exposed a wide range of demands on outcome assessment put up by different stakeholder groups. This diversity impedes the agreement on a uniform set of outcome criteria.

The next step in the structured research process on quality indicators and dimensions in palliative care from the research network of the German Association of Palliative Care is a systematic review of the outcome instruments used in the scientific literature as systematic reviews help to improve the evidence for quality measurement [14]. Starting from a discourse analysis of the definition of ‘palliative care’ to clarify the goals of palliative care as a basis for the systematic review, we will then use the results from the focus group to evaluate results of the literature review, select suitable criteria and instruments, describe best-fit models of outcome indicators for different settings and test these models in clinical practise.

Further research is needed to test the results in other settings; considering the perspective of patients, bereaved relatives and other professional involved in palliative care. Moreover, elaboration of these ideas within interdisciplinary groups is recommended.