Development and preliminary validation of the pancreatic cancer disease impact score

Abstract

Objective

Patient-reported outcomes are important for clinical practice and research, and should reflect what patients perceive as important. The objective of this study was to develop and preliminarily validate a brief, patient-derived, disease-specific tool, the pancreatic cancer disease impact (PACADI) score.

Methods

The development was performed in two phases. Forty-one patients with confirmed pancreatic cancer (PC) selected dimensions of health related to the impact of the disease. A weighting of the eight most frequently reported dimensions was performed in a second sample of 80 PC patients who also rated the impact on eight numeric rating scales (NRS, range 0 to 10). The relative weights and the scores from the NRS were used to compute the PACADI score (range 0 to 10). The patients also completed Edmonton Symptom Assessment System (ESAS) and EQ-5D.

Results

Dimensions reported by more than 20 % of the patients were included in the PACADI score (relative weights in parenthesis): pain/discomfort (0.16), fatigue (0.16), anxiety (0.15), bowel/digestive problems (0.14), loss of appetite (0.13), dry mouth (0.11), itchiness (0.08), and nausea (0.07). The PACADI score in the 80 PC patients had a mean (SD) value of 3.26 (2.06) (95 % CI 2.80, 3.71), was moderately to strongly correlated to ESAS sense of well-being (r = 0.69) and EQ-5D (r = −0.52), and discriminated significantly between patients with and without PC.

Conclusion

The PACADI score is a new eight-item, patient-derived, disease-specific measure. Preliminary validation regarding construct validity and discrimination encourages further validation in independent patient samples.

Introduction

Pancreatic cancer (PC) is the seventh most frequent cause of cancer deaths worldwide [1, 2]. Data published from the International Agency for Research on Cancer suggest that the mortality rate is 96 % of the incidence rate [3]. The short time from diagnosis to death in most patients makes research challenging due to limited disease-specific follow-up before transition into a general palliative stage. Frequently used outcomes in clinical and epidemiological research include survival, assessment of pancreatic tumors and metastases, soluble biomarkers, symptoms, function, and health-related quality of life. PC presents with symptoms that are most inflicting to patients. Thus, there is a need to measure, document, and monitor the level of health status in dimensions of high relevance and importance to patients in a feasible manner.

Patient-reported outcomes (PROs) have become increasingly emphasized in both clinical research and practice over the last decades and include generic and disease-specific measures [4, 5], as well as tools to measure utilities [6]. The Food and Drug Administration (FDA) and the European Medicines Agency recommend that pharmaceutical industry researchers provide sufficient evidence supporting benefit on PROs before granting approval of a new therapeutic agent [7, 8].

The use of PROs is based on the idea that the patient is the best source of information about the experience of their own condition. This is particularly important when the primary goal is to improve health-related quality of life [9]. However, there are some general concerns related to the feasibility of PROs. The questionnaires are often extensive which may limit the number of patients who provide their response, especially for longitudinal follow-up [10]. The lack of relevance of items to patients may also negatively influence the respondent rates.

The implication is that patients also are the best source of information to identify the questions of importance and how they should be presented. In rheumatology, the Rheumatoid Arthritis Impact of Disease (RAID) score, based on patients’ selection of dimensions where the disease has the most important impact, has been developed and validated [11, 12]. The RAID score has also been used to identify cut points for acceptable symptom state, as perceived by patients [13], as well as evaluating onset of response to medication [14].

A number of self-reported measures have been used in research in patients with PC, but until now, only the European Organisation for Research and Treatment in Cancer (EORTC) has developed a disease-specific instrument for PC [5]. The Quality of Life module for pancreatic cancer (EORTC QLQ-PAN26) has 26 questions and is meant for use in conjunction with the EORTC Quality of Life Questionnaire-C30 (EORTC C-30) which is generic for patients with malignant disease. Eventually, there are 56 questions to complete, which limits the feasibility of the instrument both in research and clinical practice, especially in patients with a severe disease [10].

The NIH PROMIS initiative [15] includes a number of items with multiple follow-up questions, which provide detailed and relevant information. Each question has Likert scale response options. PROMIS is a dynamic, promising tool to measure health outcomes, but it has to our knowledge not been validated in PC.

The objectives of this study were to develop a brief, patient-derived, disease-specific tool—pancreatic cancer disease impact (PACADI) score—and to perform a preliminary validation.

Methods

The methods for this project were modeled after the RAID score methodology but with some modifications based on the differences between the diseases.

The RAID score

An international group of ten patients with rheumatoid arthritis (RA) from ten countries was selected in the first phase (17 dimensions) where the disease had important impact. The second step was performed by 100 patients from the same countries, who ranked the dimensions for importance to reduce the number of dimensions to seven [11].

In the second phase, 500 patients from the same countries were asked to distribute 100 points between the 7 dimensions. Weights were ranked within each individual, and the average ranks were used for the final weight of each dimension. These ranks formed the basis for the final weights and were linearly transformed to a 0–100 scale. These weights were multiplied by the results of the numeric rating scales (NRS) and added together into the RAID score with a range from 0 to 10 [12].

An international multicentre cross-sectional and longitudinal study of consecutive RA patients from 12 European countries was conducted in the third phase to examine the psychometric properties of the combinations of instruments that might be included within the RAID and the use of NRS for each dimension. Construct validity was assessed cross-sectionally.

The PACADI score

First phase: selection of dimensions and follow-up examinations

Different from the patients with RA, who were familiar with their disease, the patients with PC only recently faced a severe disease at time of inclusion. For the PACADI score, based on an ethical consideration, individual settings were chosen to replace the group format. The study is a single center study on a tertiary level.

The goal was to include 30 patients for the identification of dimensions. There is no definitive answer to decide the sample size in qualitative approaches. However, the principle of saturation, i.e., to reveal the full range of important perceptions, is regarded as an indicator. A critical review from Yamazaki et al. found a median sample size of 36 [16]. The first sample consisted of 52 consecutively referred patients between November 2008 and July 2009 with symptoms and findings indicating PC. After the diagnostic procedures, 41 had confirmed diagnoses of PC (ICD10 C25*) based on cytology or histology in 92.7 % of the patients. Three patients (7.3 %), in whom all clinical, biochemical, and radiological data were consistent with pancreatic adenocarcinoma, were included based on findings by imaging modalities. Demographic and disease characteristics at baseline of the 41 patients with confirmed PC are shown in Table 1 (for the entire enrolled population (n = 52) see online supplementary Table S1).

Table 1 Demographic and disease characteristics at baseline for sample 1 for selection of dimensions and sample 2 for weighting and preliminary validation

With a qualitative approach, the patients were asked an open-ended question to identify up to 10 important dimensions of health related to the impact of the disease. The patients were shown a list of 56 dimensions (Table S2) from frequently used and relevant generic and disease-specific PRO instruments, and were then asked to report as free text the most important dimensions according to their personal opinion, but not in prioritized order. The selected dimensions were given priority according to importance by the patients immediately after selection, giving 1 to the most important, 2 to the next, etc.

Age, gender, body weight, and height at baseline were recorded. Body mass index (BMI) was computed. Regular weight prior to disease onset was recorded, and weight loss to baseline was calculated. Date of surgery was registered, if performed.

The patients also completed Edmonton Symptom System (ESAS) [4] and EuroQol-5D (EQ-5D) [6]. ESAS consists of ten NRS (scales 0–10). The Norwegian ESAS version was used with the following ten items: pain at rest, pain at movement, fatigue, nausea, dyspnea, dry mouth, loss of appetite, anxiety, depression, and sense of well-being [10]. EQ-5D is a five-item instrument that can be used in cost–utility analyses. Three of the items concern physical function, and the two last items address anxiety and pain. Each question has three response options (none/minor, moderate, or major problem). A single score from zero to one can be calculated.

We repeated the selection of important dimensions of health, after 1 and 2 months in a longitudinal design, to examine if the selection of important dimensions changed during the disease course. Data were available in 29 out of the 41 patients. Three patients died before the follow-ups were completed. Three dropped out, and six were too weak and died between 4 and 8 months after inclusion.

Second phase: weighting

Patients with PC (sample 2) were included between September 2009 and September 2011 based on the same inclusion criteria as in sample 1. Out of the 110 patients with suspected PC (online supplementary Table S1), 80 had confirmed PC (Table 1). Selection of important dimensions of health and data collection were similar to sample 1 (demographics, ESAS, and EQ-5D), but included one NRS (scales 0–10) for each of the selected dimension of the PACADI score (online supplementary Table S3, showing the eight NRS in Norwegian and in English translation (see Table S3 for description of the translation process)) [17]. Each NRS addressed the impact on each dimension during the last week. The last week was chosen as a time frame for overall impression of the health dimensions in the PACADI score, since patients often report fluctuating severity of their health problems. The 1-week time frame is similar to the RAID score and different from the ESAS (same day). Time frame varies between instruments, and 1 week has been found as valid as momentary assessment [18].

For the weighting exercise, the following question was asked: “Considering your entire disease duration, please distribute 100 points among the following eight dimensions of health based upon the impact of the disease on these dimensions.” This methodology is identical to the methodology for assigning weights to the dimensions of the RAID score [9] and as described by Ruta et al. [23].

The weights of the PACADI profile were ranked from 8 to 1 in each patient (8 was given to the dimension with highest weight). If two dimensions had identical weights, they were given the same rank. The mean of the rank of weights for each dimension was calculated and was then normalized to a scale from 0 to 1 (i.e., the sum of all weights was equal to 1.0). NRS were also used in the RAID score and were performed, as well as other and more comprehensive tools within the same dimension. In addition, NRS is faster and more feasible to complete for patients than more comprehensive questionnaires and visual analogue scales [19]. The final step in computing the PACADI score was to multiply the NRS values with the normalized final weights and then to provide a sum of these products in each individual patient.

Third phase: preliminary validation

The preliminary validation in sample 2 included a comparison of the PACADI scores across gender, between patients undergoing versus not undergoing surgery, between patients with and without confirmed PC in sample 2, and to examine correlations between the PACADI score and age, ESAS items and EQ-5D.

Statistics

Parametric statistical methods were preferred since data appeared normally distributed based on Q–Q plots, and mean and median values were similar. Parametric methods were performed robustly [20] and were used in the development of the RAID score. Demographic continuous variables as well as PACADI score, ESAS and EQ-5D are presented as mean (SD) values, and associations between PACADI and continuous demographic variables, ESAS and EQ-5D were examined with Pearson’s correlation coefficient. Two-sample t tests were used for the comparison of the PACADI score across dichotomized variables. All analyses were performed in SPSS version 18. Two-tailed p values <0.05 were considered statistically significant.

Regulatory and ethical aspects

All data were stored with a numeric identifier in a separate, secured research database detached from the identifier code list. The code list was properly stored with accessibility for the principal researchers only. The regional ethical committee approval of the protocol and patient information/consent form was obtained separately for phases 1 and 2. The privacy protection supervisor evaluated and accepted the data-handling procedures. All participating patients signed a written informed consent.

Results

Development of the PACADI score

First phase: selection of dimensions at baseline and longitudinal assessment

Table 2 shows the number and percentage of patients in sample 1, who at baseline, reported dimensions of health where the disease had important impact. The right column in Table 2 shows the single dimension of health that each of the 41 patients regarded to have most important impact (highest priority (number 1)). Dimensions reported less than twice or not among the dimensions of highest impact (n = 22 (50 %)) are not included in Table 2. With the aim of brevity, the eight most frequently reported and highly prioritized dimensions were selected for the PACADI profile. The dimensions of health that were reported by more than 20 % of the patients were included as candidate dimensions for the PACADI score: pain/discomfort (70.0 %), fatigue (63.4 %), bowel and/or digestive problems (58.5 %), loss of appetite (46.3 %), anxiety (39.0 %), dry mouth (31.7 %), itchiness (24.4 %), and nausea (24.4 %) (Table 2).

Table 2 Number (percentage) of patients with PC reporting dimensions of health with important disease impact/most important impact (highest priority) at baseline

The baseline analyses were repeated in the original sample of 52 referred patients (including 11 without PC) with similar results (online supplementary Table S4). Pain/discomfort, fatigue, bowel and/or digestive problems, and loss of appetite were reported with stable importance. Anxiety and itchiness were reported less frequently over time, whereas the proportion reporting nausea increased (Fig. 1).

Fig. 1

Percentage of patients (n = 29) with PC reporting dimensions of health with important disease impact at baseline and after 1 and 2 months

Second phase: weighting of the eight selected dimensions and construction of the PACADI score

The mean weights attributed to each of the eight dimensions are shown in Table 3. Pain/discomfort, fatigue, bowel and/or digestive problems, and anxiety had the highest weights. This table also shows the mean rank of the weights for each dimension based on ranks from 8 to 1. The means of each of these ranks were normalized to a scale from 0 to 1 to create the final weights to be used in the PACADI score. The sum of these normalized weights was 1.0 (Table 3).

Table 3 Results from the weighting exercise across the eight selected dimensions of health in patients with PC (n = 80). Mean (SD) weights, mean (SD) ranks of the weights, and the mean ranks normalized to a scale from 0 to 1 (i.e., final weights)

The weights were examined in the 110 referred patients with suspected PC in whom 30 had other diagnoses than PC (see online supplementary Table S5). The weights in this total sample were similar to the weights shown in Table 3.

The mean values of the reported NRS for the eight dimensions are shown in Table 4. The PACADI score for each patient was computed according to the following formula (NRS values multiplied by the final weight): pain/discomfort NRS × 0.16 + fatigue NRS × 0.16 + anxiety NRS × 0.15 + bowel and/or digestive problems NRS × 0.14 + loss of appetite NRS × 0.13 + dry mouth NRS × 0.11 + itchiness NRS × 0.08 + nausea NRS × 0.07 (see also online supplementary Table S2).

Table 4 NRS and weighted NRS scores (NRS multiplied by the final weight) within the proposed PACADI score dimensions

The mean (SD) baseline PACADI score in sample 2 was 3.26 (2.06) (95 % CI 2.80, 3.71, median 3.27, interquartile range 3.45), whereas the mean of the scores from eight NRSs without any weight was lower (3.07).

Third phase: preliminary validation

The PACADI score was similar in females and males (females/males, n = 39/41, mean (SD) 3.49 (2.05)/3.03 (2.06), p = 0.32); it did not correlate to age (r = 0.09) and was not different in patients undergoing versus not undergoing subsequent surgery (no surgery/surgery, n = 38/41 mean (SD) 3.62 (2.06)/2.86 (2.01), p = 0.10). However, the PACADI score was significantly higher in patients with confirmed PC (n = 80) versus patients with suspected but not confirmed PC (n = 30) (mean difference (95 % CI) 0.80 (0.34, 1.56), p = 0.04). ESAS appetite score was also significantly worse in patients with versus without PC, whereas the other ESAS items and EQ-5D did not discriminate between the two patient groups.

Correlations between the PACADI score and EQ-5D and ESAS were examined. The correlation coefficients varied from 0.21 (dyspnea) to 0.72 (fatigue). Correlations to the global measure of sense of well-being in ESAS and EQ-5D were 0.69 (p < 0.001) and −0.52 (p < 0.001), respectively, supporting that the PACADI has moderate to strong correlations to other global measures (Table 5).

Table 5 Associations between the PACADI score, EQ5D, and ESAS items in patients with PC in sample 2 (n = 80)

Discussion

The aim of this study was to develop a tool to facilitate a seamless assessment and monitoring of PROs in patients with PC in a research setting and in clinical practice. Clinicians have their perspective on collecting information for decision making. The patient perspective is one out of four key issues for decision making in evidence-based medicine [21]. The PACADI score may help clinicians to identify and monitor dimensions and symptoms that are highly relevant to patients. The score is a global measure that has the potential to be used in both clinical and research settings.

The decision to limit the inclusion of dimensions of health that were reported by more than 20 % of the patients as candidate items for the PACADI was based on the important goal to obtain a feasible score. This aspect is especially important in patients feeling unwell. The number of dimensions was similar to the RAID score (eight versus seven).

The idea behind the longitudinal assessment was to examine whether the selection of domains was stable over time. Overall, the selection was stable but with a reduced number of patients selecting anxiety and itchiness during follow-up (Fig. 1). The reduced focus on anxiety may be related to the adaptation to the disease after proper information and that reduced reporting of itchiness may be explained by procedures like stenting [22]. However, both these dimensions remained in the PACADI score since they obviously are important in the early phases of the disease, which may be the phases of largest interest when considering current and future therapeutic options.

It was expected that some patients might have problems understanding what was meant by dimensions of health. Therefore, the probe list of 56 dimensions from frequently used questionnaires (Table S2) was prepared and offered at baseline to exemplify dimensions of health. About half of the selected dimensions for PACADI were in the probe list; this list was mostly used for the initial illustration of examples at a glance. The use of this probe list could potentially have been a limitation to the study by influencing the selection. During follow-up, the probe list was available, but it was rarely used by the patients. The selected dimensions were overall similar at all three time points, even if the patients did not have access to the previous selections (Fig. 1).

An alternative approach to the model of ranks of the weights of each dimension could had been to use the mean weights to construct the final weight (Table 3). However, the weighting accentuates the patient perspective of importance. The aim was to reach a number of patients, which was ten times the number of dimensions [23]. Even so, only 80 patients from one single center is considered a limitation, since the weighting procedure of the RAID score included more than 500 patients across Europe. The intention for the development of the PACADI instrument was to make it disease-specific, and hence, only patients with PC were included in the primary analyses. However, results were similar both for selection of dimensions and weights when both groups of patients were analyzed (Tables S1 and S4).

A particular strength of the PACADI score is that both selection of dimensions (i.e., what to measure) and the weighting procedure (i.e., how important is the dimension to the patients) are based on the patients’ opinion. Inherent in this design is the relevance of the tool to patients.

Several of NIH PROMIS items may be of relevance to patients with pancreatic cancer. However, it seems that only pain, fatigue, and anxiety are covered in both tools, since PACADI reflects only the most important impact on health. While PROMIS is measured on Likert scales, PACADI uses NRS. NRS and Likert scales are often used interchangeably. Likert scales seem to be more frequently used to measure strength of agreement in attitude, whereas NRS seem to be more frequently used for measuring severity/intensity, which suits PACADI. Electronic versions of both tools would increase both feasibility of completing and errors in datasets [24].

It seems that the three approaches related to the eight selected dimensions of health (selection and giving priority between dimensions of importance, weights, and symptom severity) gave rather consistent results on a group level. For example, the five dimensions reported as important by more than 39 % of the patients (pain/discomfort, fatigue, bowel and/or digestive problems, loss of appetite, and anxiety; Table 1) also had the highest weights (Table 3) and the worst scores on NRS (Table 4). However, the data also indicated that patients may differentiate between severity (the value on the NRS) and the importance (the frequency of reporting dimensions of importance to patients). Even if fatigue is reported as more severe than pain/discomfort on NRS (Table 3), pain/discomfort is still the most important dimension to patients (Table 1). This observation implies that asking patients “what is important to you?” may give complimentary responses to asking patients “how are you?” A similar differentiation between severity and priority in patients with rheumatoid arthritis has previously been reported [25].

In sample 2, the PACADI profile was tested (n = 80) in a preliminary validation. The moderate to strong correlations to the global measure of well-being in ESAS and EQ-5D supported that the PACADI score may perform as a global PRO in patients with PC. Further, the weighted PACADI score showed a slightly higher value than the mean of the NRS profile (Table 4), which may support that the inclusion of the weights may accentuate the patient perspective. Interestingly, the PACADI scores differentiated between patients with and without confirmed PC, which may support its role as a disease-specific tool, since a similar differentiation was not seen for ESAS well-being and EQ-5D.

The PACADI score is a brief, patient-derived, disease-specific measure. Preliminary validation regarding construct validity and discrimination encourages further validation in independent patient samples. It has the potential similar to RAID [13, 14] to be used in both clinical and research settings, but additional testing of reliability and a larger scale validation in a population of at least 200 patients according to requirements from the FDA [26] should be performed. Sensitivity to change should be examined in intervention studies, for example, by longitudinal comparisons of patients undergoing versus not undergoing surgery [27]. It is also important that further validation is performed in an international setting.