Study Participants and Data Collection
The validity study was performed at 11 site hospitals across the Netherlands over a period of 1 year (July 2016–June 2017) using a convenience sample of IPF patients using pirfenidone or nintedanib and aHUS patients receiving eculizumab treatment. In addition, a sample of patients was recruited that had not been involved in the development process of the PESaM questionnaire. Patients using a once-a-day oral formulation of tacrolimus (brand name Advagraf) after kidney transplantation were asked to complete the generic module only.
Inclusion criteria for the validity study were (1) being an adult (aged > 15 years) diagnosed with IPF, aHUS (first diagnosis or recurrence), or a history of kidney transplantation; (2) taking any of the following medications: pirfenidone or nintedanib, eculizumab, or tacrolimus, respectively; and (3) being able to read Dutch. There were no restrictions to the time on medication or treatment regimen. IPF patients were recruited in nine hospitals across the Netherlands. Atypical HUS patients received their treatment in the Radboud University Medical Center in Nijmegen, the national expertise centre for aHUS. Patients using tacrolimus were attending follow-up visits in the Maastricht University Medical Centre. The appropriate modules of the PESaM questionnaire were either handed out (on paper) to patients during outpatient clinic follow-up visits or emailed via an online survey system.
Written informed consent was obtained prior to completing the questionnaire. The study protocol for the development and testing of the measure was reviewed and approved by the Medical Ethics Committee of Erasmus Medical Centre (MEC-2015-265). The study is registered in The Netherlands National Trial Register (trial code 5860).
Generic Module of the PESaM Questionnaire
The generic module of the PESaM questionnaire consists of 18 items related to the domains of effectiveness (4 items), side effects (4 items), ease of use (3 items) and satisfaction (4 items) (Table 1). The final three items concern the perceived importance of the effectiveness, side effects and ease of use of the medication. These items were included for validation purposes only. Items within the domains of effectiveness and side effects relate to the impact of the medication on aspects of physical health, emotional health and daily life (social and work activities) . These items all relate to subjective experiences of the respondent (e.g. to what extent did the side effects of the medication negatively influence your daily life?). The ease-of-use items focus on the administration mode and the (potential) interference of administering the medications with daily life. Each domain has one item on satisfaction (e.g. how satisfied or dissatisfied were you with the ease of use of the medication?). In addition, there is an ‘overall satisfaction’ item asking the respondent to take all domains of the medication into account. A 5-point (Likert-type) scale with the following anchor levels was chosen as the response format for items evaluating experiences: ‘not at all’, ‘a little’, ‘reasonable’, ‘a lot’ and ‘very’. In the domains of effectiveness and side effects, a ‘don’t know’ response category was added. Items regarding satisfaction were scored using a (horizontal) thermometer, ranging from − 5 (very dissatisfied) to + 5 (very satisfied).
Mean scores for experiences in each domain of the generic module were calculated if at least two items of the domain were completed. Response categories were coded from 0 (‘not at all’) to 4 (‘very’) and the domain scores can therefore range between 0 and 4. The ‘don’t know’ response category was considered a missing value for the purpose of calculating mean scores. Higher scores in the domain of effectiveness represented higher positive experiences regarding the effectiveness of the medication, while in the domains of side effects and ease of use, higher scores represented higher burden from side effects and lower ease of use, respectively. To facilitate interpretation, original scores in the domains of side effects and ease of use were recoded so that higher scores represented low burden of side effects and ease of use (i.e. for all domains, higher scores represent positive experiences). The items related to satisfaction (i.e. items 5, 10, 14 and 15) were not recoded; the reported scores (ranging between − 5 and 5) were used.
Disease-Specific Modules of the PESaM Questionnaire
IPF and aHUS patients completed the applicable disease-specific module of the PESaM questionnaire in conjunction with the generic module. Similar to the generic module, the disease-specific modules focus on experienced effectiveness, side effects and ease of use of the medication, but do not include items regarding satisfaction. The items on effectiveness assume a positive influence on health, the items on side effects a negative influence and the items on ease of use focus on the potential inconvenience of the specific mode of administration and whether patients have skipped medication. The difference between these and the generic module is that the disease-specific modules evaluate the effectiveness of the medication on specific disease symptoms. For example, the module for aHUS asks about the influence of eculizumab on energy levels, the ability to participate in society and fear of infection (meningitis), and provides a checklist of potentially experienced side effects. The module for IPF focuses on its perceived ability to slow down disease progression, reduce coughing, feeling tired, out of breath and whether respondents experienced side effects such as photosensitivity, nausea and diarrhoea. Details on the contents and response levels (e.g. which side effects are included) can be found in our earlier paper . Response levels and the scoring of domain scores for effectiveness and ease of use are similar to the generic module and range between 0 and 4. For each experienced side effect, respondents are asked to rate how bothered they are by that side effect and a sum score for this domain is calculated by multiplying the number of side effects with the respondent’s average rating of bothersomeness. Thus, a higher score represents a higher level of bothersomeness experienced due to one or more side effects.
Participants completed the EQ-5D (3-level version) to measure HRQoL . The EQ-5D consists of two components: a descriptive system of health and a visual analogue rating scale (VAS). The descriptive system consists of five items (mobility, self-care, usual activities, pain/discomfort and anxiety/depression), each with three response levels (no problems, some problems and severe problems). Reported health states on the descriptive system are converted to an EQ-5D index score where 1 represents full health and 0 represents death . The EQ-VAS records the respondent’s self-rated health on a 0–100 scale, with the endpoints respectively labelled ‘worst imaginable health state’ and ‘best imaginable health state’.
Demographic and Clinical Data
Date of birth, gender, date of diagnosis, medication and time on medication were collected from electronic medical files. For IPF patients, forced vital capacity (FVC) at diagnosis and around completion date of the questionnaire were collected, expressed in percentage and litres.
Due to a limited number of repeated measurements, a cross-sectional dataset was used for testing psychometric properties of the generic module. In case participants had completed the questionnaires at multiple time-points during the study, only their first completed questionnaire was used for this analysis.
A stacked bar chart graphically presents the distribution of scores, percentage of missing values, and ‘don’t know’ responses. Floor and ceiling effects were considered present when at least 15% of respondents scored the lowest or highest possible score, respectively .
Internal consistency as a measure of the extent to which the items in the domains of effectiveness, side effects and ease of use are correlated (homogeneous), thus measuring the same concept, was assessed by calculating the Cronbach’s alpha for each domain separately. A low Cronbach’s alpha indicates a lack of correlation between the items, which makes summarizing the items unjustified. The internal consistency was considered good when Cronbach’s alpha was between 0.70 and 0.95 .
Confirmatory factor analysis (CFA) with robust maximum likelihood estimation was used to test the factor structure of the generic module. We hypothesized that items 1–4 measure the latent construct ‘experienced effectiveness’, items 6–9 ‘bothersomeness of side effects’, and items 11–13 represent ‘ease of use’. First, goodness-of-fit indices were used to evaluate the adequacy of the model’s fit to the data, including the Chi-square value, comparative fit index (CFI), Tucker–Lewis index (TLI), root mean square error of approximation (RMSEA) accompanied by its 90% confidence interval (CI) and standardized root mean square residual (SRMR). CFI and TLI values exceeding 0.95, and SRMR and RMSEA values close to (or less than) 0.08 and 0.06, respectively, represented a good fit . Second, convergent validity was examined by assessing the size and significance of the factor loadings (using the standardized regression coefficients) and the average variance extracted (AVE) for each factor, which should exceed 0.50 . Finally, discriminant validity (i.e. whether each one of the domains has enough discriminant validity from the other domains) was evaluated and considered good when the correlation coefficients between the three domains were < 0.8.
Construct validity was further assessed by investigating whether scores on the domains (i.e. constructs) of the PESaM generic module (e.g. experiences and satisfaction) relate to other constructs in a manner that is consistent with a priori hypotheses [30, 36]. The following hypotheses were tested:
It was a priori expected that patient experiences have a medium to high correlation with satisfaction; more positive experiences (e.g. low burden of side effects) are associated with higher levels of satisfaction [4, 29, 37].
It was expected that the effectiveness of the medication was considered the most important domain of medication use to patients and therefore that ‘satisfaction with effectiveness’ and ‘experiences with effectiveness’, respectively, have the largest independent contribution to overall satisfaction, relative to the contributions of the other domains (i.e. side effects and ease of use).
Positive patient experiences in the PESaM refer to an experienced positive impact on physical, emotional and social health, respectively. It was therefore expected that patient experience scores for the domains effectiveness and side effects of the medication were moderately correlated with HRQoL [4, 38, 39].
Patient experiences of IPF and aHUS patients regarding effectiveness, side effects, and ease of use, as reported in their respective disease-specific modules, were expected to have a strong (positive) correlation with the corresponding experiences as reported in the generic module.
The strength and direction of the associations were measured using the Pearson product-moment correlation or Spearman’s rank-order correlation coefficient, depending on the distribution of the mean scores or measurement scale (hypotheses 1, 3 and 4). A correlation coefficient between 0.90 and 1.00 would indicate a very high positive correlation, 0.70–0.90 a high (positive) correlation, 0.50–0.70 a moderate (positive) correlation, 0.30–0.50 a low (positive) correlation, and 0–0.30 a negligible correlation .
Multiple regression analyses (enter method) were conducted to examine the relationship between overall satisfaction (item 15) and several proposed predictors (hypothesis 2). Unstandardized regression coefficients (B) and explained variance (adjusted R2) were estimated for two multiple regression models: (i) satisfaction with effectiveness (item 5), satisfaction with side effects (item 10), and satisfaction with ease of use (item 4) as predictors of overall satisfaction; and (ii) experiences with effectiveness (domain score), experiences with side effects (domain score), and experiences with ease of use (domain score) as predictors of overall satisfaction. Mean scores on the patients’ rating of importance of each domain (items 16, 17 and 18 of the generic module) were compared using repeated measures ANOVA and post hoc testing (Bonferroni). Subsequently, the ranking of the importance of the domains (in case of significant differences between mean scores) were compared with the results of the regression models (i.e. the size and significance of the independent contribution of a domain to overall satisfaction). It was hypothesized that a direct measurement of the importance of domains (items 16–18) would produce a ranking that is similar to results of the regression analyses (where the domain with the largest regression coefficient is considered to be the most important domain to the patient).
Known-groups validity (i.e. whether an instrument shows different scores for groups that in theory should have different scores) was tested by comparing mean scores of the different medications.
We expected varying scores between the drugs for different diseases due to varying therapeutic effects, side effects and modes of administration. More specifically, we tested the following hypotheses:
It was expected that patient experiences regarding effectiveness were more positive for eculizumab and tacrolimus compared with pirfenidone or nintedanib, since the latter therapies aim to reduce disease progression, which may be difficult to experience by patients over short periods (i.e. ‘the past 4 weeks’).
It was expected that eculizumab users report less positive experiences with ease of use, due to the intravenous administration requiring inpatient hospital visits.
No major differences in ease of use were expected between pirfenidone, nintedanib and tacrolimus, all requiring oral administration.
A final hypothesis is that long-term users of a medication (i.e. > 2 months on medication) had more positive experiences and were more satisfied with a medication compared with new users (i.e. ≤ 2 months) .
Analysis of variance (ANOVA) with post hoc (Tukey) testing was used for comparing scores between medications. The independent sample t test was used when two groups (i.e. new users vs long-term users) were compared. A p value of 0.05 was used as the cut-off for significance.
A subsample of patients in stable health, as determined by their healthcare provider, were asked to complete the generic module a second time, 2 weeks after their initial completion (test–retest). The Intraclass Correlation Coefficient (ICC) was used to test the reliability of the questionnaires between the two measurements . ICC estimates and their 95% confident intervals were calculated based on absolute agreement and a 2-way mixed-effects model. Following recommendations by Terwee et al., the reliability is positively rated when the ICC is at least 0.70 .
Statistical analyses were performed in SPSS statistical package version 23 (IBM SPSS, IBM) and R version 3.5.1 using the lavaan package .