Background

An Intensive Care Unit (ICU) is a place full of uncertainty and stress for both patients and for their relatives. Many researchers have suggested that relatives of ICU patients have a need for accessibility, support and information [1–3]. Nurses usually respond to these needs intuitively, based on individual experience, rather than in an evidence-based way [4]. Furthermore, relatives rarely ask for support and attention which may, in the long term, result in psychological distress [5, 6]. Therefore, it is necessary to better tailor the quality of care to relatives. Furthermore, according to the quality standards of the Dutch Society of Intensive Care (NVIC), every ICU needs to monitor the quality of care, including the satisfaction of the relatives with respect to the care [7, 8]. However, an evidence-based, valid, and reliable Dutch instrument that evaluates concrete experiences and perceived quality of care from the perspective of the relatives is still lacking.

To bridge the gap of this lacking instrument, we had some thoughtful considerations translating and adapting existing questionnaires. The ‘Critical Care Family Needs Inventory’ (CCFNI) is a questionnaire with 45 items to measure the needs of relatives in the ICU [9] in a French and an English version. However, need as a unique starting point is not sufficient to confirm simply and straightforward which interventions may have positive effects in the support of relatives [10]. Thus, the CCFNI does not adequately assess the quality of care as perceived by relatives. Another frequently used questionnaire to evaluate the satisfaction of the ICU patients’ relatives is the ‘Family Satisfaction in the ICU survey’ (FS-ICU) [11, 12]. Their items are based on an existing framework that measures patient satisfaction, in combination with items related to end-of-life care. The FS-ICU 24 seems a valid, reliable and feasible instrument for determining the satisfaction of relatives in ICU. Quite apart from the fact that satisfaction of patients might not at all correlate with the satisfaction of the relatives [13], it is preferable to measure experiences rather than satisfaction as they give more objective and specific information for quality improvement [14]. The utilized concept of satisfaction might raise some bottlenecks such as ceiling effects, cognitive dissonance and socially desirable answers. A discrepancy model, which describes satisfaction as a result of expectation minus the perceived experience, could overcome these problems [15]. Because of this conceptual difference, the FS-ICU was not used to translate and adapt the items of the questionnaire. The ‘Critical Care Family Satisfaction Survey’ (CCFSS) was assessed as a reliable and valid tool to measure the satisfaction of relatives as well [16]. Yet, both instruments, the FC-ICU 24 and the CCFSS, have a disadvantage when being implemented in the Netherlands, as they have been developed and used in a non-Dutch situation. Therefore, it is likely that some items will be rated as being more or less important by relatives in different countries or even on different continents [17]. For example, perceptions related to decision making might have fundamental culture specific differences on overall responsibilities of the medical team or the relatives. In addition, questions in this domain seemed multi-interpretable and difficult to translate in the exact meaning of the original questionnaire. Therefore, it was desirable to develop a measurement instrument that specifically evaluates the quality of care from the perspective of relatives in ICUs in the Netherlands in a logical follow-up of all previous studies.

This paper describes the development of a valid, reliable and feasible measuring instrument in the quality of care for practical use in ICUs in the Netherlands. The development process was based on standards for determining the experiences with provided care from a client group’s perspective, according to the Consumer Quality Index (CQI) method [18]. The CQI instruments are theoretically founded by the CAHPS® instruments and QUOTE® methodology, both based on a discrepancy model. To meet a sufficient quality of care, the expectations regarding the quality should be in accordance with the perceptions of the actual experiences according to these methodologies [15]. This questionnaire, the CQI ‘Relatives in Intensive Care Unit’ (CQI ‘R-ICU’), has been developed in a close cooperation between the University of Applied Sciences of Arnhem and Nijmegen, the Open University of the Netherlands and three hospitals (Erasmus University Medical Centre Rotterdam and the regional medical centers Kennemer Gasthuis Haarlem and Ziekenhuis Gelderse Vallei Ede). The Medical Ethics Committee of Erasmus MC judged that the research proposal (MEC-2011-189) complied with the Dutch law on Medical Research in Humans (WMO).

The strength of the CQI questionnaire is that it addresses the conceptual and methodological problems associated with satisfaction surveys, and that relatives were directly involved in the instrument’s development. The questionnaire focuses on “reports” of facts and experiences of the quality of care rather than on subjective ratings of satisfaction [14, 15, 19]. An important step in the development of a CQI is determining the measurable aspects of care (quality indicators), whereby many authors have adopted a structure, process and outcome indicator [20–22]. The aim of this study is to develop an appropriate set of quality indicators which measures all the domains in the quality of care relating to relatives in the ICU.

Methods

Questionnaire development of the CQI ‘R-ICU’

The overall research plan, based on the CQI Manual [18], consisted of qualitative and quantitative surveys. The method included four phases: 1) preparation, 2) performance, 3) psychometry, and 4) discriminatory. This CQI method has been described extensively in a previous study [15]. The current article, is limited to the first three phases. The Strobe guidelines have been used for preparing current article [23].

Preparation phase

The aim of the preparation phase was to detect relevant quality aspects of healthcare performance in the ICU in the support of relatives. A literature review was completed in Pubmed, Embase, Cinahl and Invert, including a search for existing questionnaires published in the period 2000–2011. Subsequently, five experts composed a topic list for focus group discussions with relatives of ICU-patients on their experiences in the ICU. Focus group discussions were conducted with a total of 18 Dutch speaking relatives of ICU patients aged 18 and older; in Erasmus MC (n = 7), Kennemer Gasthuis (n = 4), and Ziekenhuis Gelderse Vallei (n = 7). In each hospital, the same researcher (IM) acted as moderator during the focus group discussion. Each discussion was audiotaped; the transcriptions were coded and analyzed through thematic analysis [24] by four researchers.

The performance phase

Based on the results of the preparation phase, a questionnaire was drafted in the performance phase. Following the CQI Manual guidelines [18], the questionnaire consisted of two parts. Part one consisted of three types of questions covering experience, problems, and general judgment. The experience questions were either scored on a 4-point Likert scale (ranging from 1 = never, 2 = sometimes, 3 = often to 4 = always) or they were dichotomous (yes/no), as prescribed in the CQI Manual guidelines and forcing a non-neutral answer. The problem questions were scored on a 3-point scale (ranging from 1 = big problem, 2.5 = small problem, 4 = no problem), also according to the development guidelines and meant to gather insights of the size of the problem [18]. The general judgment questions were scored on a scale from 0 to 10 (0 = very bad, 10 = excellent). Demographic variables and questions regarding general health were added.

Part two determined the relative importance of the items in the questionnaire. Ordinary 5-point Likert scales tend to be highly skewed towards the ‘important’ dimension. Therefore, a greater differentiation on the positive side of the continuum was realized by a Likert scale using 4-point response choices: 1 = not important, 2 = of some importance, 3 = important, 4 = extremely important. This non-neutral solution was proven to be workable [25].

This first draft was sent to 11 relatives of ICU patients and to 21 ICU experts (doctors, nurses, social workers, spiritual counselors, psychologists) in the three hospitals where the cognitive test (pilot test) was held. The relatives should not experience any problems self-completing the questionnaire, and the items had to be relevant, unambiguous, understandable and useful. Based on their response, any unclear items in the CQI ‘R-ICU’ were rephrased.

Psychometric phase

In the third phase, the questionnaire was tested to assess the psychometric properties. In the period October 2011 - July 2012, the test was conducted in six hospitals: 4 ICUs in one university hospital, and 5 single ICUs in general hospitals. Inclusion criteria of the relatives were: age >18; Dutch language speakers; relative is partner or child/brother/sister, parent or other important relative of the ICU patient; patient stayed at the ICU ≥ 24 h; and the relative was present at admission at the ICU. Relatives were excluded when the patient died within 24 h after admission to the ICU. Most of the eligible relatives were approached to participate in the study. No data were available of relatives who were not met to include in the study, for example because the time of visiting was not matching to researcher’s working hours. After obtaining informed consent the questionnaire and cover letter were sent by post within 1 week of the patient’s discharge from the ICU. Up to two reminders were sent to non-responders after 1 and 4 weeks [26]. The recipients could return the questionnaire in a stamped, addressed envelope.

Data analysis

The data were analyzed in order to identify item response rates and frequency distributions. Questionnaires completed by someone other than the relative, and questionnaire items with more than fifty percent of the answers missing, were excluded. Questionnaire items were excluded from further analysis if they had an item non-response of >5% of expected responses or an extreme skewness score (>90% of responses in the same category, i.e., a ceiling or floor effect). Spearman’s correlation coefficient was calculated to check for correlations between items (r > 0.70). Items with a negative wording were recoded to ensure comparability in the analysis. The answers on both the experience and the importance items were calculated in means and standard deviations to compare differences in the scores per item and to analyze significant differences between groups in the demographic variables.

Quality improvement scores

Quality improvement scores (QIS) were calculated by multiplying the importance scores with the percentages of the negative response categories ‘never’, ‘sometimes’, ‘big problem’ or ‘no’ on the corresponding experience questions. The improvement scores form an estimate of the potential improvement of quality of care and are useful for internal monitoring. Scores > 1.0 may potentially improve quality of care (range: 1–4).

Factor analysis

An exploratory factor analysis (EFA) was used to cluster the experience questions. In the first EFA, only the 27 experience items with a 4-point Likert scale were included. Preliminary requirements were Kaiser-Maier-Olkin Measure of Sampling Adequacy (KMO-value) ≥ 0.60 and the Barlett’s test of sphericity significant (p > .05), so that the number of respondents was sufficiently large and the correlations between the variables high enough to detect a factor loading. All factors with Eigenvalue ≥ 1.0 and factor loadings ≥ 0.30 were retained. The internal consistency of the different subscales was analyzed using Cronbach’s alfa, using α ≥ 0.70 as criterion for being reliable. In addition, we used the Item Total Correlation (ITC) ≥ 0.30 to define whether or not an item belonged to a certain subscale. All statistical analyses were performed using SPSS 18.0.

Results

Preparation and performance phase

The literature search in PubMed and EMbase resulted in 284 and 285 hits respectively; a search on PsycInfo, Invert and CINAHL did not lead to new aspects. Subsequently, adding ‘questionnaire’ in the title and abstract resulted in 43 articles. To explore relevant topics, we studied the Critical Care Family Needs Inventory (CCFNI) [9, 27], the Family Satisfaction in the Intensive Care Unit (FS-ICU 24/34) [12], the Critical Care Family Satisfaction Survey (CCFSS) [16], the Parent Satisfaction Instrument [28] and the CQI-palliative care relatives [29]. The resulting topic list with relevant quality aspects consisted of: support at first entrance in ICU, information and communication, attitude of the caregivers, (multidisciplinary) support, participation, organization of ICU, discharge to a general ward, and aftercare. Table 1 shows how the various aspects of the focus group discussion were divided among these quality indicators.

Table 1 Quality indicators and aspects: results from the qualitative phase
Full size table

The draft of the CQI ‘R-ICU’version 3.0, with the cognitive test feedback from relatives and the critical view of healthcare professionals, consisted of 74 questions divided into 12 categories: (1) general; (2) support at time of admission; (3) information and communication; (4) attitude; (5) ICU organization; (6) support; (7) transfer; (8) support from peers; (9) after care; (10) general judgment; (11) demographic variables, and (12) final questions. Of the total of 74 questions, 57 were constructed as so called ‘experience questions’. This resulted in a temporary set of 44 importance questions in part two of the CQI ‘R-ICU’(Fig. 1).

Fig. 1
figure 1

CQI ‘R-ICU’development process

Full size image

Psychometric phase

In total, 564 relatives received the CQI ‘R-ICU’. Of these, 455 returned the questionnaire (response rate = 81%); four uncompleted questionnaires were excluded, as were five questionnaires which has been filled in by someone other than the relative. The dataset for the analysis contained 446 questionnaires. Relatives’ characteristics are presented in Table 2.

Table 2 Characteristics of the study sample (n = 446)
Full size table

Results of the experience questions on the ICU

Table 3 shows the results of the five highest and five lowest scores on the experience questions calculated in mean scores. The highest scores were calculated for the most positive category, and the lowest for the most negative category or the dichotomous response category. The mean score for the question General judgement for Care and Support in the ICU was 7.56 (SD = 1.83). A paired samples t-test showed a significant difference in the scores for Communication with nurses (M = 8.23, SD = 1.46) and Communication with doctors (M = 7.39, SD = 2.20) conditions; t(424) = −9.38, p = 0.000.

Table 3 Five of the highest and lowest mean scores in the experience questions
Full size table

We used ANOVAs to test the differences between gender and age groups, relation to patient, length of stay at the ICU, planned or unplanned admission, deceased or not, and level of education, on the items ‘General judgment of the ICU’ and the two items ‘Communication with nurses and with doctors’. There was a significant difference between the different age groups on ‘General judgment of the ICU’ [F(2, 425) = 6.46, p = .002]. Ratings for the group ≥ 65 were higher (M = 8.02, SD = 1.6) than for the group 18–54 (M = 7.25, SD = 2.04). For the two overall items ‘Communication with doctors and nurses’, relatives rated a mean score of 7.39 (SD = 2.20) and 8.23 (SD = 1.46) respectively on a scale 0–10. We found a significant difference between the group survivors and non-survivors on the item ‘Communication with doctors’ [F(1, 419) = 5.13, p = .024]. The non-survivors group rated ‘Communication with doctors’ higher (M = 7.89, SD = 2.13) than the survivors group (M = 7.28, SD = 2.19). There was a significant effect on the item ‘Communication with nurses’ at the p < .01 level [F(2, 425) = 2.95, p = .053]. The ≥ 65 group rated ‘Communication with nurses’ higher (M = 8.41, SD = 1.43) than the 18–54 group (M = 8.03, SD = 1.55).

Results of the importance questions

In the first step of analysis of the importance study to select items for construction of the CQI ‘R-ICU’, the scores were calculated for questions with the response category ‘extremely important’ (score = 4) and the categories ‘not important/of some importance’ (score = ≤2). Respondents on the importance questions considered 12 items to be ‘extremely important’ and 8 items to be ‘not important/of some importance’. Table 4 represents the five most and the five least important items. In the next step, the quality improvement scores (QIS) were calculated by multiplying the mean of the importance scores (IS) with the percentages of the negative response categories ‘never/sometimes/big problem or no’ on the corresponding experience questions (EQ). The QIS gave an estimate for the potential improvement of quality of care and were useful for internal monitoring. Table 5 shows the top-10 of scores > 1.0.

Table 4 Five of the highest and lowest scores in the importance scores (%)
Full size table
Table 5 Top 10 of Quality Improvement Scores (QIS) >1.0
Full size table

Factor analysis and internal consistency

According to the CQI guideline [18], 6 items were left out of the factor analyses. Five experience questions had a nonresponse >5%: accessibility fixed contact person; doctors listen carefully to relatives; family room fits needs; sufficient facilities to stay at ICU; and attention to mutual contact relatives. One item, ‘relatives get information through digital means’, was extremely skewed (>90%).

Factor analysis was carried out on 27 experience items with a 4 point Likert scale to determine the underlying structure of the newly developed questionnaire. The PCA met all the requirements, KMO excellent (0.88) and Bartlett’s test of sphericity significant (N = 364 en p < .000). The EFA (Table 6), based on 27 experience questions, showed a 2-factor-solution with an explained variance of 34.4%, covering all items (KMO 0.88, Bartlett’s test p < 0.000, N = 364). Two factors that best show the quality of care resulted from the EFA: Communication (α = .80) with 14 items, and Participation (α = .84) with 13 items.

Table 6 Domains, items and internal consistency of the second factor analysis (27 items)
Full size table

CQI ‘R-ICU’ 4.0

Based on the results of the psychometric test, the CQI ‘R-ICU’, version 3.0 was revised following a pilot test and discussion by experts. Nine experience questions were reformulated: information about a fixed person; information which professionals contributed to care; presence of family waiting room in ICU; attention to relatives and their needs; coming in contact with other disciplines; information about transfer; and need for after care personal interview. The questions on the following 5 issues were deleted: nuisance of noises equipment; attention to contact between relatives; transfer a problem from ICU to other ward; information about support group; and need for contact with peers. In addition, the 5 ‘skip and go’ questions were deleted and the response category ‘inapplicable’ was added for the questions to which they belonged.

The CQI ‘R-ICU’ version 4.0 (see Additional file 1) consisted of 55 questions divided into 8 categories (Fig. 1): general (5 items); support (12 questions); attitude (8 items); information (8 items); organization (6 items); general judgment (5 items); demographic variables (10 items); and final questions (3 items). The CQI ‘R-ICU’ version 4.0 will be tested on a large scale in 20–30 ICUs in the discriminating phase of the study.

Discussion

The aim of this study was the development and psychometric evaluation of a new Dutch questionnaire to measure experiences of ICU patients relatives’ with the quality of care, the CQI ‘R-ICU’. At first, the underlying aspects of the total care quality assessment for relatives of ICU patients were determined. From the focus group discussions it appeared that relatives found information on the patient’s situation of utmost importance. Furthermore, they considered the following aspects to be essential: to be involved, honest communication, way of approaching, attention, time of the health care providers, emotional support, participating in the process of caring, and being present during medical visit rounds. The participants ranked second in importance: care at first entrance, visiting hours, privacy, and waiting room.

The a-priori grouping, with the structure indicator ‘Organization’ and the process indicators ‘Communication’, ‘Care for relatives’ and ‘Participation’ were not found in the factor analyses. Alternatively, a clustering of items was found on two process factors; Communication and Participation. We need to note that these factors include the same items as preliminarily defined in the aspects of the quality indicators, although they also contain items of the structure indicator Organization (privacy, noise, waiting room, possibilities to visit). The items ‘noise’, ‘possibility to visit’, and ‘privacy’ showed a limited internal consistency (ICT < 0.3), and were consequently removed from the factor. The item ‘waiting room’ was excluded due to the number of missing values. The item ‘information about parking and parking fees’ was classified in the factor Participation, but it also had an ICT < 0.3. All these items seemed to fit with the factor Organization. When we performed a factor analysis with four factors, the factor Organization only included 2 items ‘privacy’ and ‘possibilities for visiting’ with a very low alpha (α = .49).

Following the CQI guideline, [18] several items (digital information, having fixed person, contact and support with other professionals, waiting room, transfer from ICU, support group, after care appointment) were not included in the factor analysis. These items were either not categorized as an experience question, skewed, had a high number of missing values because of ‘skip and go’ answers. Although some items (e.g., doctors listen carefully) had too many missing values, we had to include the item as they were required questions. Most of these items were reformulated and have been included in the CQI ‘R-ICU’ 4.0 for the discriminatory phase study, which will be more appropriate to determine its underlying structure.

The short length of stay of the patient at the ICU might have been another important reason for missing values. In these cases, relatives often indicated that they were not able to answer all the questions. In current study, relatives were included based on being admitted for ≥ 24 h at the ICU. In other studies, family of patients were included only when admitted for ≥48 h at the ICU [26] or >6 h [30]. This variable length of stay is important for the analysis of the discriminatory phase, in order to decide whether it is necessary to design a special CQI ’ R-ICU’ for the short-stay group.

In current study, the overall score of relatives’ experiences with the quality of care at the ICU was high, which matches findings from other studies [29–32]. Relatives were most satisfied about organizational and environmental aspects (e.g., no noise of staff, visiting hours, shift in medical staff, accessibility fixed person) and aspects of communication (e.g., no conflicting information, understandable) and they were dissatisfied about informational aspects (e.g., digital and written information, support groups, meals), participation aspects (e.g., care for patient, keeping diary) and supportive aspects (after care personal interview). Both in current and another study [29], relatives wanted the physician to be more available for regular person-to-person calls. It is noteworthy that the question ‘do doctors listen carefully’ had a high rate of missing values (6.1%); relatives noted that they had not seen the doctor. A possible explanation was that these patients were only admitted for 1 day and the relatives had missed the doctor’s visit. This item will be revised with a response category ‘did not see a doctor’.

The oldest group (>65) had a significantly higher overall score and a significantly higher score regarding the communication with nurses. The relatives of the non-survivors had a significantly higher score regarding the communication with doctors. These results correspond to results from other studies [33, 34], that recommend that younger patients and their families may need more support around end-of-life preparation and discussion of treatment preferences.

Current study showed that items related to contact with professionals for psychosocial care, aftercare and support groups, were relatively less important. In recent literature, the opposite was observed [35–37]. It may be possible that relatives only recognize the impact of the stressful period and the need for psychosocial care at a later stage [35]. Possible explanations for this may be the of measurement, the kind of relationship, severity of illness, and current situation of the patient.

Practical implications

Two strengths of the development of the CQI ‘R-ICU’ questionnaire are that it attempts to overcome the conceptual and methodological problems associated with previous satisfaction surveys and that the relatives were directly involved in the instrument’s development. The questionnaire focuses on “reports” of the quality of care rather than on highly subjective ratings of satisfaction [15]. These individual-reported measures are essential to quality improvement programs as they will provide feedback regarding person-centeredness in daily practice to healthcare professionals and policy makers. It is therefore essential to involve the individual as an active partner in professional care and treatment [38]. This means seeing them as valuable persons, working alongside professionals to get the best outcome. The CQI ‘R-ICU’ is a helpful instrument to learn from the relatives as partners in caring for the ICU patient and use these reports to advance quality improvement effort in the ICU. Although this instrument has been developed in the Netherlands, the method could be applied in all healthcare settings in an international perspective. It seems a general and robust measurement instrument, even more applicable if the discriminative phase has been reported. This study, already conducted in 21 hospitals nationwide, will provide further knowledge on the applicability in different settings such as cardiology, general mixed, and specific ICUs. The translated English version of the CQI ‘R-ICU’ 5.0 can be found on http://blog.han.nl/acute-intensieve-zorg/files/2009/07/Engelse-vertaling-CQI-5.0-Naasten-op-de-IC-1.pdf.

Limitations

Although the study had a strong design, with qualitative and quantitative result to rely on, some limitations to the results could be made. First, the respondents might have provided socially desirable answers, which were more positive than actually experienced. Second, a selection bias may have occurred, only the most satisfied individuals returned the questionnaire. Third, this study was performed in six hospitals resulting in a relatively small number of respondents to evaluate the whole population of ICU patients relatives’ in the Netherlands. Therefore, the results are not generally applicable. A nationwide multicenter study, which has already been conducted, is necessary to confirm or disapprove the identified results.

Conclusion

The development and psychometric evaluation of the questionnaire-based Consumer Quality Index ‘Relatives in Intensive Care Unit’ (CQI ‘R-ICU’) resulted in a draft questionnaire that was sufficient to justify further research into the reliability, validity and the discriminative power of the questionnaire.