Methods part 1: development of the item pool
In order to develop a sound questionnaire with high content validity, a protocol based on recommendations by Haynes (Haynes et al. 1995) and by Terwee (Terwee et al. 2007) was followed. The development of the item pool comprised of three phases: the preparation phase, the item generation phase and the revision phase, is described in detail below. Figure 1 presents an overview of the study design with the methods and results for each step.
Procedure of the preparation phase: In the first phase, we conducted two systematic literature searches in four databases: PubMed, PsycINFO, Embase, and Cinahl. We aimed to inventory all literature about effects of CMDs on work functioning in general (first search) and nurses and allied health professionals in particular (second search) (Gartner et al. 2010). Subsequently, five focus group interviews were held. Following a multiple category design (Krueger and Casey 2000), three focus groups were held with nurses and allied health professional and two with experts on work functioning in the health sector.
The focus group interviews with a duration of 2 hours were conducted by two researchers (FG & KN) who alternately moderated or observed. The group interviews were structured by three cases, which were presented to the participants. The cases, written in the second person, described, respectively, an employee with fatigue and stress, depression and anxiety, and alcohol abuse. Participants were asked to reflect on aspects of the work that might be affected by the mental health complaints described. By working with these cases, participants of the employee focus groups were not forced to disclose whether mentioned examples were derived from own experiences or from the behavior of colleagues. In the beginning of each focus group, the discussion was explorative in nature. Later on, aspects of impaired work functioning derived from our literature review were validated and supplemented with illustrative examples. The moderator ensured that for each aspect of impaired work functioning mentioned, the different occupations and specialties present gave concrete examples. The moderator explicitly asked for differences in experiences between the various occupational groups present. Also, the moderator asked to clarify any ambiguities in the examples of participants.
Each focus group discussion was audio taped. The Medical Ethics Committee of the Academic Medical Center Amsterdam decided that approval of the research protocol by the committee was not required.
Textbox: Cases used for the focus group discussion
Case1: Try to imagine yourself in the following situation:
Due to conflicts at home you have not been feeling well the past weeks. You have much less energy than usual and after a long day at work you feel too exhausted to do your everyday activities and to relax. This morning you arrive at work feeling stressed already, today will be a very busy day again. Just the idea of all the work you have to do makes you tired.
What difficulties do you expect to face during this workday?
Case 2: Try to imagine yourself in the following situation:
Since a few months you have not been feeling very well. In the last few weeks you have been feeling especially bad. You feel depressed, there is nothing you want to do or what excites you. The only thing you feel like doing is to stay in your bed all day long. At work you sometimes feel anxious without any reason; you can’t tell where the anxiety comes from, the feelings just comes over you. In the past weeks you have had more and more difficulties to accomplish your tasks at work.
Can you describe how your working day goes in these circumstances?
Case 3: Try to imagine yourself in the following situation:
You have a nice team you work with, with many different people and you get along with each other very well. Since a while you have noticed that one of your colleagues behaves differently. Regularly, you have the feeling she smells of alcohol.
What has changed in the behavior of your colleague?
Subjects of the preparation phase: Focus group members were recruited from one academic medical center using a purposive sampling procedure, with variation in wards and occupations as a major criterion. Nurses and allied health professionals for the three employee focus groups were invited via head nurses. For the selection of participants in the focus groups, we asked for a mix between healthy participants and participants with current or past mental health complaints. We assumed that every employee can deliver input on the research question either based on own experiences with mental health complaints in the presence or past or otherwise based on observations in co-workers with mental health complaints. Employees from the same ward were assigned to different focus groups. Information was collected about the participants’ history of mental health complaints. Of the 19 participants, 16 had experienced a difficult period in life with effects on their mental health in the past and three currently experienced problems. Nine participants had (mild) mental health complaints in the past and one currently had.
Participants for the expert focus groups, such as senior nurses and occupational physicians, were personally invited. Informed consent was obtained from each participant, and all participants were compensated with a 25 Euro voucher for their 2-h participation.
Analysis of the preparation phase: Audiotapes of the focus groups were transcribed verbatim. The analysis of the focus group interviews followed a purpose-driven approach, aiming to distinguish as many different signals of impaired work functioning as possible and to organize all signals into themes (Krueger and Casey 2000). First, each interview was open coded. In this inductive step, all examples of impairments in the work functioning were indexed. During the coding procedure, we aimed to be as inclusive as possible. Therefore, in case of inconsistencies between codes, no exclusion or broadening of codes was performed but inconsistent codes were preserved. Second, codes were refined and reduced within a process of re-reading and constant comparison (Pope et al. 2000). Third, the obtained codes were categorized into themes covering related aspects of work functioning. One researcher (FG) performed the coding of the data; subsequently, a second researcher (KN) checked the coded data of each interview. For the analysis of the literature review, see Gärtner et al. (2010).
Item generation phase
Procedure of the item generation phase: In the second phase, items were formulated based on the results of the literature search and focus groups. For each theme that resulted from the preparation phase, sufficient items for possible subscales were formulated (minimum of seven). Each item had to refer to a clear, concrete single action or behavior. To connect with the actual behavior and perception of nurses and allied health professionals, item formulation had to reflect expressions from focus group participants as much as possible. Where possible, items had to be applicable for the different tasks and jargons of the various occupations and specialties as well. A four-week timeframe was chosen for all items. Response formats were chosen according to the content of the associated themes with a minimum of five and maximum of seven categories (Streiner and Norman 2008). Response scales utilized were the following: Likert-type scales (from 0 = totally disagree to 6 = totally agree), an adjectival scale asking for difficulty (from 0 = no difficulty to 6 = great difficulty), relative frequency categories (from 0 = almost never to 6 = almost always), or absolute frequency categories from (0 = not once to 6 = on average more than 1x per day).
Analysis of the item generation phase: The results of the focus groups together with the information derived from the literature reviews were synthesized into themes and all signals of impaired work functioning were translated into items. These were discussed several times by all of the authors, which resulted in the first pool of items. In this phase, we adhered to the principle of being as inclusive as possible (Terwee et al. 2007).
Procedure of the revision phase: As part of the revision phase, the first pool of items was submitted for an expert check. Six experts (head nurses and occupational health professionals) were asked to identify items that were unclear or irrelevant. They were asked to rate the relevance of each theme and the completeness of the questionnaire as a whole on a 5-point Likert scale ranging from 1 = not at all relevant/complete to 5 = highly relevant/complete. On item level, the relevance was rated on a 2-point scale (yes, no). In addition, participants were invited to suggest supplementary themes and items.
Subsequently, verbal probe interviews were conducted with six nurses and allied health professionals who reviewed the individual items in a 1-hour interview (Willis 2005). Participants were asked to identify any item that was unclearly formulated, difficult to respond to, or not applicable to all nursing wards and allied health professions. Additionally, the preference for response formats was discussed.
Subjects of the revision phase: For the expert checks, six key persons (head nurses and occupational health professionals) were invited. For the verbal probe interviews, six nurses and allied health professionals were invited personally. The sampling in this phase was again purposive and we aimed to have as many different professions represented, e.g., also (head) nurses form anesthetic and surgical nursing wards and allied health professionals. The experts, nurses, and allied health professionals invited were partly already participated in the focus group interviews and partly were newly recruited.
Analysis of the revision phase: Possible changes in the item pool resulting from the expert checks and verbal probe interviews were proposed by one researcher (FG) and discussed by the research team until consensus was reached. Items and response categories that were reworded where when possible checked in subsequent interviews. Expert comments on missing signals of impaired work functioning led to the formulation of additional items. In order to draw conclusions on the content validity, the quantitative results about the relevance and clarity of themes and items were summarized by frequencies of the given answers.
Methods part 2: item reduction and subscale generation
Procedure part 2
The second part of our study has a cross-sectional design. Respondents were contacted by e-mail and asked to fill out an electronic version of the item pool, which took approximately 45 min for completion on a computer. It was possible to log out half way through the survey and to continue after logging in again later on. However, the questionnaire had to be fully completed within 3 days. It was not possible to skip questions. Two reminders to complete the questionnaire were sent by e-mail. For each completed questionnaire, we donated 2.50 Euro to a charity that the respondents could select from among three options.
Subjects part 2
A random sample of 1,200 nurses and allied health professionals in one Dutch academic medical center was taken, as we expected a response rate of 25% and strived to recruit 300 respondents. This sample was stratified by age, gender, and occupation.
Information was collected about the participant’s gender, age, and the history of their mental health complaints. Mental health status was measured using two questionnaires. First, the General Health Questionnaire (GHQ-12) was used, a 12-item self-report questionnaire developed to detect common mental disorders in the general population (Goldberg et al. 1988). Following earlier studies in the working populations, a cut-off point of ≥4 was applied to identify individuals reporting sufficient psychological distress to be classified as probable cases of minor psychiatric disorder (Bultmann et al. 2002). Second, the 16-item distress subscale of the Four-Dimensional Symptoms Questionnaire (4DSQ) was used (Terluin 1998; Terluin et al. 2006). For case identification, a cut-off point of ≥11 was applied (van Rhenen et al. 2008).
Analysis part 2
A first reduction in items was based on the variation in answers. In the case of minimal variation (≥95% of answers given in one response category), exclusion of the item was discussed in the research team (Streiner and Norman 2008). Further reduction in items and determination of the underlying factors were based on explorative factor analysis with an orthogonal rotation approach, using principal component analysis (PCA) and Varimax Rotation (Stevens 2002; Tabachnick and Fidell 2001). To determine the optimum number of factors, we considered Catell’s screetest (1966). Kaiser’s criterion (retain factors with Eigenvalue >1) (Kaiser 1960), and parallel analysis, following the criterion that the PCA Eigenvalue of our dataset had to exceed the mean Eigenvalue of 100 random datasets with the same number of items and sample size (Horn 1965). In cases where these methods led to different numbers of components, we preferred the most interpretable component structure, with the least number of components.
Subsequently, we performed a sequence of PCA Varimax rotations and the analysis of internal consistency (using Cronbach’s alpha coefficients), to give meaning to the selected factors, to distribute items to the factors, and to further reduce the number of items (Ruiz et al. 2008; Stevens 2002). Items were assigned to a factor if their factor loading was 0.40 or greater (Stevens 2002). In case of cross-loadings, they were assigned to the factor with highest factor loading. The selection of items forming the definite subscale was based on the following considerations:
The content of the items: selected items should clearly represent the subconstruct with as many different facets as possible.
Factor loading: items with higher factor loadings were preferred.
Cronbach’s alpha: items with highest contribution to the scale’s overall alpha were proposed for selection.
The analyses were repeated after each deletion of items until the unidimensional structure of each subscale was stable without further improvement in the alpha coefficient. A Cronbach’s alpha of at least 0.70 was regarded sufficient and above 0.80 as good (Nunnally 1978; Streiner and Norman 2008).
Since the item pool was too large (231 items) to analyze in one PCA, we analyzed four clusters of themes that are related to each other from a theoretical point of view. This division is in line with existing models of job performance (Viswevaran and Ones 2000). Our first cluster, “cognitive aspects of work functioning”, corresponds with the idea of task performance. The second cluster, “causing incidents”, corresponds with counterproductive behavior, although we do not regard causing incidents as voluntary, which is part of the definition of counterproductive behavior. Our third cluster, “interpersonal behavior”, and fourth cluster, “energy and motivation”, are in accordance with organizational performance and the extra effort needed to perform the work, respectively. See Table 2 for the allocation of themes to the clusters.
Finally, to test whether the selected subscale structure remained stable, a confirmatory factor analysis with all remaining items from all clusters was carried out, using the Oblique Multiple Group Method (Stuive et al. 2008; Stuive et al. 2009). Based on the highest item test correlations for each item on each subscale, it can be determined for which subscale the individual items have the best fit. Possible incorrect assignments of items to subtests were corrected in this step.
All statistical analyses were performed using SPSS version 16.0, except for the Parallel Analysis, which was conducted using Monte Carlo PCA for Parallel Analysis (Watkins 2006).