Background

Mental illness refers to a wide range of mental health conditions which can range in degree of severity (mild, moderate, or severe), such as depression, anxiety disorders, and schizophrenia [1]. Mental illness related stigma manifests itself in different ways. For instance, Corrigan and Bink (2005) refer to public stigma as “the process by which individuals in the general population first endorse the stereotypes of mental illness and then act in a discriminatory manner” [2]. Internalized stigma or self-stigma refers to a process of internalizing these expressed negative stereotypes by applying them to oneself which can lead to feelings of devaluation, lower self-esteem, marginalization, shame, withdrawal, and lower self-efficacy [3,4,5]. Mental illness related stigma can have numerous negative effects, like lower intentions to help-seeking, being more socially isolated, reduced psychiatric medication adherence, higher unemployment rates, and income loss [5,6,7,8]. More validated measures are urgently needed, especially in the employment setting where stigma has shown to be an important and underestimated contributing factor to unemployment [9, 10]. For example, both stigma by employers and internalized stigma have been found to be a problem for work participation [11,12,13], and the number of scientific publications on stigma and work has doubled in the past six years [10].

In the Netherlands, The Gatekeeper Improvement Act and the Extended Payment of Income Act are protecting disabled workers. The employer, the occupational physician, and the worker are responsible for the reintegration when a worker drops out due to sickness [14]. In the first two years of sickness absence the employers pay at least 70% of the income [15]. Employers are not allowed to ask about a health issue of a worker. The Netherlands has adopted a disability quota system which is called the Job Agreement. Government, municipalities, labor unions, and employers have agreed that they must supply 125,000 jobs for disabled persons by 2026. However, the labor participation rate among Dutch people with mental illness is low (21%) [16] and Dutch workers with mental illness are vulnerable to mental illness related stigma. For example, 64% of Dutch line managers are reluctant to hire workers with mental illness [11]. Also, a majority of Dutch workers expect lower chances of contract renewal and lower chances of getting a promotion after mental illness disclosure at work [17].

To get a better understanding of self-stigmatization there is a need for brief, valid, and reliable instruments that measures internalized stigma. The understanding and standardization of scales measuring internalized stigma is beneficial because it can lead to the development of effective (therapeutic) strategies to diminish internalized stigma. Worldwide, the most commonly used scale to measure internalized stigma is the Internalized Stigma of Mental Illness (ISMI) scale [18]. The ISMI is a self-report instrument which consists of 29 items and 5 subscales, with each item rated on a 1 (strongly disagree) to 4 (strongly agree) Likert scale. There are at least 55 known versions of the ISMI, these versions from all over the world showed reliability and validity across a diverse range of cultures and languages [19]. The ISMI has not yet been validated in the Netherlands. Because of the need for a shortened version to counter survey fatigue without losing quality, the developers of the ISMI created a brief version with 10 items, i.e. the brief version of the ISMI (ISMI-10) [20]. The ISMI-10 contains two items from every subscale of the original scale. The ISMI-10 questionnaire has already been translated, and the psychometric properties have been tested in countries like the Czech Republic, Greece, and Japan, however the questionnaire is not yet translated and tested in the Netherlands [21,22,23].

Therefore, the aim of this study is to translate the ISMI-10 into a Dutch version and to assess the reliability and validity of this Dutch version in a sample of employees with mental illness. Like in the original ISMI studies, a systematic review, and several validation studies, to evaluate convergent validity, it was hypothesized that the ISMI-10 will be negatively correlated with self-esteem and empowerment, and positively correlated with anxiety and depression [18,19,20,21,22,23]. It was also hypothesized that ISMI-10 will be negatively correlated with the concept of hope because internalized stigma is known to diminish self-esteem which has an adverse effect on hope [24]. To evaluate divergent validity, following the example of Van Gestel-Timmermans et al. (2010) [25], it was hypothesized no or only small correlations will be found with (physical) health.

Methods

Study design and participants

Data were collected using the Longitudinal Internet Studies for the Social Sciences (LISS) panel. This panel, administered by research institute CentERdata, consists of a representative and random selection of Dutch panel members who participate in monthly internet surveys covering different domains like work, education, housing, income, time use, political views, values, and personalities [26]. The LISS panel is based on a true probability sample of Dutch households drawn from the population register. All panel members gave written informed consent to participate in the surveys. More information about the LISS panel can be obtained at www.lissdata.nl.

The online questionnaire was sent in September 2021 to a selection of 322 panel members who indicated (1) to have mental illness (based on self-identification) in a previous LISS-survey, and (2) to have paid work. Having a mental illness was assessed by the question: “Can you indicate whether you have a mental illness or not?”, (1) yes or (2) no. For the present study, respondents were first asked whether they were still having mental illness, otherwise, they were excluded. The retest questionnaire, to assess the test-retest reliability, was sent five weeks after the data collection of the first questionnaire closed to a random selection of 75 respondents who had completed the first questionnaire (October/November 2021). Prior to its start, the Ethics Review Board of the School of Social and Behavioral Sciences of Tilburg University gave Ethical approval for this study (registration number: RP606). The Strengthening the Reporting of Observational Studies in Epidemiology guidelines were followed during the reporting of this study, all methods were carried out in accordance with these guidelines [27].

Measures

Internalized stigma (Dutch ISMI-10)

ISMI-10

The ISMI-10 was developed to measure internalized stigma of mental illness. The ISMI-10 has 10 Likert scale items, with scores ranging from strongly disagree (1), disagree (2), agree (3), to strongly agree (4). Item 2 and item 9 are reversed scored. The original English version of the ISMI-10 showed adequate internal consistency (α = 0.75) [4]. There are three ways of interpreting the mean total scores. First, the score can be interpreted as a continuous variable, like in the original ISMI-10 paper [20]. Second, the 2-category method can be used to divide the scores into: does not report high internalized stigma (1.00-2.50) and reports high internalized stigma (2.51-4.00) [28]. Third, the 4-category method can be used to separate the scores into: minimal to no internalized stigma (1.00–2.00), mild internalized stigma (2.01–2.50), moderate internalized stigma (2.51-3.00), and severe internalized stigma (3.01-4.00) [29]. In this paper the score was interpreted the same as in the original ISMI-10 paper, as a continuous variable.

Translation and adaption of Dutch ISMI-10

The original English ISMI-10 was translated into Dutch following the forward-backward translation procedure [30]. First, the forward translation was performed by two subject matter experts (EB and MJ) with a good command of English and Dutch. They translated the English ISMI-10 scale into Dutch. This version was translated back into English by two native English speakers (who both also have a good command of Dutch). This backward translation was compared with the original English ISMI-10 based on conceptual balance. Any differences were discussed in detail between the subject matter experts (EB and MJ), native speakers, and authors (IB and MB) and necessary adjustments were made. Second, the preliminary validity was tested by conducting a pilot test with five (other) subject matter experts and five persons with mental illness to make sure that the Dutch version of the ISMI-10 was well understood in clarity and simplicity. The subject matter experts and the persons with mental illness were approached via the network of the researchers. During this procedure, only small additional changes were made to the initial translation, for example changing a word for a more relevant one. See Supplementary Material 1 for the final version of the Dutch ISMI-10.

Hope (HHI-Dutch)

The Herth Hope Index (HHI) is used for assessing the overall hope level of people with mental illness [31]. The HHI has 12 Likert scale items, ranging from strongly disagree (1) to strongly agree (4). Item 3 and item 6 were reversed-coded. The total scores range from 12 to 48, with higher scores indicating a higher level of hope. The original scale contains three factors: (1) temporality and future, (2) positive readiness and expectancy, and (3) interconnectedness with self and others. The overall reliability (α = 0.84) and validity of the Dutch version of the HHI (HHI-Dutch) was verified by Van Gestel-Timmermans et al. (2010) [25]. They advised to use the total scale scores instead of the subscales.

Anxiety and depression (HADS)

The Hospital Anxiety and Depression Scale (HADS) measures anxiety and depressive states [32]. The two subscales (anxiety and depression), each contain 7-items with Likert scores ranging from 0 to 3. The total scores of the two subscales range from 0 to 21, with higher scores indicating higher levels of anxiety and depression. The Dutch version of the HADS, used in this paper, has been validated for various groups of Dutch subjects and is stable across medical settings, and the internal consistency of the total scale for the different groups is good (α = 0.82–0.88) [33].

Self-esteem (Dutch RSES)

The Rosenberg Self-Esteem Scale (RSES) assesses global self-esteem [34]. This scale was also used in the original English ISMI-10 validation paper [20]. The RSES is designed as a unidimensional scale containing 10 Likert scale items ranging from strongly disagree (0) to strongly agree (3). Items 2, 5, 6, 8 and 9 were reversed scored. The total scores range from 0 to 30, with higher scores indicating a higher global self-esteem. The Dutch version of the RSES (Dutch RSES) proved to have good internal consistency (α = 0.86) and the validity was verified [35].

Empowerment (BUES-17)

The Boston University Empowerment Scale (BUES) measures empowerment [36]. In accordance with the original English ISMI-10 validation paper, the selection of the 17 items of factor 1 (self-esteem) and factor 2 (power) items were used (BUES-17), because the BUES-17 showed more support compared to the use of the whole scale (BUES-28) [20]. The BUES-17 contains items with scores ranging from strongly disagree (1) to strongly agree (4). Item 10 and item 17 were reverse coded. The total scores range from 0 to 64, with higher scores indicating more empowerment. Although not formally validated, the BUES-17 proved to have good internal consistency (α = 0.85) [20]. The BUES-17 was not formally tested for the Dutch context, there was one validation study that focused on the Dutch version of the original BUES with 28 items [37]. The Dutch version of the BUES-28 had a good internal consistency (α = 0.82). In the current study, the BUES-17 had a good internal consistency (α = 0.83).

Health (Dutch RAND-36)

The RAND-36 assesses health and contains eight subscales: physical functioning, social functioning, role limitations due to physical problems, role limitations due to emotional problems, emotional well-being, energy/fatigue, pain, and general health [38, 39]. The scale consists of items on 3-point to 6-point Likert scales and two subscales with 2-point Likert scales. Items 15, 17, and 19 were reversed-coded. The total scores range from 0 to 100, with a higher score indicating a more favorable health state. The internal consistency of the subscales and validity of the Dutch version of the RAND-36 (Dutch RAND-36) was verified by Van der Zee and Sanderman (2003) [40], and in accordance with Van Gestel-Timmermans et al. (2010) [8], the current study only used the 19 items of the subscales: physical functioning (α = 0.92), role limitations due to physical problems (α = 0.90), and general health (α = 0.81).

Statistical analyses

Cronbach’s alpha was used to evaluate the reliability of the Dutch ISMI-10 by testing the internal consistency of the scale (α < 0.50 – unacceptable; α = 0.60–0.69 – questionable; α = 0.70–0.79 – acceptable; α = 0.80–0.89 – good; α > 0.90 – excellent) [41]. In addition, the internal consistency of the other scales used in this study was also evaluated. The test-retest reliability of the Dutch ISMI-10 was evaluated five weeks later after the data collection of the first questionnaire closed by assessing the scale with 75 respondents using Pearson’s correlation coefficients (r) between total scores (r = 0.40–0.59 – fair; r = 0.60–0.74 – good; r > = 0.75 – excellent) [42].

The construct validity was evaluated by computing the Pearson’s correlation coefficients (r) between the Dutch ISMI-10 and the HHI-Dutch, HADS, Dutch RSES, BUES-17, and Dutch RAND-36. Convergent validity was indicated by medium to high correlations with the HHI-Dutch, HADS, Dutch RSES, and BUES-17. Divergent validity was assumed if correlations were non-significant or small between the Dutch ISMI-10 and the RAND-36 subscales physical functioning, role limitations (due to physical problems), and general health. In line with Cohen [43], correlations of 0.10 to 0.29 were interpreted as small, 0.30 to 0.49 as medium and 0.50 to 1.0 as high.

Data of five respondents were excluded from analysis due to one or more missing values on the Dutch ISMI-10 items or the previous mentioned scales. All p-values below 0.05 were considered significant. All analyses were performed using SPSS Version 24.

Results

Sample characteristics

The online questionnaire was filled out by 256 of the 322 LISS-panel members who indicated (1) to have a mental illness in a previous LISS-survey, and (2) to have paid work (response rate = 79.5%). Of this group, 166 respondents belonged to the target group (employees who were still having mental illness) and 161 of them completed the questionnaire. The employees (60.2% female) had a mean age of 40.94 (SD = 12.14) years and had mostly a secondary (41.6%) or higher (45.3%) educational level. Other sample characteristics can be found in Table 1. The respondents mean internalized stigma score was 2.05 (SD = 0.44) and ranged from 1.00 to 3.20.

Table 1 Sociodemographic characteristics of the sample (N = 161)
Full size table

Reliability analysis

Internal consistency

The Dutch ISMI-10 had a good internal reliability of α = 0.83. Table 2 shows the Cronbach’s alpha of the overall scale of the Dutch ISMI-10 compared to the overall scales of the short original English version (ISMI-10) and the original English version (ISMI-29) [18, 20]. In this present study, this were the Cronbach’s alpha coefficients for the other used scales: HHI-Dutch (α = 0.81), HADS (α = 0.90), Dutch RSES (α = 0.89), BUES-17 (α = 0.83) and RAND-36 subscales (physical functioning: α = 0.92; role limitations – physical problems: α = 0.83; general health: α = 0.80).

Table 2 Cronbach’s alpha coefficients in relation to the internal consistency of the Dutch ISMI-10, original English ISMI-10, and original English ISMI-29
Full size table

Test-retest reliability

The test-retest study involved 63 of 75 respondents (response rate = 84.0%) who completed the questionnaire after 37 to 82 days (average 56.2 days). No significant differences were found on sociodemographic characteristics between the retest sample compared to the total sample. The test-retest study showed good test-retest reliability of r = 0.73 (p < 0.01).

Construct validity

Convergent validity

Convergent validity was assessed by computing the correlating between the Dutch ISMI-10 sum scores and the HHI-Dutch, HADS, Dutch RSES, and BUES-17. Table 3 shows the correlations. High significant correlations were found between the Dutch ISMI-10 and HHI-Dutch, HADS, Dutch RSES, and BUES-17 (p < 0.01).

Table 3 Pearson correlations of the Dutch ISMI-10 with HHI-Dutch, HADS, Dutch RSES, and BUES-17 to assess convergent validity
Full size table

Divergent validity

Divergent validity was evaluated by correlating Dutch ISMI-10 sum scores with the RAND-36 subscales physical functioning, role limitations due to physical problems, and general health. Table 4 shows that small significant correlations were found between the Dutch ISMI-10 and the physical functioning subscale and the role limitation subscale (p < 0.01). Medium significant correlations were found between the Dutch ISMI-10 and the general health subscale (p < 0.01).

Table 4 Pearson correlations of the Dutch ISMI-10 with three RAND-36 subscales to assess divergent validity
Full size table

Discussion

The aim of this current study was to translate the ISMI-10 into a Dutch version and to assess the reliability and validity in a sample of employees with mental illness. A Dutch version of the ISMI-10 was translated using the forward-backward translation procedure [30]. The Dutch ISMI-10 showed good internal consistency and good test-retest reliability. The Dutch ISMI-10 showed excellent convergent validity and acceptable divergent validity.

Analysis of the reliability of the Dutch ISMI-10 supported the reliability of the questionnaire. The internal reliability of the Dutch ISMI-10 was better compared to the original English ISMI-10, but slightly lower than the original English ISMI-29. One explanation for this difference is that the alpha is mostly higher in scales with more items [44]. This was also seen in the Czech and original English ISMI-10 papers [20, 21]. The Dutch ISMI-10 showed similar test-retest reliability compared to the results from the Japanese ISMI-10, which is the only version that studied test-retest reliability [23].

As expected, we found high negative correlations between the Dutch ISMI-10 and hope (HHI-Dutch), self-esteem (Dutch RSES), and empowerment (BUES-17), and positive correlations with anxiety and depression (HADS). These convergent validity findings are in line with the findings of the papers concerning the original English ISMI-29, the original English ISMI-10, and the Japanese version of the ISMI-10 using self-esteem, empowerment, and depression measures [18, 20, 23]. Additional to other ISMI-10 papers, this current study also used an extra measure, a measure for hope, to test convergent validity thoroughly.

Analysis of the divergent validity showed that, as expected, small correlations were found between the Dutch ISMI-10 and the RAND-36 subscales physical functioning and role limitations due to physical problems, but unexpected medium correlations were found with the general health subscale. The medium correlation with the general health subscale may be explained by the fact that general health consists of both a physical and a mental element. Especially in a sample of respondents with mental illness, this mental element might be experienced as more prevailing [25]. For future psychometric research on the ISMI-10 and divergent validity, this subscale may be less suitable in respondents with mental illness.

In contrast to several ISMI-29 studies [18, 45, 46], this study did not include a factor analysis to investigate the factor structure of the scale, because the ISMI-10 is meant to be an unidimensional scale [20]. Although the ISMI-10 includes all of the five dimensions in two-item subscales, the developers of the original English ISMI-10 did not intend to make five subscales. The developer mentioned that two-item scales are likely to be unstable [20]. Other research showed that for multidimensional scales, scales need at least four items to use Confirmative Factor Analysis for validation [47]. A later study on the factor structure of different versions of the ISMI confirmed the recommendation of the developers and underlined the one-dimensionality of the ISMI-10 [48]. Thus, we interpreted the ISMI-10 as a unidimensional scale.

Strengths and limitations

A strength of this study, compared to the original ISMI-10 paper, was the test-retest design to describe the stability of the Dutch ISMI-10. Another strength was the study sample, which consisted of a selection of employees with mental illness from a representative panel of the Dutch population. Some of the psychometric validation studies of the ISMI have used more specific samples, such as the original ISMI-10, which included a male military veterans sample who received Veterans Affairs services [20]. However, the current sample did not include unemployed people with mental illness, which may be a limitation because this group has been found to have higher levels of self-stigma compared to their employed counterparts [49]. Other ISMI-10 validation papers from Greece, Japan, and Czech Republic focused on mental illness outpatients in general [21,22,23]. The overall Dutch ISMI-10 mean score (2.05, SD = 0.44) is indeed slightly lower than the reported overall ISMI-10 mean scores of the original ISMI-10 paper (2.32, SD = 0.39) and the Czech ISMI-10 paper (2.16, SD = 0.54) [20, 21]. Another limitation of this study was the use of self-report data to determine if the respondent experienced mental illness at the time of the data collection, and the use of data that did not include more background information on participants’ diagnoses, symptom severity or participants’ working situations. An additional limitation was the use of the BUES-17 scale for assessing convergent validity. This shorter version of the BUES-28 scale was not formally tested for the Dutch context. Nevertheless, the translated Dutch version of the BUES-17 scale proved to have good internal consistency in this study. Additionally, the HADS-scale is originally developed for medical out-patients, in this study the HADS-scale was used for workers following the example of other studies [50,51,52]. To conclude, a limitation could be the use of simple correlations to check convergent and divergent validity. A more rigorous analytical procedure could have been used, e.g. examining latent construct correlations [53].

Conclusion

In conclusion, the results of this study revealed that the Dutch ISMI-10 demonstrated adequate psychometric properties. The Dutch ISMI-10 showed good internal consistency, good test-rest reliability, excellent convergent validity, and acceptable divergent validity. The ISMI-10 had already been shown to be a good and brief alternative for the original ISMI-29 [20]. The understanding and standardization of scales measuring internalized stigma like the Dutch ISMI-10 can be used by researchers in Dutch speaking countries to get a better understanding of self-stigmatization among people with mental illness. The Dutch ISMI-10 is valuable for the development of effective (therapeutic) strategies to diminish internalized stigma.