Psychometric Testing of Three COVID-19-Related Scales Among People with Mental Illness

Abstract

Fear of novel coronavirus 2019 (COVID-19) may result in psychological health problems among different populations. Moreover, believing COVID-19 information and preventive COVID-19 infection behaviors are relevant constructs associated with fear of COVID-19. Therefore, the present study validated three instruments assessing fear, beliefs, and preventive behaviors related to COVID-19 among individuals with mental illness. Moreover, relationships between the three constructs were examined. Individuals with mental illness (N = 400; 178 females; mean age = 46.91 years) completed the Fear of COVID-19 Scale (FCV-19S), Believing COVID-19 Information Scale (BCIS), Preventive COVID-19 Infection Behaviors Scale (PCIBS), and Depression Anxiety Stress Scale-21 (DASS-21). The FCV-19S, BCIS, and PCIBS demonstrated a single-factor structure with satisfactory fit indices. Moreover, believing COVID-19 information positively and significantly associated with fear of COVID-19, and fear of COVID-19 negatively and significantly associated with preventive behaviors and positively and significantly associated with psychological distress. The FCV-19S, BCIS, and PCIBS may assist healthcare providers in assessing COVID-19-related information among individuals with mental illness. Consequently, relevant programs may be designed to help individuals with mental illness going through the period of crisis.

With the increasing novel coronavirus desease 2019 (COVID-19) infection and climbing mortality due to COVID-19 worldwide (World Health Organization 2020a), fear of COVID has been reported, especially in the countries with a great number of confirmed cases. For example, an Iranian study reported that 717 Iranians recruited from general population reported their fear as being 3.9 out of 5 utilizing a 5-point Likert scale (Ahorsu et al. 2020). More recently, a Bangladeshi study found that 8550 Bangladeshi individuals from general population reported their fear as 3.1 out of 5 (Sakib et al. 2020) using the same instrument as that in Ahorsu et al.’s (2020) study. The higher levels of fear found in Iran may be due to COVID-19 situation being more serious in Iran: as of April 26, 2020, Iran was the seventh leading country in total reported cases of COVID-19 with a total of 89,328 confirmed cases and 5650 deaths compared to 5142 cases and 145 deaths in Bangladesh (Worldometer 2020). Although a lower level of fear was found among Bangladeshi individuals, some extreme cases with great fear of COVID-19 have been observed. For instance, a 36-year-old Bangladeshi man committed suicide because of suspected COVID-19 infection that he did not actually have (Mamun and Griffiths 2020). A similar situation was described in India where a 50-year-old father with three children committed suicide because he had COVID-19-like symptoms but again did not actually have the virus (Goyal et al. 2020). Therefore, a validated instrument to assess fear of COVID-19 among different populations at the present time is of great utility.

To the best of the present authors’ knowledge, only one specific instrument has been developed to assess fear levels toward COVID-19, i.e., the Fear of COVID-19 Scale (FCV-19S), at the time of writing this paper. The FCV-19S was developed by Ahorsu et al. (2020) who carried out testing using exploratory factor analysis and demonstrated the scale had very good psychometric properties. Sakib et al. (2020) translated the FCV-19S into Bangla and verified its robust psychometric properties using confirmatory factor analysis (CFA). Moreover, Soraci et al. (2020) and Satici et al. (2020) translated the FCV-19S into Italian and Turkish respectively and found that these translated versions had satisfactory psychometric properties. Apart from the four aforementioned studies (Ahorsu et al. 2020; Sakib et al. 2020; Satici et al. 2020; Soraci et al. 2020), no other psychometric information concerning the FCV-19S has been reported at the present time. Moreover, the four FCV-19S psychometric testing studies all focused on general populations (Ahorsu et al. 2020; Sakib et al. 2020; Satici et al. 2020; Soraci et al. 2020) and it remains unclear whether the FCV-19S can be applied to non-normative populations (e.g., individuals with mental illness). As a robust instrument needs to have satisfactory psychometric properties across different populations (Lin et al. 2017, 2019a, b) to accumulate its scientific rigor, the present authors believe that validating the FCV-19S among individuals with mental illness is therefore essential. More specifically, individuals with mental illness are more vulnerable than general population with regard to their psychological health (Chang et al. 2016; Su et al. 2014).

In addition to the issue of fear of COVID-19, COVID-19 information and preventive COVID-19 infection behaviors are also of great concern. Moreover, fear of COVID-19, COVID-19 information, and preventive COVID-19 infection behaviors appear to be interrelated. Fear may blind an individuals’ logical thinking and facilitate their unreasonable/maladaptive behaviors (Lin 2020; Pakpour and Griffiths 2020; Ren et al. 2020) such as the two aforementioned suicide cases (Goyal et al. 2020; Mamun and Griffiths 2020). However, an underlying reason for individuals to engage in extreme behavior (e.g., suicide) may be the misinformation they receive. More specifically, individuals are keen to obtain information about dangerous health conditions from different sources, especially social media, during outbreak periods (Shin et al. 2016; Wang et al. 2020a, b). Unfortunately, some information reported in social media is incorrect. A recent study utilizing the general public in two countries (UK and the USA) found that some misconceptions and falsehoods are believed and circulated on social media (Geldsetzer 2020). Moreover, even accurate information reported on the social media can be frightening (e.g., the rapid growth of death cases). Therefore, misconceptions and misbeliefs concerning COVID-19, or even accurate COVID-19 information, may trigger and amplify individuals’ fear, subsequently leading to psychological distress and inappropriate behaviors. Consequently, the present authors hypothesized that individuals who have a greater level of fear concerning COVID-19 will be more prone to engaging in inappropriate behaviors rather than engaging in preventive COVID-19 infection behaviors.

Although beliefs about COVID-19 information and preventive COVID-19 infection behaviors appear to be important factors underlying the fear of COVID-19, no validated instruments have been developed to assess either beliefs concerning COVID-19 information (especially information obtained from social media) or preventive COVID-19 infection behaviors. Therefore, the present study also developed and validated instruments assessing beliefs in COVID-19 information from social media and preventive COVID-19 infection behaviors for future studies’ use. More specifically, commonly used social media (as well as other online and offline media) were used in the present study to develop an instrument to assess an individual’s belief in COVID-19 information. The preventive COVID-19 infection behaviors proposed by the World Health Organization (World Health Organization 2020b) were used to construct an instrument to assess an individual’s preventive behaviors. With the development and validation of the two instruments, healthcare providers and researchers would be able quantify the beliefs and preventive behaviors of an individuals’ psychological and behavioral reactions to COVID-19.

In order to better capture the impacts of COVID-19 among individuals with mental illness, the present study validated three COVID-19-related instruments, i.e., the FCV-19S, the Believing COVID-19 Information Scale (BCIS), and the Preventive COVID-19 Infection Behaviors Scale (PCIBS). All three instruments were hypothesized to be unidimensional and their single-factor structures were tested using CFA. Item properties and internal consistency of the three instruments were also examined. In addition to the psychometric properties, the present study proposed a preliminary path model to examine the relationships between fear of COVID-19, beliefs in COVID-19 information from social media, preventive COVID-19 infection behaviors, and psychological distress. In the preliminary path model, several hypotheses were proposed: (i) believing COVID-19 information from social media will lead to greater fear of COVID-19; (ii) fear of COVID-19 will lead to fewer preventive COVID-19 infection behaviors; and (iii) fear of COVID-19 will lead to greater psychological distress, including depression, anxiety, and stress.

Methods

Participants and Recruitment Procedure

The present cross-sectional study was conducted after many hospitals and clinics in Taiwan had already tightened their regulations to avoid unnecessary contact between medical personnel and patients as well as outside visitors because of the COVID-19 pandemic. It was approved by the Institute of Review Board (IRB) of the Jianan Psychiatric Center (JPC), Ministry of Health and Welfare (IRB numbers: 20-004). To facilitate protection of privacy for patients with mental illness, the participants were recruited at their most convenient locations, namely, the JPC. The JPC is the sole psychiatric teaching center in a city of southern Taiwan (i.e., Tainan). The JPC has a psychiatric treatment network in southern Taiwan and therefore provides a service to the entire southern Taiwan region with more than three million residents. Thousands of individuals with mental illness receive different program services from the JPC, including inpatient rehabilitation programs, daycare programs, and outpatient services. At the start of the present study’s recruitment, the agency implemented dynamic measures including fever screening, health declarations, contact, and travel history check of both patients and medical personnel before entry into the JPC. Also, visits to patients, including relatives, being hospitalized even for psychiatric rehabilitation were banned in order to prevent a cluster outbreak of COVID-19. All eligible participants were recruited from the daycare, outpatient units, and inpatient rehabilitation programs between March 23 and April 23, 2020. The inclusion criteria included the following: (i) at least one diagnosis of mental disorder by at least one psychiatrist using the DSM-IV (fourth edition of Diagnostic and Statistical Manual of Mental Disorders) criteria; (ii) had sufficient mental capacity to provide consent and complete the assessments; (iii) admission to the daycare or inpatient rehabilitation programs or regular follow-up during outpatient treatment, indicating that the patient had a relatively stable condition (e.g., psychiatric symptoms were residual or non-active); and (iv) being aged over 20 years. The exclusion criteria were having (i) a severe and unstable medical disease or a history of neurological disease and (ii) a history of head injury. The total number of participants that took part in the study was 400 (178 females).

Measures

Fear of COVID-19 Scale

The FCV-19S includes seven items with a 5-point Likert scale (1 = strongly disagree; 5 = strongly agree) to assess how an individual fears COVID-19. A higher score of the FCV-19S represents a greater level of fear toward COVID-19. An example item of the FCV-19S is “I cannot sleep because I’m worrying about getting coronavirus-19.” The satisfactory psychometric properties of the FCV-19S have been demonstrated in a general Iranian population (e.g., Cronbach’s α = 0.88; Ahorsu et al. 2020). A higher score indicates a greater fear of COVID-19. The translation of FCV-19S was performed following cognitive interviews. More specifically, the translated FCV-19S was modified after several experts (including a psychiatrist, a public health expert, and an orthopedist) reviewed it with comments. Therefore, the scale’s linguistic validity was ensured.

Believing COVID-19 Information Scale

The BCIS comprises six items responded on a 5-point Likert scale (1 = strongly disbelieve; 5 = strongly believe) that assesses how individuals believe the COVID-19 information they obtain. A higher score of the BCIS indicates a greater level of believing in the obtained COVID-19 information. The BCIS items share the same item stem of “How much do you believe in the COVID-19 information on/in…” with different media sources added to the item stems. The six sources (with a much heavier emphasis on social media sources) were LINE chat room, LINE news page, Facebook, online news, television, and traditional newspaper. LINE and Facebook were used in the BCIS because they are the most frequently used online social media among Taiwanese (NapoleanCat 2018; Statista 2018).

Preventive COVID-19 Infection Behaviors Scale (PCIBS)

The PCIBS was developed according to the preventive behaviors recommended by the World Health Organization Q&A on coronaviruses (COVID-19) (2020b). The WHO advises individuals worldwide to engage in these behaviors to avoid COVID-19 infection. The PCIBS comprises five items responded to on a 5-point Likert scale (1 = almost never; 5 = almost always) and assesses how individuals perform preventive COVID-19 infection behaviors. A higher score of the PCIBS indicates performing preventive behaviors more frequently. An example item of the PCIBS is “How often do you avoid touching eyes, nose, and mouth.”

Depression Anxiety Stress Scale-21 (DASS-21)

The DASS-21 comprises 21 items that are responded to on a 4-point Likert scale (0 = never; 3 = almost always) and assesses three types of psychological distress among individuals (depression, seven items; anxiety, seven items; and stress, seven items). Higher scores on each subscale of the DASS-21 indicate a greater level of depression, anxiety, or stress. Example items in the DASS-21 include “I felt that I had nothing to look forward to” (depression), “I felt that I was close to panic” (anxiety), and “I tended to overreact to situations” (stress). Satisfactory psychometric properties of the DASS-21 have been demonstrated in different populations, including individuals with mental illness (e.g., Cronbach’s α = 0.81 to 0.85) (Lee et al. 2019). The internal consistency of the DASS-21 subscales in the present study was satisfactory: α = 0.90 (depression subscale), 0.85 (anxiety subscale), and 0.89 (stress subscale).

Background information

The participants were also asked to provide information concerning their age, gender, educational year, and where they obtained COVID-19 information (with selections of LINE chat room, LINE news page, Facebook, online news, television, traditional newspaper, and healthcare providers). Participants’ mental illness diagnoses were retrieved from their medical records.

Data Analysis

Background information of the participants was analyzed using descriptive statistics, including means, SDs, frequencies, and percentages. Internal consistency of the three COVID-19 instruments (i.e., FCV-19S, BCIS, and PCIBS) was examined using Cronbach’s α. Item properties of the three COVID-19 instruments were checked by skewness, kurtosis, percentage of missing responses, floor effect, ceiling effect, corrected item-total correlation, and factor loading. Factor loadings of the instruments’ items were retrieved from the CFA for each instrument.

The three COVID-19 instruments were all unidimensional. Consequently, a single-factor structure was tested for each instrument when performing the CFAs. Diagonally weighted least square estimation was used because all the instruments adopted a Likert-type scale for item responses. Fit indices of comparative fit index (CFI > 0.9 indicates acceptable), Tucker-Lewis index (TLI > 0.9 indicates acceptable), root mean square error of approximation (RMSEA < 0.08 indicates acceptable), and standardized root mean square residual (SRMR < 0.08 indicates acceptable) were used to examine whether the single-factor structure of each instrument was supported (Lin et al. 2019a, b; Yam et al. 2019).

Apart from the CFA, a hypothesized path model was investigated using structural equation modeling (SEM). The hypothesized path model proposed that believing in COVID-19 information from social media was likely to increase an individual’s fear of COVID-19 because prior research reports misconceptions and misinformation among social media (Geldsetzer 2020). Moreover, fear of COVID-19 was hypothesized to lead to decreased preventive behavior and increased psychological distress because prior research reports that fear may cause inappropriate behaviors (Lin 2020; Ren et al. 2020) and psychological distress (Wang et al. 2020a, b; Xiao et al. 2020). In the hypothesized path model, no measurement structure was used to satisfy the principal of parsimony. Therefore, total scores of BCIS, FCV-19S, and PCIBS and subscale scores of depression, anxiety, and stress in the DASS-21 were used as observed variables in the path model via SEM. Age, gender, and educational year were controlled for in the model. All the statistical analyses were performed using R software, including the R packages of lavaan (Rosseel 2012) and psych (Revelle 2019). More specifically, the lavaan package was used for CFA and SEM.

Results

The mean age of the participants was 46.91 years (SD = 10.92) (Table 1). Slightly less than half of the participants (n = 178; 44.5%) were females and the majority diagnosis for the participants was schizophrenia (n = 242; 60.5%). Most of the participants obtained information about COVID-19 via television (n = 347; 86.8%).

Table 1 Participants’ characteristics (N = 400)

The three COVID-19 instruments had very good to excellent internal consistency (α = 0.93 for FCV-19S, α = 0.91 for BCIS, and α = 0.82 for PCIBS). Table 2 presents the item and scale properties of the FCV-19S, BCIS, and PCIBS. All the item and total scores were nearly normal distributed: FCV-19S (skewness = − 0.14 to 0.47; kurtosis = − 1.19 to − 0.63), BCIS (skewness = − 0.61 to − 0.06; kurtosis = 0.43 to 1.36), and PCIBS (skewness = − 0.84 to − 0.31; kurtosis = − 0.92 to 0.08). Almost no missing values were found in the three instruments. The floor and ceiling effects were acceptable for all three COVID-19 instruments: FCV-19S (floor effect = 12.0 to 27.8%; ceiling effect = 1.3 to 14.3%), the BCIS (floor effect = 1.8 to 7.8%; ceiling effect = 4.3 to 16.0%), and the PCIBS (floor effect = 2.3 to 13.0%; ceiling effect = 10.0 to 31.8%). The corrected item-total correlations for all items were strong: (FCV-19S = 0.69 to 0.85; BCIS = 0.71 to 0.83; PCIBS = 0.66 to 0.70) and factor loadings (FCV-19S = 0.68 to 0.84; BCIS = 0.69 to 0.83; PCIBS = 0.65 to 0.71).

Table 2 Item and scale properties of the Fear of COVID-19 Scale (FCV-19S), Believing COVID-19 Information Scale (BCIS), and Preventive COVID-19 Infection Behaviors Scale (PCIBS)

In addition to the strong loadings, all the fit indices support the uni-dimensionality of all three COVID-19 instruments: FCV-19S (CFI = 0.994, TLI = 0.992, RMSEA = 0.056, and SRMR = 0.057), BCIS (CFI = 0.994, TLI = 0.989, RMSEA = 0.042, and SRMR = 0.064), and PCIBS (CFI = 1.000, TLI = 1.006, RMSEA = 0.000, and SRMR = 0.027). Therefore, the single-factor structure for all three instruments was confirmed (Table 3). After ensuring that the psychometric properties of the FCV-19S, BCIS, and PCIBS were satisfactory, the hypothesized path model examined the relationships between believing in COVID-19 information, fear of COVID, preventive behaviors, and psychological distress (Fig. 1). The hypothesized directions were supported in the path model. Believing COVID-19 information positively and significantly associated with fear of COVID-19 (standardized coefficient [β] = 0.28, p < 0.001), and fear of COVID-19 negatively and significantly associated with preventive behavior (β = − 0.11, p = 0.03) and positively and significantly associated with psychological distress (β = 0.31 for depression; = 0.33 for anxiety; = 0.29 for stress; p values < 0.001).

Table 3 Confirmatory factor analysis fit indices of the Fear of COVID-19 Scale (FCV-19S), Believing COVID-19 Information Scale (BCIS), and Preventive COVID-19 Infection Behaviors Scale (PCIBS)

Fig. 1

Hypothesized paths in associations between believing COVID-19 information, fear of COVID-19, preventive COVID-19 infection behavior, and psychological distress. p values of all path coefficients were < 0.001, except for that of fear of COVID-19 to preventive behavior (p = 0.03). Age, gender, and educational year were controlled in the path model

Discussion

The present study found that the three newly developed COVID-19-related scales (i.e., FCV-19S, BCIS, and PCIBS) had satisfactory psychometric properties among a sample of individuals with mental illness. Their single-factor structures were confirmed by the CFAs and their item properties were promising in terms of the normal distribution, strong corrected item-total correlations, high factor loadings, and acceptable ceiling and floor effects. Moreover, the results of preliminary path model fully supported the present study’s hypotheses. More specifically, the model showed that among the present sample, (i) beliefs in COVID-19 information from social media had significant positive correlation with fear of COVID-19, and that (ii) the fear of COVID-19 was significantly associated with fewer preventive COVID-19 infection behaviors and greater psychological distress, including depression, anxiety, and stress.

Compared to previous studies psychometrically testing the FCV-19S (Ahorsu et al. 2020; Sakib et al. 2020; Satici et al. 2020; Soraci et al. 2020) the present findings demonstrate similar results, especially the single-factor structure of the FCV-19S. The participants’ characteristics were different between these studies. The present study recruited individuals with mental illness (mean age = 46.91 years [SD = 10.92]) in Taiwan, whereas Ahorsu et al. (2020) studied the Iranian general population (mean age = 31.25 years [SD = 12.68]), Sakib et al. (2020) studied the Bangla general population with a relative younger age (mean age = 26.53 years [SD = 9.09]), Soraci et al. (2020) studied the Italian general population predominantly females (mean age = 34.5 years [SD = 12.21]), and Satici et al. (2020) studied the Turkish general population also predominantly females (mean age = 29.47 years [SD = 10.54]). Therefore, the FCV-19S can be tentatively concluded to be validated across populations with different features and FCV-19S may be used for cross-cultural comparisons. However, it is unclear whether measurement invariance across cultures is supported for FCV-19S. Additional evidence is needed for this psychometric property.

In general, the level of fear of COVID-19 was relatively low in the present study’s sample (mean = 2.63 out of 5) as compared with Iranian (mean = 3.9) and Bangladeshi (mean = 3.1) samples. The relatively low score in FCV-19S for the present study’s sample may be due to Taiwan’s successful policy concerning COVID-19 infection control (Wang et al. 2020a, b). More specifically, the Taiwan government implemented their infection control policy before the World Health Organization recognized the worldwide danger of COVID-19. For example, by January 5, 2020, any individuals who had traveled to Wuhan in the previous 14 days were screened to see if they had symptoms of an upper respiratory tract infection (e.g., coughing) or fever at the point of entry. Quarantined policy was also implemented and applied to the travelers displaying fever or coughing symptoms (Wang et al. 2020a, b). Later on, when the World Health Organization acknowledged the seriousness of the COVID-19, the Taiwan National Health Command Center, an institution established after the SARS outbreak (Wang et al. 2020a, b), collaborated with various ministries (e.g., the ministries of transportation, economics, labor, and education) to counteract the global public health crisis in a comprehensive effort. Therefore, Taiwan is one of the few countries that did not officially cancel schooling or shut down business and had relatively few confirmed cases (cumulative total 428 cases as of April 24, 2020) (Taiwan Centers for Diseases Control 2020). Therefore, the present participants, even though they were a vulnerable population, might have felt safer during the COVID-19 pandemic period (compared to other countries) and subsequently reported relatively low levels of fear.

The hypotheses tested in the preliminary path model were fully supported by the SEM results. The relationship between fear of COVID-19 and psychological distress is aligned with prior findings (Ahorsu et al. 2020; Sakib et al. 2020; Satici et al. 2020; Soraci et al. 2020). However, to the best of the present authors’ knowledge, the present study is the first to identify the relationship between believing more in COVID-19 information from social media, greater fear of COVID-19, and fewer preventive COVID-19 infection behaviors. Therefore, the present study provided empirical evidence to support prior claims that COVID-19 information irrespective of whether it was correct or incorrect information may have negative effects on an individual’s mood and emotions (Ahorsu et al. 2020; Lin 2020). With a stronger belief in COVID-19 information, an individual may have increased fears and tend to use inappropriate behaviors rather than positive behaviors (i.e., preventive COVID-19 infection behaviors) to cope with their fear. However, the beliefs in COVID-19 information assessed in the present study were not solely based on misinformation (because the present study assessed to what extent an individual believes in COVID-19 information irrespective of whether it is correct or incorrect). Future studies should therefore design specific instruments assessing beliefs in COVID-19 misinformation and beliefs in COVID-19 correct information separately to distinguish the effects of believing in COVID-19 information.

There are some limitations in the present study. First, both the BCIS and PCIBS were newly developed adopting a relatively simple method (e.g., without a thorough literature review to identify all potential items). Therefore, the BCIS and PCIBS may not capture the entire concept of beliefs and preventive behaviors related to COVID-19. However, with the urgent need to assess beliefs and prevent behaviors related to COVID-19, the present authors believe that both BCIS and PCIBS have promising psychometric properties that are suitable instruments at the present time. Nevertheless, future studies using rigorous instrument development methods are needed to strengthen the BCIS and PCIBS. Second, test-retest reliability, an important psychometric property for instruments, was not examined for the FCV-19S, BCIS, and PCIBS. Therefore, further psychometric testing on the test-retest reliability of the three instruments is needed. Third, the instruments used to construct the hypothesized path model all relied on self-report data. Therefore, single-rater bias and memory recall bias may present in the model tested. Finally, the present study utilized a cross-sectional design so no causal relationships can be determined in the preliminary path model. Future longitudinal studies are warranted to verify the causality between the variables examined here.

Conclusion

The present study demonstrated satisfactory psychometric properties in three new instruments assessing constructs related to COVID-19. More specifically, the FCV-19S, the BCIS, and the PCIBS were found to have a single-factor structure with high internal consistency. Moreover, a path model was constructed and showed a positive association between believing in COVID-19 information and fear of COVID-19, a negative association between fear of COVID-19 and preventive COVID-19 infection behaviors, and a positive association between fear of COVID-19 and psychological distress.

Appendix

Believing COVID-19 Information Scale	Strongly disbelieve	Disbelieve	Neither believe nor disbelieve	Believe	Strongly believe
How much do you believe in COVID-19 information in LINE chat rooms?
How much do you believe in COVID-19 information in LINE news pages?
How much do you believe in COVID-19 information on Facebook?
How much do you believe in COVID-19 information in online news?
How much do you believe in COVID-19 information on television?
How much do you believe in COVID-19 information in traditional newspapers?
Preventive COVID-19 Infection Behaviors Scale	Almost never	Rarely	Sometimes	Often	Almost always
How often do you regularly and thoroughly clean your hands with an alcohol-based hand rub or wash them with soap and water?
How often do you avoid touching eyes, nose, and mouth?
How often do you cover your mouth and nose with your bent elbow or tissue when you cough or sneeze?
How often do you maintain at least 1-meter distance between yourself and others?
How often do you stay home when you feel unwell?