Discomfort with emotion has gained increasing scientific attention in efforts to better apply clinical interventions (Sass et al., 2013). Researchers have focused on the duality of emotions, known as “meta-emotion,” which suggests that emotions can be experienced in two distinct ways. Individuals experiencing greater discomfort with emotion may have less benefit from psychotherapeutic interventions, as their discomfort with emotion can lead to reduced willing to engage their emotional self-awareness. It is important to consider clients’ levels of discomfort with emotion before applying an intervention with an emotion-based approach such as acceptance-based commitment therapy or mindfulness-based intervention, where emotional regulation ability is a key factor for the treatment mechanism (Momeni et al., 2016; Takebe & Sato, 2023).

Among various types of discomfort with emotions, discomfort with anger (or anger discomfort) refers to fear, worry, or anxiety about anger and attempts to control uncomfortable feelings about the anger experience (Sass et al., 2013; Sharkin & Gelso, 1991; Williams et al., 1997). Sharkin and Gelso (1991) investigated the degree to which people feel uncomfortable with their own anger. Those who feel anger discomfort will tend to “keep it in” and suppress its expression. Anger discomfort is an inner, subjective experience with intrapsychic and interpersonal elements. Typical items on the anger discomfort scale capture elements such as “I am troubled by my anger” and “I fear that my anger will hurt other people,” highlighting the coexistence of uncomfortable feelings with anger experiences and expressions, often accompanied by emotions such as anxiety and guilt (Sharkin & Gelso, 1991).

To date, a scale for measuring anger discomfort has not been developed or validated in South Korean culture. South Korea has a unique culture regarding anger. The connection between repressing anger and Confucianism is noteworthy, as Confucianism, which permeates Far Eastern culture, emphasizes harmony through self-control and the cultivation of virtuous behavior. Because South Koreans value interpersonal harmony, they tend to suppress their anger and not express it. Han (한) is a unique Korean word for rage and regret, which refers to anger or feelings of unfairness that are suppressed and accumulate after exposure to stressful life events. Koreans tend to suppress their anger and develop these feelings of Han, which result in an illness such as Hwabyung (화병), a culture-related somatic anger syndrome in Korea (Min et al., 1986). Since anger suppression is associated with anger discomfort, Koreans’ anger experiences like Han or Hwabyung might be related to anger discomfort. Because of the uniqueness of South Koreans’ anger experiences, a reliable and valid measurement to assess anger in interpersonal contexts is crucial to identify Korean individuals with anger regulation issues and to provide interventions in a timely manner.

The Anger Discomfort Scale

The Anger Discomfort Scale (ADS; Sharkin & Gelso, 1991) is a 15-item self-report measure developed in the USA, assessing the affective component of anger experience and anger expression. Anger discomfort is conceptualized as an internal experience based on one’s perception and attributions. The authors (Sharkin & Gelso, 1991) identified four factors: (1) intrapersonal discomfort (fear of one’s own anger); (2) positive views of and comfort with anger; (3) interpersonal discomfort with one’s own anger (fear of others’ reactions to one’s anger); and (4) outcomes associated with being angry. Individuals scoring high on the ADS may experience elevated levels of internalized anger and concerns about expressing their anger in social contexts. Such individuals, uncomfortable with expressing their anger in social interactions, often withhold their anger, potentially impacting their mental health and social relationships social relationships negatively. Originally, the ADS was originally developed to measure a counselor’s or psychotherapist’s own discomfort about anger when meeting their clients and how that discomfort impedes therapeutic processes and outcomes. A tool like the ADS can also be particularly useful in screening participants for readiness and motivation to enter anger treatment programs.

Evidence of the validity of the ADS was also provided through associations found with other key variables: trait anxiety, anger suppression, anger expression, and anger control. Anger discomfort is associated with trait anxiety because it connotes an uncomfortable feeling of anger experience (Sharkin & Gelso, 1991). Researchers also reported the shared features of anger discomfort and three different anger regulation strategies (i.e., anger suppression, expression, and control; Sharkin & Gelso, 1991). Participants experiencing high levels of anger discomfort are more likely to suppress their anger (Sharkin & Gelso, 1991). On the other hand, inappropriate anger expression may induce anxiety, which in turn leads to participants experiencing anger discomfort. Furthermore, participants with high levels of anger discomfort may experience difficulty controlling their anger because they tend to suppress or deny their uncomfortable emotions, making it difficult for them to effectively control their anger (Sharkin & Gelso, 1991).

Previous studies investigated cross-cultural differences in anger discomfort levels and relationships between anger discomfort and mental health disorders using the ADS. For example, in an examination by Edman and Yates (2004), notable differences in anger discomfort levels emerged among ethnic groups, with Chamorro female college students showing significantly higher levels of anger discomfort (M = 2.34) compared to their Caucasian counterparts (M = 1.97). Anger discomfort was positively related to all three Eating Disorder Inventory (EDI) sub-scales for both ethnic groups, including: (1) drive for thinness (r = 0.34 for Chamorros; r = 0.34 for Caucasians); (2) feelings of worthlessness/lack of control over achievements (r = 0.33 for Chamorros; r = 0.51 for Caucasians); and (3) general level of difficulty and confusion in responding to emotional states (r = 0.54 for Chamorros; r = 0.45 for Caucasians). Furthermore, Edman et al. (2005) expanded upon those findings by exploring the correlation between overall ADS levels and specific aspects of EDI sub-scales for both genders. Edman et al. (2005) also used the ADS as a single factor. They found that the overall ADS level was positively related to all two EDI subscales for both gender groups, including: (1) drive for thinness (r = 0.48 for males; r = 0.40 for females) and (2) interoceptive awareness (r = 0.40 for males; r = 0.49 for females). Edman and Yates (2005) found that anger discomfort was positively associated with body dissatisfaction (r = 0.27 for Caucasian females; r = 0.061 for Filipino females) and drive for thinness (r = 0.42 for Caucasian females; r = 0.27 for Filipino females). Moreover, Laye-Gindhu and Schonert-Reichl (2005) delved into self-harming behavior among adolescents and found a compelling link between self-harming tendencies and elevated levels of anger discomfort. They examined self-harm in a community sample of adolescents and found that those who engaged in self-harm had higher levels of anger discomfort. The results indicate that adolescents who self-harm have more discomfort with angry feelings compared to non-self-harmers.

Examining the relationship between discomfort with anger and its suppression, Leenaars and Lester (2011) examined the relationship between feeling discomfort with anger (and suppressing it) and anger expression. They found that anger discomfort was associated with indirect aggression. Expanding on this theme, Aruguete et al. (2012) examined the associations among anger, eating pathology, and perfectionism in college women. They found that anger discomfort was positively associated with self-loathing (r = 0.31) and perfectionism (r = 0.35). Adding to this body of research, Laye (2002) found that anger discomfort was positively associated with self-harm (r = 0.20 for females; r = 0.14 for males), emotional distress (r = 0.46 for females; r = 0.46 for males), anger control (r = 0.30 for females; r = 0.028 for males), and negative self (r = 0.40 for females; r = 0.24 for males). Collectively, the strong correlations between anger discomfort and eating disorders, as well as non-suicidal self-harm, suggest that difficulty in emotional regulation is a key feature of anger discomfort. In essence, the ADS serves as a valuable tool for assessing challenges in emotional regulation, as it highlights the fundamental and unique aspect of discomfort with anger. As individuals experience heightened discomfort with their anger, they may be more inclined to choose unhealthy ways of anger expression, such as engaging in disordered eating or self-harming behaviors.

These findings collectively highlight the multifaceted nature of anger discomfort and its significant implications for mental health and well-being. However, several gaps exist in the current literature. First, the hypothesized four-factor scale structure was not tested (with confirmatory factor analytic techniques). Although previous studies have used the ADS as a one-factor model (Aruguete et al., 2005, 2012; Edman & Yates, 2004, 2005; Edman et al., 2005; Laye, 2002; Laye-Gindhu & Schonert-Reichl, 2005; Leenaars & Lester, 2011), none of these studies have tested the structure of the ADS. Second, although the ADS (Sharkin & Gelso, 1991) is a scale that assesses various forms of psychological distress, it has several methodological limitations, such as the inappropriate use of principal components analysis (PCA) with varimax rotation. For example, PCA assumes that the principal components are orthogonal (uncorrelated). In practice, this may not always be appropriate because factors can be correlated with each other (Jolliffe & Cadima, 2016). Lastly, it is important to examine the existence of cultural differences, especially for anger measures because research has shown that there are vast cultural differences in anger appraisal (Mesquita & Walker, 2003) and anger regulation (Kim & Zane, 2004).

The Present Study

The goal of this study is to determine and verify the factor structure of the ADS (Sharkin & Gelso, 1991) among Korean college student and community-based samples of adults. Therefore, the research question is whether the Korean version of the ADS (ADS-K) is a valid measurement tool for assessing anger discomfort. The aims are (1) translate the ADS into Korean and examine its psychometric properties (e.g., reliability and validity); (2) determine the construct validity of the ADS-Kusing confirmatory factor analysis techniques; and (3) determine the convergent validity of the scale with anger regulation strategies (i.e., anger suppression, anger expression, and anger control.) as well as fear of negative evaluation for others and expectation of negative interpersonal consequences.

Study 1

In Study 1, the items of the Korean version of the Anger Discomfort Scale were examined using exploratory factor analysis.

Method

Participants and Procedure

Data were collected from two Korean samples (college students and a community sample). A total of 378 participants (211 women and 167 men; M age = 29.43 years; SD = 12.37; range = 18–77 years) took part in the study. Among these participants, 215 college students were recruited through introductory psychology classes and media production classes at a large private university located in Suwon, South Korea. The students participated in this study in partial fulfillment of their research participation credit through introductory psychology and media classes. Additionally, 163 participants were recruited through a survey panel company, which matches online respondents with researchers’ target audiences. Participants received a payment of 3000 Korean won (approximately $3 US) as an incentive. Both sets of participants voluntarily completed the questionnaires. Sample sizes of 300 are generally sufficient for an EFA (Worthington & Whittaker, 2006).

Measures

Development of Korean Version of the Anger Discomfort Scale

The Korean version of the ADS (Sharkin & Gelso, 1991) was developed to assess its applicability in Korean culture. First, the use of the ADS for this research was approved by Charles Gelso, the corresponding author of the scale. The authors assembled a team of bilingual experts, who are familiar with both Korean and US cultures. The translators were bilingual English-Korean speakers who are graduate students at Korea University’s Department of English Language and Literature in South Korea. The four translators translated the English version of the ADS into Korean. During the translation process, translators reviewed Korean culture’s norms, values, and language to understand the cultural nuances and potential variations in the way concepts are understood. Based on the final translation, a counseling psychology professor who has lived in an English-speaking country for more than 10 years made a back translation. A back-translation helps ensure that the essence of the scale is retained and that any discrepancies are identified. The author also conducted a pilot testing to administer the final scale to two counseling psychology professors to further assess its cultural relevance. The professors provided feedback indicating that they found the scale to be acceptable. The completed items are shown in Table 1.

Table 1 English original and Korean translation of the 15-item ADS
Full size table

The Anger Discomfort Scale

The English Version of the Anger Discomfort Scale (ADS) developed by Sharkin and Gelso (1991) is a 15-item questionnaire that measures discomfort with one’s anger. The original English version of the ADS includes four factor subscales of anger discomfort, namely (a) intrapersonal discomfort, (b) positive views of and comfort with anger, (c) interpersonal discomfort, and (d) concomitants of anger, which refers to emotions or outcomes associated with being angry. Items are rated on a 4-point Likert-type scale, with response options ranging from 1 (almost never) to 4 (almost always). Cronbach’s α for the total scale and for the test–retest reliability was 0.81 and 0.87, respectively (Sharkin & Gelso, 1991). In the present study, the Cronbach’s α for the translated version of the ADS was 0.87.

For each sample, a principal axis factoring (PAF) extraction was conducted on the 15 items using an oblique rotation because it was assumed that the sub-factors of the ADS (Sharkin & Gelso, 1991) would correlate with each other (Tabachnick et al., 2013). SPSS 25.0 was used to conduct the exploratory factor analysis for the ADS and to assess descriptive statistics and internal consistency (Cronbach’s α). We separately examined the internal consistency for the college student sample and the community-based sample of adults because inter-correlations between items can be affected when there are significant differences in item means between samples (Gaskin et al., 2017). We also conducted a parallel analysis (PA) using SPSS syntax provided by O’Connor (2000). Factor loadings were assessed, and problematic items were removed sequentially using the following criteria: Several criteria determined a stepwise item reduction throughout EFA. First, items with communalities (h2) ≤ 0.20 were excluded from the unrotated factor solution (Child, 2006; Samuels, 2017). After conducting a direct oblimin rotation, we retained items with factor loadings higher than 0.50 (Costello & Osborne, 2005) and cross-loadings equal to or less than 0.32 (Tabachnick & Fidell, 2001). Items were excluded throughout the EFA if their exclusion considerably increased Cronbach’s α (Field et al., 2013).

Results

The EFA results indicate that all 15 items met the criteria satisfactorily. Bartlett’s test of sphericity (Bartlett, 1954) yielded significant results for both the student samples, χ2 (105) = 1323.328 (p < 0.001) and the community-based samples of adults, χ2 (105) = 986.699 (p < 0.001). The Kaiser–Meyer–Olkin measure of sampling adequacy was 0.90 for the student sample and 0.87 for the community-based sample of adults, confirming the suitability of the data for factor analysis in both samples. Values greater than 0.80 for the KMO index are considered appropriate for factor analysis (DeVellis, 1991). The EFA revealed two primary factors, “discomfort with anger” and “comfort with anger,” with distinct item compositions and explained variances. These two factors were identified based on the eigenvalue rule and the observed scree test, which indicated the presence of three factors with eigenvalues greater than 1.0. The first factor, “discomfort with anger,” consisted of 12 items and accounted for 40.60% of the variance in the student sample and 38.43% in the community-based sample of adults. The Cronbach’s alpha coefficients were 0.90 for the student sample and 0.89 for the community sample. The second factor, “comfort with anger,” consisted of three items and accounted for 10.62% of the variance in the student sample and 12.22% in the community sample.. According to the eigenvalue rule (Kaiser, 1960) and the observed scree test (Cattell, 1966), three factors with eigenvalues that were greater than 1.0 were observed, accounting for 59.10% of the total variance in the student sample and 58.64% in the community sample. For the student sample, the initial eigenvalues were 6.10, 1.60, and 1.18. Based on 100 random data sets, the first two factors had raw data eigenvalues (6.10 and 1.60) greater than the simulated random eigenvalues (1.59 and 1.42). For the community sample, the initial eigenvalues were 5.77, 1.83, and 1.20. Based on 100 random data sets, the first two factors had raw data eigenvalues (5.77 and 1.83) greater than simulated random eigenvalues (1.67 and 1.51). The alphas were 0.61 for the student sample and 0.61 for the community sample. The PAF and PA indicated that a two-factor solution would be appropriate. Factor loadings are displayed in Table 2.

Table 2 Exploratory factor analysis of the ADS-K
Full size table

Factor 1 comprised the following items (i.e., items 1, 2, 3, 4, 6, 7, 8, 9, 12, 13, 14, and 15), revealing that the 12 items loaded strongly on Factor 1 with all items referring to affective or cognitive evaluations of oneself when angry in a social situation. For affective evaluation of oneself, other emotions are accompanied, such as guilt and fear about having anger in social interactions (e.g., “I feel guilty about being angry at others”). For the cognitive evaluation of oneself, items reflect on one’s negative thoughts about oneself regarding anger in social situations, such as attempting to read others’ minds (e.g., “People do not seem to like me when I am angry”), predicting negative consequences of anger (e.g., “I create more problems for myself when I get angry”), or using words like “should” (e.g., “I should not be angry as often as I am”). Factor 1 was therefore named “Anger Discomfort.” The three items loading on Factor 2 comprise the following items (i.e., items 5, 10, and 11), and they refer to positive views or beliefs about feeling angry (e.g., “I believe that it is natural and healthy to feel angry”). Factor 2 was therefore named “Comfort with Anger.” Table 1 reports the factor structure, loadings, mean, and standard deviation. In order to test for content validity, two experts on anger research were requested to check whether each item fit the present definition of anger discomfort. They confirmed the consistency of each item with the present definition.

Study 2

In Study 2, the EFA results indicated that a two-factor model is appropriate. A two-factor model consists of a general factor and an item-specific factor. In this model, every item is required to be associated with both the general factor and one of the two factors (Brown & Moore, 2012). A two-factor model is often more appropriate when the variables being analyzed represent distinct concepts or a two-factor model provides a better fit to the data compared to a one-factor model.

In Study 2, the two-factor structure of the Korean version of the Anger Discomfort Scale (ADS-K) was tested using confirmatory factor analysis (CFA) to examine whether a two-factor model has better goodness-of-fit indices compared to a one-factor model. Additionally, the convergent validity of the ADS-K was evaluated by examining the correlation between the ADS-K and other scales measuring theoretically related constructs: anger suppression, anger expression, trait anxiety, fear of negative evaluation, and expectation of negative interpersonal relationship.

First, it was expected that the ADS-K would be positively related to anger suppression and anger expression as measured by the State-Trait Anger Expression Inventory (STAXI; Spielberger et al., 1983). Second, the ADS-K was compared with a measure of trait anxiety because anger discomfort describes the degree to which a person’s level of anxiety is affected by both experiencing and expressing anger (Sharkin & Gelso, 1991). Third, since anger discomfort measures one’s levels of discomfort about anger experience and anger expression specifically in social or interpersonal contexts, it was predicted that anger discomfort may be related to anxiety in social situations such as fear of negative evaluation and expectation of negative interpersonal relationships. Given previous studies suggesting that the self-report scale for anger measurements may have been influenced by the bias of social desirability (Selby, 1984), the association between anger discomfort and social desirability was examined to estimate the discriminant validity for the ADS-K scores. It was predicted that anger discomfort measured by the ADS-K would not be related to social desirability. Moreover, the predictive validity of ADS-K designed for use in mental health contexts was investigated by comparison of ADS-K with other established measures of psychological well-being, including measures of anxiety and somatization.

Method

Participants and Procedures

Data were collected from two Korean samples: college students and a community sample. A total of 387 participants (193 women and 194 men; M age = 30.32 years; SD = 10.21; range = 18–61 years) were recruited. Among the 387 participants, 187 were college students recruited through introductory psychology and media production classes at a large private university located in Suwon, South Korea. The students participated in this study in partial fulfillment of their research participation credit through introductory psychology classes and media classes. Additionally, 200 adults were recruited through a survey panel company, which matches online respondents with researchers’ target audiences. Participants received a payment of 3000 Korean won (approximately $3 US) as an incentive. Both sets of participants voluntarily completed the questionnaires. Sample sizes greater than 100 are required for an CFA (Worthington & Whittaker, 2006).

Measures

Korean Version of the Anger Discomfort Scale

The Korean version of the Anger Discomfort Scale (ADS-K) is a 15-item questionnaire designed to measure discomfort with one’s own anger. The ADS-K includes two factor subscales: (a) anger discomfort and (b) comfort with anger. Items are rated on a 4-point Likert-type scale, with response options ranging from 1 (almost never) to 4 (almost always). In the present study, the Cronbach’s α for Factor 1 (anger discomfort) of the ADS-K was 0.90 for the student sample and 0.92 for the community sample. The Cronbach’s α for Factor 2 (comfort with anger) was 0.62 for the student sample and 0.60 for the community sample.

Anger Suppression/Anger Expression/Anger Control

The State-Trait Anger Expression Inventory (STAXI; Spielberger et al., 1983) is a 44-item questionnaire designed to assess anger expression with three subscales: (a) anger suppression (8 items), (b) anger expression (8 items), and (c) anger-control (8 items). Items are rated on 4-point Likert-type scale, with response options ranging from 1 (not at all) to 4 (very much so). Evidence for convergent validity was supported by a positive association between anger suppression and anger discomfort (r = 0.32) and between anger expression and anger discomfort (r = 0.20) among college students (Sharkin & Gelso, 1991). Evidence for discriminant validity was supported by a significant but weak correlation between anger-control and anger discomfort (r = –0.16) among college students (Sharkin & Gelso, 1991). The Korean version of State-Trait Anger Expression Inventory (STAXI-K; Chon et al., 1998) demonstrated adequate internal consistency with a Cronbach’s alpha (α) of 0.67 for anger-suppression, 0.71 for anger-expression, and 0.82 for anger-control among college students and community samples. In the present study, the estimated internal consistency for the three subscales was: 0.82 (student sample) and 0.78 (community sample) for anger-suppression, 0.80 (student sample) and 0.85 (community sample) for anger-expression, and 0.79 (student sample) and 0.84 (community sample) for anger-control.

Trait Anxiety

The State-Trait Anxiety Inventory (STAI-Y; Spielberger, 1983) was used to measure trait anxiety. The STAI-Y is a 40-item questionnaire designed to measure (a) state anxiety (20 items) and (b) trait anxiety (20 items). Items are rated on a 4-point Likert-type scale, with response options ranging from 1 (almost never) to 4 (almost always). Higher scores indicate greater levels of trait anxiety. Evidence for convergent validity was supported by a positive association with anger discomfort (r = 0.51) among college students (Sharkin & Gelso, 1991). The Korean version of the Trait Anxiety Inventory (STAI-Y-K; Hahn et al., 1996) demonstrated excellent internal consistency with a Cronbach’s alpha (α) of 0.90 among college students and community sample. In the present study, Cronbach’s α for trait anxiety was 0.93 for the student sample and 0.95 for the community sample.

Fear of Negative Evaluation by Others/Expectation of Negative Interpersonal Relationship

The Anger-In Attitude Scale (Seo et al., 2004) assesses fear of negative evaluation and expectation of negative interpersonal relationships. It comprises 25-items designed to measure anger-in attitudes, consisted of four scales: (a) positive attitudes toward anger suppression (6 items), (b) fear of others’ negative evaluation (10 items), and (c) expectation of negative interpersonal relationship (9 items). Items are rated on a 7-point Likert-type scale, with response options ranging from 1 (strongly disagree) to 7 (strongly agree). Evidence for convergent validity was supported by previous research which has shown that socially anxious persons often encounter challenges with anger regulation among adults (Weber et al., 2004). In the study by Seo et al. (2004), the Anger-In Attitude Scale demonstrated robust internal consistency, with a Cronbach’s α of 0.86 for fear of negative evaluation, 0.86 for expectation of negative interpersonal relationship, and 0.79 for total score among college students. In the present study, Cronbach’s α was 0.89 (student sample) and 0.90 (community sample) for fear of negative evaluation, and 0.84 (student sample), and 0.91 (community sample) for expectation of negative interpersonal relationship.

Social Desirability

The Social Desirability Scale-17 (Stöber, 2001) was employed to measure discriminant validity. Prior research has indicated that anger inventories typically exhibit negative correlations with measures of social desirability (Biaggio, 1980), and there was no significant relationship with social desirability (Sharkin & Gelso, 1991). The SDS-17 is a 16-item questionnaire designed to measure (a) self-deception (10 items) and impression management (6 items). Items are rated on a binary answer 1 (true) and 0 (false). Higher scores indicated higher levels of social desirability. Stöber (2001) found a Cronbach’s α for the total scale was 0.80. The Korean version of SDS-16 (Bae et al., 2015) demonstrated a Cronbach’s α of 0.72 for the total measure among college students. In the present study, the estimated internal consistency for the total scale was 0.88 (student sample) and 0.72 (community sample).

Anxiety/Somatization

The Symptom Checklist-90-Revision (Derogatis, 1994) was used to measure anxiety and somatization. The SCL-90-R is a 90-item questionnaire designed to measure psychological symptoms and consists of 9 subscales, namely (a) somatization (12 items), (b) obsessive–compulsive (10 items), (c) interpersonal sensitivity (9 items), (d) depression (13 items), (e) anxiety (10 items), (f) hostility (6 items), (g) phobic anxiety (7 items), (h) paranoid ideation (10 items), and (i) psychoticism (10 items). Items are rated on a 5-point Likert-type scale, with response options ranging from 0 (not at all) to 4 (extremely). Higher scores indicate greater anxiety or somatization severity. Evidence for convergent validity was supported by previous research showing that participants with high levels of anger are also high on anger suppression, which makes them more likely to experience anxiety, tension, nervousness, and somatization (Liu et al., 2011). The Korean version of the SCL-90-R (Won, 1978) demonstrated good internal consistency with a Cronbach’s α of anxiety (0.86) and somatization (0.82) among college students. In the present study, the estimated internal consistency for anxiety was 0.93 (student sample) and 0.95 (community sample). The estimated internal consistency for somatization was 0.89 (student sample) and 0.91 (community sample).

Results

Construct Validity

Using Mplus software 8.8, a confirmatory factor analysis (CFA) was conducted with maximum likelihood (ML) on the variance–covariance matrix (REF). Model fit was evaluated using several fit indices: (a) The chi-square to df ratio (χ2/df), where a value of no more than 3.0 indicates a good fit (Byrne, 2013), (b) Comparative Fit Index (CFI), with values greater than 0.95 indicating good fit, between 0.92 and 0.94 indicating adequate fir (Hu & Bentler, 1999), (c) Tucker-Lewis Index (TLI), with values greater than 0.90 indicating good fit (Hu & Bentler, 1999), (d) root mean square error of approximation, with values less than 0.05 indicating very good fit, 0.05 to 0.08 indicating good fit, 0.08 to 0.10 indicating poor fit, and over 0.10 indicating bad fit (Byrne, 2013; Fabrigar et al., 1999), and (f) standardized root-mean-square residual (SRMR), with values less than 0.05 indicating good fit, close to 0.08 indicating acceptable fit (Byrne, 2013).

First, a confirmatory factor analysis (CFA) was conducted to assess the one-factor model. The chi-square of the one-factor model was significant, χ2 (90) = 474.56, p < 0.001, with a χ2/df of 5.27, which exceeds the recommended threshold. Additionally, the standardized root mean square residual (SRMR) was 0.08, and the root mean square error of approximation (RMSEA) was 0.11 [0.096, 0.114], indicating an inadequate fit. The comparative fit index (CFI) was 0.83, and the Tucker-Lewis index (TLI) was 0.81, also suggesting inadequate fit. Second, a CFA was conducted to assess the four-factor model proposed by the authors (Sharkin & Gelso, 1991). The chi-square of the four-factor model was significant, χ2 (84) = 293.36, p < 0.001, with a χ2/df ratio of 3.49, which did not meet the threshold. The SRMR was 0.05 and the RMSEA of 0.08 [0.070, 0.090], indicating inadequate fit. However, the CFI was 0.91 and the TLI was 0.89, suggesting slightly better fit but still inadequate. Third, a CFA was conducted to assess two-factor model of anger discomfort (item 1, 2, 4, 5, 6, 7, 10, 12, 13, 14, 15) and comfort with anger (item 3, 8, 9) identified in Study 1. The chi-square of the two-factor model was significant, χ2 (89) = 317.39, p < 0.001, with a χ2/df of 3.57, which did not meet the threshold. The SRMR was 0.05, and the RMSEA was 0.08 [0.072, 0.091], indicating a reasonable fit. The CFI was 0.90, and the TLI was 0.88, suggesting acceptable fit. Lastly, a CFA was conducted to assess a bi-factor model. The chi-square of the bi-factor model was significant, χ2 (75) = 177.83, p < 0.001, with a χ2/df of 2.37, which is considered acceptable. The SRMR was 0.04, and the RMSEA was 0.060 [0.048, 0.071], indicating good fit. The CFI was 0.96, and the TLI was 0.94, suggesting good fit. The results are summarized in Table 3. Table 4 provides the standardized factor loadings of one-factor and bi-factor models of the ADS-K.

Table 3 Model comparison for ADS-K
Full size table
Table 4 Standardized factor loadings of one-factor and bi-factor models of the ADS-K
Full size table

Convergent Validity

We used SPSS 25.0 to evaluate descriptive statistics, item-total correlations, internal consistency (Cronbach’s α), and the validity of the ADS scores (see Table 5). Internal consistency was assessed by Cronbach’s α.

Table 5 Means, standard dviations, skewness, kurtosis, Cronbach’s alpha scores, and correlations among latent variables
Full size table

Correlation analyses were conducted between the ADS-K scores and each score of the other constructs to examine convergent validity. As hypothesized, the ADS-K scores exhibited relatively high positive correlations with scores for all the other scales measuring related constructs: anger suppression (r = 0.52, student sample; 0.60, community sample), anger expression (r = 0.34, student sample; 0.67, community sample), and trait anxiety (r = 0.52, student sample; 0.47, community sample). Additionally, the ADS-K was also positively associated with scales measuring interpersonal relationships, such as fear of negative evaluation (r = 0.57, student sample; 0.47, community sample), and expectation of negative interpersonal relationship (r = 0.61, student sample; 0.48, community sample). These results emphasized the role of anger in interpersonal relationships, suggesting that individuals with high anger discomfort may experience greater levels of fear of negative evaluation by others or anticipate negative outcomes in interpersonal relationships.

Discriminant Validity

To examine the discriminant validity of the ADS-K scores, correlations between the ADS-K scores and the anger control score were calculated. As expected, no significant correlations (p > 0.05) were observed. These results provide evidence that the ADS-K is a valid measure of anger discomfort in Korean samples.

Criterion Validity

Considering the negative mental health implications of discomfort with anger, we hypothesized that the ADS would significantly predict each of our mental health indicators. As predicted, the ADS scores were positively correlated with anxiety (r = 0.52, student sample; 0.47, community sample) and somatization (r = 0.47, student sample; 0.36, community sample).

Discussion

The aim of the present research was to provide evidence of the psychometric properties and evaluate the validity of the Korean version of the ADS based on data from Korean college students and a community sample of adults in South Korea.

In Study 1, EFA results indicated the ADS-K supported a two-factor model which is inconsistent with the original version of the four-factor solution of the ADS. The original ADS distinguished between intra- and interpersonal anger discomfort. However, the ADS-K did not delineate the distinctive characteristics between intrapersonal and interpersonal factors of anger discomfort. Several reasons may account for this discrepancy.

First, it is possible that cultural differences in lexical gaps between Korean and English contributed to this discrepancy. Choi and Kim (2003) noted that Koreans conceptualize the self differently from Westerners. In Korean culture, self-awareness is not as highly emphasized in social relationships, with Koreans more likely to emphasize collective identity over individual self. For instance, instead of saying “my country” or “my family,” Koreans say “our country” or “our family.” The Korean word Na is analogous to “self” in English, but it carries additional connotations related to relational aspects of the self, promoting harmonious social relationships (Choi & Kim, 2003). Therefore, participants in our study might not have distinguished between intra- and interpersonal aspects of anger discomfort.

Second, most items in the original ADS include focus on interpersonal anger rather than intrapersonal anger. Intrapersonal anger refers to feeling of upsetness when one’s goals have been hindered, whereas interpersonal anger pertains to the impact of one person’s show of rage on another person (Van Kleef et al., 2008. For example, item 7 (“People do not seem to like me when I am angry”) is labeled as intrapersonal anger discomfort, although it is closely related to the notion of interpersonal anger discomfort. Therefore, the ADS may not adequately distinguish between intrapersonal anger discomfort and interpersonal anger discomfort, or it might not adequately capture intrapersonal anger discomfort all together.

In Study 2, contrary to the original authors’s empirical assumption (Sharkin & Gelso, 1991), we found that a bi-factor model, rather than a one-factor model of anger discomfort, provided the best fit for the data. This suggests that it is beneficial to examine anger discomfort and comfort with anger separately rather than using an overall anger discomfort score. Confirmatory factor analysis (CFA) results confirmed that a bi-factor model primarily yielded the strongest support among three alternative models (i.e., one-factor, two-factor, and four-factor models) for the general factor and the two factors: (1) anger discomfort—reflecting discomfort levels when one experiences anger in interpersonal situations; and (2) comfort with anger—re[resenting positive beliefs and views about anger. Furthermore, the second factor (comfort with anger) corresponded to the three items of the original ADS. When a scale has two opposite concepts, such as discomfort and comfort in the case of the ADS, a bi-factor model is more suitable for a factor structure (Reise et al., 2010). This finding underscores the importance of considering both dimensions separately to gain a comprehensive understanding of individuals’ experiences with anger.

Furthermore, not only did the ADS-K demonstrate high internal reliability, but correlational analyses also provided support for its construct, convergent, discriminant, and criterion validity. Initially, the ADS-K scores exhibited strong correlations with anger suppression, although they did not completely overlap. While the ADS-K primarily contains items related to the reasons individuals may decide to suppress their anger, it does not explicitly inquire whether participants engage in anger suppression. Essentially, the ADS-K measures anger discomfort itself, which may subsequently lead to anger suppression. This finding aligns with prior research indicating that individuals who experience discomfort with their anger tend to encounter difficulties expressing it and often resort to suppression (Sharkin & Gelso, 1991). Additionally, anger discomfort displayed a positive association with anger expression. Individuals who express their anger inappropriately may experience self-conscious regarding their behavior’s acceptability to others, potentially leading to ruminative thoughts like people who suppress their anger (Linden et al., 2003).

Moreover, anger discomfort as measured by the ADS-K exhibited a positive correlation with trait anxiety, consistent with previous findings linking trait anxiety to anger discomfort (Sharkin & Gelso, 1991). Essentially, trait anxiety appears to be a significant contributor to anger discomfort, suggesting that individuals with inherent anxiety may experience heightened discomfort when confronted with their anger. This association underscores the ADS-K’s ability to capture the multifaceted nature of anger, as it can assess emotions beyond anger alone, such as anxiety and anger.

Additionally, our study revealed a positive association between anger discomfort and fear of evaluation from others as well as negative expectations of interpersonal relationships. This suggests a potential link between the interpersonal aspects of anger discomfort and its correlation with social anxiety. Cognitive-behavioral theorists have highlighted fear of negative evaluation from others as a core feature of social anxiety (Wells et al., 1995). Individuals with high social anxiety levels often tend to suppress and avoid expressing their anger in interpersonal situations to alleviate their anxiety (Moscovitch et al., 2008). Similarly, negative expectations of interpersonal relationships, such as anticipating rejection and negative evaluations from others, are characteristic of social anxiety (Baldwin & Main, 2001). Those with elevated levels of anger discomfort may harbor negative expectations in their interpersonal relationships, leading to avoidance of expressing anger constructively, containment of one’s anger, and reduced assertive behaviors (Weber et al., 2004). Specifically, anger discomfort encompasses both self-consciousness regarding one’s anger and discomfort in social settings. As expected, the ADS-K score demonstrated no significant correlation with anger-control, supporting discriminant validity. This finding aligns with previous research indicating that individuals experiencing high levels of anger discomfort may struggle to control their anger effectively, or conversely, individuals lacking control over their anger may experience heightened levels of anger discomfort (Sharkin & Gelso, 1991). Furthermore, our study found a correlation between anger discomfort and psychological health, specifically anxiety and somatization. This finding is consistent with previous research suggesting that Koreans may tend to express psychological distress using somatic language rather than openly expressing anger, in contrast to North American samples (Choi et al., 2016).

Limitations and Future Directions

These preliminary findings are promising, but several limitations present opportunities for further research. First, while the Korean version of the ADS demonstrates good to adequate model fit, the absence of a model fit comparison with the original scale in North American samples, due to the original study only including EFA results, limits the extent to which the findings can be generalized. Second, the present study only included Korean samples. To validate these findings, conducting a cross-cultural scale validation study would be beneficial to determine if the two-factor model holds across other collectivistic cultures. Given that the original scale of the ADS was developed in the USA, it remains unclear whether the items accurately capture anger experience and expression specific to Korean culture. The Hwa-Byung Scale (HB scale), for example, includes the subjective anger construct, uk-wool and boon, which represent Korean culture-related sentiment of unfairness (Min et al., 2009). This emphasizes the importance of including culture-specific items in future studies. In terms of anger expression, there is currently no explicit evidence establishing a close relationship between anger discomfort and state or trait anger. Further studies into these associations are needed.

Our findings indicate significant associations between anger discomfort and trait anxiety, anger suppression, and anger expression. However, it remains unclear which variable precedes experiencing anger discomfort. Future studies are needed to explore the predictors of the underlying mechanisms that contribute to experiencing anger discomfort. Variables such as guilt and shame may play a role in shaping the interactive relationship between anger discomfort and expression strategies. Additionally, individuals with high level of anger discomfort may resort to suppressing their anger to avoid interpersonal consequences and could potentially have negative implications for their mental health, such as somatization. Therefore, mediation analyses or regression analyses would be valuable in elucidating the consequences of anger discomfort.

Expanding the scope of the study to include clinically referred populations would provide valuable insights into the role of anger discomfort in mental health issues among individuals seeking professional help. Additionally, determining a cut-off score for the ADS-K would enhance its utility as a screening tool for identifying individuals at risk for experiencing significant levels of anger discomfort. Furthermore, while our study focused on the relationship between anger discomfort, anxiety, and somatization, future research should explore additional psychological constructs as guilt and shame. Given the association between anger discomfort and feelings of guilt about expressing anger, as well as the potential link to self-harm behaviors, investigating the interplay between anger discomfort, guilt, and depression is needed (Laye-Gindhu & Schonert-Reichl, 2005). This expanded understanding would contribute to a more comprehensive assessment of the impact of anger discomfort on mental health outcomes.

Lastly, there can be potential issues with the imbalance of items loading onto Factor 1 versus Factor 2 in a two-factor model. In the current ADS model, there are 12 items loading onto Factor 1 and 3 items loading onto Factor 2. When one factor has a significantly larger number of items loaded onto it compared to the other factor, it can lead to several problems and limitations. An imbalance in the number of items on each factor can make it challenging to interpret the meaning of each factor. The factor with more items may represent a broad and multifaceted construct, while the factor with fewer items may be less clear in its interpretation. Furthermore, the imbalance of items can also raise questions about the validity and reliability of the model. If one factor has a much larger number of items than the other, it may dominate the model’s results, potentially overshadowing the contributions of the smaller factor. The reliability of the factors can be affected. A factor with fewer items may have lower internal consistency compared to the factor with more items, making it less robust as a measure of the underlying construct.

Implications for Practice and Research

The results of the present study have significant implications for practice and research. We identified a bi-factor model that is different from the original version of the ADS. Specifically, our findings suggest that participants did not differentiate between intra- or interpersonal anger discomfort. This suggests that using anger measures validated in collectivist cultures would be more appropriate than those developed in individualist cultures. Therefore, for Korean clients and clients from other collectivists, it may be more suitable to use the bi-factor model of the ADS-K than the one-factor or the four-factor subscales.

The ADS-K can support therapists in monitoring clients’ anger discomfort within their interpersonal relationships. Given that anger discomfort is often subtle and internal, the ADS-K provides counselors with a valuable tool for understanding clients who may have high anger discomfort, which may not be readily observable. Moreover, since anger discomfort typically manifests in interpersonal contexts, it can be challenging to detect during traditional psychotherapy sessions. Therefore, therapists can utilize the ADS-K to explore the dynamics of the therapeutic relationship in a here-and-now manner. For example, if a client has high anger discomfort in their interpersonal relationships outside of therapy, they may also experience difficulty expressing anger towards their counselor or therapist, potentially impacting the rapport or therapeutic alliance.

In a college counseling center setting, group counseling would be beneficial for clients struggling with anger discomfort, given its association with both anger expression and anger suppression. Clients exhibiting high levels of anger suppression or expression may particularly benefit from cognitive behavioral group therapy (CBGT; Kuo et al., 2021). Conversely, individuals scoring low on anger expression may benefit from mindfulness-based stress reduction (Kuo et al., 2021). Given that anger discomfort is linked to trait anxiety, mindfulness-based stress reduction may be particularly helpful for clients experiencing high levels of anger discomfort, as they address underlying anxiety issues. Therapists can consider prioritizing anxiety reduction techniques before addressing anger-related concerns in these clients.

Family and marriage therapists, as well as couple counselors, can utilize this tool to assist clients with anger issues within their relationships. By exploring into the client’s familial and cultural backgrounds, counselors can gain insight into the client’s experience of anger discomfort. Anger experience and anger expression are often shaped by internalized familial and cultural rules (Deffenbacher et al., 2002). For instance, individuals who raised in emotionally repressed family cultures may exhibit heightened levels of anger discomfort (Burrowes & Halberstadt, 1987; Sharkin & Gelso, 1991).

In conclusion, this study introduces an initial adaptation of the Korean version of the Anger Discomfort Scale (ADS-K) for Korean college students and a community sample of adults. The two-factor structure of the ADS-K was confirmed as having satisfactory validity and reliability. The ADS-K provides an opportunity to explore the nuanced emotions associated with anger; that is, the ADS-K can measure more than one feeling (i.e., fear of anger). Through examining the relationship between anger and other psychological constructs, the present study helps to understand the overall concept of anger discomfort. Counselors are encouraged to consider the interpersonal dynamics of clients when working with those who have high levels of anger discomfort.