The COVID-19 pandemic placed significant strain on our society. While millions became ill from the virus itself, the pandemic additionally disrupted daily life for everyone, creating new stressors and increasing risk for mental health problems. New financial, work-related, school-related, health-related and social stressors arose due to the virus and the stay-at-home orders that aimed to curb the spread of the disease. Critically, the pandemic significantly affected mental health: 45% of Americans reported that coronavirus-related stress and worry negatively impacted their mental health (Kirzinger et al., 2020) and calls to crisis hotlines dramatically increased (American Psychiatric Association, 2020).

College students have been at high risk for negative mental health impacts of the pandemic. Emerging adulthood is already a critical time for the onset and maintenance for internalizing psychopathology (Kessler et al., 2012), with 20–45% of college students experiencing a mental health disorder each year (Auerbach et al., 2016; Blanco et al., 2008). Internalizing symptoms predict poor academic performance and college dropout (Bruffaerts et al., 2018) and impact professional trajectories, making it critical to understand risk and protective factors during this time. Stress is a strong, transdiagnostic risk factor for mental health symptoms, especially anxiety and depression (for review see Grant et al., 2014). Importantly, the COVID-19 pandemic introduced myriad new stressors into the lives of college students. The sudden and unpredictable closure of campuses interrupted the semester, causing classes to move online and many students to move home and reintegrate into family life. This was additionally accompanied by loss of peer and social interactions, and loss of college milestones such as graduation. This major disruption to college life came on top of ongoing stressors experienced by college students, such as pressure to succeed academically. Thus, heightened stress exposure created by the pandemic was likely to put college students at even greater risk for rises in internalizing symptoms.

However, how individuals respond to stressors can significantly moderate psychopathology risk. Diathesis-stress theories posit that relatively stable individual differences (diatheses) frequently moderate associations between stressful life events and psychopathology (e.g., Monroe & Simons, 1991). Individuals who employ response styles that are not effective in managing their emotional responses to stressors experience longer and more severe periods of distress which may lead to elevated depression and anxiety, whereas response styles that enable successful emotion regulation may be protective against stress effects (e.g., for review see Aldao et al., 2010).

Based on prior research primarily on effects of everyday stressful life events, a number of response styles have been identified as generally maladaptive or adaptive in coping with stressors. However, there are significant gaps within this research. First, to date, little response style research (Aldao & Nolen-Hoeksema, 2010) has taken a transdiagnostic approach to assessing internalizing risk, and, to our knowledge, no prior studies have used an analytical approach that separates out specific internalizing dimensions (i.e., anhedonic depression [loss of interest], anxious arousal and anxious apprehension [worry]) from general internalizing (i.e., transdiagnostic general distress), which is important for assessing whether response styles are risk or protective factors for transdiagnostic, common internalizing versus specific internalizing symptom dimensions. Second, much of this research (e.g., Cox et al., 2012) focuses on stress response styles individually, despite evidence (e.g., Garnefski et al., 2001) suggesting that individuals engage in multiple response styles, introducing potential confounds from unmeasured co-occurring response styles. Third, although some prior research (e.g., Chen & Hong, 2010; Cox et al., 2012; D’Zurilla & Nezu, 2009; Felsten, 1998; Troy et al., 2010; Zahniser & Conley, 2018) supports the diathesis-stress model, empirical evidence for moderation across the various response styles is either limited (e.g., Chen & Hong, 2010; Zlomke & Jeter, 2014), inconsistent (e.g., Burton et al., 2004; Pakenham et al., 2007), or has yet to be tested. Finally, although there is evidence for associations between stress response styles and internalizing symptoms, research also suggests that response style effects can vary a great deal based on context and that response styles that are adaptive in one context may be maladaptive in another and vice versa (e.g., strategy-situation fit hypothesis [Aldao et al., 2015; Folkman et al., 1986a, b; Kato, 2020; Nolen-Hoeksema & Watkins, 2011). It is thus important to test hypotheses based on evidence from typical life stressors across other contexts, such as the new, uncontrollable stressors during the beginning of the COVID-19 pandemic, to better understand when particular response styles are adaptive, maladaptive, or neither.

Thus, the current study addresses these gaps. First, we use a bifactor model to separate out specific dimensions from general internalizing, to provide novel insights into specific versus transdiagnostic risk and protective pathways. Second, we assess multiple putatively adaptive and maladaptive response styles within the same models and use the LASSO method to identify the most parsimonious set of predictors to optimally account for variance in each internalizing dimension, providing information for more efficient risk screening and identifying potential targets for prevention and intervention. Here, we focus on six response styles most frequently studied and found to be risk (brooding, intolerance of uncertainty, and escape/avoidance) or protective (positive reappraisal, planful problem solving, seeking social support) factors for internalizing psychopathology. Third, we test both additive and diathesis-stress models to better understand how stress and response styles are associated with internalizing symptoms. Finally, we studied participants at the start of the COVID-19 pandemic to improve understanding of how stress response styles work in the context of new, uncontrollable stressors. Better understanding the role of response styles in coping with the sudden onset of uncontrollable stressors has implications beyond the COVID-19 pandemic, to understanding responses to other such stressors, including natural disasters, war and conflict, and of course future pandemics.

Maladaptive Response Styles

Brooding

Brooding is the maladaptive subtype of rumination that involves repetitive and negative self-focused attention in response to past events or stressors (e.g., “Why can't I handle things better?”; Treynor et al., 2003). A meta-analysis found that rumination is consistently associated with risk for depression (r = .55) and anxiety (r = .42) symptoms (Aldao et al., 2010), and it has been associated with broad, transdiagnostic psychopathology (e.g., McLaughlin & Nolen-Hoeksema, 2011; Snyder et al., 2019). Brooding rumination has been shown to exacerbate the effects of stress on internalizing symptoms (Cox et al., 2012; Mezo & Baker, 2012), potentially by increasing negative affect and decreasing problem solving (Wisco & Nolen-Hoeksema, 2008).

Intolerance of Uncertainty

Intolerance of uncertainty (IU) is a dispositional fear of the unknown, or difficulty in enduring the aversive response triggered by the perceived absence of sufficient information (e.g., about what will happen in a particular situation) (e.g., Carleton, 2016). Individuals high in IU perceive uncertainty as aversive, threatening, and something to be avoided or controlled (e.g., for review see McEvoy et al., 2019). IU was first proposed as a risk factor for uncontrollable worry (Dugas et al., 2001), but a meta-analysis has since found that IU is approximately equally associated with multiple internalizing symptoms and syndromes, including depression, worry/GAD, social anxiety, and panic (rs ~.4–.6), suggesting it is a transdiagnostic risk factor (McEvoy et al., 2019). Based on the theory that IU in response to uncertain negative events leads to internalizing psychopathology (e.g., Carleton, 2016), it seems plausible that IU should moderate the effects of stressors on internalizing symptoms, but little research has directly tested this theory: One study found that IU increased effects of daily hassles on increases in anxiety symptoms (but not worry) in undergraduates (Chen & Hong, 2010), but another study in undergraduates unexpectedly found that daily hassles were associated with lower worry in those with higher IU (however, this study found no main effect of hassles on worry, suggesting the hassles measure may have been flawed) (Zlomke & Jeter, 2014). Thus, although IU clearly is associated with internalizing psychopathology, it remains unclear to what extent it moderates stress effects.

Escape/Avoidance

The suppression or avoidance of thoughts, emotions, and experiences related to stressors, and attempts to escape stressors by behaviorally withdrawing from engaging with them (e.g., sleeping too much, using substances) are considered maladaptive because it can prevent individuals from taking necessary steps to solve problems, and paradoxically increase negative thoughts (e.g. for review see, Aldao et al., 2010). A meta-analysis found that avoidance was associated with both higher depression (r = .48) and anxiety (r = .37) (Aldao et al., 2010). However, this research has largely examined avoidance as a general tendency in response to stressors, and has not directly tested whether avoidance moderates the effects of recent stressors on symptoms.

Adaptive Response Styles

Planful Problem Solving

Planful problem solving can be an adaptive response style because it allows individuals to manage stressors and reduce repercussions (Aldao et al., 2010). A meta-analysis found that problem solving is associated with lower depression (r = –.33) and anxiety (r = –.27) (Aldao et al., 2010) and there is an abundance of evidence that problem-solving ability can buffer the relation between stressors and internalizing symptoms (D’Zurilla & Nezu, 2009). Thus, stress may be less likely to contribute to internalizing symptoms for individuals who use more planful problem solving in response to stressors.

Seeking Social Support

Individuals may also seek social support in response to stressors in the form of advice and information (i.e., informational support) or for the provision of reassurance, love, care, and acceptance (i.e., emotional support) (Schaefer et al., 1981). Social support has a robust negative association with internalizing symptoms (e.g., Heerde & Hemphill, 2018) and seeking social support can reduce the relation between stress and internalizing symptoms (e.g., Felsten, 1998), perhaps by providing individuals with resources to manage stressors or regulate associated emotions (Cohen & Wills, 1985). However, other research has found only main effects of social support, with no evidence for stress buffering (e.g., Burton et al., 2004). Thus, the question of whether social support has only main versus stress buffering effects on internalizing psychopathology risk remains open.

Positive Reappraisal

Positive reappraisal involves generating more positive reinterpretations or perspectives on stressors in order to reduce distress (e.g., Gross, 1998). A meta-analysis found that reappraisal was associated with lower depression (r = –.17) and anxiety (r = –.13), although effect sizes were modest (Aldao et al., 2010). However, most research has studied effects of reappraisal on affect in lab tasks or main effects of reappraisal tendencies on psychopathology risk, while little research has directly tested stress buffering (moderation) effects of reappraisal on internalizing psychopathology. Results from those studies that have tested such moderation have been mixed with some finding evidence of stress buffering (Troy et al., 2010; Zahniser & Conley, 2018), but others finding moderation only for participants also at high genetic risk (Ford et al., 2014), or experiencing uncontrollable stressors (with exacerbating effects for controllable stressors; Troy et al., 2013), or no buffering effect (Pakenham et al., 2007). Thus, evidence that reappraisal can reduce effects of stressful life events on internalizing psychopathology remains somewhat tenuous.

Role of Situational Factors in Response Style Effects

The response styles reviewed above have been robustly found to be associated on average with higher (brooding, intolerance of uncertainty, escape/avoidance) or lower (positive reappraisal, seeking social support, planful problem solving) internalizing psychopathology, with some evidence of stress moderating effects. However, some responses can be either adaptive or maladaptive depending on the situation. The strategy-situation fit hypothesis proposes that emotion regulation strategies are only adaptive when used in appropriate contexts (Aldao et al., 2015; Folkman et al., 1986a, b). In particular, problem-focused coping (e.g., planful problem solving) may be more adaptive when situations are controllable, whereas emotion-focused coping (e.g., reappraisal) may be more adaptive when situations are uncontrollable (but may actually be harmful in situations that are controllable, by decreasing efforts to change the situation) (Folkman et al., 1986a, b; Ford & Troy, 2019; Haines et al., 2016). In addition, when situations make a strategy hard to apply successfully, attempting to do so may be counterproductive (e.g., Ford & Troy, 2019). Other research (Nolen-Hoeksema & Watkins, 2011) also suggests that maladaptive strategies like brooding pose risk for different internalizing dimensions depending on the context: in the context of loss, brooding may be a greater risk for anhedonia, while brooding in threatening situations may increase anxiety symptoms. Thus, given the ways in which stressors caused by the COVID-19 pandemic differed from typically occurring stressors (e.g., less controllable), and the situational availability of some coping responses (e.g., limited ability to obtain in-person social support outside of the household unit), the effects of various response styles may be different than those found for typical stressful life events.

In sum, an abundance of evidence suggests response styles influence internalizing symptoms, but effects can vary by context, knowledge of how these responses affect specific internalizing dimensions is limited, and evidence for moderation effects is either limited, inconsistent, or lacking.

Current Study

Thus, based on evidence from responses to typical stressful life events, positive reappraisal, planful problem solving and seeking social support, which are generally found to be adaptive response styles, may be positively associated with, and buffer against the effects of COVID-19 pandemic related stress on, internalizing symptoms in college students; whereas brooding, intolerance of uncertainty and escape/avoidance behaviors, which are generally found to be maladaptive response styles, may be negatively associated with, and exacerbate the effects of stress on, symptoms. However, given the many ways in which COVID-19 stressors differed from typical stressful life events experienced by college students and the limitations of past research, the effects of different response styles remain uncertain. In addition, most research has studied each response style in relation to individual disorders or symptom dimensions, without taking into account the high co-occurrence of psychopathology, especially anxiety and depression symptoms (e.g., for review see Hankin et al., 2016). Thus, relatively little is known about how different response styles may be transdiagnostically related to internalizing psychopathology versus differentially related to specific internalizing symptom dimensions.

Identifying the response styles profiles which are most and least adaptive in helping college students cope effectively with pandemic-related stress provides an “experiment in nature”, enabling us to better understand responses to severe, unpredictable and uncontrollable stressors which may arise in individual’s lives or during future public health, natural disaster, or other emergency situations. It also has important implications for screening, prevention and intervention. As students have returned to college campuses following the pandemic shutdowns, lingering effects from the pandemic and the need to re-adjust to the “new normal” mean colleges must be prepared to meet increased mental health needs of students, including helping students develop more resilient response styles.

Thus, the current study tested whether response styles moderate the effects of pandemic-related stress on internalizing symptom dimensions in college students during the highly stressful period when students were transitioning off campus and adjusting to online instruction and social isolation (early April 2020). We do so across two universities that differ geographically and demographically, in order to test robustness of findings to different student populations. To examine associations with general and specific internalizing symptom dimensions, we used a bifactor model that separates general internalizing from anhedonic depression, anxious arousal and anxious apprehension specific symptom dimensions. We also analyzed traditional manifest symptom sum scores for comparison.

We hypothesized that higher levels of brooding, intolerance of uncertainty and escape/avoidance behaviors would predict a stronger association between COVID-19 stress severity and general internalizing symptoms, whereas higher levels of positive reappraisal, planful problem solving and seeking social support would predict a weaker association between COVID-19 stress severity and general internalizing symptoms. These hypotheses were based on what the majority of evidence from past research indicated, but were tentative given some of the inconsistencies, limitations in moderation testing, and unknown effects of the pandemic. We also focused on the general internalizing factor given evidence that most response styles are broadly related to internalizing psychopathology. However, in addition, some response styles may moderate stress effects on the specific symptom dimensions (e.g., intolerance of uncertainty may be particularly linked to worry) or have different effects on different symptom dimensions (e.g., escape/avoidance behaviors could decrease anxious arousal but increase general internalizing). Thus, although we did not have strong hypotheses about these potential specific effects, we tested for moderation of stress effects by both composite adaptive and maladaptive response styles factors and individual response styles on both the general and specific internalizing dimension factors.

Methods

The hypotheses, methods, and data analysis plan for the current study were pre-registeredFootnote 1.

Participants

Brandeis University

Participants were 154 undergraduates at Brandeis University recruited via online advertisements posted to Brandeis-specific platforms (e.g., student class year Facebook pages). Participants were ages 18 to 23 years (M = 20.05, SD = 1.28) and the majority identified as female (77.3% female, 19.5% male, 3.3% other). Participants identified as 61.0% White, 26.2% Asian, 6.1% Black, 1.2% American Indian/Alaskan Native, 1.8% Native Hawaiian/Other Pacific Islander, and 3.7% Other; 9.2% identified as Hispanic/Latino. Inclusion criteria were being a Brandeis undergraduate student aged 18 years or older and speaking fluent English (self-reported). The study was approved by the Institutional Review Board at Brandeis University, and all participants provided informed consent and were compensated with Amazon gift cards for their time.

University of Colorado Boulder

Participants were 297 undergraduate students at the University of Colorado Boulder recruited via online advertisements posted to CU Boulder-specific platforms (e.g., student e-newsletter, Facebook pages). Participants were ages 18 to 38 years (M = 20.91, SD = 2.55), the majority identified as female (76% female, 21% male, 3% non-binary), and the sample was representative of the Boulder area racial/ethnic distribution (0.4% Native American, 8.3% Asian, 83.1% White, 8.3% Biracial; 13.3% Hispanic or Latinx). Inclusion criteria were being a CU student aged 18 years or older and speaking fluent English (self-reported); the sample reported here is specific to undergraduates. The study was approved by the Institutional Review Board at CU Boulder, and all participants provided informed consent and were compensated with Amazon gift cards for their time.

Procedures

Brandeis University

Data were collected as part of an online longitudinal study across 8 weeks of the COVID-19 pandemic (April–May 2020). Data from the first time-point of the study were used to test the research questions described above. At this time-point, participants completed a Qualtrics survey with questionnaires assessing stress, coping, and psychopathology during the COVID-19 pandemic. Participants also completed additional questionnaires which were not included at the CU Boulder site and/or were not relevant to the current study’s hypotheses, and will be reported elsewhere.

University of Colorado Boulder

Data were collected as part of an online longitudinal study across 8 weeks of the COVID-19 pandemic (April–May 2020). Data from the first time-point of the study were used to test the research questions described above. Participants were later randomized into control and intervention arms of the study, but baseline data were collected prior to any intervention. At this time-point, participants completed a Redcap survey with questionnaires assessing stress, coping, and psychopathology during the COVID-19 pandemic. Participants also completed additional questionnaires which were not included at the Brandeis site and/or were not relevant to the current study’s hypotheses, and will be reported elsewhere.

Measures

COVID-19 Stress Questionnaire (CSQ, see Appendix A)

We created items to assess 12 stressors related to the COVID-19 pandemic (e.g., stress related to school disruption). Participants were asked if they experienced the event as a result of the COVID-19 pandemic, and if they have: how often the event happened, how stressful it was, and how controllable it felt. The CU site excluded the “how often the event has happened” item in the CSQ by error. Thus, for our main analyses, the stress experience of participants was measured by the average severity (“how stressful it was”; 1 (“not very stressful”) to 5 (“very stressful”)) of the 12 stressors. See Supplementary Materials for model results using the stressor frequency data on the Brandeis subsample (for which frequency ratings were available).

Mood and Anxiety Symptoms Questionnaire (MASQ, Watson et al., 1995b)

The MASQ evaluates symptoms of anxious arousal and anhedonic depression. In the current study, the CU site used an abbreviated 24-item version, which omits the low positive affect subscale; thus, the analyses here use only those items for consistency across sites. The 24-item version assesses anhedonic depression with the loss of interest subscale (LI; 8 items; e.g., “Felt really bored”). The anxious arousal subscale (AAr; 17 items) assesses symptoms of panic and physiological hyperarousal (e.g., “Hands were shaky). CU Boulder participants rated each item on how much they felt or experienced it during the past two weeks, while Brandeis participants rated each item on how much they felt or experienced it during the worst two weeks since the pandemic began, from 1 (“not at all”) to 5 (“extremely”). It has good internal consistency, test–retest reliability, and convergent and discriminant validity in relation to depression and anxiety disorders (e.g., Bredemeier et al., 2010; Nitschke et al., 2001; Watson et al., 1995a).

Penn State Worry Questionnaire (PSWQ, Meyer et al., 1990)

The PSWQ is a 16-item questionnaire assessing the degree to which an individual worries. CU Boulder participants were asked to indicate how true each statement is for them, while Brandeis participants were asked to indicate how true each statement was for them since the pandemic began (e.g., “Many situations make me worry”) from 1 (never true) to 4 (always true). It has good internal consistency, test–retest reliability, and convergent and discriminant validity in relation to anxiety disorders (e.g., Brown et al., 1992; Kertz et al., 2014; Meyer et al., 1990).

Ways of Coping Scale (WOCS, Folkman et al., 1986a, b)

The WOCS is a 66-item questionnaire containing a wide range of thoughts and acts that people use to deal with the internal and/or external demands of specific stressful encounters. For the current analyses, we used the escape/avoidance (e.g., “Had fantasies or wishes about how things might turn out.”), planful problem solving (e.g., “I made a plan of action and followed it”), positive reappraisal (e.g., “Changed or grew as a person in a good way.”), and seeking social support (e.g., “Accepted sympathy and understanding from someone.”) subscales only, as the remaining subscales (e.g., taking responsibility) are less applicable to the pandemic. Items are rated from (0) not used to (3) used a great deal. The scale instructions were modified: Brandeis participants were asked to indicate the extent that they used each item in response to the COVID-19 pandemic, while CU Boulder participants were asked to indicate the extent that they used each item in response to the COVID-19 public health stressor in the past two weeks (the WOCS is designed to be customized to ask about responses to a particular stressor). It has good internal consistency, and convergent and discriminant validity (Clark et al., 1995; Folkman & Lazarus, 1988; Greenaway et al., 2015).

Intolerance of Uncertainty Inventory (IUI, Gosselin et al., 2008)

The IUI is a 45-item questionnaire that measures an unacceptability of uncertainty and negative manifestations of uncertainty. CU Boulder participants rated how typical each item is of them, while Brandeis participants rated how typical each item is of them since the pandemic began (e.g., “I have difficulty tolerating life’s uncertainties”) from 1 (“not at all typical”) to 5 (“very typical”). The current analyses use only Part A, which assesses overall intolerance of uncertainty (Part B assesses different responses to uncertainty). It has good internal consistency, test–retest reliability, and convergent and discriminant validity (Gosselin et al., 2008; Lauriola et al., 2018).

Ruminative Responses Scale (RRS, Treynor et al., 2003)

The RRS is a questionnaire assessing the degree to which an individual ruminates when feeling sad. Participants completed only the 5-item brooding subscale. CU Boulder participants rated how often they generally do each item, while Brandeis participants rated how often they generally do each item since the pandemic began (e.g., “Think about a recent situation, wishing it had gone better”) from 1 (“almost never”) to 4 (“almost always”). The brooding subscale has good internal consistency, test–retest reliability, and convergent and discriminant validity (Griffith & Raes, 2015; Treynor et al., 2003).

Data Analysis

Measurement Models

To assess general and specific internalizing symptom dimensions, we fit a bifactor confirmatory factor analysis (CFA) measurement model to the MASQ and PSWQ, following a model developed in a prior study (Banich et al., 2020). The bifactor model parses symptom covariance into a common factor on which all indicators load, capturing what is shared across symptom dimensions (common internalizing psychopathology), and unique, orthogonal, factors on which each scale/subscale loads, capturing what is specific to each symptom dimension (MASQ anxious arousal [AAr] and loss of interest [LI], and PSWQ anxious apprehension [AAp]), after accounting for the common factor. Factors were constrained to be orthogonal to one another because what is shared between factors is already captured by the common factor (e.g., Chen et al., 2012). We compared the model fit of the bifactor model to simpler one factor and correlated factor models. However, we emphasize that model fit was not our primary criteria for choosing between models. Rather, we selected bifactor models a priori based on their suitability for addressing our research questions as described above.

To reduce multiple comparisons and examine links with overall use of more adaptive versus maladaptive response styles, we also fit CFA models for response styles factors. The RRS brooding, IUI Part A and WOCS escape/avoidance subscale were loaded on a maladaptive response styles factor, and the WOCS planful problem-solving, seeking social support and positive reappraisal subscales on an adaptive response styles factor, and these factors were allowed to correlate (note that model fit cannot be tested for each factor individually as they are each just-identified with three indicators).

CFAs were conducted in Mplus version 8 using full information maximum likelihood estimation. As χ2 is sensitive to sample size, good model fit was defined as: CFI > .95, RMSEA < .06 and SRMR < .08 (Hu & Bentler, 1999). If good fit was not achieved, modification indices were examined, and residual correlations between indicators were added if they were justified based on similar item content/wording. Statistical indices of factor reliability and validity (Omega (ω), ECV, H) were also calculated (Rodriguez et al., 2016). We tested invariance of the final models across samples (Brandeis and CU Boulder). Because χ2 difference tests can be significant with large samples even when the absolute differences between model estimates are marginal in size (Cheung & Rensvold, 2002; Meade et al., 2008), we also compared model fit between unconstrained and constrained models, where △CFI less than or equal to .01, and △RMSEA less than or equal to .015 suggests measurement invariance (Chen, 2007; Cheung & Rensvold, 2002; Meade et al., 2008). If our models showed at least metric invariance, we proceeded with using the full combined sample for SEMs.

SEM Main Effect and Moderation Models

We estimated SEM models with stress severity, response styles factors (maladaptive factor only, adaptive factor only, and combined models), and their interactions predicting factors in the bifactor internalizing model, controlling for age, gender, and site location. Exploratory analyses repeated these analyses with individual response styles (e.g., brooding). Models were tested using the residual method (Koch et al., 2018), which provides unbiased estimates of the relation between the predictor and the general factor, free of influences of the specific factors, and vice versa. We used factor scores for the response styles factors and residual predictor variables to enable the residual method to be used in models including interaction terms. We corrected for multiple comparisons using the two-stage sharpened method of FDR correction with two thresholds: within each of the models (hypothesis-wise correction) and across all models (excluding exploratory models; study-wise correction).

Regularized Regression Model

Finally, we used a data-driven approach (LASSO) to determine the most parsimonious set of variables for predicting each internalizing symptom factor. To identify the optimal (sparse) collection of predictors that maximizes predictive power, we performed predictor selection and regularization using the glinternet 1.0.10 package in R 3.5.1. This package uses a form of group LASSO (least absolute shrinkage and selection operator), in which related covariates (here, main effect and interaction terms) are selected as a single unit for subset selection or shrinkage. One-hundred model iterations were tested using tenfold cross-validation to select the optimal set of predictors. Of note, the glinternet package requires strong hierarchy, i.e., main effects must be significant for interaction terms to be selected to survive in the model, since these terms are only interactions in the presence of the main effects.

Results

Data and analysis syntax is available at [https://osf.io/fbvmx/].

Descriptive Statistics

Descriptive statistics, internal consistencies, and correlations of study variables are provided in Tables 1 and 2. Participants at CU Boulder were older and had higher scores on the MASQ loss of interest subscale and the WOCS positive reappraisal subscale compared to participants at Brandeis University; no other variables differed significantly across sites (Table S1).

Table 1 Descriptive Statistics
Full size table
Table 2 Bivariate Correlations
Full size table

Internalizing Bifactor Measurement Model

The internalizing bifactor model (Table S2) had good-to-adequate fit (CFI = .941, RMSEA = .035, SRMR = .047, χ2(694) = 1066.35, p < .001). All indicators loaded significantly onto their factors except for one indicator on the common internalizing factor (MASQ item 21) and two indicators on the anxious apprehension factor (PSWQ items 14 and 15). Six significant residual correlations were added between questionnaire items with similar content (e.g., “Hands were shaky” and “Was trembling or shaking”). The bifactor model achieved metric invariance across site location (△χ2(76) = 59.98, p = .911), thus one model with participants across both sites was used. The bifactor model was better fitting than one factor and correlated factors models based on AIC and BIC indices. Omega reliability (ω) coefficients were high for all factors (ωs = .818–.943). The Omega hierarchical (ωH) value, or the proportion of reliable variance in total scores explained by the common factor, was moderate for the common internalizing factor (ωH = .643), indicating that there is substantial common variance but that the measures cannot be considered unidimensional (Reise et al., 2013). Omega hierarchical specific values (ωHS), or the proportion of reliable variance in scale scores due to specific factors after controlling for the common factor, were moderate for the anxious arousal (ωHS = .627) and loss of interest (ωHS = .706) factors and lower for the anxious apprehension factor (ωHS = .279), indicating that much of the reliable variance in the anxious apprehension items was accounted for by the common factor. Likewise, the explained common variance (ECV), or the proportion of common variance explained by each factor relative to all explained variance of the items that load onto that factor, indicated an ECV of 48.5% for the common internalizing factor, 72.0% for the anxious arousal factor, 84.2% for the loss of interest factor and 29.9% for the anxious apprehension factor. Thus, while all the internalizing dimensions had both common and specific variance, anxious arousal and loss of interest had more specific variance, whereas anxious apprehension variance was more strongly accounted for by the common factor. The H values, which indicate the reliability and replicability of a factor, were above the minimum benchmark of .70 (Rodriguez et al., 2016) for all factors (Hs = .768–.942; Table S3).

Response Styles Measurement Model

The response styles correlated factors model (Table S4) had good-to-adequate fit (CFI = .954, RMSEA = .073, SRMR = .054, \({\chi }^{2}\)(15) = 431.78, p < .001). All indicators loaded significantly onto their factors, and the maladaptive and adaptive factors were significantly correlated (r = .20, p = .013). The response style model achieved metric invariance across site location (\(\Delta {\chi }^{2}\)(4) = 1.97, p = .742), thus one model with participants across both sites was used.

Structural Equation Modeling

See Supplemental Materials for full model output of the SEMs reported here (Tables S5–S8), for results using manifest sum scores as the dependent variables, and for results with response styles factors/individual response styles in separate models, which found similar results to the main analyses reported here.

Response Styles Factor Models (Fig. 1)

Fig. 1
figure 1

Response Styles Factor Model. Note. ***p < .001, **p < .01, *p < .05. Associations between response styles factors and perceived stress severity and internalizing factors in the same model. Main effect model is shown

Full size image

With both response styles factors and stress severity in one model, stress severity was significantly associated with the common internalizing (β = 0.228, p < .001), anxious arousal-specific (β = 0.170 p = .009), and loss of interest-specific (β = 0.303, p < .001) factors, but not anxious apprehension-specific (β = 0.037, p = .590) factor. The maladaptive response styles factor was significantly associated with the common internalizing (β = 0.637, p < .001), anxious arousal-specific (β = 0.168, p = .002), anxious apprehension-specific (β = 0.188, p = .017), and loss of interest-specific (β = 0.373, p < .001) factors. The adaptive response styles factor was significantly negatively associated with the loss of interest-specific factor only (β = –0.120, p = .012; all other ps > .055). The independent variables explained (R2) 55% of the common internalizing factor variance, 32% of the loss of interest-specific factor, 15% of the anxious apprehension-specific factor, and 8% of the anxious arousal-specific factor. All effects remained significant after FDR hypothesis-wise and study-wise corrections. Additionally, the adaptive response styles factor was significantly associated with the common internalizing factor after FDR hypothesis-wise correction (q = .027), although it was not significant after study-wise correction (q = .051). Adding the interaction terms, neither the maladaptive (ps > .105) nor the adaptive (ps > .198) response styles factor moderated the effect of stress severity on any of the internalizing factors.

Individual Response Styles Models (Fig. 2)

Fig. 2
figure 2

Individual Response Styles Model. Note. ***p < .001, **p < .01, *p < .05. Associations between individual response styles and perceived stress severity and internalizing factors in the same model. Main effect model is shown

Full size image

With all individual response styles and stress in one model, stress severity was significantly associated with the common internalizing (β = 0.219, p < .001), anxious arousal-specific (β = 0.163, p = .010), and loss of interest-specific (β = 0.287, p < .001) factors. Brooding was significantly associated with the common internalizing (β = 0.292, p < .001), anxious apprehension-specific (β = 0.141, p = .032), and loss of interest-specific (β = 0.170, p = .001) factors. Intolerance of uncertainty was significantly associated with the common internalizing (β = 0.398, p < .001), anxious apprehension-specific (β = 0.173, p = .009), and loss of interest-specific (β = 0.132, p = .005) factors. Escape/avoidance was significantly associated with the common internalizing (β = 0.163, p < .001), anxious arousal-specific (β = 0.200, p = .001), and loss of interest-specific (β = 0.355, p < .001) factors. Planful problem solving, positive reappraisal, and seeking social support were not associated with any factors (ps > .059). Multicollinearity was not a concern for any of the predictor variables (VIFs < 1.72). All significant effects remained after FDR hypothesis-wise correction.

Adding the interaction terms, intolerance of uncertainty significantly negatively moderated the effect of stress severity on the anxious apprehension-specific factor (β = –0.123, p = .039) and brooding positively moderated the effect of stress severity on the common internalizing factor (β = 0.08, p = .049). However, the moderation effects did not remain significant after FDR hypothesis-wise correction. No other response styles showed significant moderating effects (ps > .080). Multicollinearity was not a concern for any of the predictor variables (VIFs < 1.99).

LASSO Modeling (Fig. 3)

Fig. 3
figure 3

LASSO Scatterplots. Note. LASSO modeling identified (A) perceived stress severity, (B) intolerance of uncertainty, and (C) brooding as optimal predictors of the common internalizing factor, and (D) perceived stress severity and (E) escape/avoidance as optimal predictors of the loss of interest-specific factor

Full size image

All continuous variables were standardized, and missing values were imputed with the median. The LASSO models predicting the anxious apprehension-specific and anxious arousal-specific factors failed to converge (model inconsistency) when more than 10 model iterations were tested, so only predictors of the loss of interest-specific and common internalizing factors could be tested. For the loss of interest-specific factor, stress severity (β = 0.032) and escape/avoidance (β = 0.091) were selected as predictors. There were no other nonzero estimated coefficients for other main effects, or for any interactions. For the common internalizing factor, stress severity (β = 0.031), brooding (β = 0.138), and intolerance of uncertainty (β = 0.249) were identified as the optimal predictors. There were no other nonzero estimated coefficients for main or moderated effects.

To complement results from the LASSO analyses, standard multiple regression models were performed using the predictors selected in LASSO. In the first regression, the loss of interest-specific factor significantly predicted stress severity (β = 0.207, SE = .041, p < .001) and escape/avoidance (β = 0.264, SE = .041, p < .001). These predictors explained 17% of the loss of interest-specific factor variance (Adjusted R2). A second multiple regression confirmed that the common internalizing factor significantly predicted stress severity (β = 0.164, SE = .037, p < .001), brooding (β = 0.253, SE = .038, p < .001), and intolerance of uncertainty (β = 0.358, SE = .040, p < .001). These predictors explained 40% of the common internalizing factor variance (Adjusted R2).

Discussion

Overall, we found that both the perceived severity of COVID-19 pandemic stressors, and stress response styles, were associated with internalizing symptoms during the beginning of the pandemic as hypothesized, but that contrary to our hypotheses, response styles largely did not moderate stress effects. The findings support strong associations between internalizing symptoms and maladaptive response styles, weak associations with adaptive response styles, and medium-sized associations with stress severity.

The current study builds on past research finding that perceived stress is a risk factor for internalizing symptoms in undergraduates (e.g., Cohen & Wills, 1985; Zahniser & Conley, 2018) by testing how stress severity is associated with common and specific dimensions of depression and anxiety. Our results suggest that although perceived stress severity may be transdiagnostic, it is also uniquely associated with a loss of interest-specific factor and anxious arousal-specific factor. LASSO analyses also identified stress severity as an optimal predictor for the loss of interest-specific factor. Although stress severity was associated with anxious apprehension overall (based on the bivariate correlation), the lack of association with the anxious apprehension-specific factor suggests that the common internalizing factor accounts for this association and that perceived stress severity is not associated with a worry-specific component of internalizing symptoms.

The tripartite model suggests that the common internalizing factor captures general distress that is common across internalizing symptoms (Clark & Watson, 1991). Thus, stress severity may be associated with the distress that is common across depression and anxiety symptoms, but also additionally associated with loss of interest- and anxious arousal-specific factors potentially through separate, independent risk pathways. Future research is needed to test potential mediating processes, which may include neurophysiological (e.g., stress activating the sympathetic nervous system, leading to anxious arousal [e.g., Friedman et al., 1992]), behavioral (e.g., stress induced withdrawal, leading to loss of interest [Peterson et al., 2021; Prevot et al., 2019]) and cognitive (e.g., negative thinking about stressful events leading to general distress [e.g., Snyder et al., 2019]). However, it is important to note that since the current analyses are cross-sectional, they cannot differentiate between perceived stress severity as a risk factor versus consequence (i.e., those with greater internalizing symptoms may also perceive stressors as more stressful [e.g., Foland-Ross & Gotlib, 2012; Galaif et al., 2003]), or both.

Consistent with past work (e.g., McEvoy et al., 2019; McLaughlin & Nolen-Hoeksema, 2011; Snyder et al., 2019), we found that maladaptive response styles, and in particular brooding and intolerance of uncertainty, were strongly transdiagnostically associated with internalizing symptoms. Brooding and intolerance of uncertainty were not only identified as optimal predictors for the common internalizing factor in the LASSO analyses, but, in combination with stress severity, explained a large portion of its variance. The current results further clarify that these maladaptive response styles are associated with common internalizing as well as specific dimensions of internalizing symptoms above and beyond the association with the common internalizing factor.

Although past research did not address potential associations with specific dimensions of depression or anxiety, brooding in the context of loss may be more likely to contribute to the development of depression symptoms, while brooding in the context of threat may be more likely to develop anxiety symptoms (Nolen-Hoeksema & Watkins, 2011). During the pandemic, individuals have experienced significant loss (e.g., typical social interaction) and threats (e.g., health), which could explain why brooding has unique associations with both loss of interest- and anxious apprehension-specific factors. Additionally, brooding and worry have the same underlying process of repetitive negative thinking (Taylor & Snyder, 2021) that could further explain the association with the anxious apprehension-specific factor. Although brooding was correlated with anxious arousal, it was not associated (or had a weak association) with the anxious arousal-specific factor, likely because this association is captured by the common internalizing factor.

Similarly, the associations between intolerance of uncertainty and the loss of interest- and anxious apprehension-specific factors also suggest different risk mechanisms. Intolerance of uncertainty has been strongly linked to worry in previous research (e.g., Dugas et al., 2001), and may directly lead to higher anxious apprehension as individuals high in intolerance of uncertainty attempt to use worry to anticipate, and thus reduce uncertainty, about uncertain outcomes (Freeston et al., 1994), which were prevalent during this period of the pandemic. On the other hand, intolerance of uncertainty may be indirectly associated with loss of interest via the tendency to avoid more uncertain situations (Mahoney & McEvoy, 2012).

LASSO analyses indicated that escape/avoidance (and not intolerance of uncertainty or brooding) is an optimal predictor for the loss of interest-specific factor. In the main analyses, escape/avoidance was also associated with all internalizing factors except anxious apprehension-specific, suggesting it is transdiagnostic, but like other response styles, has multiple risk mechanisms. Avoiding thoughts and emotions related to stressors can lead to withdrawal and may subsequently lead to loss of interest (Spielberg et al., 2011), thus potentially explaining the association between escape/avoidance and the loss of interest-specific factor. Although individuals may use escape/avoidance because it provides relief, this strategy is often only partially helpful for acute stressors and the relief is only temporary (Aldao et al., 2010). Escape/avoidance may also generate more stressors, thus triggering more internalizing symptoms (Holahan et al., 2005), and further physiological symptoms via perceived stress severity. In the context of chronic stress, like the pandemic, escape/avoidance can make negative thoughts and affect worse, and potentially trigger physiological symptoms because individuals have not learned to adjust to stressors (Aldao et al., 2010). Conversely, escape/avoidance may not be associated with the anxious apprehension-specific factor because the association is captured by the common internalizing factor, though the bivariate correlation was also somewhat small, potentially because individuals are avoiding worrying thoughts when using escape/avoidance, but only partially succeeding in doing so. Like the stress associations, it is important to note that all these associations may be bidirectional, as in addition to maladaptive response styles being risk factors for internalizing psychopathology, internalizing psychopathology also prospectively predicts more maladaptive responses to stress (e.g., Whisman et al., 2020).

Unlike most of the associations with maladaptive response styles, the adaptive stress response style associations were weak and generally non-significant. Although the adaptive response styles factor was associated with lower loss of interest-specific factor, suggesting adaptive response styles may be protective, consistent with some prior research (e.g., Aldao et al., 2010; Heerde & Hemphill, 2018; Martin & Dahlen, 2005), the effect size was small and associations with all individual response styles were nonsignificant. These findings are consistent with other research (Aldao & Nolen-Hoeksema, 2010) that suggests these typical adaptive response styles are not particularly protective. In general, this may be because individuals are reporting using these strategies but may not be effective in their execution (Aldwin & Revenson, 1987). However, even if individuals are typically adept at certain adaptive coping strategies, unique pandemic stressors may have made this more difficult. The strategy-situation fit hypothesis proposes that strategies are potentially useful in some situations, but not others (Aldao et al., 2015; Folkman et al., 1986a, b). Reappraisal and problem solving may not have been useful strategies for pandemic stressors, or perhaps were more difficult to implement for situations that seem uncontrollable or lacking positive aspects.

On the other hand, seeking social support may demonstrate weak associations because it can be maladaptive as well as adaptive even in non-pandemic conditions. Social support has been linked to greater negative affect (e.g., Scholz et al., 2012), potentially because individuals may engage in excessive reassurance-seeking, potentially harming support providers’ willingness to help (Starr & Davila, 2008), and co-brooding (i.e., excessively discussing negative affect, thoughts, and consequences surrounding stressors with a partner), which can increase internalizing symptoms (Bastin et al., 2014). Thus, the harmful effects of social support may reduce its potential benefits. Although strategies considered to be adaptive in the current study did not exhibit protective effects, other strategies that we did not test might have been protective during the beginning of the pandemic, such as mindfulness (Blanck et al., 2018) which could be addressed in future research. Alternatively, more recent research suggests that rather than specific strategies being adaptive, how flexible individuals are in coping may contribute to mental health outcomes (Kato, 2020). The ability to flexibly switch between strategies when particular ones do not work may have been especially important for coping with rapid changes during the pandemic and unprecedented pandemic stressors.

Overall, results supported additive, rather than interactive, effects of stress and response styles, generally not supporting moderation effects found in prior research (e.g., Chen & Hong, 2010; Cox et al., 2012; D’Zurilla & Nezu, 2009; Felsten, 1998; Troy et al., 2010). However, the lack of moderation results is not entirely surprising given inconsistent moderation effects in past research (e.g., Burton et al., 2004; Zlomke & Jeter, 2014). Response style effects can also vary across different contexts (e.g., Aldao et al., 2015; Folkman et al., 1986a, b; Nolen-Hoeksema & Watkins, 2011), and lack of moderation effects could potentially be due to the unique situation the pandemic presented. Alternatively, it is also important to note that the current study assessed perceived stress severity cross-sectionally, which may have affected the results. Some theories propose that rather than response styles strengthening or weakening the effect of stress exposure on internalizing symptoms, coping responses change how individuals appraise stressors and it is these perceptions that buffer or exacerbate the effect of experiencing stressors on internalizing symptoms (e.g., Cohen & Wills, 1985). Evidence that perceived stress severity moderates the association between stress frequency and internalizing symptoms, namely anxiety symptoms (Fassett-Carman et al., 2019), supports this possibility. Thus, the effects of response styles in response to stress exposure may already be reflected in perceived stress severity, and thus do not further moderate its associations with internalizing symptoms. Although supplementary analyses suggest that these stress response styles also do not moderate associations between stressor frequency and internalizing symptoms, longitudinal analyses are needed to investigate whether stress response styles predict changes in perceived stress severity, and whether those changes moderate stressor frequency effects.

Limitations and Future Directions

Thus, because these analyses were cross-sectional, we could not address the extent to which stress severity and response styles increased or decreased internalizing symptoms during the pandemic versus how preexisting internalizing symptoms influenced coping and stress perceptions. Thus, future research should use longitudinal data to test for bidirectional effects and whether response styles contributed to changes in stress perceptions during the pandemic and changed stress perceptions strengthen or weaken the effect of stress occurrence or frequency on internalizing symptoms, in addition to applying this stress model to other high-stress situations. Future research would also benefit by including measures of coping flexibility, ability to effectively use strategies, and maladaptive components of social support (e.g., co-brooding) that may explain inconsistent findings in coping research. The present research also did not include a measure of low positive affect and was thus unable to address associations with all internalizing dimensions. The measures included in the study were also self-report and thus susceptible to biases and inaccuracy (e.g., Burton & Blair, 1991). Lastly, the sample was primarily White and only included students from two universities; the study also did not collect data on culture, geographic background, or socioeconomic status. Thus, the sample may not be representative of the general undergraduate population, non-student populations, and clinical populations.

Conclusions

This study provides evidence for strong additive associations between internalizing symptoms and both perceived stress severity and maladaptive response styles in undergraduates during the peak of the COVID-19 pandemic. The high percentage of variance that stress severity and maladaptive response styles explained overall in internalizing symptoms may have important clinical implications, suggesting that a relatively brief set of questionnaires may provide effective screening to identify individuals at risk in high-stress situations, and that maladaptive response styles may be important targets for prevention and intervention programs in these situations. Results also suggest that focusing only on the use of adaptive strategies (or at least simply encouraging their use, rather than training on their effective use) may not be particularly helpful in combating internalizing symptoms. This study also makes an important contribution to understanding how response styles are transdiagnostically associated with internalizing symptoms, with some response styles associated with common and specific internalizing dimensions, suggesting both converging and divergent risk pathways which can be probed in future research.