1 Introduction

The spread of COVID-19 across the U.S. during Spring 2020, caused many states to enact unconventional policies as a means of limiting the impact of the virus, a process that has become colloquially known as “flattening the curve.” Of these new policies, the most controversial has been statewide stay-at-home (SAH) orders, which effectively shutters non-essential businesses and severely limits social interaction. Between mid-March and early April 2020, 43 states implemented statewide SAH orders, while 7 never implemented such orders.

The goal of this paper is to estimate the treatment effects of SAH orders on economic conditions, focusing attention on the change in weekly new jobless claims in the period immediately following the adoption of the policy.Footnote 1 The connection between statewide SAH orders and the labor market is natural. Many opponents of the policy express concerns that such restrictions may severely hurt the labor market, increase unemployment, and result in economic costs that out-weight the health benefits.Footnote 2 If this is true, then we should observe large spikes in new jobless claims around the time SAH orders are enacted. However, it is also possible that other factors associated with the spread of COVID-19, such as a rapid increase in the number of infections and a general reduction in consumer confidence, may also result in a sharp uptick in new jobless claims around the same time. As such, determining the treatment effects of SAH orders on the labor market requires careful consideration of both effects.

In this paper, we use the synthetic control method (our primary specification) to determine the impact of SAH orders on new jobless claims. The synthetic control method developed by Abadie and Gardeazabal (2003) and Abadie et al. (2010), by combining elements from matching and difference-in-differences (DiD) techniques, is able to provide reliable estimates of treatment effects in much more general contexts.Footnote 3 The main advantage of the synthetic control method is that it estimates the treatment effect by creating a hypothetical or “synthetic” control unit, a weighted average of all other untreated units (donor pool), that resembles the treatment unit in the pre-treatment period. This avoids choosing any individual unit from the untreated group to match the treated unit. As for our research question, the presence of 7 states that never implemented SAH orders provides a natural donor pool from which we can construct a synthetic control unit to best match each of the 43 states with SAH orders. Our findings suggest that the negative impact of SAH orders on the labor market may be overstated. Specifically, while we do observe a spike in new jobless claims following the start of SAH orders, we observe a similar spike in the synthetic counterfactual states where no policy was implemented. In short, our findings suggest that over two-thirds of the spike in new jobless claims observed in the data is a direct result of the virus, not the policy response. We also re-estimate our treatment effects using DiD and find consistent results.

Given the importance and topical nature of the research question, there are several recent working and published papers that directly relate to our present work. While our paper focuses on new jobless claims as the variable of interest, several previous works have examined the economic impact of COVID-19 and the subsequent policy responses using output as a primary measure (e.g., Luo and Tsang 2020 and Makridis and Hartley 2020). One exception is the work of Baek et al. (2021), which also estimates the causal effects of SAH orders on unemployment insurance claims. The authors construct an employment-weighted measure of the duration of exposure to SAH orders for each state using county-level data and estimate the marginal effect of an additional week of SAH exposure on state-level weekly initial new jobless claims. While the present paper and Baek et al. (2021) share a similar research question, the methodologies employed differ in three important ways: (1) (Baek et al. 2021) estimate the marginal effect of an additional week of SAH exposure, while we focus on the implementation of the SAH order as an event or treatment. (2) While (Baek et al. 2021) include important controls in their analysis, their main empirical specification still relies on a state-level cross-sectional regression. In contrast, we use the synthetic control method, which provides substantial advantages as a research design in social sciences with the aim to estimate the causal effects of aggregate interventions; see Abadie (2021). (3) Baek et al. (2021) use an employment-weighted measure of the duration of exposure to SAH orders constructed from county-level data, which allows them to consider the impact of both county-level and state-level SAH orders. In contrast, we focus on state-level policy interventions that are universal for all counties in a state. Interestingly, while the two papers approach the research question in very different ways, both reach a similar conclusion. Specifically, our results suggest that SAH orders account for only about 32 percent of the increase in new jobless claims during the early stages of the COVID-19 pandemic, while Baek et al. (2021) find that SAH orders account for only 24 percent of the increase. The fact that two studies using very different empirical strategies, reach similar conclusions regarding the impact of SAH orders on the labor market provides additional support to the individual findings.

2 Data and method

In response to COVID-19, most U.S. states issued statewide SAH orders. Table 1 shows the start date of the order for each state. To estimate the impact of SAH orders on the number of jobless claims, as a percentage of total population of each state, we adapt the synthetic control method proposed by Abadie and Gardeazabal (2003) and Abadie et al. (2010). The method has been applied to the fields of political science, health policy, criminology, and economics to evaluate the effect of a treatment. It involves the construction of a synthetic control, which is a weighted combination of units used as potential controls, to which the treated unit is compared.

We borrow the notation from Abadie (2021) and briefly illustrate the synthetic control method here. Let j = 1 denote the unit affected by the policy intervention of interest and j = 2,...,J + 1 denote the donor pool, i.e., a set of potential comparisons that are not affected by the intervention. The sample spans T periods with the first T0 periods prior to the intervention. For each unit j = 1,...,J + 1 and time t = 1,...,T, we observe the outcome variable of interest, Yjt. We also observe k predictors of the outcome, denoted X1j,...,Xkj, for each unit j. Note that these predictors may include the pre-treatment values of the outcome variable. Let Xj, j = 1,...,J + 1, denote the k × 1 vector of the predictors for the unit j and X0 denote the matrix collection of the predictors for the J untreated units, i.e., X0 = [X2,...,XJ+ 1]. The treatment effect of interest for the affected unit j = 1 in period t, with t > T0, is defined as:

$$\tau_{1t}=Y_{1t}-Y_{1t}^{N},$$
(1)

where Y1t is observed and \(Y_{1t}^{N}\) is the potential response of unit j in the absence of the treatment, which is not observed.

Table 1 Statewide SAH orders in response to COVID-19
Full size table

The synthetic control method approximates the unobserved outcome \(Y_{1t}^{N}\) of the treated unit using a weighted average of the units in the donor pool, i.e.,

$$\widehat{Y}_{1t}^{N}=\sum\limits_{j=2}^{J+1} w_{j} Y_{jt},$$
(2)

where W = (w2,...,wJ+ 1)′ denotes a J × 1 vector of non-negative weights. The weights, which sum to one, are chosen so that the resulting synthetic control best reproduces the values of the predictors for the treated unit, i.e., to minimize the following distance:

$$||\mathbf{X}_{1}-\mathbf{X}_{0}\mathbf{W}|| = \left(\sum\limits_{h=1}^{k} v_{h} \left(X_{h1}-w_{2}X_{h2}-...-w_{J+1}X_{hJ+1}\right)^{2}\right)^{1/2},$$
(3)

where the positive constants v1,...,vk reflect the relative importance of each of the k predictors. While each potential choice of V = (v1,...,vk) produces a synthetic control, these values are usually selected to minimize the mean squared prediction error of the synthetic control with respect to \(Y_{1t}^{N}\) for the pre-treatment periods, \(\frac {1}{T_{0}}{\sum }_{t=1,...,T_{0}}\left (Y_{1t}-w_{2}(\mathbf {V})Y_{2t}-...-w_{J+1}(\mathbf {V})Y_{J+1t}\right )^{2}\).

Table 2 Variables and definitions
Full size table

We refer to the treatment as the implementation of a statewide SAH order. The donor pool includes the 7 states that have never implemented such orders, namely Arkansas, Iowa, Nebraska, North Dakota, South Dakota, Utah, and Wyoming. For each of the 43 states that have issued SAH orders, we construct the synthetic state as the convex combination of states in the donor pool that most closely resembles the focal state in terms of the pre-treatment outcomes, averaged over each quarter,Footnote 4 and a set of predictor variables. The predictor variables chosen for our analysis follow from Partridge (1997) and include 6 demographic, 2 institutional, and 10 market variables which are among the factors that underline the dispersion in U.S. state unemployment (see Table 2 for a list of these variables). State-level data on weekly initial claims for unemployment insurance from the beginning of 2018 to April 25, 2020 are obtained from the U.S. Department of Labor.Footnote 5 The 2018 annual data on demographic variables are obtained from the U.S. Census Bureau and those on market variables are obtained from the Bureau of Economic Analysis. The 2019 annual data on institutional variables are obtained from the U.S. Bureau of Labor Statistics and the U.S. Department of Labor.

3 Empirical results

3.1 Main results

The synthetic control algorithm estimates the missing counterfactual as a weighted average of the outcomes of potential controls. We refer to the outcome as the number of jobless claims as a percentage of the population of each state. The weights are chosen so that the pre-treatment average outcomes and the covariates of the synthetic control are very similar to those of the treated state. Table 3 presents the average jobless claims (with standard deviations in parentheses) for each focal state and its synthetic counterpart before and after the SAH order. Table 3 also reports the weights of untreated states in the donor pool of the synthetic control for each treated state.

Before the start of an SAH order, the synthetic state resembles each focal state very well, with the exception of Pennsylvania; the difference in the average jobless claims between the treated state and its synthetic counterpart varies between -0.04 (New Hampshire) and 0.05 (Alaska and Rhode Island), with an average of 0.01. The number of jobless claims significantly increases after the SAH order. For example, the number in Washington, the state first hit by COVID-19, increases from 0.10 to 1.92. New York, a state with an extreme level of cases, experiences an increase from 0.08 to 1.37. However, the increase in the jobless claims is not merely driven by the implementation of the SAH order. Even without such an order, there would have been an increase in new jobless claims as a direct result of the pandemic. For example, the increase in synthetic Washington is from 0.07 before the start of the SAH order to 1.06 after and the increase in synthetic New York is from 0.06 to 1.10. Figure 1 shows the weekly jobless claims in actual and synthetic New York and Washington over the entire sample period.Footnote 6 The vertical dotted line denotes the start date of an SAH order. The figure clearly shows that each synthetic state well resembles its actual counterpart prior to the start of an SAH order. After the treatment, a small fraction of the increase in jobless claims in the state of Washington is driven by the SAH order, and this fraction is even smaller for the state of New York. These findings suggest that there would have been an a sizable increase in new jobless claims even without a SAH order. On average, a state experiences an increase of 1.45, as a percentage of the total state population, in average jobless claims after the start of a SAH order (the difference between columns 4 and 1 in Table 3), whereas the increase in the synthetic state is 0.98 (the difference between columns 5 and 2 in Table 3). This latter number provides a lower bound for the counterfactual change in the number of jobless claims if the state had not imposed a SAH order because, in the absence of such an order, the state would have had more jobless claims with a faster spread of the virus. The impact of SAH orders on jobless claims is therefore expected to be smaller than the difference of these two numbers, 0.46, which accounts for less than 32% of the observed change.Footnote 7 Using different levels of aggregation and vastly different empirical methodologies, we reach effectively the same conclusion as Baek et al. (2021) that the direct effect of SAH orders accounts for a minority share of the overall rise in unemployment benefit claims.

Table 3 Average jobless claims before and after the SAH order
Full size table
Fig. 1
figure 1

Average jobless claims in actual and synthetic New York and Washington

Full size image

3.2 Placebo test

In this section, we conduct a placebo test to provide further evidence in support of our estimated treatment effects reported in Table 3. For this placebo test, we effectively flip the assignment of donor and treated states and follow the procedure outlined above. Given that SAH orders were never implemented in the placebo, hypothetical SAH orders are assumed to start on April 1st, 2020. Table 4 presents the results of this placebo test which shows that each synthetic control well resembles the associated hypothetical treated unit in the pre-treatment period. On average, the difference between Y_treated and Y_synth is zero over the entire pre-treatment period. In the post-treatment period, the discrepancy between Y_treated and Y_synth is also small, with an average of 0.02. These results confirm that our estimated treatment effects for states that have issued SAH orders are significantly large relative to that obtained when we apply the same analysis to the states in the donor pool.

Table 4 Placebo test
Full size table

3.3 DiD robustness check

While we use the synthetic control method to estimate the impact of SAH orders on new jobless claims, other empirical strategies could be implemented. In the section, we consider a robustness check where we re-estimate the treatment effects using the DiD method from the following specification:

$$Y_{it}=\alpha+\beta\text{Treated}_{i} + \gamma\text{After}_{t} + \delta \text{Treated}_{i}\times \text{After}_{t} + {\Theta}\textbf{X}_{it} + \epsilon_{it},$$
(4)

where Yit denotes new job less claims in state i at time t, Treatedi is a dummy variable that equals one for a treated state and zero otherwise, Aftert is a dummy variable that equals one in the post-treatment periods and zero otherwise, and Xit is a vector of control variables, including the demographic, institutional, and market variables listed in Table 2. The parameter γ captures the change in the outcome of untreated units and δ captures the treatment effects on the treated units.

Table 5 presents the DiD estimation results. On average, treated states experience an increase in new jobless claims of 1.45 following an SAH order. However, new jobless claims also increase by about 1.35 in untreated states without SAH orders. The treatment effects are estimated to be 0.09, which only accounts for 6.3% of the observed change in treated states. Therefore, the DiD estimates are consistent with the synthetic control estimates presented in Table 3, suggesting that most of the increase in new-jobless claims during the early stages of the COVID-19 pandemic followed from factors outside of SAH orders. In fact, the DiD estimates suggest an even smaller economic cost of SAH orders.

Table 5 Robustness check
Full size table

4 Concluding remarks

The COVID-19 pandemic forced policymakers to consider adopting policies that simultaneously limited the spread of the virus and reduced economic activity. Understanding the extent of these trade-offs is essential for improving our response to subsequent public health crisis, should one arise. In this paper, we focus on measuring the economic costs associated with non-pharmaceutical interventions, specifically the relationship between statewide SAH orders and new jobless claims during the early phase of the COVID-19 pandemic. Our findings suggest that statewide SAH orders account for only 32 percent of the increase in new jobless claims during the beginning phase of the pandemic, with the remainder of the increase being driven by other factors.

Our finding is remarkably close to that reported in Baek et al. (2021). Specifically, Baek et al. (2021) also estimate that statewide SAH orders only account for about 24 percent of the increase in new jobless claims observed in the early phase of the COVID-19 pandemic, which is very close to our estimate of 32 percent. However, we use a synthetic control approach to recover a causal estimate of the impact of SAH orders, while Baek et al. (2021) consider the variation across both space (e.g., states) and the duration of exposure (e.g., duration under SAH orders) to determine the cause impact of the policy. The fact that two very different empirical strategies reach very similar conclusions regarding the economic impact of statewide SAH orders provides additional support in favor of our findings. While our present analysis does not measure the health benefits of SAH orders, others have found large positive effects. Taken together, these findings point to the efficacy of statewide SAH orders as a means to significantly limit the spread of the virus while only moderately increasing new jobless claims.