Assessing the Impacts of Birmingham’s Clean Air Zone on Air Quality: Estimates from a Machine Learning and Synthetic Control Approach

Abstract

We apply a two-step data driven approach to determine the causal impact of the clean air zone (CAZ) policy on air quality in Birmingham, UK. Levels of NO₂, NO_x and PM_2.5 before and after CAZ implementation were collected from automatic air quality monitoring sites both within and outside the CAZ. We apply a unique combination of two recent methods: (1) a random forest machine learning method to strip out the effects of meteorological conditions on air pollution levels, and then (2) the Augmented Synthetic Control Method (ASCM) on the de-weathered air pollution data to isolate the causal effect of the CAZ. We find that, during the first year following the formal policy implementation, the CAZ led to significant but modest reductions of NO₂ and NO_X levels measured at the roadside within (up to 3.4% and 5.4% of NO₂ and NO_X, respectively) and outside (up to 6.6% and 11.9%) the zone, with no detectable changes at the urban background site outside the CAZ. No significant impacts of the CAZ were found on concentrations of fine particulates (PM_2.5). Our analysis demonstrates the short-term effectiveness of CAZ in reducing concentrations of NO₂ and NO_X.

1 Introduction

Air pollution is one of the largest risks to the health of urban populations. Short and long-term exposure to air pollution has been documented to negatively affect human health, including increased risk of acute and chronic disease outcomes and reduced life expectancy (Chen and Hoek 2020; Orellano et al. 2020). As the urban population continues to grow it becomes increasingly important to mitigate urban air pollution, and traffic related emissions are a key policy target for governments trying to improve air quality. Widely applied interventions intended to reduce traffic-related air pollution emissions include Low Emission Zones (LEZs) or Clean Air Zones (CAZs), which relate in general terms to restrictions on, or charges for, the use of specified vehicles in particular areas of cities. Restrictions may be applied to different vehicle types, ages, fuel types, or emissions classifications (such as the Euro standards). A number of cities have implemented CAZs in the last thirty years including Munich, Milan and Amsterdam, with mixed results (Holman et al. 2015).

On 1st June 2021, Birmingham City Council introduced a Clean Air Zone policy around central Birmingham. Birmingham became the second major UK city to implement such a policy at scale, after London’s Low Emission Zone in 2008 (and Ultra Low Emission Zone in 2019).¹ The Birmingham Clean Air Zone refers to ‘a designated area where heavily polluting vehicles are discouraged from entering’ (Birmingham City Council).Although no vehicles are banned from entering the zone, drivers of non-compliant vehicles must pay daily charges depending on vehicle type; for example, for cars and light goods vehicles that do not meet the required emission standards the charge is £8 per day while the charge for non-compliant coaches and HGVs is £50 per day.²These charges are applied and operate 24 h a day, 7 days a week, with a range of exemptions for certain users, resident groups and others, dependent upon status and income. The Birmingham Clean Air Zone is a ‘Class D’ policy meaning it is the most restrictive under current UK national legislation (see Table A1 for details).

Fig. 1

The “gap” plot- the difference between weather normalised NO₂ and NO_X and their synthetic counterfactuals for 8 Birmingham sites. Note: the plot consists of weekly data points between 1 Jan 2021 to 31 May 2022 for each site, (data for Acocks Green site only available until 30 Sep 2021), the vertical line refers to the week starting on 24 May 2021. The shaded area illustrates the 95% Jackknife + Confidence Interval calculated from the Ridge ASCM. Roadside sites within CAZ include St Chads Queensway, Colmore Row, Lower Severn Street, urban site within CAZ site include Ladywood. Roadside sites outside CAZ include A4540 Roadside, Selly Oak, Stratford Road, urban site outside CAZ include Acocks Green

Table 1 The estimated CAZ impact on NO₂ at eight Birmingham sites

The Birmingham CAZ was introduced following a ministerial directive to Birmingham City Council, due to its breaches of the air quality objective (AQO) for annual mean concentrations of nitrogen dioxide (NO₂) (40 μg m⁻³). A class D (all vehicle charging) CAZ was mandated as necessary to bring NO₂ levels into compliance with the AQO in the shortest possible time. Now that the CAZ has been implemented it is important to understand its efficacy on air quality.

The purpose of this paper is to develop and apply data-driven techniques based on machine learning and synthetic control methods to assess the impact of the introduction of the CAZ on air quality in Birmingham (as measured at the available fixed monitoring stations) from June 2021 to May 2022. First, the paper assesses the impacts of the CAZ on concentrations of specific air pollutants. Second, it investigates whether CAZ had different impacts at different air quality monitoring sites (i.e., urban/roadside). Third, we investigate whether there is evidence from the available automated air pollution monitors for CAZ spillover effects which may arise if air pollution is displaced from within the CAZ to areas outside the CAZ, rather than simply reducing total traffic emissions.

Our study answers these questions by applying an interdisciplinary approach based on combining weather normalisation techniques (a random forest-based Machine Learning algorithm), as developed by Atmospheric Scientists, and the Augmented Synthetic Control Method (ASCM) more familiar to econometricians. This paper provides the first causal estimation of the impacts of the CAZ intervention on air quality in Birmingham. Using hourly measured pollution and meteorological data from 39 surface monitoring sites across the UK between 1 Jan 2021 and 31 May 2022, we first apply a weather normalisation technique (Grange and Carslaw, 2019, Shi et al. 2021) to ‘deweather’ the observed air pollutant concentrations. Second, under the ‘quasi-natural experiment’ design, we apply the newly developed ASCM approach (Ben-Michael et al. 2021) to simulate the counterfactual air quality trend for each monitoring site in Birmingham (treatment group) using sites from other UK cities, with no CAZ, as the control group.

To briefly summarise our results, we find statistically significant reductions in NO₂ and NO_x (up to 4% and 7% reductions respectively) over the first 7-months (June to Dec 2021) post implementation period at two out of three roadside sites within the CAZ. As expected, the biggest reductions in NO₂ and NO_x were at busy roadside locations within the CAZ area, while no statistically significant changes were detected for NO₂ and NO_x levels at the urban background sites outside of the CAZ. No significant impacts were detected for PM_2.5. We also find reductions in NO₂ levels at the roadside site located outside the CAZ suggesting that rather than displacing traffic to areas outside the CAZ there are potential short term positive spill-over effects, such as behavioural changes that also contribute to reduced air pollution in surrounding areas.

Determining the effectiveness of urban air pollution control policies, like a Clean Air Zone or Low Emission Zone, is challenging due to the many confounding factors (Holman et al. 2015). For example, observed air pollutant concentrations are co-influenced simultaneously by emission strength, atmospheric chemical processes, and meteorological factors such as wind direction, wind speed, pressure, precipitation, temperature, and humidity, including non-linear effects and interaction effects between these variables. The presence of these varied and often site-specific meteorological effects considerably complicate policy evaluation efforts (Cole et al. 2020).

Air pollution levels in a city can also be affected by a range of other factors such as changes in traffic volumes, in fleet composition, economic policies, social trends and (during this time period) COVID-19 effects on social and economic activity. Decoupling the direct and causal impacts of a CAZ intervention from other confounding polices can be difficult and may lead to erroneous conclusions. It is difficult, or can be wholly misleading, to draw robust inferences regarding the effect of a measure such as CAZ implementation on emissions from simple comparison of ‘before and after’ pollution levels. Similarly, even if a more sophisticated Differences-in-Differences approach is used, failing to adequately control for the complex and varied effects of meteorological factors can provide equally unreliable results. The data-driven method used in our paper addresses these challenges providing robust and reliable quantification of the CAZ evaluation that can also be applied to the evaluation of other environmental policies.

In this context, the contribution of this paper is twofold: (1) we provide the first comprehensive evaluation of the Birmingham Clean Air Zone, an internationally significant policy to improve urban air quality in the UK’s second largest city; (2) we illustrate the benefits of applying a two-step methodological approach to isolate the causal effect of clean air zone policy implementation, thereby providing (a) a more rigorous evaluation of clean air zone policies than can be found in the extant literature and (b) a blue print for future attempts to undertake similar policy evaluations for CAZs in the UK and globally.

Although policies like Low Emission Zones have been widely applied internationally (Ferreira et al. 2015), the UK government only began to consider applying Clean Air Zone frameworks relatively recently. The Department for Environment Food & Rural Affairs (Defra) and Department for Transport (DfT) published the ‘Clean Air Zone Framework’ in May 2017 setting out the principles for local authorities to gradually implement CAZs across major UK cities (Defra, 2017). A number of other UK cities have plans for Clean Air Zones in the near future. The method we apply to Birmingham can be extended and reapplied to other contexts including other urban and regional policy interventions and to inform the future design of CAZ policies, as well as any other environmental policies.

The paper is structured as follows. Section 2 discusses the relevant literature while Sect. 3 describes the data and research method used in this study. Section 4 presents the results while Sect. 5 presents discussion. The final section concludes.

2 Background: Clean Air Zones and Policy Evaluation Methods

Understanding the efficacy of Clean Air Zones (CAZs) and Low Emission Zones (LEZs) on city-region air quality levels has been a topic addressed by both atmospheric scientists and environmental economists. These are two very different literatures and there have been few attempts to link the two by convincingly addressing meteorological influences on pollution within causal policy evaluation techniques from economics (Grange and Carslaw, 2019; Cole et al. 2020). The difference-in-differences method has been widely used to assess the effect of German LEZ on air quality and infant health (Gehrsitz, 2017), as well as the effectiveness of London LEZ on air quality, however the impact of weather on observed pollutant concentrations were addressed by controlling for key weather variables with various fixed effects misses the complex and non-linear nature between various weather conditions and concentration levels.

Using five-year air quality data from monitoring sites in Lisbon, Ferreira et al. (2015) applied time series de-trended analysis and compared air quality levels before and after a LEZ intervention in July 2011. They identified that the Lisbon LEZ failed to reduce NO₂ levels. Using PM₁₀ data from monitoring sites in Munich, Fensterer et al. (2014) applied a semiparametric regression model and found larger and significant reductions in roadside sites than urban background sites. Focussing on PM₁₀, PM_2.5, NO₂ and NO_X from five cities in the Netherlands, Boogaard et al. (2012) compared air pollution levels at street and urban sites with suburban locations acting as a potential control group between 2008 and 2010 and found that LEZs led to modest reductions in traffic related concentrations.

The first attempt to use a machine learning based weather normalisation technique able to decouple the effects of meteorology from observed air pollutant concentrations was by Grange and Carslaw (2019) who developed and applied this approach to analyse NO_X and NO₂ data for London Marylebone Road. The results reveal that the implementation of the London LEZ in 2008 reduced weather normalised NO₂ levels. Compared to traditional statistical models or air quality models, the machine learning based weather normalisation method provided an alternative (arguably better) tool for real world air pollution control policy evaluation. This method has been applied to other air pollution control policy studies including Forster et al. (2020), Shi et al. (2021), Singh et al. (2022) and Song et al. (2023).

Applying the weather normalisation method to assess the impacts of CAZ is still challenging given the difficulties of attributing air pollution reduction to a specific mitigation (Vu et al. 2019). In the real world many changes occur contemporaneously making it difficult to separate policy impacts. This is especially the case for evaluation of the Birmingham CAZ as behaviour (travel/shopping/leisure) was affected by the Covid-19 lockdown restriction changes, further complicating any assessment of the CAZ impact.

Causal policy evaluation has been the central focus of many economic studies (Athey and Imbens 2017) with an emphasis on quasi-natural experiment design, and through using identification strategies including Regression Discontinuity (RD), Difference in Difference (DID) and Synthetic Control Methods (SCM). Developed by Abadie et al. (2010), it has been argued that the SCM is “arguably the most important innovation in the policy evaluation literature in the last 15 years” (Athey and Imbens 2017). The principal concept of the SCM is to construct the counterfactual movement of the outcome variable of a treatment unit (exposed to the intervention) using a weighted average combination of a set of control units (not exposed to the intervention), and if the ‘synthetic unit’ is able to closely simulate the trajectory of the movement for the treatment unit before policy intervention, then the difference of the outcome variable movement after the point of policy implementation can be regarded as the causal impact of the policy intervention (Abadie 2021). Thus, this paper applies a weather normalisation model as the first stage in the evaluation of the Birmingham CAZ with the second stage based on the application of a SCM approach on de-weathered concentrations to provide a synthetic control, and hence causal estimation of the CAZ impacts.

3 Data and Methods

Hourly air pollution data (NO₂ and NO_X) from 39 air quality monitoring sites across the UK were collected, combining data from sites operated under the Defra ‘UK Automatic Urban and Rural Network (AURN)’ and sites operated by Birmingham City Council.³ We collected air pollution data from both urban background and roadside sites to provide a comprehensive analysis of the impact of CAZ implementation.⁴ To construct the Business As Usual (BAU)/counterfactual air pollution level for each CAZ site in Birmingham, we also collected air quality data from sites in other UK cities as control groups.⁵ Since the CAZ was implemented in June 2021, data were collected between 1 Jan 2021 and 31May 2022 to allow for sufficient time to simulate the air pollution trend (five months) before CAZ and identify the impact (one year) after the CAZ was implemented.

Table 2 The estimated CAZ impact on NO₂ using different methods

Table 3 Types of clean air zones

Following Grange et al. (2018), we collect hourly meteorological data to match the hourly air quality data and inform the weather normalisation algorithm. More specifically, surface observed weather data including wind speed, wind direction, temperature, and relative humidity from the nearest airport weather monitoring sites in each city were obtained from the National Oceanic and Atmospheric Administration (NOAA). Hourly data of surface pressure, boundary layer height, surface net solar radiation, total cloud cover, and total precipitation were retrieved from the ERA5 reanalysis dataset from the European Centre for Medium-Range Weather Forecasts (ECMWF). Aside from meteorological data, time variables including date-unix, week, weekday, hour, month, and Julian day were derived from the hourly data.⁶ The air pollution data are the dependent variables, and the rest of the meteorological and time variables are used as independent variables (predictors) to perform the weather normalisation analysis.

We apply a two-step approach to estimate the causal impact of the implementation of the Clean Air Zone (CAZ) on air quality in Birmingham. The first stage is to apply a machine learning based weather normalisation technique (developed by Grange et al., 2018, Grange and Carslaw, 2019) to decompose the impact of weather from observed air pollutant concentrations obtained from each monitoring site independently. The second stage is to apply the Ridge Augmented Synthetic Control Method (Ben-Michael et al. 2021) to estimate the causal impact of CAZ on weather normalised concentrations.

3.1 Step 1. Machine Learning Based Weather Normalisation Technique

First, a predictive random forest (RF) model is built using the meteorological and time variables discussed in the previous section. Data for dates between 1 Jan 2021 and 31 May 2022 were fed into the random forest model to predict hourly concentrations of NO₂ and NO_x for each of the 39 air monitoring sites across the UK.

According to Grange and Carslaw (2019), the key principle of the weather normalisation technique is to reduce the fluctuations and variability of trends in air pollution using statistical modelling. If the model (i.e., the random forest model built up using all independent variables) can explain the variation of the dependent variable (i.e., trends in air pollution) to a large extent, then the impact of the predictors on the outcome variable can be removed by repeatedly sampling and predicting. Since we intend to infer short term changes in emission strength from the implementation of the CAZ in Birmingham city centre, we apply the weather normalisation method by Shi et al. (2021).⁷

Weather normalisation was conducted for each pollutant at each air quality monitoring site, separately. More specifically, taking a random hour (e.g., 11:00 am on 1 May 2021) from the Birmingham Colmore Row site as an example, the previously trained and validated random forest (RF) model was used to predict the NO₂ concentration at this specific hour at this site 500 times. For each prediction, the meteorological variables were randomly resampled without replacement from the original whole data period (1 Jan 2021 and 31 May 2022) and fed into the RF model; the arithmetic mean of the 500 predicted NO₂ concentrations were then calculated which we refer to as the weather normalised (de-weathered) concentration of NO₂ at that specific hour for that site. We use the same code to run the weather normalisation model for each pollutant and finally obtain the hourly weather normalised concentration for all the sites during the study period (1 Jan 2021 and 31 May 2022). Both hourly observed and weather normalised pollutant concentrations were converted into weekly averages to run the second-stage analysis, i.e., the Ridge Augmented Synthetic Control Method (ASCM).

The random forest-based weather normalisation method can more flexibly identify and remove the impact of weather on pollution levels, compared to the usual way of controlling for weather variables in regression analysis. For example, the data-driven machine learning model is non-parametric and has no restrictions on data structure, especially in the air quality trend analysis where atmospheric processes can be challenging due to their complex, often non-linear nature. In addition, interaction effects can be better handled by machine learning algorithms, instead of hand picking different functional forms for different variables (Mullainathan and Spiess 2017; Athey and Imbens 2019;) Moreover, given the large heterogeneity in the impact of weather on pollution levels for different pollutants at different sites, the RF model can more effectively and flexibly remove the weather impacts, based on each individual RF model built for each pollutant at each site (Grange and Carslaw 2019).

3.2 Step 2. Ridge Augmented Synthetic Control Method

Although the impact of weather on observed pollutant concentrations can be addressed using the weather normalisation technique, it is still challenging to estimate the causal impact of the CAZ on air quality level by simply comparing the de-weathered pollution level before and after the CAZ implementation (Shi et al. 2021). For example, the concentrations of NO₂ are higher in winter and lower in summer and this trend will interact with other confounding factors, making it difficult to disentangle one from another.

Under the quasi-natural experiment design of the Birmingham CAZ, we can evaluate the potential causal impacts of the CAZ on air pollution in Birmingham, where the air monitoring sites within the CAZ were assigned as the treatment group, and sites from other UK cities that had no similar intervention were used as the control group. The Synthetic Control Method has been applied by researchers from various disciplines to understand policy evaluation problems (Guettabi and Munasib 2018; Doerr et al. 2020). According to Abadie (2021), the principal idea of the SCM is analogous to the ‘comparative case study’ approach with the efficacy of an intervention being identified through comparing the movement of the outcome variable between a treatment group and a control group. Ben-Michael et al (2021) extended the original SCM by proposing the Ridge Augmented SCM, which applies Ridge regression as the outcome model to control for inexact pre-intervention fit, which can address the problem of the original SCM if the pre-treatment fit is infeasible in practice.⁸

The causal impact of the CAZ on air pollutant concentrations is given by:

$$Y_{1T} - \hat{Y}_{1T}$$

where $Y_{1T}$ and $\hat{Y}_{1T}$ refer to the weather normalised and synthetic counterfactual weather normalised air pollutant concentrations in the treatment group (air quality monitoring sites within the CAZ) at time $T$, and $\hat{Y}_{1T}$ is calculated by:

$$\hat{Y}_{1T} = \sum \hat{w}_{j}^{scm} Y_{jT} + (X_{1} - \sum \hat{w}_{j}^{scm} X_{j} )\cdot\hat{\eta }^{Ridge} = \sum \,\,\hat{w}_{j}^{aug} Y_{jT}$$

where $\hat{w}_{j}^{scm}$ refers to the estimated weight from the original SCM, $Y_{jT}$ refers to the weather normalised air pollutant concentrations in control group at time $T$, $X_{1}$ and $X_{j}$ refer to the pre-policy outcomes vector in treatment group and control group, respectively. $\hat{\eta }^{Ridge}$ are the coefficients of a ridge regression of post intervention outcomes $Y_{0T}$ on centered pre intervention outcomes $X_{0}$, and finally $\hat{w}_{j}^{aug}$ refers to the estimated element of the targeted weights vector $W^{aug}$ (see Ben-Michael et al. 2021 for a detailed explanation). In our paper, NO₂ (and NO_X) data from 15 urban (roadside) sites of other UK cities are used as control sites to construct the ‘synthetic’ treatment sites in Birmingham urban (roadside) sites, and we use data between 1 Jan 2021 to 25 May 2021 as the pre intervention period.⁹

4 Results

4.1 Weather Normalised Air Pollution Concentrations

Before presenting the main results it is useful to include information on the weekly plots of both observed and weather normalised NO₂ and NO_x at the eight available Birmingham automated measurement sites between 1 Jan 2021 and 31 May 2022 (see Figures A2 and A3 in the appendix).¹⁰ Both pollutants show clear reductions in weather normalised levels from St Chads Queensway, and slight reductions at Colmore Row, Lower Severn Street and A4540 Roadside in the week starting on 24 May (the data point on 24 May represents the weekly average level between 24 to 30 May 2021). This effect occurred in the week right before the 1 June 2021 official implementation date. For this reason, we used the week starting on 24 May (to 30 May) as the CAZ ‘starting to take effect’ date and as the treatment start date for the later ASCM analysis.¹¹ In contrast, clear increases in weather normalised NO₂ and NO_x levels were observed around September 2021 for roadside sites including St Chads Queensway, Colmore Row, Lower Severn Street and A4540 Roadside, with a particularly sharp rise at the St Chads Queensway site. Sharp elevations in observed NO₂ and NO_x can also be seen in January and March 2022 for all sites, most likely reflecting weather events which caused local air pollution to spike.

To provide a broader picture, Figures A6 and A7 illustrate the weekly weather normalised NO₂ trends at roadside and urban background sites, respectively, including both treatment and control sites. During the 3-month period after CAZ implementation (June to Aug 2021), the trend at the control sites behaved similarly to those in Birmingham. However, a significant and clear increase in weather normalised NO₂ levels were observed around Sep 2021 for some roadside sites (both treatment and control sites), suggesting that the elevated emissions occurring around early Sep 2021 might complicate the CAZ evaluation during the 3–6 months post CAZ implementation (Sep to Nov 2021).

4.2 Changes in Air Quality Relative to Synthetic Control Sites

Figure 1 illustrates the difference in weather normalised NO₂ (the top eight panels) and NO_X (eh bottom eight panels) between the Birmingham air quality monitoring sites and their synthetic control equivalents. The data used to generate Fig. 1, the weather normalised NO₂ and NO_X trends and the synthetic counterfactual/business-as-usual trends, are shown in Figures A8 and A9 of the appendix. Note that ‘wn’ refers to weather normalised so NO₂wn is weather normalised NO₂. The counterfactual trend is constructed as a weighted average of the air quality trend from control sites of other UK cities using the Ridge ASCM approach (Ben-Michael et al. 2021). For all eight sites, the synthetic NO₂wn (or NO_Xwn) closely approximates the trajectory of NO₂wn (or NO_Xwn) before the CAZ implementation, suggesting that the Ridge ASCM is able to simulate the trajectory of NO₂wn/NO_Xwn that would have been experienced if there were no CAZ intervention. The analysis confirms that the method can accurately construct a ‘business as usual’ counterfactual to the (weather normalised) NO₂/NO_X levels post CAZ implementation.

Figure 1 presents the ‘gap’ plot (i.e. the CAZ impact), which refers to the difference between weather normalised NO₂ (NO_X) and the synthetic counterfactuals. The shaded area refers to the 95% point-wise confidence intervals calculated using a Jackknife + procedure (Ben-Michael et al. 2021). Figure 1 suggests that, during the period June to Aug 2021 the weather normalised NO₂ and NO_X levels for roadside sites within the CAZ (St Chads Queensway, Colmore Row) were significantly reduced relative to the synthetic counterfactual. Other sites also experienced reductions relative to the synthetic control, in some cases after a slightly delay (e.g. Selly Oak and Ladywood). No reductions are observed for Acocks Green (which is outside of the CAZ). What is striking is that in almost all cases, pollution levels rebounded to pre-CAZ levels or higher within 6–12 months.

Table 1 provides estimates of the magnitudes of the CAZ effect. For example, on average and relative to a no-CAZ scenario, for St Chads Queensway, Colmore Row and Lower Severn Street, weather normalised NO₂ was reduced by 3.23 µg m⁻³ (7.32%), 1.1 µg m⁻³ (3.96%) and 0.81 µg m⁻³ (2.85%) during the 7-month post-CAZ period, relative to the synthetic counterfactual. Similarly, the corresponding reductions in weather normalised NO_x at the three locations, relative to a no-CAZ scenario, were 4.5 µg m⁻³ (4.83%), 2.49 µg m⁻³ (6.01%) and 4.46 µg m⁻³ (6.91%), respectively (see Table A4). For the urban background site of Ladywood (within the CAZ), relative to the counterfactual, weather normalised NO₂ was reduced by 0.94 µg m⁻³ (5.6%) and weather normalised NO_x was reduced by 2.99 µg m⁻³ (11.67%).

Table 1 also shows that the reductions in NO₂ concentrations at the roadside sites within the CAZ were smaller over the post 1-year period, than the post 7-month period with the reduction in NO₂ concentrations at St Chads Queensway falling from 3.23 µg m⁻³ (7.32%) (for post 7 months: 25 May to 31 Dec 2021), to 0.25 µg m⁻³ (0.64%) (for post 1 year: 25 May 2021 to 31 May 2022), Colmore Row was reduced from 1.10 (3.96%) to 0.96 (3.38%) and Lower Severn Street was reduced from 0.81 (2.85%) to 0.16 (0.57), respectively. Similar reductions were found for NO_x concentrations (Table A4), where the average reductions fell from 2.49 µg m⁻³ (6.01%) for post 7 months, to 1.57 µg m⁻³ (3.73%) for the post 1 year at Colmore Row, and the average reductions at the Lower Severn Street fell from 4.46 µg m⁻³ (6.91%) for post 7-month to 1.53 µg m⁻³ (2.44%) for post 1 year.

Figure 1 and Table 1 include the impact of the CAZ on four sites outside the zone, including three roadside sites (A4540 Roadside, Selly Oak and Stratford Road) and one urban background site (Acocks Green). The A4540 Roadside site is located adjacent to Birmingham’s A4540 ring road (thus just outside the CAZ). Figure 1 and Table 1 show that the impact of the CAZ on average NO₂ levels at the A4540 Roadside site reduced by 0.84 µg m⁻³ (2.52% equivalent) during the first 7-month post-CAZ implementation, relative to the counterfactual scenario.

Furthermore, Fig. 1 and Table 1 also show the results for Selly Oak (roadside site), Stratford Road (roadside site) and Acocks Greens (urban background site) respectively. In terms of the 7 months post CAZ period, we find significant reductions in both NO₂ and NO_x (Table A4) for both the Selly Oak and Stratford Road sites (possibly due to reduced emission strength from less traffic volume). As with the roadside sites within the CAZ, we find reductions of changes in weather normalised NO₂ and NO_x levels for Selly Oak and Stratford Road (both roadside site), respectively.

To provide the full picture of the CAZ impact on site types as a whole, Figure A10 presents the gap plot of site type average results. The average NO₂ level of roadside sites within the CAZ was reduced immediately and significantly after the CAZ implementation and rebounded gradually after Jan 2022. The average NO₂ level of roadside sites outside the CAZ and urban site within the CAZ showed similar trends (with slightly lower reductions compared with roadside sites within the CAZ), with no significant changes detected for urban sites outside CAZ.¹²

5 Discussion

5.1 Causal Impact of the CAZ

Table 2 compares the estimated NO₂ results from three different intervention evaluation methods: (i) simple comparison of observed concentrations before and after the implementation of the CAZ; (ii) comparison of weather-normalised concentrations before and after the implementation of the CAZ, and (iii) comparison of weather-normalised concentrations with a synthetic control, post-implementation (i.e., the main approach used in this paper).¹³

The first observation is that the ‘observed (column 1)’ and ‘de-weathered (column 2)’ approaches tend to overestimate the impact of the CAZ for sites both within (Colmore Row, Lower Severn Street, and Ladywood) and outside the CAZ targeted zone (A4540 Roadside, Stratford Road, and Acocks Green). The overestimation of intervention impacts can also be observed from the air quality trend plots in Figures A6 and A7, since concentrations of NO₂ will follow a seasonal pattern in which concentrations are higher in winter and lower in summer. For example, the NO₂ level is lower in the summer than the winter (i.e., Feb to June 2021) for almost all sites in our study irrespective of whether they were treatment or control sites. This arises from the changing seasonal meteorological conditions (e.g. they tend to be more favourable for pollution dispersion in the summer) and emissions (fewer NO_X emissions due to reduced heating needs). Simply comparing pollution levels before after the CAZ (for example, 1 Jan to 25 May vs. 1 June 2021 to 31 May 2022) without properly ‘detrending’ will lead to overestimation of an intervention (Shi et al. 2021).

Our results (Fig. 1, Table 1, Figures A8, A9, and A10) suggest that, during the seven months post CAZ implementation period (June to Dec 2021), the significant reductions of both pollutants at roadside sites within the CAZ (St Chads Queensway for post 3-month, Colmore Row, Lower Severn Street) and outside the CAZ (Selly Oak, Stratford Road) may be attributed to the CAZ implementation. While emissions of both NO₂ and NO_X at all the above-mentioned sites decreased during the 1-year post CAZ implementation (June 2021 to May 2022), no direct and significant changes were detected at the urban background sites within (Ladywood) and outside (Acocks Green) the CAZ. Our estimation of the magnitude of the CAZ impact is in a similar range to those found in Ma et al. (2021), who found that the introduction of the London Ultra Low Emission Zone (ULEZ) resulted in around a 3% reduction in NO₂ concentrations, and an insignificant impact on PM_2.5.

Our results in Fig. 1 and Table 1 suggest that the CAZ impacts are localised and tend to be restricted to roadside locations. Unlike the roadside sites that directly reflect the traffic going past the monitors, the Ladywood and Acocks Green site (urban background) has limited local traffic activity and no significant CAZ impact was detected for these locations. We also found that the CAZ implementation led to larger reductions in NO_X than NO₂ in these locations. This is consistent with Shi et al (2021) who found similar trends during the COVID lockdowns. This is due to the fast NOx-O₃ photochemistry that shifts the NO₂/NO_x ratio in favour of NO₂. Moreover, we found that substantially larger declines in NO_x and NO₂ were detected at roadside sites than at urban background sites, indicating that the NO₂ reductions are primarily driven by changes in vehicle emissions. Besides NO₂ and NO_X, we also estimated the impact of the CAZ on PM_2.5. Unlike NO₂ which largely originates from vehicle emissions and is the target pollutant for the Birmingham CAZ, PM_2.5 is a complex pollutant that originates from various sources, including but not dominated by primary road transport emissions. Figure A11 indicates no significant impact of the CAZ on weather normalised PM_2.5 for these sites (A4540 Roadside, Acocks Green and Ladywood stations). PM_2.5 has a significantly longer atmospheric lifetime than NO₂, and PM_2.5 in the UK is primarily formed of secondary aerosols from regional sources.¹⁴ Therefore, it is not surprising, and indeed it is reassuring, to see limited or no significant impact from local interventions, such as the Birmingham CAZ, on PM_2.5.

Our argument that changes in weather normalised NO₂ and NO_x at roadside site locations within the CAZ are likely to be primarily driven by reduced traffic strengths is supported by the CAZ report from Birmingham City Council, who issued the Birmingham CAZ Fact Sheet on 23 Sep 2021 showing that the average daily valid charges (both collected fees and the compliance rate) fell between June and Aug 2021.¹⁵ In addition, the CAZ Six Month Report provides the A38M road traffic counts (June 2019 to November 2021) and the ring road and inner city (CAZ) traffic flow (January 2021 to January 2022).¹⁶ These indicate that the traffic flow within the CAZ was reduced after the CAZ launch, which is consistent with our estimation of NO2 and NOx at most roadside sites. The report also suggests there has been no corresponding increase in traffic flows on the ring road (CAZ boundary, outside), which is also consistent with our estimation of NO₂ and NO_X at A4540 Roadside site. Although the change in traffic flow information supports out results, a more rigorous analysis would be to use ASCM to construct the counterfactual trend of traffic flow within Birmingham using other UK cities as controls. Unfortunately, this analysis would require detailed traffic flow data which is not currently available.

5.2 Sensitivity Analysis

To confirm the robustness of our main results on NO₂, we carried out two sensitivity analyses. First, we undertake an in-time placebo test with the results shown in Figure A12. In this test we assume that the CAZ was implemented from June 2019 (instead of June 2021) in Birmingham. As expected, Figure A12 shows that there was no significant reduction of weather normalised NO₂ detected at the two roadside sites within the CAZ, suggesting that our main results are not observed by accident. Interestingly, for St Chads Queensway, we also found a significant increase of NO₂ around August and September 2019. Table A5 reports the CAZ impact of the main results (2021 data) and the in-time placebo tests (2019 data) for St Chads Queensway, suggesting that St Chads Queensway might have experienced greater NO₂ elevation (as in 2019) around Aug/ Sep in 2021 if the CAZ were not implemented in Birmingham.¹⁷

For a second sensitivity analysis we also undertook an in-place placebo test which was designed to randomly assign the CAZ to roadside sites within the control group (that had no CAZ in reality) by selecting nine sites (from a group of large cities) from 15 control sites and applying the same code and data to conduct the same analysis as our main results. As shown in Figure A13, no significant impacts (and no similar pattern to Fig. 1) were detected immediately after the CAZ on the nine randomly assigned sites, suggesting that our main results for Birmingham are robust.

5.3 No Evidence of Negative Spill-Over Effects

A potential concern for all policies that use geographically limited restrictions such as the Birmingham CAZ is the potential for “negative spill-over/displacement effects”. For example, to avoid daily charges, people may drive vehicles into areas surrounding or just outside the CAZ, which could increase emissions in those areas. In the case of Birmingham, the city ring road (A4540) marks the CAZ perimeter but is not itself part of the charging zone. Since we have data from sites surrounding and outside Birmingham CAZ, we were able to investigate whether such spill-over/displacement effects had emerged as one negative impact of the CAZ policy, at those measurement locations.

Figure 1 and Table 1 show that, on average weather normalised NO₂ levels at the A4540 Roadside site reduced by 0.84 µg m⁻³ (2.52% equivalent) during the first 7-month post CAZ, and by 0.53 µg m⁻³ (1.58% equivalent) during the 1-year post-CAZ implementation, relative to the counterfactual scenario. Thus, we find no evidence of a negative spill-over effect. Unfortunately, we do not have traffic data near the A4540 site to confirm whether there is a shift of traffic from inside the CAZ to the A4540. Interestingly, we found significant reductions of NO₂ and NO_x for roadside sites outside CAZ (Selly Oak and Stratford Road), suggesting the “positive spill-over effect” of the CAZ, showing the possibility of traffic and journey reductions overall as a result of behavioural changes. However, more detailed traffic and pollution data from more sites is needed to provide a definitive understanding of the potential spillover effects for other areas outside the CAZ.

5.4 Health Impact

Although no the primary focus of the paper, we present a simple back of the envelope calculation to derive the potential health impact of reduced pollution levels due to CAZ. Here we focus on the NO₂-mortality relationship (from single pollutant studies) for simplicity. Based on our estimation that the CAZ causally led to 1.23 μg/m³ reductions for three roadside sites (on average) during the post 1 year period, we assume the overall annual reduction of NO₂ for CAZ area is 1.23 μg/m³. We draw upon the NO₂ concentration-mortality relationship summarised by Public Health England (COMEAP 2018) “1.023 (95% CI: 1.008, 1.037) per 10 μg/m3 NO₂ as an annual average”. We calculate the reduced mortality from reduced NO₂ (due to CAZ) is: 1245.4/100,000 * 67,000 * (1.023%/10 *1.23) = 1.05 (death), where 1,245.4 refers to the 2020 age-standardised mortality rate (per 100,000 population) in Birmingham (Birmingham Demographic Brief 2022), 67,000 is the CAZ area population (Clean Air Fund 2021). The simple back of the envelop calculation suggests that 1.05 deaths were avoided due to CAZ implementation in a year, thus the reduced NO2 impact on mortality is likely to be negligible.¹⁸

5.5 Implications

This study provides the short-term (seven months to one year post intervention) causal estimation of the impacts of the CAZ intervention on air quality in Birmingham. Our results (on weather normalised NO₂ and NO_x) suggest that the CAZ impact was relatively small and primarily at the roadside.¹⁹ As traffic is not the only emission source of air pollution, small emission reductions from traffic sources might not be sufficient to significantly reduce the overall pollution levels within a city.²⁰

From the policy perspective, our results suggest that the CAZ did reduce NO₂ (and NO_X) levels but the actual change is modest. This suggests that a more stringent form of CAZ would be needed to achieve larger and more sustained pollution reductions in Birmingham. This could take the form of increased fees for non-compliant vehicles or a widening of the definition of non-compliance to include a great proportion of vehicles.²¹ Compliance is currently based on whether a vehicle’s engine meets the Euro 4 emissions standard for petrol or Euro 6 for diesel. Since Euro 6 is the most recent emissions standard, the only scope for widening non-compliance would be to raise the requirement for petrol engines to Euro 5 or 6 i.e. to deem cars registered before September 2011 (Euro 5) or September 2015 (Euro 6) as non-compliant. Birmingham City Council would need to ascertain what proportion of cars this would affect and hence whether any gains to air quality would be sufficient to justify further restrictions to road users and potentially to those on the lowest incomes who are more likely to own older vehicles.

The positive spillover effects found at roadside sites outside CAZ also suggest that expanding the CAZ coverage area bring further pollution reductions, including to areas outside the actual zone. We also confirm that the CAZ has no detectable effects on PM_2.5. There exists a larger epidemiological evidence base for long-term health effects associated with PM_2.5 than for NO₂ exposure, therefore interventions which target multiple pollutants will deliver greater health benefits (for a given population) than traffic interventions which are targeted primarily at NO₂ (NO_X) emissions. Thus, future policy would benefit from a systematic approach that attempts to tackle multiple pollutants. We also illustrate how simpler statistical analyses significantly overestimate the effect of the CAZ on air pollutant levels potentially leading to misleading interpretations of CAZ costs and benefits. It is important that policy reviews have as accurate a picture as possible of the impact of any pollution reduction policy on emissions and the relative costs and benefits.

5.6 Limitations

Data availability, including for both air quality, traffic and behavioural changes is critical for monitoring the impacts of air pollution on city-regions and for assessing the impacts of CAZ interventions. One limitation of this study is that it relies on air pollution data from a relatively small number of air quality monitoring sites within Birmingham. While more monitoring sites are being commissioned, the critical pre-CAZ-implementation measurement envelope is modest. The whole CAZ covers a large area and ideally more air pollution data is required to provide a more comprehensive analysis of its impact. Similarly, air pollution data from more (especially urban background) sites (both treatment and control) will also increase the representativeness and reduce the uncertainties of the analysis. More detailed data regarding traffic, economic and behavioural changes will also help us to better interpret the results. Given the importance of understanding the impact of CAZs future city implementations should place monitoring stations in a wide range of locations well in advance of any implementation.

6 Conclusions

We apply a data-driven two-step approach, first using a random forest-based machine learning model to strip out the effects of weather and second to apply the Augmented Synthetic Control Method to isolate the causal effect of the CAZ. We demonstrate the value of this approach to air quality intervention evaluation drawing on insights from different disciplines.

We find significant reductions in NO₂ (and NO_x), relative to the non-CAZ control scenario, over a 7-month post implementation period at most roadside sites within the CAZ. As expected, the biggest reductions of NO₂ are at busy roadside locations within the Clean Air Zone, while no significant changes were detected for the urban background site outside CAZ. We also find no evidence of negative spill-over effects with no increase in concentrations at a site bordering the CAZ, whereas we find positive spillover effects for roadside sites outside the CAZ. Overall, the Birmingham CAZ led to modest reductions in NO₂ and NO_X levels but no detectable change in PM_2.5. Crucially, the NO₂ and PM_2.5 levels for the sites in Birmingham still regularly exceed the 2021 WHO NO₂ and PM_2.5 guidelines (annual averages of 10 and 5 µg m⁻³, respectively). This suggests that more rigorous policy interventions such as increased fees for non-compliant vehicles, a widening of the definition of non-compliant vehicles, and/or an expansion of the CAZ coverage, will be needed to realise significant health gains for Birmingham.

Appendix

See Figs. 2, 3, 4 , 5 , 6, 7, 8, 9, 10, 11, 12, 13, and 14.

Fig. 2

Map of Birmingham clean air zone. Note: for completeness, we include all the sites with continuous measurements available during the study period (Jan 2021 to May 2022), including three roadside sites (St Chads Queensway, Colmore Row and Lower Severn Street) and one urban background (Ladywood) within the CAZ, we also include three roadside sites outside of the CAZ: A4540 Roadside (just located outside of the CAZ), Selly Oak and Stratford Road (located near the main road connecting to the CAZ), and one urban background location (Acocks Green)

Fig. 3

Observed and weathered normalised weekly-averaged NO₂ for Birmingham sites. Note: Two vertical lines refer to the week starting on 24 May 2021 and 31 Dec 2021 respectively. The plot consists of weekly data points between 1 Jan 2021 to 31 May 2022. Data for Acocks Green site stops after 30 Sep 2021

Fig. 4

Observed and weathered normalised weekly-averaged NO_x for Birmingham sites. Note: Two vertical lines refer to the week starting on 24 May 2021 and 31 Dec 2021 respectively. The plot consists of weekly data points between 1 Jan 2021 to 31 May 2022. Data for Acocks Green site stops after 30 Sep 2021

Fig. 5

Random Forest model performance for NO₂ and NO_X in selected sites

Fig. 6

Variable importance of random forest model for NO₂ and NO_X in Colmore Row

Fig. 7

Weather normalised NO₂ for roadside treatment and control sites. Note: Time period is 1 Jan 2021 to 31 May 2022. Three vertical lines refer to the week starting on 24 May 2021, 1 June 2021 and 14 June 2021 respectively

Fig. 8

Weather normalised NO₂ for urban background treatment and control sites. Note: Time period is 1 Jan 2021 to 31 May 2022. Three vertical lines refer to the week starting on 24 May 2021, 1 June 2021 and 14 June 2021 respectively

Fig. 9

Observed and synthetic weather normalised NO₂ for 8 Birmingham sites. Note the plot consists of weekly data points between 1 Jan 2021 to 31 May 2022 for each site, (data for Acocks Green site only available until 30 Sep 2021), the vertical line refers to the week starting on 24 May 2021. ‘NO₂wn’ (solid red line) refers to the weather normalised NO₂ level of each site, and ‘SNO₂wn’ (dashed blue line) refers to the synthetic weather normalised NO₂ level of each site (i.e., the counterfactual level had the CAZ intervention not been implemented)

Fig. 10

Observed and synthetic weather normalised on NO_X for 8 Birmingham sites. Note: the plot consists of weekly data points between 1 Feb 2021 to 31 May 2022 for each site, (data for Acocks Green site only available until 30 Sep 2021), the vertical line refers to the week starting on 24 May 2021. ‘NO_Xwn’ (solid red line) refers to the weather normalised NO_X level of each site, and ‘SNO_Xwn’ (dashed blue line) refers to the synthetic weather normalised NO_X level of each site (i.e., the counterfactual level had the CAZ intervention not been implemented)

Fig. 11

The CAZ impact on NO₂ for different site types. Note: the plot consists of weekly data points between 1 Jan 2021 to 31 May 2022, the vertical line refers to the week starting on 24 May 2021. ‘Traffic_Avg_In’ refers to the average pollution level of, Colmore Row and Lower Severn Street (St Chads Queensway was dropped due to its abnormal trend during the post 1-year priod); ‘Traffic_Avg_Out’ refers to the average pollution level of A4540 Roadside, Selly Oak and Stratford Road; ‘Urban_Avg_In’ refers to the pollution level of Ladywood, ‘Urban_Avg_Out’ refers to the pollution level of Acocks Green. ‘NO2wn’ (solid red line) refers to the weather normalised NO2 level of each site type, and ‘SNO2wn’ (dashed blue line) refers to the synthetic weather normalised NO2 level of each site type (i.e., the counterfactual level had the CAZ intervention not been implemented)

Fig. 12

The CAZ impact on PM_2.5 for 3 Birmingham sites. Note: PM_2.5 data was only available for the three sites in Birmingham, the plot consists of weekly data points between 1 Jan 2021 to 31 May 2022 (data for Acocks Green site stops after 30 Sep 2021), the vertical line refers to the week starting on 24 May 2021, the shaded area illustrates the 95% Jackknife + Confidence Interval calculated from the Ridge ASCM

Fig. 13

Result of in-time placebo test in 2019. Note: the vertical lines refer to the week starting on 24 May 2019, the shaded area illustrates the 95% Jackknife + Confidence Interval calculated from the Ridge ASCM

Fig. 14

Result of in-place placebo test. Note: The vertical lines refer to the week starting on 24 May 2021, the shaded area illustrates the 95% Jackknife + Confidence Interval calculated from the Ridge ASCM. We randomly select 9 sites from our control group and assign a “fake” CAZ in these sites

See Tables 3,

Table 4 Air quality monitoring sites information in Birmingham

4,

Table 5 Air quality monitoring (NO₂) sites information in other UK cities

5,

Table 6 The estimated CAZ impact on NO_X in eight Birmingham sites

6, and

Table 7 The CAZ impact on NO₂ (St Chads Queensway) in 2021 and the in-time placebo testes results using 2019 data

7.