Pitfalls in comparing Paris pledges

The Paris pledges are unique documents in climate politics that outline what each country intends to do to combat climate change. The documents—called (Intended) Nationally Determined Contributions (INDCs)—are often oriented around a headline national greenhouse gas (GHG) emissions reduction target expressed in percentage terms.¹ For example, the Canadian government announced its intention to cut Canadian GHG emissions by 25% by 2030, while the Russian government announced its intention to cut Russian emissions by 25–30% in the same time period. However, even though these two targets may appear similar at face value, they mask actual contributions to mitigation since each country benchmarks their percentage cuts using a different historical reference point. Canada benchmarked its Paris GHG target to 2005 emissions levels, while Russia benchmarked its target to 1990 emissions levels. Any actual GHG mitigation depends on both the percentage reduction and the reference point that emissions are to be cut from.

In Fig. 1, I visualize each country’s emissions trajectory in the 1990–2015 period, at the end of which all countries submitted their INDCs ahead of the Paris climate conference. The figure illustrates that both the Canadian and Russian governments’ choice of reference year exerts a strong influence on the percentage terms that their targets are expressed in. Specifically, the chosen reference years make the percentage reduction appear larger than alternative reference years, since national GHG emissions fluctuate throughout this period. If each country had selected the other’s reference year but still chosen to target the same absolute GHG emissions level in 2030 as in their original INDC, Canada’s headline percentage emissions reduction would be only 16.5% from 1990 levels and Russia’s would be an increase of 8.7–16.6% from 2005 levels. The choice of reference GHG emissions levels has enormous consequences for actual GHG emissions flows under the Paris Agreement and measuring each country’s contribution to mitigating climate change.

Open image in new window

This point about measuring mitigation in INDCs would be purely semantic were it not distorting climate research. The Paris Agreement’s institutional design creates a unique opportunity for researchers to observe each country’s willingness to contribute to collective mitigation through the content of their INDCs. However, the Agreement’s design also left an opportunity for governments to communicate their pledges in ways that mask the true nature of their contributions, since states were not obligated to submit documents in a common format. Changing the reference years or other parameters of GHG accounting allows governments to submit ambitious sounding GHG targets that are disconnected from the actual GHG emissions reductions that are necessary to keep global mean temperatures below 1.5^∘C or 2^∘C.

I illustrate the nature of this problem by re-visiting a recent study by Aaron Drummond and co-authors in Climatic Change (Drummond et al. 2018). This study finds that countries with higher public awareness of climate change adopted more stringent GHG targets in the Paris Agreement than countries with lower public awareness of climate change, controlling for a number of covariates. This finding suggests that efforts to raise public awareness may help catalyze more ambitious climate policies and overcome persistent collective action problems that have plagued climate politics. However, errors in interpretation and measurement in the study produce misleading inferences about the relationship between public opinion and GHG targets. Essentially, the study takes each country’s headline GHG emissions reduction percentage as its measure of the stringency of mitigation without accounting for the fact that countries expressed their targets in fundamentally different ways. As countries pledged to reduce their emissions from different reference years or forecasts of future emissions growth, the naive percentage target may be disconnected from actual GHG emissions reductions and is therefore not directly comparable across countries.

I propose two alternative measures of mitigation targets that better account for the structure of the Paris targets and use these to re-visit the findings in Drummond et al. (2018). My two indicators standardize each country’s target relative to their own historical emissions. The indicators do not address the distributional implications or equitable mitigation across countries. Instead, they purge opportunistic choices of reference levels that allow governments to declare large percentage cuts while actually pledging minor changes in absolute emissions. In this sense, they re-focus measurement on emissions flows and remove elements of political communication from target-setting. Using these new indicators, I do not find that public opinion predicts the stringency of countries’ Paris targets.

This paper makes contributions for those interested in climate politics, public opinion, and international cooperation. For climate scholars, this paper demonstrates the challenges in comparing countries’ Paris targets since the pledge documents vary dramatically in form as well as in substance. Future studies of countries’ Paris pledges ought to be very attentive to the manner in which countries express their climate targets. For scholars of public opinion, this paper shows that public opinion does not appear to be related to the ambition of governments’ GHG targets in the Paris Agreement. Future research should explore when and where public awareness of climate change filters through domestic political institutions to affect governments’ climate policies. Finally, for scholars of international cooperation, this paper cautions that the “bottom-up” model of international cooperation—whereby governments deliberate nationally and put forward cooperative pledges—may create an opportunity structure for governments to mislead interested publics about the scale of their contributions. If the Paris Agreement is a new model for future international cooperation, then negotiators and academics should be aware of this challenge to measuring and comparing cooperative efforts.

The INDC model of target setting reflects a compromise that allowed for maximum participation in the Paris Agreement at the expense of transparency and precision (Keohane and Oppenheimer 2016). Instead of negotiating country-level GHG targets multilaterally—as was done with the 1997 Kyoto Protocol and was attempted at the 2009 United Nations climate conference in Copenhagen—governments selected their own, nationally determined, contributions to addressing climate change before meeting in Paris in December 2015. However, many states had reservations about accepting costly mitigation commitments given that there was uncertainty about their counterparts’ willingness to contribute as well. Therefore, while the 2014 Lima climate conference outcome document invited parties to include “quantifiable information on the reference point (including, as appropriate, a base year), time frames and/or periods for implementation, scope and coverage, planning processes, assumptions and methodological approaches including those for estimating and accounting for anthropogenic greenhouse gas emissions and, as appropriate, removals,” this information was not mandatory (UNFCCC 2014, para. 14). Governments interpreted this guidance loosely and submitted INDCs that vary tremendously in their content, detail, as well as ambition.

Since domestic GHG mitigation imposes costs in the present and creates uncertain future benefits, political leaders must navigate difficult intertemporal and distributive bargains (Jacobs 2011). In contexts where domestic publics are more aware of climate change, politicians have incentives to appear to be taking responsible action to mitigate risks, while remaining sensitive to the political risks if policy reforms are costly (Harrison and Sundstrom 2010). When policymakers have access to complex policy instruments, such as with the largely unstructured INDC system, these incentives may lead actors to increase the complexity of policy measures rather than to match public preferences for policy outcomes (Kono 2006). Similarly, national governments may engage in “symbolic regulation” that seeks to reassure domestic regulatory advocates, without providing the actual policy reforms advocates desire (Peacock 2018). In sum, recent political science research suggests that when governments face pressure to impose new regulations but also fear the costs of policy reforms, they may adopt policies that mask their true impacts from public audiences.

In climate mitigation, this obfuscation may take the form of a divergence between stated GHG emissions reductions and actual reductions. If such a divergence exists, then using the headline percentage emissions reduction to measure GHG emissions reductions—as Drummond et al. (2018) do—may be uninformative about actual emissions reductions. Governments may pledge ambitious sounding policy targets to appear engaged, while cushioning the impact of such policies by changing the terms to minimize the actual emissions reductions associated with a stated percentage point change. I now explain how governments could use the vagueness of the INDC design to disguise their true GHG mitigation commitments.

The lack of common reporting requirements has led the INDCs to vary along many dimensions and the documents therefore require further interpretation to make them comparable. The World Resources Institute has undertaken an exhaustive data collection and synthesis project on the INDCs (WRI 2016; Ross et al. 2016). Political scientists have already begun analyzing this data (Tobin et al. 2018). For the purposes of this paper, I outline the different categories of targets that countries adopted in their INDCs and use these to explain how different types of targets create opportunities for headline percentage reductions to diverge from actual emissions reductions. Interested readers should consult Ross et al. (2016), Climate Watch (2016), and Tobin et al. (2018) for more in-depth treatments of target types.

INDCs may contain three types of non-mutually exclusive climate targets. First, countries may declare policy goals that have mitigation co-benefits, such as increasing energy efficiency by 20%. Second, they may announce specific mitigation of adaption actions without linking these explicitly to GHG emissions flows, such as improving coastal management to reduce flood risk. Finally, countries may express economy-wide GHG emissions reduction targets. The above non-GHG targets and actions targets cannot be compared directly with GHG targets without a host of non-trivial modelling decisions. Ultimately, not all countries have directly comparable targets, which may lead to misleading inferences for the raw data.

Economy-wide GHG emissions reductions targets can also be expressed in different ways. Base year targets specify GHG targets in terms of reductions from GHG emissions in a historical base year. This GHG target format is adopted by Annex 1, industrialized countries, as well as a few developed countries. As noted above, Canada has pledged to reduce GHG emissions by 30% from 2005 GHG emissions levels by 2030, while Russia will cut by 25–30% from 1990 GHG emissions levels by 2030. Canada and Russia’s targets may be similar in headline percentages (30% and 25–30%, respectively), but the choice of base years (2005 and 1990, respectively) informs the size of a percentage reduction. Russia’s emissions fell substantially during its prolonged recession at the end of the Cold War, but have been very stable since the mid-1990s and never recovered to their earlier levels. In this situation, choosing an earlier base year makes a percentage reduction seem larger than choosing a later base year. Canada’s emissions trajectory is reversed, as its emissions have risen consistently over the past twenty years, making percentage reductions seem larger if a later base year is chosen. Governments may choose base years strategically to make an emissions reduction seem larger and so they ought to be standardized with common base years.

Scenario targets specify emissions reductions from a hypothetical “business as usual” (BAU) GHG emissions growth scenario. This type of target is common among non-Annex 1, developing countries. For example, Brazil announced its intention to cut GHG emissions by 37% relative to BAU levels by 2025. However, governments are not always transparent about how they calculate BAU emissions growth and some countries forecast much quicker GHG growth in the 2015–2030 period than in the previous 20 years. The BAU forecasts are not simple extrapolations of historical growth rates. Just as countries may select base years opportunistically, they may also forecast high exceptionally BAU emissions growth to make a nominal emissions cut seem larger. Furthermore, some countries do not specify a future BAU emissions level that they intend to cut emissions from, thereby making their targets fundamentally non-quantifiable. Without an emissions reference point, it will not be possible to assess whether actual GHG emissions in 2030 are below “BAU” levels by the target amount.

Countries may also index their GHG targets to another variable, such as national gross domestic product (GDP) or population. These intensity targets require forecasting the index variable (i.e., GDP or population). A 30% reduction in the emissions intensity of GDP is not the same as a 30% reduction in absolute emissions even if both can be evaluated at the expiry of the Paris Agreement. Trajectory targets outline intentions to peak emissions by a certain year. However, if no hard cap on absolute GHG emissions is given to this peaking level, then peaking cannot be compared with other targets ex ante and will require many additional time periods after the target year to verify that emissions have actually peaked.

The INDC design allowed countries high levels of discretion in drafting their Paris pledges. This led to wide variation in the form and substance of the documents. Comparing GHG emissions reductions targets across countries requires knowing the percentage reduction, the reference absolute emissions level from which emissions are to be cut, the target year, and historical emissions levels. The headline percentage reduction target is uninformative without this complementary information, which may be omitted by governments or manipulate to inflate perceptions of their mitigation contributions. In the next section, I outline two alternative measures of countries’ targets that are less sensitive to opportunistic choice of reference levels and that allow all countries with quantifiable GHG targets to be compared on a common scale.

I now present two alternative measures of GHG emissions mitigation that better account for the structure of GHG targets than the headline percentage term that Drummond et al. (2018) use. The headline percentage indicator is highly sensitive to the choice of reference levels, since governments may make their target appear more ambitious by selecting reference levels with higher emissions. The two new indicators that I evaluate are not sensitive to opportunistic choices of reference levels, since I standardize countries’ targets using their own historical emissions and using common reference points for all countries. As such, the measures are transparent, available for all countries with explicit GHG targets, and remove aspects of political communication that may lead headline percentages to diverge from actual emissions reductions. These attributes allow discussion to focus on GHG emissions flows in the near term and relative comparisons across states. The first measure compares each country’s annual GHG emissions growth rate under compliance with their selected Paris target to their annual emissions growth rate in the preceding two decades. The second measure re-scales each country’s Paris target in terms of a percentage difference from every year between 1990 and 2014 to evaluate countries in common.

However, neither indicator that I evaluate perfectly summarizes national climate policy effort. As Aldy et al. (2017) explain, measures of implicit carbon prices or the marginal cost of abatement better capture government policy but are not directly observed or are not widely available cross-nationally. Furthermore, the two indicators I evaluate make no adjustments for national-level mitigation costs, equitable mitigation, or the consistency of national targets with the Paris Agreement’s goal of limiting temperature rise to 1.5^∘C or 2^∘C. Instead, they highlight the importance of identifying emissions baselines and changes in emissions flows. They allow for relative comparisons across countries without distortions that may arise from the vagueness of the bottom-up design of the Paris Agreement. Future work could consider equitable mitigation more explicitly (Robiou du Pont et al. 2017).

Positive values indicate slower annual GHG growth under compliance with the Paris target than historical growth; negative values indicate faster annual GHG growth under compliance with the Paris target than historical growth; and values of zero represent parity between the two periods. Countries with more ambitious targets will have positive values indicating intentions to force their GHG emissions trajectories downward. I plot the relationship between standardized growth rates (y-axis) and headline percentage reductions (x-axis) in Fig. 2. The two measures diverge considerably. The Pearson and Spearman correlation coefficients are 0.16 and 0.22, respectively. Headline Paris GHG targets reveal little about GHG emissions trajectories under the Paris Agreement.

Open image in new window

With this new indicator, we can consider each country’s headline emission reduction in relation to the range of possible percentage reductions from different reference years between 1990 and 2014. Figure 3 plots countries’ headline percentage reduction targets and each vertical bar summarizes the range of percentage changes consistent with the fixed absolute emissions level expressed in their INDC. The edges of the vertical bars indicate the smallest possible emissions reduction percentage (bottom limit) and the largest possible emissions reduction percentage (upper limit) consistent with this absolute emissions level. The size of the bars is a function of the variability of a country’s emissions between 1990 and 2014, with more stable emissions having a smaller range. The marker points indicate the chosen headline percentage target. They are all positive (e.g., + 30% reduction), except for Bosnia and Herzegovina, indicating that countries announced they were reducing their emissions, even though their absolute emissions target level may actually entail increasing emissions from one of the years between 1990 and 2014. In the comparison of Canada and Russia from above, Russia’s headline target sits at the peak of its range, while Canada’s lies more toward the middle. For scenario targets, the headline percentage emissions reduction is often inconsistent with the entire range of possible emissions reductions from a historical base year. Summarizing countries’ targets using only the headline percentage reduction target (the marker point)—as Drummond et al. (2018) do—is a misleading way to measure emissions reductions.

Open image in new window

I use these two new indicators to re-estimate the relationship between public opinion and national GHG targets that Drummond et al. (2018) analyzed previously. To measure public opinion, Drummond et al. (2018) use survey data from Lee et al. (2015), who studied the determinants of public awareness of and perceptions of risk from climate change in 119 countries in 2007–2008. In this respect, Drummond et al. (2018) join a growing research program exploring public support for international climate cooperation (Tingley and Tomz 2013; Bechtel et al. 2017; McGrath and Bernauer 2017). To study national mitigation targets, Drummond et al. (2018) compile each country’s unconditional GHG emissions reduction target in the Paris Agreement, using summary information contained in the Supplementary Information of a study by Rogelj et al. (2016). The original study uses only the headline percentage reduction to measure stringency, so a 30% reduction is always more stringent than a 20% reduction. As such, it does not account for how countries have used different reference levels in calculating these reductions. The study regresses country-level mitigation commitments on country-level public opinion, controlling for a number of confounding variables.

I replicate model 4 from table 2 in Drummond et al. (2018) with my new measures. The data are compiled from Climate Watch (2016) and cover only INDCs submitted before the Paris climate conference. I make three further changes to the data. First, the original study scores countries lacking unconditional GHG targets as “0”. This implies that these countries actually have headline GHG targets to hold emissions constant under the Paris Agreement. As shown in the previous section, many countries with GHG targets allow their GHG emissions to grow under compliance with their Paris targets, which makes the assumption that lacking an unconditional target implies holding emissions constant tenuous. I recode these as missing data, since they are missing unconditional targets. Second, since European Union (EU) countries actually have differentiated GHG emissions reductions targets as set forth in a 2016 effort sharing decision (EU 2018/842), I disaggregate the EU and include member states individually in my analyses. Finally, I cross-reference the unconditional target measure in Rogelj et al. (2016) and Drummond et al. (2018) with data from the World Resources Institute and identify a handful of countries coded as having unconditional targets in the original paper, but whose targets are actually communicated conditionally (Climate Watch 2016).

For comparison, Table 1 model I re-estimates Drummond et al. (2018)’s main specification with their data. I find the same relationships for awareness and perceived threat as in the original study. In model II, I use my re-coded headline target outcome variable, described above. The sample composition differs slightly from model I, as a number of developing countries that did not communicate unconditional targets are removed from the sample and EU countries are added in.³ I recover the same relationship as in the original study and model I.

In model III, I estimate the relationship between public opinion and changes in countries’ GHG growth rates under their Paris targets. I find a negative coefficient for public awareness, suggesting that countries with greater awareness of climate change adopted less stringent Paris targets than countries with lower levels of awareness. Nonetheless, this coefficient is not statistically significant at the p < 0.05 level. I interpret this as evidence for a disconnect between states’ headline GHG percentage targets and the underlying stringency of their targets.

Next, I consider the relationship between public opinion and GHG targets standardized using common base years. The choice of any reference year has non-trivial implications for individual countries and, accordingly, I evaluate all years between 1990 and 2014. I re-estimate Drummond et al. (2018)’s original model and plot the coefficients for public awareness of climate change and perceived threat from climate change along with their 95% confidence intervals in Fig. 4. I find coefficients in the same direction as the original study, but that are much stronger and much less precisely estimated. Every re-estimation also contains the point estimates and uncertainty intervals from models I and II. The results suggest that the effect of public awareness on countries’ GHG targets may be positive and strong, as in the original study, but the uncertainty interval does not exclude the possibility of a null or negative association between public awareness and target stringency. In every model, the recovered coefficients are not statistically significant, as their 95% confidence intervals include zero. This analysis shows that characterizing the Paris targets by their headline percentage target creates a false sense of certainty about the relationship between public opinion and the stringency of governments’ climate policy.

Open image in new window

The discrepancy between the headline percentages and standardized GHG targets merits closer scrutiny. One explanation for the relationship between nominal targets and public opinion could be that governments respond to public awareness of climate change, but that they rely on symbolic headline reductions to appear to be taking action while actually deferring costly mitigation policies. If this were the case, then the relationship should be stronger in countries where citizens perceived greater threats from climate change, but this relationship is not borne out in the data. Future work could consider more explicitly how public opinion is filtered through domestic political institutions to influence government climate policy.

Another explanation for the divergence between different measures of targets may stem from the fact that not all countries expressed quantifiable and comparable GHG targets in their INDCs. While nearly every country submitted an INDC before COP21 in Paris, not all of these documents contained explicit GHG targets, and even fewer contained economy-wide GHG mitigation targets that governments could be held accountable to in 2030. Saudi Arabia, for example, pledged to pursue economic diversification with mitigation co-benefits in its INDC, but did not specify an emissions target. Lee et al. (2015) estimate only 48.8% of Saudi citizens are aware of climate change, roughly 1.5 standard deviations below the mean level of awareness worldwide. If countries with lower levels of awareness or perceived threat systematically drop out of the analysis because they lack quantifiable GHG targets, then this could introduce bias into the models. A simple difference of means test shows that roughly 69.7% [64.1, 75.2] of citizens in countries with quantifiable targets were aware of climate change, while only 54.5% [49.7, 59.4] of citizens in countries with non-quantifiable targets were aware of climate change, a difference that is statistically significant at the p < 0.01 level. There may be a relationship between public awareness of climate change and the stringency of governments’ GHG targets, but since we can only compare states with a certain level of commitment—states that propose quantifiable GHG targets—the relationship may be obscured by the censoring in the data.

Drummond et al. (2018) argue that greater public awareness of climate change is associated with more stringent national GHG emissions reductions targets. However, I show that this finding does not hold when targets are standardized using common base years or historical emissions growth rates. In the Paris Agreement on Climate Change, countries pledged to reduce their GHG emissions by a percentage chosen by their national government. Furthermore, governments also chose the reference emissions levels used to benchmark their reductions. For governments worried about the costs of domestic GHG mitigation and worried that domestic or transnational publics would hold them accountable for their targets, this target structure created opportunities for obfuscation. Since actual emissions mitigation is a function of both reference emissions levels and percentage changes, the headline emissions reduction percentage contained in national pledges can be a misleading summary of mitigation. When mitigation pledges are standardized in terms of historical GHG emissions growth rates or using base years common to all countries, I find no relationship between public opinion and GHG mitigation targets using conventional standards of statistical significance.

Changes in absolute emissions or relative to historical emissions may be disconnected from notions of equity and fairness that underpin fragile existing climate cooperation (Climate Watch 2016; Robiou du Pont et al. 2017). Countries have made very different contributions to global warming and have very different abilities to respond. Failing to consider other priorities, such as development and the eradication of poverty, may bias findings against poorer countries (Kartha et al. 2018). Future work should consider the role of equity more explicitly in evaluating national climate policies.

This paper demonstrates that studies of INDCs ought to pay close scrutiny to the manner in which governments articulate their commitments. Headline percentages do not map clearly onto absolute or relative emissions mitigation. INDCs are a novel treaty design strategy, but they may provide opportunities for governments to feign cooperation.

This has implications for the durability and effectiveness of the Paris Agreement. It is widely held that robust climate cooperation depends on reciprocity, with counterparts making roughly equal contributions to collective governance and with provisions for differentiation based on responsibility and capabilities (Tingley and Tomz 2013; Keohane and Oppenheimer 2016). Transparency is key for comparing states’ contributions and holding them accountable for their pledges (Aldy et al. 2017). If the INDC system rewards vagueness, then accountability, reciprocity, and ultimately effectiveness are likely to suffer. In this respect, the current impasses at the United Nations climate conferences relating to transparency in the “Paris Rulebook” is unsurprising given the imprecision of national pledges.