Introduction

The United Nations Framework Convention on Climate Change (UNFCCC) recognizes that transparency and accountability are essential elements for the negotiations (Kinley et al. 2021).

Generally, agreements under the UNFCCC typically prefer “public shaming” over “policing” to ensure compliance (Kinley et al. 2021). In such a governance-by-disclosure approach, self-reporting serves as a basis to hold governments accountable (Gupta and van Asselt 2019; van Asselt et al. 2015), especially for countries seeking and receiving international funding (Biermann and Gupta 2011; Rai et al. 2019; UNEP 2021). Transparent reporting is also key to assessing progress to achieve the UNFCCC targets, improving policy by learning from prior experiences (Aldy and Pizer 2014; Jacoby et al. 2017) and gaining insights into to legitimacy, equity and justice issues (Bäckstrand et al. 2018).

The central role played by reporting within the UNFCCC system has resulted in a multitude of different reporting structures. Countries were already asked to regularly provide information in the original Convention (UNFCCC 1992, see article 12), and requirements increased under the subsequent Kyoto Protocol and the Paris Agreement. This information is provided through various plans and documents, including National Adaptation Plans for Action, Nationally Appropriate Mitigation Actions, Biennial (Update) Reports, National Communications, Nationally Determined Contributions (NDCs), and Adaptation Communications, among others.

Reporting under the Convention is typically considered a technical exercise where governments follow guidelines to provide requested information; in practice, however, this reporting is inherently political. Most reporting requirements are designed to offer a substantial amount of flexibility to countries (Weikmans et al. 2020). Given the substantial stakes in the UNFCCC negotiations, governments have both the motive and the opportunity to highlight national priorities and position themselves in the international arena (Weikmans et al. 2020; Tørstad et al. 2020). For example, some countries might highlight their structural vulnerabilities, whereas others might frame their reporting around progress on reducing greenhouse gas (GHG) emissions.

Recent studies have seen the proliferation of reports and policy documents as a promising data source to study progress on climate action, for instance, regarding ambition levels, alignment to national policies, measurement, reporting and verification, and climate education (McKenzie 2021; Morgan and Fullbrook 2019; Rosenstock et al. 2019; Tørstad et al. 2020). Yet most studies using UNFCCC reporting are qualitative in nature and focus on limited topics (e.g. (McKenzie 2021; Rosenstock et al. 2019). Given the large number of documents available, attempting a comprehensive assessment would be too time consuming using traditional qualitative approaches. Moreover, reporting requirements to the UNFCCC are only increasing, with additional information to be submitted under the Paris Agreement’s Enhanced Transparency Framework (see: UNFCCC 2015); similarly, submissions to the Global Stocktake have not yet closed, but they number over a thousand documents, many of which are hundreds of pages long (see: UNFCCC 2023). In short, whilst reporting can be a useful source of data, its sheer volume is making it increasingly difficult to use this reporting for global-level assessments using established manual review methods.

Computer-based quantitative approaches can form part of the solution here, but those methods so far are rarely used in the context of UNFCCC reporting. In only the last few years, a handful of studies have started using machine learning approaches to analyse reports such as National Communications and NDCs (Berrang-Ford et al. 2014; Biesbroek et al. 2022; Hsu et al. 2020; Lesnikowski et al. 2019). These studies have shown that computer-based methods can broadly be useful to distinguish patterns in reporting. However, they also typically use reporting as a proxy to assess progress on a given topic, without addressing political narratives. In our view, this is an oversight, given the aforementioned political nature of this reporting.

In this paper, we aim to increase our understanding of what countries consider important in reporting to the UNFCCC. More specifically, we test four prevalent hypotheses in the scientific literature about vulnerability, emissions and the impact of the Paris Agreement that help us to better understand the political messaging of these reports (see Policy attention to climate change: hypotheses).

To explore messaging, we focus on the Executive Summaries of National Communications. An Executive Summary is highly visible, intended to be read by a broad range of stakeholders, such as donors, potential partners and governmental actors. This means governments have a large incentive to not only summarise the whole document’s narrative and content, but also to emphasise their main political messages in the Executive Summary. To analyse the political nature of the Executive Summaries, we apply natural language processing (NLP) methods to test our hypotheses.

Our study represents an important step in assessing how governments frame their political messages and what they choose to report on. These aspects of self-reporting are key to understanding the usefulness of governance-by-disclosure in two directions: on the one hand, one would expect international norms such as the Paris Agreement to influence action and, by extension, reporting; on the other hand, these documents are the product of a political process and quantifying vestiges of this political process helps the climate community appropriately assess the value of these reports. As such, our work can inform the international negotiations on climate change, especially given the new transparency requirements under the Paris Agreement for which countries will submit their first reports at the end of 2022.

Policy attention to climate change: hypotheses

We focus here on two groups of hypotheses: national and international level.

National-level hypotheses: vulnerability and emissions

Although we acknowledge that exact definitions in the field are sometimes contested (e.g. Dewulf 2013), vulnerability to climate change can broadly be understood as the propensity to be adversely affected by the impacts of climate change, and typically include measurements of exposure, sensitivity and adaptive capacity (IPCC 2022a). Many non-climatic drivers such as poverty and governance influence human vulnerability to climate change; the most vulnerable countries are already experiencing severe impacts attributed to climate change (IPCC 2022a).

Vulnerability to climate impacts forms an integral part of international climate policymaking; adaptation is even named explicitly in the original Convention. Other early examples include the Adaptation Fund (5/CP.7) and the National Adaptation Plan of Action process (4/CP.7), both established at COP7, 2001. The importance of adaptation was emphasised also in key agreements such as the Bali Action Plan (1/CP.13) and the Cancun Adaptation Framework (1/CP.16), which nominally placed adaptation on equal footing with mitigation (Article 2b; see also Singh and Bose (2018)). Notably, many of these agreements emphasise the vulnerability and adaptation needs of the Global South, calling on Annex I countries to provide support.

Governments can prioritise a high vulnerability framing by mentioning more topics around vulnerability and mentioning each of these topics more often in their Executive Summaries. This can have benefits for countries in future negotiations by effectively emphasising the need for international finance and support (Betzold and Weiler 2017). Placing much emphasis on vulnerability can also have negative implications, however. For example, it may discourage investments in the region by calling attention to the risks posed by climate change; similarly, it may call into question the success of prior adaptation investments. Yet, given the sustained call for “new and additional finance” for adaptation within the UNFCCC (Donner et al. 2016; Khan et al. 2020), the benefits may outweigh the negatives for vulnerable countries. We therefore hypothesise that:

  • Hypothesis 1: Highly vulnerable countries focus more on impacts, adaptation and vulnerability (IAV) than less vulnerable countries.

Previous studies have shown the dominance of mitigation topics over adaptation throughout National Communications (Biesbroek et al. 2022). Nonetheless, it seems plausible that there are significant differences within country groups as the current and historical GHG emissions by countries are widely diverse. Typically, the most vulnerable countries are least responsible for current and accumulated GHG emissions (IPCC 2022b). Conversely, countries with high emissions have the moral (Knutti and Rogelj 2015) and legal responsibility to drastically reduce emissions. Given this difference in responsibility, it seems likely that the highest emitting countries will spend a larger share of their National Communications highlighting their mitigation efforts.

There are, however, political reasons why this may not be the case: if a country’s mitigation efforts are seen as insufficient, the Executive Summary could emphasise other issues instead to distract from this topic. In other words, countries that have cut emissions most have an incentive to emphasise mitigation compared to countries that have not made much progress.

Since higher emitting countries have a greater responsibility to act, they are more likely to emphasise any actions taken to reduce emissions in their Executive Summaries. We therefore hypothesise that:

  • Hypothesis 2: Countries with high GHG emissions place more emphasis on mitigation to limit global warming than less emitting countries.

International-level hypotheses: Paris Agreement effect

Although adaptation has always been a component of global climate policy (e.g. Article 4 of the Convention), early landmark agreements such as the Kyoto Protocol largely treated adaptation as an issue for developing countries (Khan et al. 2020). Particularly during the early years, adaptation was considered admittance that global efforts to reduce GHG emissions failed (Schipper 2006). Although this discourse gradually changed, it was not until the Paris Agreement that adaptation was consistently mentioned as an integral element of climate action and thus firmly placed on equal footing with mitigation (Lesnikowski et al. 2017). To be clear, the reasons for this are not exclusively political, but also reflect the increasingly noticeable impacts of climate change around the globe—a process which has gradually increased the importance of adaptation relative to mitigation over time (Kuyper et al. 2018; Lesnikowski et al. 2017; Singh and Bose 2018; Streck et al. 2016).

Regardless, the increased attention to adaptation in the Paris Agreement in particular is notable, with countries committing to substantial text-based reporting on adaptation (Lesnikowski et al. 2017; Singh and Bose 2018). The Paris Agreement (UNFCCC 2015) Article 7.2 states that “adaptation is a global challenge faced by all” and that it “is a key component of and makes a contribution to the long-term global response to climate change”. Furthermore, Article 7.9 states that “Each Party shall, as appropriate, engage in adaptation planning processes and the implementation of actions, including the development or enhancement of relevant plans, policies and/or contributions”. The Paris Agreement also flags the importance of Loss and Damage, transboundary risks, the need for collective efforts to adapt and the involvement of non-state actors, among other issues. We would expect to see this increased attention reflected in the Executive Summaries and therefore hypothesise that:

  • Hypothesis 3: Countries paid more attention to climate change impacts, adaptation and vulnerability after the Paris Agreement was adopted.

In addition to placing greater emphasis on adaptation, the Paris Agreement has been characterised as an important shift in framing from problems towards solutions (Du et al. 2022; Haasnoot et al. 2020). This shift is in part a reflection of scientific progress. The IPCCs 5th Assessment Report (IPCC 2014), published just before COP21, called global warming “unequivocal” (p. 4) and found it “extremely likely that human influence has been the dominant cause” (p. 17). Meanwhile, advances in technology made adaptation and mitigation options more accessible. This meant the decisions adopted in Paris shifted from understanding the problem and exploring options, to emphasise the urgency for accelerated implementation of climate change adaptation and mitigation solutions.

Since the Paris Agreement, the above trend has continued, as is reflected in the literature (Kinley 2017; Sietsma et al. 2021), in subsequent IPCC reports using stronger language on the need for immediate climate action, and in increasing investments in climate technologies (IPCC 2022a; IPCC 2022b). We therefore hypothesise that:

  • Hypothesis 4: Countries paid more attention to climate solutions after the Paris Agreement was adopted.

Methods

Machine learning techniques are becoming increasingly popular in social and political sciences since they allow the processing of large volumes of text-based data in speed and breadth not feasible using manual methods (Berrang-Ford et al. 2021). This is also true for climate policy, where large amounts of literature are becoming rapidly available, and the number of studies applying machine learning to analyse these documents is expanding (Biesbroek et al. 2018; Ford et al. 2016; Hsu et al. 2020). Here we used two NLP methods: word-frequency comparison and topic modelling.

Data collection and pre-processing

The dataset is based on Biesbroek et al. (2022) and included all officially submitted National Communications published before and through 2019. We manually extracted the Executive Summary for all documents, creating a corpus of 606 Executive Summaries. These were annotated with the following meta-data: publication year, geographic region and Annex I status. Standard pre-processing procedures were applied using Quanteda (Benoit et al. 2018), including stemming and stopword removal. We used two lower thresholds: words needed to occur at least 120 times and in at least 30 documents. We manually removed place names to obtain topics that centre around concepts and biomes rather than nations or regions. The final vocabulary consisted of 1511 unique words. More details can be found in the methods and supplementary sections of Biesbroek et al. (2022).

Data analysis—identifying key terms and topic modelling

To identify differences in the content of the Executive Summaries, we use topic modelling for all 4 hypotheses. This is a widely used unsupervised machine learning method to discover the hidden semantic structures across a body of documents (Roberts et al. 2019). In simple terms, it assumes that each text contains a mixture of a few topics and uses an algorithm to identify clusters of words which are frequently used together (e.g. a text containing “apple” is more likely to also contain “pear” than “car” or “road”); these clusters of terms then represent topics which are labelled by the researcher (e.g. “fruit” and “transport”). Topic modelling is increasingly used in climate change contexts (e.g. Hsu and Rauber 2021; Lesnikowski et al. 2019; Sietsma et al. 2021) and is particularly useful in cases where data is unstructured and where no ex ante categories exist (for a more detailed yet accessible explanation, please refer to Lucas et al. (2015) and Grimmer and Stewart (2013)).

We ran a structural topic model using the STM package in R (Roberts et al. 2019). STMs are especially adept at creating meaningful topic models for comparative social science (Lucas et al. 2015). To determine the k-value (i.e. number of topics), we follow standard practice (e.g. Sietsma et al. 2021; Tvinnereim et al. 2017; Callaghan et al. 2020) by creating models for a range of k-values and comparing them qualitatively. Specifically, the model was run at k = 5, 10, 15, 20, 25, 30, 40 and 50 to show a wide range of results. These were qualitatively assessed for coherence, accuracy and breadth of representation of the original documents. The 25–35 range was chosen as most promising, as most topics here had a clear focus, without many “junk” topics, and with generally clear distinctions between IAV and mitigation topics. Within this, k = 33 had the highest semantic coherence (a standard quantitative measure for topic quality) and was chosen as our final model.

Topics were labelled based on keywords (see Table SM2.1) and the most closely associated documents per topic. One topic was qualitatively assessed as incoherent and was thus removed. The remaining 32 topics were then classified by the researchers in an ordinal scale ranging from “highly mitigation related” to “neutral/both mitigation and adaptation” to “highly IAV related”. Four researchers (S. J. W., A. J. S., S. K. and R. B.) each created independent classifications based on the topic correlates (Sievert and Shirley 2014) (Figure SM2.1), as well as the most-associated documents and keywords per topic, and based on the mean of these individual qualitative clusterings, each topic was assigned to one of five classes: strong adaptation, weak adaptation, strong mitigation, weak mitigation and cross-cutting themes.

Testing the hypotheses

To test the first two hypotheses, we took the mean prevalence of IAV topics and mitigation topics per Executive Summary and compared these metrics against the country’s vulnerability score (H1) and emissions data (H2) in the year preceding the report’s submission. If our first hypothesis is true, countries with a higher vulnerability should also have a higher prevalence of IAV-related topics. Similarly for the second hypothesis, high-emission countries should have a higher prevalence of mitigation topics.

As no global quantification method of country-level vulnerability is universally accepted, we used the four most established global indices of national climate risk and vulnerability: the Notre Dame Global Adaptation Initiative (ND-GAIN), the World Risk Index (WRI), INFORM and the Climate Risk Index (CRI). Earlier analyses have found considerable differences as well as overlap between the indices (Feldmeyer et al. 2021; Garschagen et al. 2021). Given space constraints, as well as data availability especially for earlier years, we present the results of the ND-GAIN (Chen et al. 2015) and the WRI (Aleksandrova et al. 2021). Further descriptions of these indices can be found in SM1. Figure SM2.2 presents additional figures using INFORM and the CRI. The SM2.2 figures also include the Environmental Performance Index (Hsu and Zomer 2016) and the physical vulnerability to climate change index (Feindouno et al. 2020), both of which focus more on policy.

For emissions data, we use the most recent version of the Global Carbon Project (Andrew and Peters 2021). The database includes both per-capita and total emissions. Given that historical emissions play a significant role in the UNFCCC negotiations, we compare these yearly emissions to the cumulative emissions per country since 1750.

We calculate r-squared and Spearman correlations for hypotheses 1 and 2, characterising the relationships between vulnerability (GAIN & WRI) vs. IAV/mitigation topic prevalence and emissions (total & per capita) vs. IAV/mitigation topic prevalence. In all cases, vulnerability and emissions data from the year preceding the report submission is used, or the first available value for National Communications published prior to the range covered by the indices. We combine this numerical baseline with qualitative observations to note broader trends in the data.

To test whether the Paris Agreement caused a shift in framing political priorities, we made use of the same mitigation-adaptation topic classification as above. Additionally, all topics were manually classified on whether they are “solution-oriented”, noting which topics were geared towards action and practical implementation, often including terms such as “program”, “policy” and verbs relating to planning and implementation. To determine solutions-oriented topic classes, two classification rounds were done independently by the researchers and the average score was taken to ensure consistency in the classification. The topic prevalence of reports submitted directly prior to the Paris Agreement (2007–2015) was then compared against the scores of reports submitted afterwards (2016–2019). The first period is longer to reflect the lower number of submissions pre-Paris and the cutoff aligns with reporting guideline changes published in 2007 (Breidenich and Schmidt 2011). We would expect to see an increase in reporting on both IAV topics (H3) and solutions-related topics (H4) after the Paris Agreement was adopted.

In addition to topic modelling, we make use of word count-based statistics. Although relatively simple, these have been shown to be highly effective at identifying how different sides in a debate frame their arguments by highlighting key terms (e.g. Risi and Proctor 2019; Supran and Oreskes 2021). Here, we used a chi-square test to identify words which are significantly under- or overrepresented in a subset of the corpus to test hypotheses 3 and 4. We divided the texts in two ways: (1) National Communications from Annex I countries compared to National Communications from non-Annex I countries, where we expected to see mitigation-related terms being overrepresented in Annex I submissions and IAV terms in non-Annex I submissions; and (2) comparing submissions post-Paris Agreement to those before, where we expected to see an overrepresentation of both IAV and solutions-related terms in the post-Paris texts. To test the robustness of this method, we include the same statistics for random subsets of the dataset, as well as using log-likelihood instead of chi-square; this can be found in the supplementary materials.

Limitations

A number of limitations arise from the methodology and data used. Shortcomings of using National Communications to track adaptation have been discussed elsewhere (Biesbroek et al. 2018, 2022; Ford and Berrang-Ford 2016). Whilst our dataset offers a global perspective, some countries are underrepresented in the submitted National Communications, for example due to different submission times or resource constraints. In order to address this, we grouped National Communication submissions in regular time frames to smooth breaks in data distribution. Additionally, the UNFCCC guidelines are not detailed enough to ensure consistent reporting between countries and over time (Ford and Berrang-Ford 2016), creating some variation in what is reported. Furthermore, National Communications represent national reporting, therefore their analysis may overlook sub-regional differences, such as urban or local particularities, as well as specific sectors or population groups (Araos et al. 2016; Ford et al. 2015). Lastly, by relying on the dataset of Biesbroek et al. (2022), NDCs in 2020 or later are missing.

Results

General results

The 32 topics that emerged were clustered into five classes (see Fig. 1). In total, 7 topics were classified as strong adaptation, 3 as weak adaptation, 7 as cross-cutting themes, 1 as weak mitigation and 14 as strong mitigation. Generally, the proportion of each class remains relatively stable over time.

Fig. 1
figure 1

a, b Proportion of the five topic classes over time, by Annex I and non-Annex I countries. Although there are some changes over time, there is a persistent general trend that non-Annex I countries (which are typically more vulnerable to the impacts of climate change) emphasise issues related to adaptation and vulnerability more in their submissions. A breakdown with individual topics and per geographical region can be found in SM2.5 (a, b) and SM2.6 (a–f). c, d Most distinctive words by Annex status. Keywords are stemmed so that different word-forms are counted together (e.g. “ vulnerable” and “vulnerability” both become “ vulner”). Terms in c occur relatively frequently in submissions by Annex I countries; terms in d occur relatively frequently in submissions from non-Annex I countries

Full size image

Using these categories, Fig. 1a–b highlight differences in class proportion by Annex countries. Annex I countries discuss strong and weak mitigation topics and cross-cutting themes more than topics in either of the adaptation classes. Non-Annex I countries place significantly (p < 0.05) more attention on adaptation and less on mitigation topics than Annex I nations. This matches the results from the term-frequency comparison shown in Fig. 1c–d, showing that the top words for Annex I are aligned with mitigation topics (e.g. greenhouse, transport) whilst top non-Annex I words are linked to IAV topics (e.g. vulnerability, capacity). A key exception here is the term mitig*, which is also overrepresented in non-Annex I reports; note however that in many cases this refers to “mitigating risks” rather than decreasing emissions.

Overall, the distribution between all topic groups has remained relatively stable over time. There are some notable shifts for individual topics. For Annex I countries, Innovations & programs and Research & observations increase the most over time, whilst Federal energy & transportation and Macroeconomics topics decrease. For non-Annex I countries, GHG reporting increases most significantly over time whilst Macroeconomics, Global conventions and Mitigation financial instruments decrease. Changes of individual topics within these groups are discussed in more detail in relation to H3-4 below.

The topic model group results broadly overlap with the key terms: some of the most-distinguishing words are related to IAV (e.g. resilience, adapt*) and others to solutions (e.g. action, plan, program). Yet overall, the recently dominant words relate to new programmes with their associated acronyms and terminology.

Topic proportion and dominance per global regions (see Figure SM2.6a-f for details) show a similar distribution as between Annex I and non-Annex I countries. For example, adaptation related topics as Rural responses and Livelihoods & water resources are extremely dominant in Africa whilst less significant in other world regions. Similarly, in North and Central America, major topics include Projected livelihood impacts, Adaptive capacity and Coastal & island impacts, but also Green programs. In contrast, Europe shows more prominence in topics such as Macroeconomics, Kyoto & GHG and Measurements. Major topics in Asia include diverse issues as Instruments & programs, Hydrological impacts and Greenhouse gases. South America is dominated by Projected livelihood impacts. Rio programs is large at the beginning of the study time period, but falling off in more recent years. Forest management & programs does the opposite, starting small and becoming dominant in the most recent time periods. In Oceania, Coastal & island impacts, Mitigation governance and Adaptive capacity stand dominant.

Results by hypothesis

Vulnerable countries broadly emphasise impacts and adaptation (H1)

In line with our hypothesis, more vulnerable countries tend to discuss IAV more extensively in their Executive Summaries, compared to less vulnerable nations who put more emphasis on mitigation. Subtracting the mitigation score from the IAV score to get one single metric for the balance between this topic, the correlation is statistically significant (p < 0.01) and moderately strong (ND-GAIN: − 0.67; for WRI: 0.46). Similarly, some of the most-distinguishing words for non-Annex I countries are related to IAV.

At the topic level, the same general trend holds (see Fig. 2a, b). The effect is especially pronounced for Small Island Developing States (SIDS), who submitted half of the top 20 most IAV-focussed Executive Summaries. By contrast, European countries tend to be among the less vulnerable countries and have a low prevalence of IAV-related topics. For Asian and South American countries, the effect is less pronounced, but more vulnerable countries in these two regions overall do discuss IAV more. Notably, some highly vulnerable African countries have low topic scores for both IAV and mitigation as these focus instead on the more process-oriented cross-cutting topics.

Fig. 2
figure 2

The mean prevalence of topics per report is calculated for two categories of topics: impacts adaptation & vulnerability (IAV) and mitigation topics. The resulting score is plotted against two vulnerability indices: the Notre Dame Global Adaptation Initiative (ND-GAIN, a) and the World Risk Index (WRI, b). Lines of best fit (least squares) are included. IAV proportion and the vulnerability indices are moderately but significantly correlated. A moderately positive correlation was found between mitigation proportion and per-capita and cumulative emissions. Note that the x-axis for b is flipped so that more vulnerable countries are plotted on the left, in line with a. In b, two reports by Vanuatu (1999 and 2016) were removed to improve legibility as the country’s WRI scores were extreme outliers (55.9 and 56.6 respectively). In c, the same topic scores are used, but the mitigation score is subtracted from the IAV score so that the colour represents the balance between the two topic groups. In other words, greener points discuss more IAV than mitigation and vice versa for blue points. The position is determined by the country’s cumulative versus per-capita GHG emissions (log scale), showing a cluster of high-emitting countries which emphasise mitigation in the top right corner

Full size image

The results are highly dependent on the vulnerability index used. ND-GAIN scores are fairly evenly distributed, making the effect more visible. Almost all low-vulnerability countries discuss more mitigation topics here; generally, more vulnerable countries emphasise IAV but this effect is less consistent. By contrast, WRI shows a large cluster of low-vulnerability countries, most of which are in Europe. Broadly, these countries report extensively about mitigation, but some also discuss IAV. The differences in topic scores appear more pronounced for the small group of countries with a very high WRI score (i.e. highly vulnerable). In part, the more clustered appearance of the WRI plot may be due to data availability: Annex I countries have reported considerably more than non-Annex I countries, so there are less data points for high-scoring countries.

The two other widely used vulnerability indicators are included in the supplementary materials, alongside plots using only sub-components of the indices (Figure SM2.2). The INFORM scores are similar to the WRI scores, though some low-ranked Asian and North American countries still emphasise IAV. The CRI scores do not appear to correlate with mitigation or IAV topics. This may be due to the lack of historical CRI scores; given that this index is based on climate-related disasters in a given year, it may also indicate that messages in the Executive Summary are not influenced by single events.

Overall, we see general support for hypothesis 1 from both the topic model results at the country level and the word-frequency differences between Annex I and non-Annex I countries.

Highest-emitters tend to emphasise mitigation (H2)

Results for the second hypothesis are similar to those of the first: at the word level, mitigation-related words are especially prevalent for Annex I countries (Fig. 1c, d). For the topic model results, the top-20 most mitigation-focussed Executive Summaries were almost exclusively submitted by European countries, with two National Communications from New Zealand (2017 and 1994) and one from Tunisia (2019) being the only exceptions.

For mitigation too, it matters which metric is used. Per-capita emissions show a moderately strong positive relationship with mitigation scores (Spearman r: 0.51, p < 0.01). Almost all countries with very low emissions emphasise IAV topics whilst mostly or completely disregarding mitigation in their abstracts. European countries especially tend to emphasise mitigation topics, even for the countries where per-capita emissions are close to the median of 4.4 MtCO2eq. Some of these countries do have fairly high cumulative emissions though, lending some support to our hypothesis.

More broadly, we see a weaker effect for cumulative emissions (Spearman r: 0.43, p < 0.01) and counting absolute emissions cumulatively or yearly does not lead to large differences (see Figure SM2.3). In both cases, the data is unevenly distributed, i.e. most countries have fairly low absolute emissions, relative to the few large outliers (notably, the USA, China, Germany and the UK). For these outliers, the mitigation scores are generally higher than the IAV scores. Within the large group of lower-emitting countries, IAV topics are overrepresented generally, but there are outliers here from all regions.

We find limited support for hypothesis 2: high per-capita emissions broadly correlate with emphasis on mitigation in the countries’ Executive Summary, but this is most apparent for the largest emitters; the effect is also less pronounced for absolute emissions compared to per capita. We do see significant differences in word-use between Annex I and non-Annex I countries, though it is unclear whether this reflects a larger focus on mitigation action or on mitigation-related procedural terminology.

No consistent growth in adaptation topics post-Paris Agreement (H3)

Figure 3 shows the size and growth rate of topics comparing submitted data before (2007–2015) and after (2015–2020) the Paris Agreement. Comparing the global average size of topics over both periods, mitigation topics are overall slightly larger than IAV topics. Largest mitigation topics comprise National GHG inventories, Mitigation governance and GHG reporting, whilst largest IAV topics are Adaptive capacity and Coastal & island impacts. Meanwhile, the largest cross-cutting topic is by far Forest management & programs.

Fig. 3
figure 3

Comparison of topic prevalence and growth over time. All countries with at least one report in either the pre-Paris (2007–2015) or the post-Paris (2015–2019) time periods are selected, using the reports furthest from the time split if there are multiple in one period. The average topic score for both periods is then calculated per region, depicted here as the size of the circle. The growth rate of these topics from pre-Paris is also calculated and determines the circle’s colour. Topics are grouped by whether they are solution-oriented; rows are coloured by topic classification (impacts, adaptation and vulnerability (IAV) or mitigation). For our hypotheses to hold, we would expect to see predominantly dark purple circles in the blue IAV rows and in the bottom rows labelled as solution topics. Instead, most circles are red or pink, indicating no change or shrinking of attention after the Paris Agreement

Full size image

Globally, the largest topics are also the ones with higher growing rates after the Paris Agreement. The highest growing rates are observed for GHG reporting and Geography, followed by Forest management & programs, National GHG inventories, Mitigation financial instruments and Kyoto & GHGs. Large topics as Adaptive capacity, Livelihoods & water resources, among other adaptation topics, have remained stable. Medium large topics as Country characteristics, Global conventions and Innovations & programs have even decreased. Overall, mitigation topics show the highest rates of growth after Paris compared to IAV topics. Cross-cutting themes tend to remain stable or decrease.

Average topic size and growth rate are driven by large regional differences. Oceania’s largest topic is Mitigation governance, which contributes heavily to its global share. Europe and Asia contribute to larger topic size of mitigation topics, such as GHG reporting and National GHG inventories. In contrast, the largest adaptation topics seem to be driven by Oceania’s Coastal & island impacts as well as Adaptive capacity, followed by Projected livelihoods & impacts in South America and Livelihoods & water resources in Africa. Cross-cutting themes also show huge regional differences. Forest management & programs is not only the dominant issue in South America and less in North America but shows additionally a high growth rate after Paris. Further regional differences stand out for Rural responses, which is the dominant topic in Africa, and Country characteristics which is larger in Europe.

Looking at the topic growth rates after Paris, it stands out that mitigation, followed by cross-cutting topics, grew more in all regions except for North America, compared to IAV topics. The latter grew more only in North America and Asia, and for individual topics, in Oceania and South America.

Looking at the most distinct words in documents before and after the Paris Agreement (as shown in Fig. 4), variations on the words adaptation and resilience prove strong in more recent documents. This supports the notion that IAV holds more prominence since Paris. Yet, mitig* and GHG, as well as terms related to sectoral emissions, are also present.

Fig. 4
figure 4

Most distinctive words pre-Paris (a) and post-Paris (b). A few terms are acronyms, mostly related to different types of greenhouse gas (GHG) emissions: IPPU stands for industrial processes and product use; AFOLU refers to agriculture, forestry and other land use; LULUCF is short for land use, land use change and forestry. Two further acronyms relate to international programmes: NAMA here means nationally appropriate mitigation actions; CDM is the acronym for clean development mechanism, a mitigation credit system originally established under the Kyoto Protocol. Like Fig. 1, terms are stemmed

Full size image

To summarise, we see mixed evidence for hypothesis 3: whilst some individual IAV-related words do tend to be used more post-Paris, overall, IAV topics do not show a consistent growth over time nor regions. Instead, mitigation topics continue to dominate the discussion.

Regional priorities, not solutions, gaining importance (H4)

Solution-oriented topics were considered those involving and pointing towards action and implementation, including decision making and funding. Of the topics strongly pointing towards solutions, 7 belong to the mitigation class, 6 to cross-cutting themes and only 1 explicitly to IAV. The non-solutions topics are composed of 5 mitigation, 5 IAV topics and 4 cross-cutting themes.

The non-solutions class shows, on average, slightly larger topic sizes than both the strong and weak solutions classes. Among the largest strong solution topics and with moderate to high growth rate after Paris, we find Forest management & programs, Mitigation governance, Rural responses and Energy efficiency. Yet, equally large but fastest growing topics are found in the non-solutions class. These comprise, for instance, Coastal & island impacts, National GHG inventories, Concepts and GHG reporting. The smallest topics overall belong to the strong solutions class, for example Federal energy & transportation and Regional mitigation planning.

Large regional differences in size and growth rate of topics are observed. Solutions largely differ per region, broadly in line with regional priorities. Whilst one or two regions tend to dominate certain topics—also with moderate to high growth after Paris—most topics remain very small. For instance, Africa and South America show only one very large solution topic (Rural responses and Forest management & programs respectively). For North America and Oceania, Adaptive capacity is dominant alongside Forest Management and Mitigation governance respectively. By contrast, in Asia and Europe, solutions topics are slightly larger in size and we see a larger diversity of topics.

Conversely, there are no large regional outliers for the non-solutions topics. Moreover, they show a moderate to high growth rate after Paris. In other words, regions tend to discuss equally and diversely on non-solutions topics. An exception are the topics related to climate impacts: they are generally non-solution topics.

To summarise, strong solutions topics are biased towards mitigation and present large regional variations, with a couple of dominant topics per region. In contrast, non-solution topics are more constant in size, growth rate and regional distribution. The most homogenous and straightforward post-Paris impact seems to be the growth of the non-solutions topics GHG reporting and Concepts. Overall, we find that regional priorities may influence the reporting on individual topics, but we see no evidence for our hypothesis that solution-focussed topics are increasing in prominence.

Discussion

Reporting to the UNFCCC is an important mechanism to capture how countries are progressing towards the global goals on mitigation and adaptation. We demonstrate how some of this reporting is used to frame issues by highlighting some topics and excluding others. Here we highlight three key findings and what they tell us about the future of climate policy tracking.

First, our results show that National Communications broadly reflect national and regional priorities in mitigation or adaptation, largely supporting our first two hypotheses. We observe that more vulnerable countries focus more attention in their National Communications towards IAV than less vulnerable countries, in line with previous studies (Biesbroek et al. 2022). We also find that high emitters tend to place more attention in their National Communications on mitigation than lower emitters. These results are perhaps unsurprising, but this work is one of the first large-scale empirical confirmation that countries highlight nationally important issues (and perhaps leave out others).

Second, we find limited evidence on the effect of the Paris Agreement on the solutions focus. The Paris Agreement stands as a key milestone in the evolution of climate policy and action, producing many aspirational targets and calling for an increase in solution-focused thinking. Whilst highly ambitious national policies are being formulated across the world, there is little knowledge on whether progress is being made towards achieving those ambitions (e.g. Meinshausen et al. 2022; UNEP 2021). Our analysis indicates that implementation post-Paris is not clearly visible across National Communications’ summaries; although our approach does not allow us to distinguish between a lack of reporting and a lack of action, both are cause for concern. In addition to providing transparency, reporting should help countries learn from each others’ experiences, but when reporting on actions is limited, this learning will also be limited. Furthermore, a lack of action would be consistent with the findings of earlier authors who noted an “implementation gap” (Roelfsema et al. 2020; Runhaar et al. 2018). This is worrisome not just because it implies that countries are failing to live up to their collectively agreed goals, but more importantly, is because of what those goals represent: they are a recognition that rapid and inclusive climate action is necessary to address future climate impacts as well as current ones.

Third, we find limited evidence towards the effect of the Paris Agreement on focusing on adaptation. Our analysis did not indicate the increase of adaptation topics and a corresponding reduction in prominence of mitigation topics in the National Communications. This is surprising, given first, the growing evidence on climate change, especially on increased frequency and severity of extreme weather events induced by climate change, as well as on their actual and potential impacts (James et al. 2019; Otto et al. 2016; IPCC 2022a); and second, the high expectations of those within the IAV community. Magnan and Ribera (2016) for example argued that the Paris Agreement may lay “foundations for a new era for climate change adaptation”.

Here, too, it is difficult to distinguish between underreporting and in action. For adaptation, this fits into a larger pattern: even large-scale collaborative efforts must rely on relatively crude heuristics to determine whether progress is being made in adaptation (UNEP 2021). Some degree of underreporting appears likely, as several authors have discussed the difficulties around setting adaptation goals and indicators, e.g. the difficulty for defining measurable and comparable indicators, as well as for building monitoring and evaluation systems and for designing a framework to take stock adaptation progress (Ford et al. 2015; Ford & Berrang-Ford 2016; Lesnikowski et al. 2016; UNEP 2021). Whilst the National Communications provide valuable insights in country progress, intra-country progress on adaptation will be difficult to extract across UNFCCC reports. The adaptation communications and further guidelines for stocktaking may play a critical role in overcoming these challenges and creating adaptation reporting that is more consistent; however, given the subjective nature and fuzzy concept of adaptation, it seems likely that countries will continue to use their reporting not just as a tool for transparency, but also for political ends (Gupta and van Asselt 2019, Weikmans et al. 2021). This raises the question how meaningful conclusions can be drawn from reporting that is large in both volume and variety.

How and when to use computer-based tools should be part of these discussions. As our results illustrate, these methods may be especially useful for high-level assessments and to identify big-picture patterns in large corpora of text data.

Computer-based tools are easiest to apply when data is available in comparable formats. The recent adoption of “common tabular formats” (FCCC/PA/CMA/2021/L.21) are especially interesting to create coherent, consistent and comparable data. These are mostly centred around emissions and mitigation, but they should provide information on progress towards nationally determined contributions as a whole, which includes adaptation as well. During the negotiations at COP26 in Glasgow, some countries appeared to fear losing flexibility as a result of standardised formats (Evans et al. 2021; see also FCCC/PA/CMA/2021/L.21 paragraph 5). The real issue however may not be one of flexibility, but of reporting capacity and funding, particularly for the Global South. This is also apparent in our database: many non-Annex I countries have submitted 3 or fewer National Communications, whereas most Annex I countries have submitted their 7th. The increases in required reporting under the Paris Agreement will fall flat if they are not matched with funding and support.

Despite these methodological limitations, tracking whether progress towards achieving the high ambitions set in the Paris Agreement is critical to hold governments accountable and to ensure timely and adequate responses to exacerbating climate change challenges. Global assessments, such as those presented here, are an important way for the scientific community to help improve transparency in global progress on climate action.