1 Introduction

In 2015, several international environmental regimes reached a new phase with agreements sharing specific features of institutional design, which can be classified as soft law arrangements based on non-binding commitments, wide-ranging provisions, voluntary action and a lack of sanctions for non-compliance, while aiming at universal membership (Biermann et al. 2017; Kelman 2017; Milkoreit and Haapala 2019). These attributes and characteristics of intuitional design give rise to new questions about regime effectiveness, which is one of the fundamental issues of international environmental regime theory (Wettestad 2001). Because of the recent adoptions of this type of institutional design—featured in prominent regimes such as the Sustainable Development Goals (SDGs), Paris Agreement and Sendai Framework for Disaster Risk ReductionFootnote 1—, soft law regimes need to be investigated more thoroughly as an important subset of international environmental regimes. For this research endeavour, however, the concepts that have been established for the assessment and analysis of regimes built on hard law regulation need to be adapted to this new soft law subset and its specific characteristics.

The aim of this study, then, is to empirically investigate the Hyogo Framework for ActionFootnote 2 (HFA) on disaster risk reduction (DRR) as a specific case of a soft law regime and its prospects of effectiveness. Since these prospects are dependent on participation and compliance with the rules (Barrett and Stavins 2003; Bernauer et al. 2013; Compton and Hart 2019; Downie 2013; Young 2013), this study examines these key factors under the HFA by analysing data and descriptive statistics based on national reports and their indicators on DRR measures and follows a three-step approach. First, the patterns of participation—operationalised as the number of reports and continuity of reporting—are analysed to see to what extent the soft law regime was successful in attracting countries to submit reports over time. Second, compliance—defined as the change in behaviour in accordance with the regime’s provisions—is explored to assess to what extent compliance was achieved. Third, the study investigates the pervasiveness of compliance across countries and periods. Thereby, providing detailed information on the national development of the DRR status over time, which further contributes to DRR research by identifying successful cases that are usually invisible (Di Baldassarre et al. 2018; Gall at al. 2014).

While we have seen quite extensive research on two of the prime examples of soft law regimes, namely the SDGs and the Paris Agreement, the Sendai Framework has not drawn as much scholarly attention. In general, DRR and its governance have been neglected in comparative studies and on a global scale, as a UN commissioned report concluded (Gall et al. 2014). This is despite the fact that the Sendai Framework and its predecessor, the Hyogo Framework for Action, share crucial similarities with the aforementioned prominent regimes and are inherently connected with the issues of sustainable development and climate change (Biermann at al. 2017; Kelman 2015, 2017; Milkoreit and Haapala 2019; Schipper and Pelling 2006b; UNISDR 2015). Whereas the Sendai Framework was tailored to precisely measure and track progress, it is still a long way off before we can expect sufficient comparable DRR data for a detailed analysis. However, the HFA, as a first-step agreement, already provides an extensive and publicly available dataset. Based on self-reported DRR status, an inventory of national activities in several priorities for action was created over four reporting periods, enabling the assessment of the status and changes over time. While being only “partially successful” (UNISDR 2017) due to missing its ultimate goal to substantially reduce disaster losses by 2015 (UNISDR 2005), the HFA provides a unique data source for systematically assessing progress in DRR. Accordingly, this study shall not only contribute to research on effectiveness of international environmental regimes by focusing on one specific instance of the specific subset of cases (soft law regimes) sharing characteristics of institutional design. It shall also further DRR research by, for the first time, comparatively evaluating DRR on a global scale from a regime effectiveness perspective, with the prospects of gaining crucial insights for the future of disaster risk reduction and its governance.

This study will proceed as follows. First, international environmental regime theory and specific considerations of institutional design are introduced followed by a discussion of regime effectiveness with respect to soft law. The study, then, provides insight into the DRR regime and the Hyogo Framework for Action along with common features of the subset of regimes under investigation incorporating similarities to the SDGs and the Paris Agreement. After reviewing the intricacies of the HFA data, the empirical findings on participation, compliance and localised change are presented. The study concludes with a discussion of the findings and an evaluation of the effectiveness of the HFA, before wider implications for institutional design, regime effectiveness and DRR are drawn.

2 International environmental regimes and their effectiveness

2.1 International environmental regimes

International environmental regime theory stems from the International Relations scholarship of the 1970s and has since then developed to a specific field of study due to the sparking interest in the phenomenon and implications of an increasing number of international regimes that were established to address a variety of environmental problems (Mitchell 2003; Pettenger 2013). Thereby, the objects of interest are “social institutions consisting of agreed upon principles, norms, rules, procedures and programs that govern the interaction of actors in specific issue areas” (Levy at al. 1995, p. 274). In the 1990s, there was a shift from theoretical debates and a focus on regime formation and persistence towards the question of whether they make a difference. Even the most extensive empirical studies could not provide clear answers to the question (Miles at al. 2002), but leading scholars agree that effectiveness depends significantly on regime design rather than on the type of problem (Young 2011), although the problem type might influence the choice of institutional design during regime formation (Hafner-Burton et al. 2011).

It is noteworthy that the design of international environmental regimes has evolved over the last decades particularly in the dimension which underlies all the institutional features that a regime can be designed with: its legal character or “the degrees of commitment, bindingness, and constraint that are imposed on members” (Böhmelt and Butkutė 2018, p. 354). Earlier, as conventional (and realist) wisdom suggested, treaties only established modest but binding commitments that were based on the lowest common denominator, leading to high compliance to the rules but little effect, since members would not want to excessively constrain themselves (Bernauer at al. 2013; Parker 2011; Victor 1997). In contrast, the latest and most prominent international environmental regimes such as the SDGs and Paris Agreement display more ambitious goals and targets founded on the guiding principle of non-binding commitments and voluntary action, which are meant to scale up to reach shared global goals while providing a platform for knowledge exchange (Biermann at al. 2017; Kelman 2015; Milkoreit and Haapala 2019; Victor 1997).

This distinction of institutional design or legal status has often been established in the binary categories or continuous scale of “hard” and “soft” law,Footnote 3 which have been operationalised in terms of the dimensions of precision, obligation and delegation (Abbott and Snidal 2000; Hafner-Burton at al. 2011). Hard law or legalisation is demarcated by strong regulations in all three aspects, and entails significant costs by restricting actors' behaviour and even their sovereignty (Abbott at al. 2000; Abbott and Snidal 2000; Stein 2008) making it less attractive to any (potential) member by making compliance more challenging (Abbott and Snidal 2000; Böhmelt and Butkutė 2018; Downs et al. 1996; Goldstein and Martin 2000; Skjærseth et al. 2006; Spilker and Koubi 2016). Soft law, on the other hand, begins once legal arrangements are weakened along one or more of the aforementioned dimensions resulting in a large variation in institutional design (Abbott at al. 2000; Abbott and Snidal 2000). Thereby, soft law can offer advantages including but not limited to flexibility when dealing with uncertainty, enabling learning processes, accommodating power differentials and retaining decision-making power, lower probability of violation and thus audience cost, as well as higher attainability and compliance (Abbott at al. 2000; Abbott and Snidal 2000; Böhmelt and Butkutė 2018; Böhmelt and Pilster 2010; Hafner-Burton at al. 2011; Koremenos 2005; Koremenos et al. 2001; Rosendorff 2005; Rosendorff and Milner 2001). Thus, the institutional design in terms of the legal character of the regime is of high importance for determining the effect of a regime.

In conclusion, the research frontier in international environmental regimes is concerned with the latest environmental regimes and their designs. Because of their prominence in the past, the majority of the attention has been paid to hard law regimes and their investigation and analysis. However, regimes are increasingly more often characterised by soft law, whereas the capabilities of soft law regimes to solve the problems they were meant to address, i.e. their effectiveness, have not been thoroughly assessed. The following section revisits the investigation and study of regime effectiveness and presents how the shift in institutional design affects the understanding and application of theoretical concepts that are necessary to evaluate the prospects of effectiveness for soft law regimes.

2.2 Regime effectiveness

There has been a plethora of definitions and operationalisations of effectiveness, depending on the interest of the researchers.Footnote 4 While there are scholars arguing for an operationalisation as the extent to which regimes contribute to solving or at least alleviating the problems they are intended to, different approaches have focused on created outputs manifesting in regulations and infrastructure, or outcomes, meaning changes in behaviour (Young 2011). Much research, including extensive and prominent research projects in the 1990s and early 2000s (Miles at al. 2002; Young 1994, 1999), indeed committed to focusing on changes in behaviour rather than purely environmental improvements (Mitchell 2003; Young 2011). The rationale behind focusing on compliance acknowledges that the alleviation of environmental problems requires this change in behaviour; it also reflects the elusiveness of the assessment of the problem-solving capacity, estimation of counterfactuals and measurement of biogeophysical effects of regimes, although these might be better indicators for regime effectiveness (Andresen 2014; Hafner-Burton at al. 2011; Helm and Sprinz 2000; Miles at al. 2002; Mitchell 2003; Sprinz and Helm 1999; Young 2001).

In the light of soft law and non-binding commitments, this change in behaviour can be understood as compliance to the rules or guidelines that the regime established including the implementation of the suggested activities. This is in contrast to the conventional tradition to define compliance as rule following which does not necessarily entail a change in behaviour. However, soft law with voluntary action and non-binding commitments renders rule following obsolete or in the absence of rules impossible. Thus, compliance has to be understood as complying with the idea and working towards the goal of the regime, which is captured in the change in behaviour. This kind of compliance analysis may entail the problem that the change in behaviour does not meet the set goals and, thus, is not sufficient for solving the problem at hand (Mitchell 2003; Underdal 2002; Young 2011). However, it can still be important for evaluating the effect of a regime, particularly in issue areas that are characterised by high uncertainty and lack reliable future projections.

For assessing effectiveness, it is in any case necessary to identify a scale for evaluation and reference points, the latter of which can be either or both of the following: no-regime baselines for relative improvements and/or desired values, for instance, in the form of goals (Mitchell 2003; Underdal 2002). Oftentimes, regimes do not provide data from the time before its inception with regards to the problems they want to alleviate (Mitchell 2003), while creating counterfactuals for the absence of the regimes are difficult to create (Hafner-Burton at al. 2011; Rittberger and Mayer 1993; Young 2001). Therefore, there is a focus on the desired values and goal achievement in the absence of ante-regime reference points. Even then, a multitude of contextual factors and determinants, which can be grouped in three categories, namely country characteristics, international context and characteristics of the environmental problem, might interfere or facilitate the effectiveness of environmental regimes (Brown Weiss and Jacobson 2000; Young 2011).Footnote 5 A summary of existing analyses concluded that there are major obstacles to analysing effects accurately, while at the same time, only a small subset of environmental regimes has been investigated in spite of existing data (Mitchell 2003).

Furthermore, effectiveness is not only dependent on compliance or the change in behaviour, but also on membership or participation it inspires or enforces (Barrett and Stavins 2003; Hafner-Burton at al. 2011), which can be seen, for instance, in the case of the Montreal Protocol (Albrecht and Parker 2019). Because most international environmental regimes are established to deal with challenges where unilateral action cannot suffice or is not desirable since the matter concerns more parties, participation and coalition building is a crucial element for success (Barrett and Stavins, 2003; Bernauer at al. 2013; Compton and Hart 2019; Gawronski and Olson 2013). While research on regime formation considers membership and participation as the involvement and inclusion of the most relevant actors (Barrett and Stavins, 2003; Bernauer at al. 2013), the study of regime effectiveness rather focuses on the ratification process or continued participation. The latter of which is necessary if ratification is obsolete after the adoption of the regime. In the case of soft law regimes, the premise holds that participation is still one key factor of effectiveness, since the challenge they are supposed to address cannot be met unilaterally. Likewise, the distinction between regimes that do or do not need ratification holds. However, the important aspects of participation for the prospects of effectiveness are then the level of participation achieved and its continuity over time.

Therefore, even for regimes following the principles of soft law with non-binding commitments and vague rules, participation and compliance are arguably still of high importance, since they are crucial for determining the effect or impact of the regime. Thus, their investigation can reveal relevant information for the effectiveness of the regime, regardless of whether the regime managed to solve or alleviate the environmental issue. Following this logic, this study acknowledges the shift in institutional design towards soft law and adapts the prevalent concepts accordingly. Thereby, it focuses on the continuity of active participation and understands compliance as change in behaviour, i.e. the intermediate outcomes of the regime in its attempt to solve or mitigate the problem it was created to address, since it is a necessary (though not sufficient) condition for its success and effectiveness.

3 The international environmental regime of disaster risk reduction

3.1 The relation of the DRR regime with climate action and the SDGs

Acknowledged by the United Nations (UN) in the Global Assessment Report on Disaster Risk Reduction (GAR) (UNISDR 2015) and researchers alike (e.g. Kelman 2015, 2017; Schipper and Pelling 2006), the issues and regimes of sustainable development, climate change and disaster risk reduction are interconnected (Albrito at al. 2015; UN 2015a, b; UNFCCC 2015). Human development affects all three determinants of disaster risk, since it can increase hazard magnitude and frequency, place human livelihoods in harm’s way and increase the vulnerability of parts of or the whole population (Aitsi-Selmi et al. 2016; Albrecht 2017; Birkmann 2006; Wisner et al. 2003). Thus, it might not be surprising that human and capital losses caused by natural hazards, which are intensifying because of the change in climate, have been on the rise (Di Baldassarre at al. 2018; Knutson at al. 2010; UNISDR 2015), highlighting the global importance of the field of disaster risk reduction and its governance.

Another UN commissioned report on the field corroborates that there is indeed a general lack of empirical and comparative research in particular on successful disaster governance, i.e. human decision-making to minimize the impact of hazards on socio-ecological systems, in spite of conceptual advances in this emerging field (Gall at al. 2014). This is in stark contrast to the extensive research on the SDGs and in particular the Paris Agreement despite their connection to DRR, Sendai Framework and Hyogo Framework for Action.

Ilan Kelman (2015, 2017) explored the connection and integration of the issues and regimes —or rather the lack thereof—, and argued that DRR and the HFA could offer valuable insights into the latest generation of international environmental regimes. However, it is not only the interconnectedness but also the similarities and commonalities in institutional design of the latest regimes that open up a variety of promising research avenues for much needed mutual and reciprocal learning. All of these regimes aim for universal membership, emphasising the importance of participation, the Paris Agreement being the only one in need of ratification. Although the Paris Agreement is meant to be legally binding and sports a peer-review process, it embraces the guiding principle of voluntary action which is at the core of the SDGs, HFA and Sendai Framework (Kelman 2017). These non-binding commitments or obligations are meant to scale up to reach shared global targets envisioning broad participation and compliance to the rules while providing a platform for knowledge exchange (Biermann at al. 2017). Thereby, Paris Agreement and HFA serve as global stocktakes, i.e. inventories, of national action with the underlying assumption that voluntary national action should and would increase in its ambition while skirting the idea of sanctioning parties for their behaviour (Milkoreit and Haapala 2019). Similarly, the three regimes incorporate soft law with respect to dimension of precision (Abbott at al. 2000; Abbott and Snidal 2000), since they are substantially vague proposing a variety of approaches and actions countries could take or diverse set of goals that could be pursued (Milkoreit and Haapala 2019). While there are partially precise activities mentioned, the vast number of them provides considerable leeway. Third, in all three instances, there is little to no delegation of authority in terms of sanctioning, and SDGs and HFA neither engage in thorough accountability mechanisms, whereas the Paris Agreement and Sendai Framework endorse a peer-review mechanism for creating accountability.

In sum, all of these regimes share characteristics of soft law along several of the dimensions of precision, obligation and delegation that Abbott et al. (2000) suggested. This sets them apart from earlier environmental agreements such as the Montreal Protocol, the hailed gold standard of environmental regimes signed in 1987, which introduced clear obligations (banning of certain substances) with precise rules (management and enforcement approach including transparency, dialogue and sanctions for non-participants), and at the same time, delegation (monitoring of compliance to the rules via reporting to the UNEP Ozone Secretariat with sanctioning mechanisms) (Albrecht and Parker 2019).

3.2 The DRR regimes and previous analyses

Due to

deep concern at the number and scale of natural disasters and their increasing impact within recent years, which have resulted in massive loss of life and long-term social, economic and environmental consequences for vulnerable societies throughout the world, in particular developing countries (UN 2006, p. 1),

two regimes were established in connection to the United Nations International Strategy for Disaster Reduction, managed by the UN Office for Disaster Risk Reduction (UNDRR)Footnote 6 and endorsed by the UN General Assembly. These were the Hyogo Framework for Action 2005–2015: Building the Resilience of Nations and Communities to Disasters (HFA), and its successor, the Sendai Framework for Disaster Risk Reduction 2015–2030. In contrast to most environmental regimes, HFA and Sendai Framework only had to be adopted, not ratified in national parliaments, proposedly because they deal with a rather benign problem in which countries strive for reducing their own disaster risk and losses. There is no extensive opportunity for free-riding (other than the benefit of international early warning systems and research), which is often inherent in international environmental challenges. In that way, the two frameworks are only to take stock of the national DRR status and suggest further activities as well as provide a forum for the exchange and diffusion of knowledge, since the problem at hand is characterised by high uncertainty concerning hazard impact and frequency in the face of complex interactions between socioeconomic and socio-ecological developments.Footnote 7 Thus, the two frameworks are self-enforcing agreements, requiring participants to do nothing beyond what is in their own self-interest (Hafner-Burton at al. 2011).

The discussion and extensive negotiation for a post-Hyogo framework highlighted several challenges. The debate on the Sendai Framework made clear that it would be a step forward, but also that not all issues had been resolved. First, the Sendai Framework included “specific measureable targets” (Cutter and Gall 2015, p.707) to enhance assessments of progress. Furthermore, it marked another step forward in the integration of DRR, climate change adaptation and sustainable development. Instead of keeping risk reduction in a silo, i.e. separate, the view was that the Sendai Framework would facilitate coordination across these areas (Aitsi-Selmi et al. 2015; Aitsi-Selmi and Murray 2015; Albrito at al. 2015; Kelman 2015). However, some researchers voiced concerns about whether crucial stakeholders, particularly local-level actors and members from the scientific community, NGOs and civil society, were sufficiently included in the negotiation process that produced the final agreement (Tozier de la Poterie and Baudoin 2015; Walch 2015).

Whereas the Sendai Framework was indeed tailored to monitor and track progress along 38 indicators and seven targets in relation to the SDGs (UNDRR 2019; Zaidi 2018), it is still long off before we can expect sufficient comparable data. However, the HFA as its predecessor was “the first plan to explain, describe and detail the work that is required from all different sectors and actors to reduce disaster losses” by “offer[ing] guiding principles and practical means for achieving disaster resilience” with the ultimate goal “to substantially reduce disaster losses by 2015” (UNDRR 2019). Thus, it already provides an extensive dataset on the status of disaster risk reduction (DRR) by inventorying national activities across several priorities for action over four reporting periods. Therefore, it enables the comparative assessment of effectiveness in terms of participation and compliance to its guidelines.

In biennial reports, UNDRR reviewed the collected data and found conflicting evidence on progress, namely higher HFA scores and at the same time an increase in damage under the runtime of the HFA (UNISDR 2017). The latest Global Assessment Reports concluded that “[u]nder the Hyogo Framework for Action […] significant progress was made in improving preparedness and response to disasters and in strengthening the institutional capacities, policy and legislative frameworks and strategies to do so” (UNISDR 2017, p.10), which is in fact not reflected in the increase in scores across reporting periods, but rather a reference to continuously high scores in these areas. The reports further concluded that the HFA has only been partially successful, since neither disaster risk nor losses have substantially decreased (UNISDR 2015). In addition, the GARs produced a global risk model providing a globally comparable metric for disaster risk making the global patterns explicit, aiming at contributing to the implementation of the Sendai Framework and SDGs. Thereby, they unveiled the unequal distribution of disaster risk to the disadvantage to lower- and middle-income countries, particularly Small Island Developing States (UNISDR 2017). Under the umbrella of the UNISDR, countries that showed some form of progress were asked to showcase practical applications for DRR, exchange experience and join in collaboration at regional platforms (UNDRR 2019). Notwithstanding the essential contribution to research and human development, the reports do not present the patterns of progress across the globe, which could substantially contribute by identifying countries and periods of progress and enabling learning processes in the long run which is often difficult due to the usual invisibility of risk prevention (Di Baldassarre at al. 2018).

Furthermore, civil society initiated a participatory monitoring programme as an accountability mechanism for reviewing the HFA-induced DRR policy changes. The reports based on cross-country surveys revealed that the changes did not materialise in local practices and left local governance institutions without authority, capacity and resources during the early periods of the HFA (GNDR 2009, 2011), thus questioning the effectiveness of the regime in general. Nonetheless, it is worthwhile to investigate the policy change facilitated by the HFA, since it indicates regime impact though it might not have permeated all structures yet.

4 The data of the Hyogo framework for action

The extensive data collected under the Hyogo Framework for Action are provided via PreventionWeb, a website hosted and managed by UNDRR. The data are largely based on the voluntary country reports and self-reported national information on DRR status, and comprises 22 areas of key activities across five priorities for action (PfAs) in the form of scores along a 5-point scale. Moreover, the reports include quantitative and qualitative questions on the key activities as well as further input on outcomes, national strategic goals, future outlook and potential drivers of progress, along with additional information on development status, geographical location and income level. While the query tool talks about the DRR status scores as progress levels, this is essentially misleading since the scores do not represent progress in itself, but the self-reported DRR status at one point in time, while only a change in scores actually demarcates progress towards a higher DRR status from one reporting period to the next. Scores are attributed to the 22 key activities which are sorted into five PfAs covering the aspects of (1) institutional strengthening, (2) risk assessment and early warning systems, (3) education, information and public awareness, (4) reducing underlying risk factors by mainstreaming DRR into other policy domains and (5) preparedness for effective response.Footnote 8

This study reviews the participation and compliance to the rules for evaluating the effectiveness of the HFA, and spatially localises change in time. Thereby, participation is operationalised as the number of reports and continuity of reporting based on the submissions by countries, since the HFA did not have to be ratified. Compliance understood as the change in behaviour, i.e. an increase in DRR measures according to the activities suggested by the HFA, is operationalised as an improvement in scores from earlier to later reporting periods, during intermissions, respectively. Effectiveness can then be understood as compliance, i.e. the change in behaviour, in relation to the achieved and continued participation.

DRR status or the so-called “progress levels” are measured in a crude 5-point scale leading to issues with ambiguity, since the same attributed value does not necessarily entail the same DRR status because scores can materialise in quite a different manner due to the crudeness of the scale. What is more, the ordinal scale needs to be considered as a quasi-interval scale for the analysis of progress. Thereby, it needs to be acknowledged that values are not equidistant on the scales, meaning a change in scores from 2 to 3 is essentially different from a change from 4 to 5. When analysing the data, special attention has to be put on participation, since most countries did not submit reports for all four reporting periods resulting in missing data and potentially distorting an aggregation of scores. The validity of self-reported data can be questioned, but in the case of the HFA, there is no advantage in providing false information.

With these caveats in mind, I argue that it is still worthwhile to utilise the HFA data, because it is indeed the most comprehensive and extensive dataset on DRR status available covering several reporting periods. It is possible and valuable to review and analyse the scores and progress towards higher levels of DRR, as it yields early information on the effectiveness of the regime. In particular, the response rates and the wide spectrum of DRR areas and activities covered enable a systematic analysis.

The study proceeds with the analysis of participation to see whether the soft law regime was successful in attracting countries to participate and submit reports, before assessing to what extent compliance was achieved. Lastly, substantial changes are localised to investigate the prevalence of compliance across countries and periods.

5 Analysis and discussion

5.1 Participation

In terms of participation and compliance, the raw numbers of countries recruited to hand in reports and the total submissions over the four periods are of relevance. It is striking that 160 of the 193 member states of the UN could be encouraged to submit reports and more than two thirds of them complied at least in two reporting periods. Remarkably, roughly the same number of countries submitted one, two and three reports (between 42 and 48), while still more than half of those countries (n = 24) did not miss one single reporting period.

When we look at the development in reporting over time, there are several observations of high importance (Fig. 1). First, the number of reporting countries rose substantially from the first to the second reporting period, namely from 61 to 105. In the two later periods, we see a slight decline in submissions, but still remaining on a high level. Striking is that UNDRR succeeded in attracting new countries to report in each period, while increasingly more countries could be encouraged to continue their submissions. Furthermore, an increasing number of countries that stopped reporting could be motivated to return and continue submitting reports again. With Sendai data to come, it might be worthwhile to look into the participation and continuation of reporting in order to check for decreasing and returning participation in comparison with the HFA. At this point, it would be speculation whether there is a trend of diminishing returns in terms of participation or a flattening due to the sunsetting of the HFA in anticipation of the Sendai Framework.

In sum, the UNDRR was successful in attracting countries to participate and comply with continued reporting, after relatively weak participation in the first reporting period.

Fig. 1
figure 1

For each reporting period, amount of countries that submitted a report, that was newly recruited, that discontinued reporting and that returned to report

Full size image

5.2 Compliance

Figure 2 shows the distribution of scores for each priority for action (PfA) and on average (HFA_av) for each of the reporting periods. Medians are generally increasing over time, which can be directly seen at the boxplots, while there is also a slight positive shift in range. Looking only at the average score across all priorities (HFA_av), the median increases over time (total of 0.26) and the spectrum of observations spans over higher scores than in earlier reporting periods, while more outliers at the lower end can be identified, leading to the suggestions that there is indeed a positive development over time. It is worth mentioning the number of negative outliers is increasing at the same time. Looking at the PfAs, we can confirm that all medians increased over the whole period, but momentary stagnation can be identified in each PfA (see Table 1 below). This could suggest that countries specifically focused on single PfAs in some intermissions, i.e. the time from one period to the next. For instance, in intermission 1, only PfA 1 shows a change in the median. However, in combination with the results on participation, it rather suggests that a more diverse set of countries was surveyed. In later periods, in which the number of submissions was higher, there is a more diverse picture with several PfAs increasing their medians. With that specific development in mind, the large increase in PfA 3 from period 2 to 4 becomes more relevant (Fig. 2). Many countries continuously reported and the median did indeed increase substantially. It would be speculation to suggest that it would be due to a low starting value.

Fig. 2
figure 2

Box plots of the distribution of scores for all reporting periods for the HFA in general and within each priorities for action (PfA) (Stata)

Full size image
Table 1 Changes in medians between reporting periods ( − : stagnation; + : increase in median)
Full size table

Similar to the figure before and to the Global Assessment Report by the UNISDR (2015), Fig. 3 shows a steady development and indeed an increase in means for all priorities for action and, thus, for the HFA on average over time. PfA 4 and 5 mark the only exceptions due to a decrease from the first to second report period. However, the changes are in almost all cases not sufficiently large to demarcate a statistically significant difference. There are two exceptions, namely PfA 3 and 4. At a 95% confidence interval, PfA 3 changes significantly over the whole runtime of the HFA, whereas PfA 4 experiences a significant positive change from period 2 to 4 (Fig. 3). While the latter might be because of the slight dump in intermission 1, the former could be partially due to a low starting value providing more room for improvement on the 1 to 5 scales of the HFA. Additionally, it is noteworthy that, while not being statistically significant at a 95% confidence interval, values increase substantially across all priorities over time and consequentially also on average.

Fig. 3
figure 3

Bar chart of the means of scores for all four reporting periods (t1–4) for the HFA in general (HFA_av) and each priority for action (PfA) with error bars representing double standard error (Stata)

Full size image

Overall, scores have increased in all PfAs and nearly from each reporting period to the next. Due to the pattern in participation, changes from period 2 to 4 are of the highest importance. Significance is hard to establish due to the 5-point scale inhibiting differentiation, but can be identified in PfA 3 and 4, i.e. education, information and public awareness and the reduction of underlying risk factors, respectively. Most substantial advances were reported in key activity P3C1, which comprises the provision of relevant information at all levels and to all stakeholders.

5.3 Localised change

For investigating how widespread progress or change actually is, we need to turn to the performance of single countries, which results in the pattern of change presented in Fig. 4. The graph displays the change in scores for each country that submitted at least two reports sorted by their starting score. In case of more submissions, the change over the longest period possible is depicted. What is striking is that most of the countries are indeed located above 0, suggesting that most countries managed to increase their HFA scores and, thus, their measures for disaster governance. Whereas most countries show minor changes (< 1), substantial positive change manifested in six countries, namely Fiji, eSwatini (formerly Swaziland), Ecuador, the Dominican Republic, the Philippines and Morocco. However, negative changes can be identified in several cases, and five countries even show a substantial decrease in scores (> 1): Brazil, Togo, Armenia, Mexico and Kenya, respectively.

For a deeper understanding of the change and its temporal pattern, it is also worthwhile to investigate the changes from one period to the next, since disaggregated data can unearth extraordinary changes that have been lumped together with rather minor or opposing changes. Figure 5 has to be handled with caution though due to missing data, since countries that suspended or discontinued their reporting are not included. Thus, it misses some information even on some outlier cases identified before. Nevertheless, the graph shows that during all intermissions scores have improved and declined, leading to no specific difference between the periods. Furthermore, Fig. 5 enables the identification of several periods in specific countries that are marked by considerable change. Table 2 lists these instances of progress and the periods when they materialised.

Fig. 4
figure 4

Change in aggregated scores for each country from one period to the next. Dashed line: maximal positive change possible. Solid line: no change (Stata)

Full size image
Fig. 5
figure 5

Change in aggregated scores for each country over the runtime of the HFA. Dashed line: maximal positive change possible. Solid line: no change (Stata)

Full size image
Table 2 Instances of progress (substantial positive change in scores) listed in declining order
Full size table

First and foremost, the scores of Fiji and Morocco drastically changed from period 3 to period 4, as did the scores of Ecuador, Senegal, the Dominican Republic and Guatemala in intermission 1. In the case of Fiji, for instance, this development could be due to the formulation and introduction of a National Strategic Plan in DRR and Climate Change Adaptation, which harmonises the Climate Change Policy (2012) and the Natural Disaster Act (1998, revised 2018) while supporting the Green Growth Framework (2014) and the Climate Public Expenditure and Institutional Review (2015) (Ratulevo 2014). However, the substantial increases in several countries during the early years contradicts the logic that regimes become more effective over time, but could possibly tell of “easy wins”, changes that could be implemented quickly, easily and without high costs or effort. However, Guatemala and Senegal experienced declines in scores during the last intermission (0.5 and 0.41, respectively). eSwatini and the Philippines are cases that should be analysed further in respect to when the changes materialised due to missing data. In contrast to the rapid changes in the countries mentioned before, they might be instances of continuous incremental changes such as Turkey, which incrementally increased its score by a total of 1.00. In most of the instances demarcated by substantial negative change, countries experienced rapid decline, Togo being the only exception due to missing data.

What the last two graphs make clear is that the 5-point scale is only a rough indicator and, thus, inherently problematic for tracking progress. Many countries start with high scores (mean = 3.15, median = 3.18) and cannot increase their scores substantially. Additionally, the issue of non-equidistance leads to the necessity of noting the scores and not only the change. However, most of the countries are indeed located above zero, suggesting that most countries managed to increase their HFA scores and, thus, strengthen their DRR status by engaging in more DRR activities, while specific countries showing substantial positive change could be identified.

6 Conclusion

This study empirically investigated the Hyogo Framework for Action and its prospects of effectiveness based on participation and compliance as a case of international environmental soft law regimes. Because regime design is a more significant determinant of effectiveness than the problem type (Young 2011), theories and concepts designed for the assessment of regimes based on hard law regulations need to be adapted to assess soft law regimes. This also contributes to the recent call for studies investigating subsets of the entire universe of cases (Young 2018).

Accordingly, soft law regimes were chosen as an important subset of international environmental regimes because of the recent adoption of this type of institutional design in prominent regimes. These regimes are characterised by voluntary action, non-binding commitments, a wide spectrum of suggested measures and a lack of sanctions for non-compliance, while aiming at universal membership (e.g. Biermann at al. 2017). These attributes and characteristics of intuitional design give rise to new questions about regime effectiveness and how established concepts need to be adjusted for the proper analyses of this subset. This study, therefore, adapted the theoretical concepts of participation and compliance that were established and commonly used in the study of the effectiveness of hard law regimes as a basis for assessing regime effectiveness of soft law regimes. Adopting a more demanding view of participation, the concept was measured as continuity of active participation and involvement instead of agreement accession or ratification. Compliance was operationalised as change in behaviour through implementation of the regime’s provisions.

This study then applied these concepts to the Hyogo Framework for Action, the regime for disaster risk reduction between 2005 and 2015. Utilizing data based on indicators on DRR measures in national reports and descriptive statistics, the investigation resulted in the following findings. First, UNDRR was successful in continuously attracting new countries to report and facilitating continued participation after facing reluctance or incapacity in the first reporting period. Second, there was indeed an increase in scores reflecting progress in the national implementation of DRR measures during the runtime of the HFA, in particular, in the priorities of action 3 (education, information and public awareness) and 4 (reducing underlying risk factors). Third, while the evidence suggests that compliance is not prevalent in all countries and periods, the study succeeded in identifying several instances indicating extraordinary positive (Fiji, Ecuador, Morocco, Senegal, Dominican Republic and Guatemala) and negative changes (Brazil, Togo, Armenia, Mexico and Kenya). This identification of outliers can inform future case selection enabling comparative research on disaster governance and success cases that has been lacking to date (Di Baldassarre at al. 2018; Gall at al. 2014). In sum, while the regime was largely successful in achieving widespread participation, compliance was rather ambiguous and multifaceted. Thus, although there are positive developments and progress, soft law did not seem to be effective in spurring compliance or change across all member states in the case of the HFA. Accordingly, it might be questionable whether soft law regimes offer indeed the prospects of effectiveness. Close attention and examination will be needed to provide answers on a larger scale.

This systematic assessment of the effectiveness of HFA revealed new information and went beyond the notion of only being “partially successful” (UNISDR 2017) by providing a more nuanced picture and understanding of the national development of DRR activities. Previous assessments failed to point out the development in specific countries, particularly the significant progress made in selected priorities for action, and neglected to account for continued active participation, both of which are needed because of the institutional characteristics of soft law regimes. To better judge the prospects of improved DRR governance, it will be necessary to provide a more detailed picture of the status and compliance with the regime’s goals and guidelines, which would require more elaborate evaluation scales in conjunction with scrutinising the involvement and participation of countries in the regime. In-depth over-time assessment of the successful cases of DRR progress could provide further insights into drivers and causal mechanisms linking this specific regime to the outcomes and attributing the effect to the regime. Moreover, the research will have to attempt to understand and explain the variation on the country level both in the status and progress of disaster risk reduction.

This study adheres to previous arguments that thorough and detailed monitoring and accountability mechanisms for tracking progress are crucial for determining regime effectiveness (Cutter and Gall 2015). This is particularly important since global performance indicators and indices are increasingly deployed and, in addition to regimes, utilised to identify and frame emerging issues as well as influence their governance globally (Kelley and Simmons 2019; Young 2018). Informed by previous research, the concepts of participation and compliance, which originate from theories on hard law regimes, were revised and adapted to the different subset of soft law regimes. The analysis provided evidence that soft law regimes might be effective solutions, however, detailed assessment is necessary on case-by-case basis until appropriate and sufficient data are available.

Although this approach was useful in investigating the prospects of the effectiveness for soft law regimes, the study suggests a more nuanced understanding of participation and compliance is warranted when adapting them to specific subsets of regimes. Therefore, future research is encouraged to further empirically explore the validity of the approach developed here for investigating the prospects of effectiveness. The next step in this research is to explore potential explanations of how the processes led to and facilitated positive outcomes. One argument could be that the institutional arrangements of the regime enabled and facilitated engagement in knowledge exchange and the diffusion of ideas (Benson and Jordan 2011; Dolowitz and Marsh 2000, 2012; Elkins and Simmons 2005; Stone 1999). Another interesting avenue of research comprises the drivers and mechanisms that link the institutional arrangements to outcomes. Concerning the subset of soft law regimes, this may place the emphasis on the ability of regimes to attract continued participation and to understand compliance in terms of the extent to which suggested measures are implemented. Moreover, the exploration of the linkage and interaction of the three environmental regimes—SDGs, Paris Agreement and Sendai Framework—should gain further attention, since it might be essential for the effectiveness of any of the regimes (Albrito at al. 2015; Kelman 2015, 2017; Kelman et al. 2015; Kelman and Glantz 2015; Schipper and Pelling 2006a). Lastly, regarding soft law regimes, there is a need to look further into their capability to “scale up” voluntary action to actually meet their goals and solve the problems they were created to address.