1 Introduction

To take urgent action to combat climate change and its impacts is one of the sustainable development goals (UN 2018). Climate change is affecting many aspects of our planet and it has progressed rapidly since the Industrial Revolution (Dukes 2011; IPCC 2014; Williston 2015; IPCC 2018). Extreme climate events present a serious risk to poverty reduction efforts and threaten to break down decades of development initiatives in developing countries. This is especially true for those populations who are living in vulnerable areas that often have the fewest resources to adapt or recover quickly from shocks (Abeygunawardena et al. 2009; Aldrich and Meyer 2015; Balica et al. 2014; Kelman et al. 2016; Kamal et al. 2018). As the effects of an extreme climate event are worsened by climate change, getting away from poverty becomes more difficult because poor populations not only face dire effects, such as decline in water quality, changes in human health, or reduction in supply of ecosystem goods and services (Scott et al. 1990), but also experience severe impacts on the necessities of life such as water, energy, and buildings (Lal et al. 2011; E-IRG 2013).

A number of changes in climate variability, such as temperature, precipitation, sea or river level rise, and extreme events will affect housing more frequently. For instance, extreme rainfall may cause deterioration or damage to the interior or exterior of homes because of seepage, flooding, erosion, and damage to building foundation (E-IRG 2013). These issues have raised climate awareness, which has motivated the implementation of risk reduction and damage prevention programs for housing in many developed and developing countries (Hochrainer-Stigler et al. 2011). These disaster risk reduction policies and measures provide the first line of defense in adaptation to climate change, and one important action is integrating disaster risk and climate change impact considerations into housing sector planning and policy (Phong and Tihn 2010). There are studies that deal with impact and cost–benefit analysis of prevention programs, because housing is considered one of the most valuable, but also vulnerable, areas impacted by climate change. The majority of these studies applies cost–benefit analysis, and the minority uses impact evaluation tools. For instance, Tran et al. (2012) and Hochrainer-Stigler et al. (2011), applying cost–benefit analysis, find positive returns from preventive measures. The former examines the costs and benefits of applying typhoon resilient housing measures in Da Nang (Vietnam), and the latter applies probabilistic cost–benefit analysis to evaluate selected disaster risk reduction measures that reduce losses to structures in hazard-prone areas in low-and middle-income developing countries. On the other hand, Gros et al. (2019) implemented a quasi-experiment to measure the impact of a forecast-based provision of cash on households in Bangladesh to reduce the flood impact (rent a truck to move their animals temporarily to higher ground), finding positive effects.

In particular, regardless of the absolute certainty of whether climate change is affecting the frequency or intensity of the El Niño-Southern Oscillation (ENSO) impacts,Footnote 1 it is clear that there are very serious repercussions associated with ENSO both on the composition and on the dynamics of the Peruvian coastal and marine ecosystem (Rossi and Soares 2017), and on the population in the northern part of the country. The ENSO occasionally increases the temperature of the surface waters of Perú’s north coast sea, causing abundant evaporation, which is added to the persistent orographic effect of the Peruvian Andes, bringing rains that in turn give rise to floods and mudslides (Galarza and Kámiche 2012).

In the last 35 years, in 1982–1983, 1997–1998, and 2017 Peru has suffered the effects of three El Niño events, considered of very strong intensity.Footnote 2 According to the National Information System for the Prevention and Attention of Disasters (SINPAD) of the National Institute of Civil Defense (INDECI), the 2017 ENSO affected more than 1.4 million people, caused 159 deaths, and collapsed 29.8 thousand homes (INDECI 2017).

Because natural hazards are a common challenge to all human beings, different actors (relevant governmental agencies, nongovernmental organizations, volunteer organizations, and research institutions) should be willing to join forces to deepen exchange and cooperation in disaster prevention and impact reduction. Tools such as natural hazard monitoring, early warning systems, information sharing, emergency rescue, scientific research, technology adoption, personnel training, and community disaster reduction practices are crucial to saving lives and mitigating damage (Zhang et al. 2012; Walshe and Nunn 2012; Jiang 2013; Rademacher 2013; Baudoin et al. 2016).

Early actions, early warning information, and clear decision criteria are all required to develop approaches for using meteorological forecasts of extreme events to trigger actions that could prevent a disaster. The risk of disaster, however, results from a combination of the probability of the hazard or extreme event, and the exposure and vulnerability of the population at risk. Therefore, the use of meteorological forecasts to trigger early action assumes that when the extreme event occurs, the impacts are high enough to cause a disaster. This requires the determination of the event threshold or danger level, that is, the magnitude and persistence of the hydrometeorological event that is linked to the occurrence of avoidable and unavoidable losses and damages (Lopez et al. 2018).

“Early warnings” of heightened risk, such as the ENSO forecasts that indicate enhanced risk of flooding, are often available well before the disaster strikes. This provides a window of time to reduce potential consequences for society. Actions such as evacuation or distribution of water purification tablets can be taken at this time window, each one with its own level of cost in terms of scope and preparation needs. A mixture of such actions can increase resilience to hazards, both prior to and during the immediate threat of a disaster. There is evidence that avoided disaster losses can at least double or quadruple the investment in risk reduction (Mechler 2005). This potential benefit buffers society against the possibility of acting “in vain” if the hazard does not occur within the forecast-projected time frame, and ensures that the long-term gains of preventative action will outweigh the costs of false alarms (Coughlan De Perez et al. 2016).

In this article, we provide an impact evaluation of the Forecast-based Financing (FbF) mechanism in Peru in preparedness for El Niño impacts. The FbF allocates resources prior to a hazard occurring based on preselected forecasts and climate information in order to trigger financial resources more efficiently and effectively to prepare communities before the disaster shock. Since 2014, the Red Cross has developed the FbF as an innovating program in order to help communities prepare and respond to shocks as well as reduce vulnerability and build lasting resilience (Coughlan De Perez et al. 2015). Even the range and extent of anticipatory actions could change with the forecast’s lead time, and thus the degree of uncertainty (for example, short-term risks such as cyclones and storms can be forecast with relatively high skill); action based on early warning systems (such as cyclones) has saved many lives and prevented damage (Galindo and Batta 2013; Rogers and Tsirkunov 2013; Harriman 2014; Gros et al. 2019).

The FbF implemented in Peru was delivered in order to enable preparedness for El Niño 2015–2016 impacts. This mechanism setting up a framework to elaborate intervention protocols of early action that could be automatically implemented when forecast thresholds were reached. The trip-wire comes from different national and international agencies (Bazo et al. 2018).

The purpose of our study is to analyze the effectiveness of implementing the FbF intervention. The information was collected at two different points in time. The first, once the early warning was activated, involved the provision of special kits to prevent damage from flooding and rain; and the other data set was collected after the coastal El Niño struck in 2017. We can estimate the impact of the intervention on specific outcomes regarding the damage level of homes by comparing those homes supported by the program with those that were unsupported. Previous literature, which performed impact evaluation of an early warning intervention such as the FbF and involved experiments or quasi-experiments, is scarce. An exception is Gros et al (2019) study in which a quasi-experimental investigation was implemented to estimate the impact of forecast-based provision of cash to help vulnerable populations take preparatory early actions. The goal of Gros and colleagues was to prevent and reduce negative disaster (flood) impacts in Bangladesh on a set of outcomes that is different than ours. Gros et al. found that a FbF cash transfer increased the regularity and quality of beneficiary households’ food intake, reduced the need to take out high-interest loans, and appears to have reduced psychosocial stress in the aftermath of the flood.

Unfortunately, in our study we do not have a baseline that has followed a randomization process to determine which home is or is not a beneficiary. Using matching techniques, we identified the most unsupported comparable homes to the households that were supported by the intervention, in order to estimate the effect of the early warning intervention on the damage level of homes. A limitation exists, however, since the post-intervention data corresponds to a period after the 2017 El Niño event. This means that the results to be found are most likely underestimated, due to the fact that the actual protection provided by the kits occurred more than a year after their installation. Nevertheless, as a methodology, it could be helpful for future interventions to provide parameters of interest for cost–benefit or cost effectiveness analysis of different alternatives in preventing effects of natural hazards in poorer areas. That is, with information on costs impact estimates will be particularly useful in determining if a specific investment is worthwhile, enabling the calculation on the net cost or benefit associated with the intervention when different options are being appraised and compared, as an option for choosing the best approach. Such analysis does provide a very useful starting point for researchers and policymakers to collaborate in assessing the efficacy of the different programs and their relevance to the particular situation (Dhaliwal et al. 2014).

Our hypothesis is that the provision of the special kits reduces the potential damage of homes as a consequence of the El Niño event in the study area, the district of Picsi (in the Lambayeque region, northern Peru). The main findings of our study suggest a positive impact of the FbF program in terms of its effectiveness in mitigating the damage caused by El Niño in Picsi, which is interpreted as a drop in the scale of house damage (less damage) of around 63% for a home that received the treatment.

The rest of the article proceeds as follows: Sect. 2 presents the study area and details of the FbF intervention in Peru, while the following three sections present the estimation strategy and methodology, variables and data, and empirical findings. Finally, Sect. 6 is devoted to the conclusion.

2 Forecast-Based Financing Intervention in Picsi, Peru

The project “Forecast-based Financing” is a global project undertaken by the International Federation of Red Cross and Red Crescent Societies and financed by the German Government. The main aim of the project is to improve the preparedness of communities and their response to disaster shock, based on knowledge and risk analysis, improved early warning, and innovative financing mechanisms.

2.1 Study Area

In the initial evaluation of this project, the community selection was made according to the needs in the Picsi District, as well as according to the experience and the implementation capacity of the Peruvian Red Cross branches. The selection criteria were: (1) severe heavy rainfall and flood impact in the past; (2) high vulnerability of the population in case of extreme rainfall, especially in the context of access to the area, depressed socioeconomic and health situation, as well as limited availability of drinking water and livelihood options; (3) the absence of other aid agencies and the scarce coverage of state services; and (4) the existing capacities of the local Red Cross. The local Red Cross trained volunteers were mobilized from eight communities in the district of Picsi: El Mango, Horcón I, Horcón II, El Faicalito, San Juan La Greda, El Pancal, Rio Dos, and Collocsi (Fig. 1).

Fig. 1
figure 1

Source FbF Team

Beneficiary communities in the Picsi District, Chiclayo Province, Lambayeque Region.

The district is located inland from the north coast of the country, in the region of Lambayeque, in the province of Chiclayo. It has an area of 56.92 km2 and an altitude of 36 m above sea level (masl), is 8.5 km away from the capital of the region (Chiclayo City), and it borders with the two other provinces of Lambayeque to the north (Ferreñafe) and to the west (Lambayeque). According to the 2007 National Census, the district had a population of 8942 people and 1916 households. Information from 2017 shows that there is an identified population of 9090 people, of whom 4573 are men and 4517 women (INEI 2017). It also has 20 population centers, of which 17 are rural and three are urban (INEI 2018a). In terms of total population, Picsi represents 1.0% of the province of Chiclayo, which makes it the fourth smallest district.

Regarding the socioeconomic aspect of the Picsi District, it is estimated that the poverty level of the population is between 21.9 and 33.9% (INEI 2013). Following the 2007 National Census, 79.7% of households have access to a public water network within the building, 78.7% have access to a public drainage network, and 89.7% have electric light from a public network (INEI 2018b). As for the local economy, the district is mainly engaged in agricultural activity, with rice and sugar cane as the most important crops.

2.2 The Forecast-Based Financing (FbF) Intervention

The northern coast of Peru (the departments/regions of Tumbes, Piura, and Lambayeque) is one of the most vulnerable regions during El Niño events; heavy rains cause flooding, which affects human activity and infrastructure in a significant way. In this context, the Peruvian Red Cross (PRC), the German Red Cross, and the Red Cross Red Crescent Climate Centre designed a project that uses scientific observations and forecasts to implement early action in the most vulnerable areas, before the onset of a potential disaster. The 2015–2016 El Niño was one of the first applications of the FbF mechanism.

The FbF aims to improve the effectiveness and efficiency of humanitarian preparedness by acting on national and international hydrometeorological forecasts. The system is based on calculations of regional impact levels (thresholds) and predefined early actions. These actions are triggered when a forecast exceeds a danger level in a vulnerable intervention area (for example, a specified amount, probability, or return period of rain that makes rivers flood and damage communities). An FbF intervention also makes financing available for the predefined actions to be taken automatically, without the need for a declaration of an emergency. Hence, actions can be taken before the impact of the disaster and can strengthen the resilience of both communities and institutions.

The triggering mechanism for this intervention was available from forecasts created by different national agencies like ENFEN (The Permanent Technical Committee for El Niño in Peru) and SENAMHI (Peruvian Meteorological and Hydrological Service) and international forecasting agencies, such as the European Centre for Medium-range Weather Forecasting (ECMWF) and the National Oceanographic and Atmospheric Administration (NOAA). This mechanism was activated if three out of four forecasts or observations reached the predefined thresholds, in a given time window between November and March (Fig. 2).

Fig. 2
figure 2

Source FbF Project—Peru 2015–2016 (https://www.forecast-based-financing.org/wp-content/uploads/2019/01/El-Nino-Norte-Peru_NotaTecnica_English.pdf)

Triggering mechanism for FbF intervention in anticipation of El Niño impacts across Perú.

Once the triggering mechanism was activated the Peruvian Red Cross implemented a logical sequence of activities:

  1. (1)

    Conformation of the technical team to evaluate and supervise housing infrastructure conditions.

  2. (2)

    Designing of a concrete and detailed tool to collect the information of the affected dwellings.

  3. (3)

    Training of volunteers in all processes of protection and house strengthening.

  4. (4)

    Technical monitoring of the provincial branches of the PRC during field work.

  5. (5)

    Collection of baseline information about houses including photos and plans to obtain an approximate standard area and house features (Fig. 3).

    Fig. 3
    figure 3

    Source FbF Team

    Household information card for intervention kit selection, Pisci District, Chiclayo Province, Perú.

  6. (6)

    Systematization and analysis of the collected information.

The criteria employed by the Peruvian Red Cross to select the beneficiaries were mainly the infrastructure conditions of the house. Then, the intervention in the study area consisted of the provision of a special kit (Fig. 4) designed to protect selected households, according to community needs and their vulnerability. Specifically, the kit’s purpose is to improve the roofs by replacing those that cannot withstand heavy rains with corrugated roofs and the structure to support it. The corrugated roof is installed so the water runs off to the street (Fig. 5). A tool kit guideline was developed in relation to the construction of the kit, and for its correct installation.

Fig. 4
figure 4

Source FbF Team

Housing kit. Note: The housing kit contains: (1) 12 corrugated roof, polypropylene wave 100 of 3.05 m × 1.10 m; (2) 08 4 mm plywood plates; (3) 01 wooden sticks of 4″ × 6 m; (4) 01 wooden sticks of 4″ × 4 m; (5) 02 wooden sticks of 4″ × 3 m; (6) 05 wooden sticks of 3″ × 6 m; (7) 02 “Cumalá” wood slats 2″ × 2″ x 3.56 m; (8) 05 “Cumalá” wood slats 2″ × 2″ × 3 m; (9) 04 “Cumalá” wood slats 2″ × 2″ × 2.95 m; (10) 06 “Cumalá” wood slats 2″ × 1″ × 2.34 m; (11) 01 kg of wire # 16; (12) 02 kg of nail for wood of 6″; (13) 02 kg of nail for wood of 4″; (14) 02 kg of nail for wood of 1″; and (15) 120 nails for corrugated roof.

Fig. 5
figure 5

Source FbF Team

Before and after pictures of the kit provision. Note: The pictures correspond to a house in the Horcón I community, Picsi District. The roof was reinforced in the bedroom, living room, and dining room.

Figure 6 shows the timeline of the activities related to the intervention and data collection. The early warning was activated in November 2015, in order to enable preparedness for the 2015–2016 El Niño impact. In the following month information was obtained for 336 households in the district of Picsi. The process of elaboration and allocation of the kits began in February 2016 and it extended up to May of the same year. The reason behind this was that the 2015–2016 ENSO had a lower intensity than expected in this region, which allowed the Peruvian Red Cross to stay longer. As a consequence, the delivered kits did not fulfill their specific protective role until the 2017 “Coastal El Niño event” (between January and May 2017, mainly), which has been considered a climate event of extraordinary intensity.

Fig. 6
figure 6

Timeline of the El Niño forecast-based intervention in Pisci District, Chiclayo Province, Perú

Later evaluation of the baseline data showed that at least 240 households actually satisfied the conditions to receive a kit. This means that the sample used in this study should have included 240 households. In November 2017, given budget and operational constraints, information regarding the physical state of the houses was only collected for 113 of the 240 households (treated households included), which means that only these observations have registered values for the dependent variable. As a result, the 113 households make up the final sample.

3 Estimation Strategy and Methodology for the Effect of a Forecast-Based Financing Intervention

Estimating the effect of the FbF program in Picsi means that one must calculate the average treatment effect on the treated households (ATET). However, in order to obtain unbiased estimates of the average treatment effect, the treatment should be randomly assigned between the observations. This is required because if individual characteristics also affect the outcome of interest, the estimation of the impact of the program would be biased. This is most likely the case for the FbF program, since the allocation of the kits was based on the evaluation of the conditions of the houses, mainly the visible conditions of the infrastructure.

As a consequence, an alternative tool is needed. Propensity score matching (PSM) techniques can be used to estimate the treatment effects. This methodology can eliminate the selection bias by comparing treated observations with untreated observations that have similar characteristics. The difference in outcome between the treatment units and their corresponding comparison units equals the ATET.

The first step of PSM is to estimate the propensity score for each observation, in other words, the probability of a particular home being treated (receiving the kit) conditioned on a set of observable characteristics. The second step is to restrict the sample to the observations that belong to a common range in the distribution of the propensity score, often called the common support. This is done because there can exist treated observations with no comparison with a similar score (or vice versa). This exclusion allows us to keep only those treated and control observations that have similar probabilities of being treated. The third step is to use the score to match the treated observations with the control observations. The PSM identifies one or more control units for each treated unit, according to the matching mechanism used. The control units with a propensity score close to the treated units score become the counterfactual. The final step is to estimate the ATET on the outcome of interest, that is, damage level of home, using the matched observations.

Formally, the PSM methodology can be presented as follows. Let \(Y_{i1}\) be the outcome for observation i, conditional on receiving the treatment \(D_{i}\) = 1, and \(Y_{i0}\) the outcome conditioned to not receiving the treatment \(D_{i}\) = 0. Strictly, the ATET is:

$$ATET = E\left( {D_{i} = 1} \right) - E\left( {D_{i} = 1} \right)$$

where the last component of the right side is not observable, since it is not possible to see the outcome of the treated observations if they were not treated. Since the best approximate available for these variables are the outcomes of the untreated observations, one could estimate the ATET as:

$$ATET = E\left( {D_{i} = 1} \right) - E\left( {D_{i} = 0} \right)$$

This is only possible if the treatment assignment is random, that is, if \(Y_{i1}\) and \(Y_{i0}\) are independent of \(D_{i}\). If the allocation depends on certain individual characteristics, the ATET estimate would be biased. Controlling for the observable characteristics through matching allows to remove this bias, and the net effect correctly estimates the ATET. Let \(X_{i}\) be the observable variables, the ATET is:

$$ATET = E\left( {X_{i} ,D_{i} = 1} \right) - E\left( {X_{i} ,D_{i} = 0} \right)$$

where the last term on the right side is the counterfactual obtained from the outcomes of the untreated observations. In principle, given a number \(k\) of observable variables, it would be necessary to perform \(k\) pairings to estimate the ATET. The PSM methodology, as shown by Rosenbaum and Rubin (1983), allows the use of the propensity score \(P\left( {X_{i} } \right) = P\left( {D_{i} = 1|X_{i} } \right)\) instead of \(X_{i}\), so that the matching is done only in function of probability. Thus, the estimate results in the following:

$$ATET = E\left( {P\left( {X_{i} } \right),D_{i} = 1} \right) - E\left( {P(X_{i} ),D_{i} = 0} \right)$$

Once the propensity scores are calculated, the ATET can be obtained as a weighted average of the difference of the outcomes of the treated observations and their corresponding matches. In a general form, the estimated ATET can be expressed as:

$$\widehat{ATET} = \frac{1}{n}\mathop \sum \limits_{i|D = 1} \left[ {Y_{i1} - \mathop \sum \limits_{j|D = 0} W\left( {i,j} \right)Y_{i0} } \right]$$

where \(W\left( {i,j} \right)\) are the weights corresponding to the counterfactuals, which depend on the chosen matching method.

In this study, as it will be seen in the empirical findings section, three matching schemes were used to estimate the ATET. The first corresponds to a nearest neighbor matching procedure, which selects the m comparison units whose propensity scores are closest to the treated unit in question (Dehejia and Wahba 2002). In this case we used one-to-one matching, which selects for each treatment observation the first control with the closest score.

The other two matching methods are radius matching and kernel matching. In the former, each treated unit is matched with all the control units whose propensity score falls in a neighborhood (radius) of the propensity score of the treated unit (Dehejia and Wahba 2002). In the latter, all control observations are used and matched with the treated ones, using a weighted average where the weights are inversely proportional to the distance between the propensity scores of the treated and control groups (Dehejia and Wahba 2002; Baser 2006).

4 Data and Variables

The data collected from the surveys consist of information about the physical condition of the house, its access to water, and some socioeconomic characteristics of the household. The data set includes 62 control and 51 treated households. Table 1 shows how the damage level in the houses (the dependent variable) is distributed within each group. On the other hand, Table 2 shows the average, standard deviation, minimum, and maximum values for all the variables, and the difference in means between the treatment and control group.

Table 1 Damage to houses, by control and treatment groups, studied in Picsi, Chiclayo Province, Perú
Table 2 Characteristics of treatment and control groups of houses studied in Picsi, Chiclayo Province, Perú

The significant difference between the means of the dependent variable (outcome) of the control and treatment group may be a first indication of the effect of the program. The positive difference suggests that, on average, an untreated household suffered more damage than a treated household. However, as shown, there are also differences in the characteristics of the treated and control observations. The main differences correspond to the variables that are related to the physical infrastructure of the home, which are almost all statistically significant. Other characteristics with a significant difference are whether (or not) the household has access to a water source, the source is a stream, the head of the household is a woman, and the house possesses good, regular, or bad sanitation.

Taking into consideration the relevant differences in the quality of the houses, it appears that these variations reflect a wider difference in socioeconomic terms between both treated and control groups. After all, it is not unreasonable to assume that families with a higher income can afford a house in better structural and aesthetic condition. This can also be inferred from the fact that these households have better sanitation conditions, since it could reflect that there exists both the means and disposition to maintain a safe and sound environment inside the home as much as possible.

Thus, these data show that the treated households are in significantly “worse” condition than the control households. This is consistent with the assignment of the kit, but the use of the PSM methodology is necessary to take into account the socioeconomic and demographic characteristics of the households in order to adequately estimate the effect of the intervention.

5 Empirical Findings

In the following sections the propensity scores for each household in the sample are estimated using a probit model. Then, these scores are used to obtain the effect of the Red Cross Program through different matching techniques.

5.1 Probit Estimations of Receiving Treatment from the Red Cross Program

Following the methodology described, the propensity score for each household was estimated using a probit model. The probability of receiving a kit depends on a group of variables, which are related to the physical conditions of the house, access to water, and the socioeconomic characteristics of the household. Table 3 shows the results of different specifications for the model.

Table 3 Probit estimates of Red Cross program

First, the effect of building integrity (condition of the roof, walls, and main door access on the probability a household receiving the kit was estimated. Columns 1 and 2 show that a household with a roof or walls in poor conditions is more likely to receive the kit. This is consistent with the assignment rule of the FbF program, which focused on giving higher priority to homes that can more easily collapse as a result of an external event (in this case, El Niño). Access to the household only has a statistically significant effect on the specification of column 3, where the variable “energy in good condition” is also included. Several households did not disclose information regarding the energy system, however, which explains the reduction in the number of observations. In addition, no energy variable was significantly relevant under different tested models, which led to its exclusion for the rest of the specifications.Footnote 3

Columns 4 to 11 introduce water access variables and their different sources. Columns 5 to 9 show that having access to a water source does not have an effect on the probability of being treated with a kit. There is also no specific effect of the main types of water sources (city or community, river, wheel, and well). Column 4 shows that obtaining water from an unspecified source increases the probability of receiving a treatment kit. Taking into account that there were nine water source options in the survey, this variable may reflect a limited or even nonexistent source. This restriction on access to water may have influenced the allocation of kits by the program. This result holds when controlling for access to water (column 10), which is now significant and with a negative sign, and for access through a well (column 11), which is the most frequent water source.

Finally, columns 12 to 16 assess whether there is an effect of the socioeconomic characteristics of the household and its members on the probability of receiving a treatment kit. Only the bad sanitation variable has a significant effect if added separately. A household with worse hygiene is more likely to be treated (column 12). This is also consistent with the significant difference of the means of this variable between the control and treatment group in Table 2. In contrast, the effect of having members who are likely to be more vulnerable—such as a female head of household, a sick person, a pregnant woman, and numerous children (column 13)—is not statistically significant. In the same way, there is no effect of the number of adults and children (column 14) and of women and men (column 15) on the probability of receiving the kit. In column 16, the effect of the most vulnerable members was estimated again, adding the rest of the variables of the category, finding no effect also on the probability of a household being treated.

Thus, almost none of the previously proposed variables had a significant effect on the likelihood of being part of the Red Cross program. Only column 16 shows a significant result: a home with a telephone is less likely to receive a treatment kit. Despite this exception, the results of all the estimates suggest that the physical aspects of households determine whether the kit is received or not. The variable about roof condition remains statistically significant in all specifications, while the variable regarding the condition of the walls does in almost all of them. The same applies to the access to water, unspecified source, and bad sanitation variables. This suggests that the program allocation was primarily focused on physical aspects, and possibly those easy to observe at first glance. On the other hand, it seems that there was no focus on other characteristics of the household or its members.Footnote 4

5.2 Average Treatment Effect on the Treated (ATET) Houses in Picsi, Perú

The ATET is calculated from the difference in the outcome variable between the matched control and treatment observations. This calculation is made on the observations that are within the common support, in such a way that one works with the treatment and control units with similar propensity scores. The expected result is a negative impact of the FbF program on the household’s level of damage. In other words, as a result of the intervention, those households that received a kit experienced less damage than those that did not.

As described in the previous section, the FbF intervention was initiated as a response to an early warning about probable El Niño events in 2015–2016; yet the kits’ protective role was not fulfilled until the 2017 Coastal El Niño event, since the anticipated natural hazards of the previous year were almost absent. This represents a limitation for the present study, because the estimated ATET could be biased by unknown factors present in the lapse of time between the installation of the kits and the actual protection they delivered.

Nevertheless, the data showed that treated households were socioeconomically worse off than the control ones, which related to their inability to improve the physical condition of their homes. It is plausible to assume that, during the time the kits remained in these houses, no significant physical improvements were made, which is the principal issue that could bias the result. As a consequence, the most probable “noise” that might alter the estimations would be a decline in the physical conditions of the kit, most likely due to wear and tear. Under this assumption, the results of this study would be underestimated, because they would reflect less than 100% of the “protectiveness” that a kit in pristine condition could deliver. In other words, the estimated ATET is a lower bound of the real effect of the intervention.

Having made this clarification, Table 4 shows the estimated ATET using three different matching techniques: one to one matching, kernel matching, and radius matching. The first column for each method—columns (a), (e), and (i)—estimates the ATET for the first estimated probit model, which includes the conditions of the roof, walls, and access. Despite being the most basic specification, the pseudo R-squared and log-likehood values of this model shown in Table 3 reflect that these variables have considerable explanatory power. This gives confidence in the fact that the variables that can be more correlated to the outcome are being taken into account, and hence the possible bias in the ATET is eliminated.

Table 4 Average treatment effect on the treated (ATET) houses in Picsi, Chiclayo Province, Perú

The estimated ATET is statistically significant and has a negative sign in the three cases. It also fluctuates between a reduction of 0.695 and 0.835. This translates into a positive impact of the Red Cross program in terms of its effectiveness in mitigating the damage caused by El Niño. These results can be interpreted as a drop in the scale of house damage (less damage) for a home that received the kit treatment.

If the statistical model used for the matching process was misspecified, then imbalances of the characteristics between the treatment and control group could still prevail, which in turn would subtract validity from the ATET findings. In order to check for balance in the matched samples, Fig. 7 presents the kernel density for the propensity score distribution before (upper graphic) and after (lower graphic) matching using the three techniques. Possible imbalances should reflect a visible difference in the distribution of the propensity scores of both groups even after matching. Nevertheless, the distribution is visibly better after pairing in the three cases, with kernel and radius matching being slightly “better” than one to one matching. This is consistent with the application of the PSM methodology, as it shows that the matching reduces the differences between the control and treated units, and reinforces the inclusion of the conditions of the roof, walls, and access variables.

Fig. 7
figure 7

Kernel distribution of propensity scores from the probit model controlling for physical conditions of households, before and after matching, in Picsi, Chiclayo Province, Perú

Estimating the ATET with other specifications of the probit model produces similar results.Footnote 5 The other three columns of Table 4 correspond to the models in column 4, 10, and 12 of Table 3, which include the access to water, unspecified source of water, and bad sanitation variables. The ATET estimated in eight of the nine cases is statistically significant and negative. If we consider the most significant results, we observe that the estimated effect of the intervention varies between − 0.6628 and − 0.9266.Footnote 6

Finally, a robustness test of the damage of the house as the dependent variable was performed using ordinary least squares (OLS). The results are presented at the bottom of Table 4. The first column—column (m)—has no controls, and the other three have the same controls that were used for the ATET estimations. The results indicate that the kit treatment is always significant with a negative sign, and the coefficients are similar to the results of the matching estimations.

6 Conclusion

We provided an impact evaluation of an early warning intervention in Peru using propensity score matching techniques, which occurred in Picsi, a region in which Forecast-based Financing was implemented in order to allocate resources before an extreme climate event struck. Special kits were provided prior to a hazard occurrence—namely, the ENSO event—based on preselected forecasts and climate information. The effectiveness of the FbF program in mitigating the negative impacts of El Niño on housing infrastructure in Picsi is supported by the findings of the study. Our findings support the notion that treated homes did better—with a noticeable drop in the scale of house damage—than untreated ones.

Our results are at some point similar to the most recent published study by Gros et al. (2019) in terms of a positive impact of an early warning intervention. Nevertheless, we only measure the effect on one outcome variable (scale of house damage) while those authors—applying the same PSM techniques as ours—were able to quantify the impact on more outcomes: quality of beneficiary households’ food intake, psychosocial stress, destitution sales of valuable assets, and financial loans taken. As far as we know there is a “climate awareness” or “prevention programs on housing assessment” literature gap, giving an opportunity for further research on early warning interventions using impact evaluation tools.

An important limitation in our study is the presence of a lapse of time between the implementation of the kits and the actual protection they delivered. This situation has been addressed under the plausible assumption that no significant physical improvement was made to the treated households, which implies that the results found in this paper would represent a lower boundary of the real mitigation effect of the kits. Further research should take into account the possibility of periods when hazards do not occur and integrate them in the process of data collection and monitoring of the intervention, in order to gain precision and reduce uncertainty in subsequent estimations.

In summary, this study offers favorable evidence of an intervention that reduces the effects of a natural hazard-induced disaster. Interventions of this kind possess the advantage of having a window of time to reduce the potential consequences of disasters for society. Further research is needed to provide more parameters of interest for cost–benefit analysis, which is crucial to evaluate and compare the alternatives that could reduce disaster risks, such as in Tran et al. (2012) and Hochrainer-Stigler et al. (2011). In this sense, the results of this study indicate a physical impact of the FbF program, a magnitude that could be used as a reference value for economic calculations in terms not only of measuring economic losses in housing infrastructure, but also for contributing to better estimations of the needed investments by developing countries—facing vulnerability to natural hazards—that make their infrastructure more resilient and more economically robust (Hallegatte et al. 2019).

We strongly suggest that relief agencies develop and implement procedures to collect complete economic data before and after their interventions, so a full economic benefit analysis could be conducted. This kind of study will provide the information that donors require to increase the value of their resources when used for relief prevention responses.