Introduction

The use of scientific information about future weather and climate offers enormous potential for society to better manage the risks associated with climate variability and change (Williams et al. 2015). By packaging complex information into a weather and climate service (WCS), knowledge of future conditions can be incorporated into decision-making at national, regional, community and household levels. WCS uses historical and/or future weather and climate information at different timescales such as short-term weather forecasts, seasonal climate forecasts proving information up to a few months ahead and long-term climate change projections up to the end of the century (Bruno Soares et al. 2018; Vaughan and Dessai 2014). As such, WCS can deliver a diversity of information to a range of beneficiaries, e.g. early warning systems for flood-prone communities, agro-met advisories to support farmers or sea level rise projections to inform coastal planning. In this way, WCS can reduce weather and climate-related losses, including the protection of lives, livelihoods and property (Vaughan and Dessai 2014).

The cumulative benefits of WCS are potentially substantial. For agriculture alone, the World Meteorological Organisation (WMO) estimates that improved forecasting could lead to a US$ 30 billion annual increase in global productivity and US$ 2 billion per year in reduced asset losses (WMO 2019). In recognition of this potential, there has been considerable financial investment in WCS. It has been estimated that the total spend for all WCS, including commercially funded research for weather and climate data, reached more than US$ 56 billion in 2014–2015 (Georgeson et al. 2017). WMO (2020) also estimates that major international development organisations, including public development banks, have invested over US$ 3 billion on WCS throughout the world’s most vulnerable regions.

Despite large-scale investment, we still lack a solid evidence base to demonstrate the socio-economic benefits of WCS. Understanding the types of socio-economic benefits that can result from WCS, including what works for who and under what circumstances, can help justify investment in WCS and also contribute toward adaptive management to ensure more effective and equitable outcomes (Bruno Soares et al. 2018). A number of high-level studies and reviews have examined the benefits of WCS across sectors and countries as well as the methodological approaches evaluators may take (see e.g. Clements et al. 2013; WMO 2015; Perrels et al. 2013, Bruno Soares et al. 2018). However, many of these studies assess the potential benefits of non-operational WCS using ex-ante approaches, i.e. estimating expected benefits of services that may be planned or hypothesised, but for which there is currently no observable evidence of the service being used on the ground (Bruno Soares et al. 2018). There are far fewer ex-post evaluations of operational WCS where a service is currently (or has previously) been in operation and for which there is observable evidence of the WCS in action. Against this backdrop, this paper systematically collates and examines the peer-reviewed and grey literature on evaluations of operational WCS in South Asia. We use the literature to demonstrate (1) the scale and scope of operational WCS that have been evaluated in the region; (2) the type of methods used in the evaluation and (3) evidence of the socio-economic benefits.

Weather and climate services in South Asia

South Asia, comprised of eight countries—Afghanistan, Bangladesh, Bhutan, India, Maldives, Nepal, Pakistan and Sri Lanka—is home to 1.89 billion people, including 530 million of the world’s poorest people (OPHI 2020) who are extremely vulnerable to the negative impacts of climate change (Ahmed and Suphachalasai 2014). The region is also a hotspot for weather and climate-related disasters such as drought, flooding and coastal erosion, which between 1990 and 2019 affected almost 1.7 billion people and resulted in economic losses of around $ 127 billion (EM-DAT 2021). It is in this context that WCS emerges as a potentially valuable tool for building resilience to climate variability and change across the region (WMO 2020).

The World Bank alone has committed millions of dollars to South Asia. For example, the Bangladesh Weather and Climate Services Regional Project is a $ 113 million investment to increase the country’s capacity to deliver reliable weather, water and climate information. Similarly, the World Bank’s Climate Adaptation and Resilience for South Asia (CARE) funds innovative technology to support resilience across Bangladesh, Nepal and Pakistan with a particular focus on agriculture, livestock, water and transport. Smaller scale projects include the $ 3.8 million Hydromet Services and Disaster Resilience Regional project in Bhutan that includes support to strengthen flood forecasting capacity. There are also a number of regional organisations who support the development of WCS. For example, the Regional Integrated Multi Early Warning System (RIMES) focuses on capacity building of national and local level institutions on all aspects of end-to-end early warning systems. Similarly, the International Centre for Integrated Mountain Development (ICIMOD) facilitates the implementation of the Global Framework for Climate Services (GFCS) within the Hindu Kush Himalayas region by bringing together providers and users of climate information. The region is also home to India’s Integrated Agro-meteorological Advisory Service (AAS) programme, which is one of the largest agrometeorological information programmes in the world (Nesheim et al. 2017).

Methodology

Systematic literature review

To demonstrate the scale and scope of WCS evaluations, methods used for evaluation and socio-economic benefits, we adopted a systematic literature review approach. This approach involves following a detailed search strategy and pre-defined plan to identify and synthesise studies relevant to a particular topic (Uman 2011). With a long history in health science, systematic reviews have more recently been used to synthesise knowledge on issues in climate change adaptation (e.g. Ford et al. 2011; Berrang-Ford et al. 2015) and international development (e.g. Mallett et al. 2012), as well as weather and climate services (e.g. Nkiaka et al. 2019; Gumucio et al. 2020; Tall et al. 2018).

To identify peer-reviewed papers that focus on the evaluation of WCS, we used ISI Web of Knowledge, the largest and most comprehensive research publication database. A keyword search was made for journal articles. A total of 168 searches were performed using a two-level search approach (Table 1). The first level specified the type of weather or climate service under investigation and the second level specified the country of interest. Truncated searches were used to allow searches for words beginning or ending with the same letters. To identify grey literature, including reports and policy documents, we contacted regional WCS experts including experts at the UK Met Office, the Regional Integrated Multi-hazard Early Warning System (RIMES), the International Centre for Integrated Mountain Development (ICIMOD), the World Bank and the Red Cross/Red Crescent Climate Centre. We provided experts with an online template in which we asked them to record the title and author of any relevant documents and to provide an electronic link where one existed. For documents without an electronic link, we asked the experts to send us this by email. We asked the experts to forward the template to colleagues they thought may be able to help. We also performed a simplified keyword search (based on Table 1) using Google to identify any documents our experts may have missed.

Table 1 Search terms to systematically search the peer-reviewed literature
Full size table

Search results and document review

All peer-reviewed and grey literature documents identified through the keyword search or by the regional experts were screened according to the inclusion/exclusion criteria in Table 2. To be included in the review, documents must be based in one of our eight South Asian focus countries and have been published since 2009, following the establishment of the Global Framework for Climate Services (GFCS). The document must also present an empirical evaluation that measures or describes the impact of a WCS that is, or has been, operational and observable on the ground. This includes evidence of a reproducible methodology for collecting empirical data. Papers that simply restate ‘common knowledge’, for example, government figures on deaths avoided as a result of a cyclone early warning system (EWS), were not included. Additionally, papers that evaluate other factors, including the extent to which WCS is understood or valued by end users, were not included unless knowledge of the benefits generated was explicitly stated.

Table 2 Inclusion and exclusion criteria
Full size table

Our peer-reviewed literature search revealed a total of 2146 papers. Once duplicate papers had been eliminated, 1706 papers remained. The remaining papers were then sorted according to our inclusion/exclusion criteria. We took a two-stage approach to the sorting process. First, we looked at the title of the paper and read the abstract to identify documents for potential inclusion. Second, we reviewed each potential document and included those that provided a substantial commentary on the evaluation of an observable WCS. We categorised papers into those focusing on physical dimensions of weather and climate science (n = 1109); the impacts and adaptation of weather and climate in social and natural systems (n = 207) and the development and implementation of WCS but without a focus of benefits on the ground (n = 41) and papers not related to the topic/geographical location or papers that were unavailable (n = 342). In total, we found seven peer-reviewed papers that explicitly set out to evaluate the socio-economic benefits of a WCS service in South Asia. We took a similar approach for the grey literature where we used the same inclusion/exclusion criteria to screen documents. In total, we identified six evaluation documents from the grey literature. Considering the peer-reviewed and grey literature together, we found a total of 13 evaluation documents that we report on in Sect. 4.

To systematically examine both the peer-reviewed and grey literature, we developed a database in Excel based on our three objectives, i.e. to understand (1) the scale and scope of evaluated WCS from the region; (2) the methodological approaches used to evaluate the WCS and (3) the socio-economic benefits of the WCS as they are reported in the literature. First, to understand the scale and scope of operational WCS that have been evaluated in the region, we recorded data on the geographical location of the WCS, the sector in which the WCS best fits, the producer and intended end user and the type of service being provided. Second, to document the types of methods used to evaluate WCS, we recorded data on methodological approach, research tools used, number of participants, type of analysis undertaken and challenges or limitations of the method as identified by the author. Finally, to demonstrate the socio-economic benefits, we take a ‘triple bottom line’ approach (Lazo et al. 2008; World Meterological Organisation (WMO) 2015) to our assessment in which the economic, social and environmental benefits are all considered. We then used thematic content analysis to combine the evaluations and identify benefits that are common across the studies (Thomas and Harden 2008).

Results

Scale and scope of weather and climate services evaluations

Of the 13 evaluations included in our analysis, there were six evaluations from India, six from Bangladesh and one from Nepal (Table 3). No examples were found of documents evaluating observable WCS in Afghanistan, Bhutan, Maldives, Myanmar, Sri Lanka or Pakistan. This is reflective of the wider lack of reporting from the region on any aspect of WCS, including the physical aspects (i.e. producing weather or climate science). Papers on the science underpinning WCS overwhelming came from India (n = 757), followed by Pakistan (n = 78), Bangladesh (n = 47) and Nepal (n = 36). Very few scientific papers came from Afghanistan (n = 9), Bhutan (n = 11), the Maldives (n = 4), Myanmar (n = 9) or Sri Lanka (n = 22). In terms of sector, seven examples focused on agriculture and food security, five on disaster risk reduction (DRR) and one on health. All of the 13 evaluations focused on weather timescales with lead times of up to 7 days. None of the evaluations that we found dealt with seasonal times scales (i.e. the provision of information over the coming months) or climate projections (i.e. the provision of information about expected impacts in the coming years). In all our examples, weather information was produced by a country’s National Meteorological and Hydrological Services (NMHS), specifically the India Meteorological Department (IMD), the Bangladesh Meteorological Department (BMD) and Nepal’s Department of Hydrology and Meteorology (DHM).

Table 3 Scale and scope of WCS under evaluation in South Asia
Full size table

Methods for evaluating weather and climate services

We identified a variety of methods used to evaluate the socio-economic benefits of WCS including quasi-experimental research design; cost benefit assessments; quantitative mortality outcomes and qualitative approaches. This section presents details of these methods and their limitations. The reported benefits from these evaluations are detailed in Sect. 4.3.

Quasi-experimental research design

We identified three evaluations that used a quasi-experimental research design to compare outcomes between communities/households who received a WCS (the intervention group) and those who did not (the counterfactual). Quasi-experiment studies estimate the causal impact of an intervention on an intervention group and a comparison group (the comparison group provides evidence of the outcome if the WCS had not been implemented) (White and Sabarwal 2014). Unlike a true experimental research, there is no random assignment in a quasi-experimental approach.

Two studies focused on the impacts of India’s Agrometeorological Advisory Service (AAS) produced by IMD. Maini and Rathore (2011) assessed the economic impact of weather forecast advisories in 15 of India’s AAS units during three summer seasons (Kalif) and three winter seasons (Rabi) in 2003–2007. The authors surveyed 40 farmers who used AAS and 40 farmers who did not. Data were analysed to revel changes in yield, production costs and profitability between the intervention and control groups. Also in India, Chakraborty et al. (2018) interviewed 60 rice farmers from ‘contact’ villages in which AAS was disseminated by email or SMS and 60 rice farmers from ‘non-contact’ villages in which AAS was not formally disseminated. Data from both groups of farmers were analysed to reveal differences in risk/awareness around climate change as well as production costs and profitability. In the third study, Gros et al. (2019) examined the impact of the Bangladesh Red Crescent Society (BDRCS) Forecast based Financing (FbF) project in which an unconditional cash grant of US$ 60 was distributed to 1039 households in advance of the 2017 flood event in the Brahmaputra river basin. Following the flood, surveys were conducted with 350 households in the intervention group, and 350 households in the control group. Qualitative data from 16 focus group discussions and 16 key informant interviews in intervention and comparison communities was also collected.

Although all three studies reported a range of benefits associated with using the service (Sect. 4.3), it is worth noting the methodological challenges associated with counterfactual studies of WCS. First, because WCS is non-rival and non-exclusionary, it is difficult to say for certain that non-users truly are non-users as weather information can be passed through social networks (Tall et al. 2018; Vaughan and Dessai 2014). This is a particular issue in the studies by Maini and Rathore (2011) and Chakraborty et al. (2018) where it is unclear to what extent participants in non-contact areas used the AAS. Second, confounding variables, factors that influence both the dependent variable and independent variable may indicate an association that does not exist (Sanderson 2000). For example, the non-users of a WCS may be excluded from using that service because they cannot afford to access the service, or because they lack the education required to understand the information (Carr and Onzere 2018). These factors may also mean that non-users are less successful farmers. Confounding variables were overcome to some extent in the Gros et al. (2019) study where the authors used a statistical matching technique (propensity score matching) to reduce bias.

Cost–benefit analysis

We found two examples of cost–benefit analysis (CBA), where the economic benefits attributable to a project are compared against the costs of funding the project (WMO 2015). Firstly, Rai et al. (2020) analysed the costs and benefits of a flood community-based early warning systems (CBEWS) in the lower Karnali River Basin in Nepal. The researchers conducted 30 focus group discussions and 40 key informant interviews to develop broad categories of the goods and assets that households may save as a result of early warning information. Following this, 453 surveys were conducted at random with households in the CBEWS area. The baseline was set against a situation in which no EWS existed. Households were asked to self-report perceived gains and losses since the CBEWS had been active. Secondly, Hasan (2018) in an unpublished Master’s thesis used a CBA approach to examine the benefits of a flood EWS in Bangladesh. The researcher randomly surveyed 50 households who lived in the EWS area and 50 households who did not. Thus, the households who did not receive EWS information became the ‘control’ group against which the baseline or counterfactual was set. In other words, the control group represented what happened in the absence of the EWS.

In terms of methodological limitations, both authors noted the difficulties with obtaining accurate data of the costs of damage at both the national and sub-national (e.g. village) levels, which is needed to validate survey data. Both note that this data was often kept confidential, was inadequate or simply did not exist. However, Hasan (2018) showed that household data on damage and recovery costs, as well as savings, was usually good enough to include without validation. Rai et al. (2020) highlighted an additional challenge of linking flood damage data to hazard-specific event data (such as the flood return period).

Quantitative mortality outcome evaluation

In an evaluation of Ahmedabad’s Heat Action Plan (HAP), Hess et al. (2018) performed a quantitative mortality outcome evaluation to examine the impact of an EWS for extreme heat. The research team used gender-stratified mortality data from crematoriums to compare deaths during the hot May season following the introduction of the HAP with deaths during the seasons before the plan was introduced. Specifically, the team created a reference period to represent normal May temperatures by taking the daily average death rate for May 2009 and 2011. To determine excess deaths during an excessively hot season, the team compared the reference period to the mean daily May mortality for 2010 when a severe heat wave affected the city. Figures were then compared to 2013 (the 1st year post-HAP launch, with relatively cool temperatures) and 2014 (the 2nd year post-HAP launch, with relatively hot temperatures). The authors note that any differences in mortality outcomes between seasons may not solely be attributed to the HAP. Comparisons of daily temperature-mortality relationships across years may be affected by a number of confounding variables. These include population increase amongst the general population and heat-vulnerable groups; increases in air pollution; outbreaks of infectious diseases and changes in access to air-conditioning. The authors state that they do not control for these confounding factors in their figures.

Qualitative approaches

We identified seven evaluations that used qualitative approaches to understand end users’ perceptions of benefits. Qualitative approaches use a variety of methods to collect data but are united by centring the analysis of benefits around the end users and their thoughts and experiences on how they benefit (Bruno Soares et al. 2018).

In the first evaluation, the Gorakhpur Environmental Action Group (GEAG) undertook focus group discussions (FGDs) with 92 farmers from four villages to understand user perceptions of the socio-economic benefits of agro-met advisories in Uttar Pradesh in Northern India (Christian Aid 2015). The FGDs discussed perceptions in changes of crop yield, avoided losses and avoided costs/reduction in spending. In the second evaluation, researchers from the Climate Services for Resilient Development (CSRD) programme, led by the International Maize and Wheat Improvement Centre (CIMMYT), provided Participatory Integrated Climate Services for Agriculture (PICSA) training to 1100 farmers in Bangladesh (Krupnik et al. 2018). The PICSA training focused on how to access and understand the agro-met advisories provided by BMD. To assess the benefits of using the advisories, a survey collected data from 280 PICSA-trained farmers (61% men, 39% women) on changes to farming decisions following the training and perceptions of their effectiveness. In addition, six FGDs split by gender took place. In the third evaluation, also based on a CIMMYT project, researchers assessed perceptions of the effectiveness of an Interactive Voice Response (IVR) System which provides rainfall weather alerts and harvesting advisories to mung bean farmers in Bangladesh (Krupnik et al. 2020). A telephone survey of 676 farmers whose phone numbers were randomly drawn from the list of 1373 recipients of IVR system was conducted. Fourthly, Mittal and Hariharan (2018) randomly selected farmers to give feedback on the usability and effectiveness of a mobile-based AAS in India. With a sample of 493, farmers were asked to self-score their awareness of weather information and improved agricultural practices, as well as the monetary benefits such as increased yields, reduced input costs, reduced losses and increased income. Fifth, Nidumolu et al. (2020) used a mixed methods approach to evaluate the benefits of community-led Climate Information Centres (CLICs) where agricultural information is provided to farmers in India using digital technology. A research team surveyed 330 farmers across eight villages using a structured questionnaire. Two gender-specific FGDs were undertaken in each of the eight villages with 15 farmers participating in each. Additionally, a semi-structured interview with four farmers in each village (a total of 32 across the eight villages) was conducted. Data collection included farmer’s self-rating of the perceived benefits. In the sixth evaluation, Tanner et al. (2019) conducted 30 interviews with recipients of cash distributed by BDRCS FbF initiative (as described in Sect. 4.2.1). Finally, in the seventh evaluation, researchers from Deltares, a European-based independent water research institute, used 91 household surveys and ten FGDs to gather perceptions on the economic benefits of communicating 5-day flood early warning messages via mobile phone voice and SMS messaging in Bangladesh (Deltares 2015).

Although qualitative data provides a rich understanding of the benefits of WCS as experienced by the end user, it is more difficult to compare benefits across services (McNie 2012). However, it is generally well-recognised that qualitative studies have advanced our understanding of what the success or failure of climate interventions might look like across scales, with many scholars calling for small-n, qualitative case research to ‘ground-truth’ local realities (see Biesbroek et al. 2018).

Evidence of weather and climate services benefits

We found multiple examples of benefits in our body of literature. We organised and analysed these benefits using the triple bottom line framing in which the economic, social and environmental benefits are considered (Table 4). The majority of the evaluations focused on economic benefits as reported in 12 of the 13 evaluations. Social benefits were mentioned in six documents and only four evaluations reported environmental benefits, although this was not the main focus of the evaluation. Using thematic analysis, we further categorise benefits into nine distinct types organised under the three criteria of the triple bottom line (Table 5). More specifically, we categorise economic benefits into four types: (1) reduced cost of cultivation and agricultural inputs; (2) increased farm production/avoidance of crop loss; (3) avoidance of loss of property from flood damage and (4) avoidance of accrued debt. We categorise social benefits into three types: (1) protection of human life, including physical and mental health; (2) increased preparedness for emergencies and increased awareness of threats and (3) household decision-making for women. Finally, we categorise environmental benefits into two types: (1) reduced irrigation and (2) efficient use of pesticide and fertiliser. The remainder of this section details evidence from the literature under each of the nine types of benefits we identify.

Table 4 Categorising the benefits of WCS under the triple bottom line
Full size table
Table 5 Triple bottom line of benefits as reported in the evaluations
Full size table

Economic

Reduced cost of cultivation and agricultural inputs

Reduced cost of cultivation and agricultural inputs was dealt with in four of the 13 evaluations. Maini and Rathore (2011) reported that farmers who use India’s AAS service reduced the cost of cultivation of grains and cash crops by 2–5%, except in the case of fruits where costs increased. The authors suggested this was due to efficient use of fertilisers and seeds and the spraying of pesticides at the most appropriate time. Also for India, Nidumolu et al. (2020) reported average annual savings of US$ 4–64 per hectare for each farmer who used contact centres where AAS information was disseminated. This was attributed to improved pest-related information and reduced household and hired labour costs. The farmers themselves highlighted the importance of information videos as well as improved access to agro-meteorological bulletins as reasons for the savings. Chakraborty et al. (2018) noted that India rice farmers who used the AAS were able to reduced costs by US$ 8/ha with most savings related to human labour and irrigation costs during the summer rice season. Finally, the Christian Aid (2015) evaluation of India’s AAS, which is enhanced by a locally employed climatologist and disseminated a local NGO, showed users of the service were able to reduce irrigation, labour and biopesticide costs by up to 50%. Specific saving on fertiliser costs for wheat was US$ 33 per acre (about 25% of costs without the service).

Increased farm production/avoidance of crop loss

Increased farm productivity and avoidance of crop loss were reported in six of the 13 evaluations. Maini and Rathore (2011) observed that the crop yield for grain, cash and fruit crops in India increased by almost 10–15% with use of the AAS. Chakraborty et al. (2018) noted that Indian rice farmers gained US$ 50–95 more profit/ha than non-AAS farmers. Amongst the farmers who took part in the Christian Aid (2015) FGDs in India, it was agreed that yield of crops including rice, wheat potato, groundnut, sugar cane and vegetables had increased by about 10–20% as a result of weather information. The farmers attributed increase yield to adjusting sowing times; irrigation management; timing of pest control, frost damage avoidance measures and compost/fertiliser application and timing the harvest to increase the likelihood of grain being stored at the optimal moisture content. For the farmers who had attended the PICSA training in Bangladesh, 86% of those surveyed thought their incomes had increased as a result of trying new crops, changing livestock management practices, changing crop/land management and reducing irrigation (Krupnik et al. 2018). A further 72% reported increased food security. In a perception-based evaluation of India’s AAS, 60% of surveyed farmers in Bihar and 60.5% of farmers in Haryana reported monetary benefits, although these were not quantified (Mittal and Hariharan 2018). Finally, farmers who took part in CIMMYT’s evaluation of IVR disseminated mung bean advisories estimated avoided losses of between 48 and 52% of their crop (Krupnik et al. 2020). This equates to between 238 and 772 kg/ha of mung bean yield, or US$ 187–607 kg/ha. Through the actions taken by farmers to avoid rainfall-induced damage, farmers saved mung beans worth an average of US$ 127,180.

Avoidance of loss of property from flood damage

Three evaluations detailed economic benefits related to avoided losses from property damaged during floods. In Rai et al. (2020) CBA of the CBEWS for flooding in Nepal, 96% of the 453 survey respondents indicated that during the most recent flood, they were able to save property. On average, households reported saving property with a value of US$ 1083. Property included cash, jewellery, motorcycles, tractors, carts, livestock and cereals. Importantly, 84% of survey participants reported that they would not have been able to save anything without the EWS. Considering avoided losses as well as possible benefits to health and lives saved, the authors suggest a CBA of 83. In a similar study, Hasan (2018) used a CBA to determine the benefits of Bangladesh’s cyclone EWS and found that following a cyclone warning, 98% of his 50 survey respondents took measures to protect educational materials whilst 96% secured household belongings. A further 52% moved their livestock to safety. When the cost of family level damage and recovery is considered together, there is an annual net benefit of US$ 285 for households under the EWS. For respondents who took part in the Deltares’ (2015) evaluation of the mobile phone–based flood EWS in Bangladesh, the average savings per household were estimated at USD$ 472. Savings by sector included US$ 590 per farming household as result of storing seed and early harvesting; US$ 707 per fishing household as a result of netting ponds to stop fish escaping, selling and preserving fish and raising dykes; US$ 613 per cattle farming household from moving cattle to embankment and raining plinths; US$ 147 for households involved in business and US$ 188 for household involved in weaving, as well as US$ 354 per household through the rapid collection of medicine and temporary sealing of tube wells.

Avoidance of accrued debt

The avoidance of accrued debt was noted by two studies that both evaluated Bangladesh’s FbF project. First, Gros et al. (2019) reported that 58% of households who received forecast-based cash assistance in advance of flooding did not take out a loan during or immediately after the floods, compared to only 40% in households who did not receive the FbF assistance. Overall, the intervention group was 30% less likely to accrue new high-interest debt during and immediately after the flood period. Tanner et al. (2019) also showed that fewer transfer beneficiaries had to borrow money and those who did took out smaller loans. The authors calculated FbF recipients who did take out loans saved US$ 7–11per person on interest payments as their loans were smaller.

Social

Protection of human life, including physical and mental health

Protection of human life, including health, was reported in three documents. In their evaluation of India’s Ahmedabad HAP, Hess et al. (2018) demonstrated a 25% decrease in excess mortality in the 2 years since the HAP was launched. Excess mortality in May, the hottest month, was 24% lower in 2014 than during the historic 2010 heat wave, and 52% lower than the less extreme 2013 heat wave. In other words, since the HAP was introduced, fewer deaths were reported during the hottest months. The authors suggested this was a result of providing early warnings, as well as awareness raising and capacity building across medical centres municipal agencies and within vulnerable communities. In terms of lives saved/protected, Gros et al. (2019) were unable to draw firm conclusions on evacuation behaviour between FbF cash recipients and non-FbF cash recipients. However, the health benefits of the intervention were clearer. Households who received FbF assistance were over three times less likely to skip meals and ten times less likely to reduce meal sizes than non-recipients. Additionally, food quality was reduced in non-FbF households. There is also evidence to show that, following flooding, non-FbF households felt anxious or depressed significantly more frequently than the intervention group. Similarly, Tanner et al. (2019) reported that FbF recipients were three times less likely to have skipped meals or reduced meal sizes. Additionally, those receiving the cash transfer were 80% less likely to report feeling miserable or unhappy following the flood.

Increased awareness of threats and preparedness for emergencies

Awareness and preparedness were explored in five evaluations. In their perception-based evaluation of AAS in India, Mittal and Hariharan (2018) reported that nearly 90% of surveyed farmers felt they benefited from the AAS through improved understanding of weather information as well as different farm practices, input use, climate-smart agriculture and technology. Similarly, Chakraborty et al. (2018) observed that Indian rice farmers in a village with an AAS dissemination service were more aware of the threats posed by climate change than those in villages without the service. This included the possible impacts on livelihoods, food security, water availability and disease. In terms of preparedness, Rai et al’s (2020) evaluation of the flood CBEWS in Nepal found that all 453 households surveyed were fully prepared to respond to a flood warning by informing neighbours, saving property and evacuating the area. Gros et al. (2019) explored awareness from an ex-post perspective and found statistically significant differences in pre-flood behaviour between FbF beneficiaries and non-beneficiaries. In particular, 57% of FbF beneficiary households purchased food compared to 38% of comparison households. Beneficiary households were also twice as likely to reinforce the roof or walls of their house compared to non-beneficiaries. Similarly, Tanner et al. (2019) reported that, although both FbF recipients and non-recipients took action, cash grants enabled greater preparations, including stockpiling of food before prices rose post-flood.

Household decision-making for women

Only one evaluation specifically explored the impact of WCS on women. In their evaluation of the socio-economic benefits of agro-met advisories in India, Christian Aid (2015) noted that women in particular had described how forecasts have helped them make decisions about household welfare. This included using forecasts to make decisions on the use of their own labour; making plans for childcare during adverse weather events to mitigate risks of colds and other disease and making strategic household decisions around where to store wood, livestock feed and household goods (including food) following a rain forecast that would reduce access to local markets and local mobility.

Environmental

There were four reported examples of potential environmental benefits although this was not the main focus of the evaluations. In Christian Aid’s (2015) evaluation of locally contextualised AAS in India, focus group respondents attributed cost savings to reduced irrigation. For example, with an accurate forecast, farmers were successfully able to hold back on irrigating before a period of significant rain. This has the dual benefit of reducing water use and prevent crop damages through waterlogging. At the same time, fertiliser costs for farmers who rely on chemical fertiliser were also reduced, suggesting a more efficient use of expensive inputs and therefore less environmental damage, e.g. through nitrogen contamination leaching into groundwater. A similar finding was described by Maini and Rathore (2011) who noted that the Indian AAS had contributed to both greater productivity and increased resilience by encouraging farmers to adopt modern agricultural production technologies and practices, weather-based irrigation management, pest/disease management and the use of improved post-harvest technologies. Chakraborty et al. (2018) noted that India rice farmers who used the AAS reduced irrigation particularly during the summer rice season. Finally, Krupnik et al. (2018) noted that PICSA-trained farmers in Bangladesh had also reduced irrigation.

Discussion

Our results highlight two key findings in relation to the evaluation of WCS benefits in South Asia. Firstly, to date, limited empirical evidence on the observable benefits of operational WCS has been documented. Whilst it is clear that benefits are achievable, and methodologies for capturing and documenting them do exist, they are not being systematically recorded in either the academic or grey literature. Secondly, where benefits are reported, there is an emphasis on reporting short-term economic benefits and limited reporting of social and environmental benefits. Without detailed evidence of the range of potential benefits that can be realised across the triple bottom line, we may risk following pathways that result in limited social and environmental benefits as well as long-term economic benefits. The following two sections explore these points in more detail, before we conclude by presenting our vision for a systematic and robust approach to documenting benefits.

Limited evidence of benefits

This study found only seven peer-reviewed papers and six documents from the grey literature that explicitly set out to empirically evaluate the observable benefits of operational WCS in the South Asia region. The overwhelming majority (n = 1109) of peer-reviewed literature that we found during our systematic review presented data on the upstream aspects of the value chain, i.e. technical and scientific development of weather and climate information. This highlights the efforts invested in the region over the past 10 years towards developing a knowledge base of the scientific underpinnings of WCS (e.g. Chevuturi et al. 2019; Deb et al. 2015; Gouda et al. 2018). Although we found a body of literature (n = 41) that used social science methods to access WCS at the community level, this largely focused on the extent to which end users access and understand complex information (e.g. Venkatasubramanian et al. 2014; Vedeld et al. 2020; Chattopadhyay et al. 2018). Whilst it is vitally important to understand the scientific underpinnings of WCS, as well as the ability of communities and households to use information, the lack of data around benefits leaves a critical gap in our understanding.

The limited effort to capture, analyse and document a range of benefits across the triple bottom line reflects the urgent need to bring interdisciplinary teams into WCS development and implementation (Bruno Soares and Buontempo 2019). By including social science experts in the development and implementation of WCS, all stages of the WCS value chain can be addressed from producing understandable science to evaluating the real benefits for the end user. This is critical not only for justifying funding but also for enabling programme managers to adopt an adaptive management approach to the development of WCS (Born 2020). Such an approach could enable a redesign of WCS to ensure services benefit end users. Our results identified a wider lack of reporting from the region, including reporting on the physical aspects of WCS, especially Afghanistan, Bhutan, the Maldives, Myanmar and Sri Lanka. There is scope for any intervention that promotes scientific capacity building in these countries to also include capacity building for carrying out evaluations.

Over emphasis on short-term economic benefits

Our analysis demonstrates that the focus of current evaluations in the region is on short-term economic benefits with little focus on longer-term benefits across the triple bottom line.

Whilst economic benefits are key to understanding what can be gained monetarily from WCS, it is critical that future analysis extends to all potential benefits, even those that are less tangible (Bruno Soares et al. 2018). Without a comprehensive assessment of benefits, we risk creating trade-offs across time-scales within and between the three pillars of the triple-bottom line. The climate adaptation and DRR literature provide examples of how these trade-offs may occur. For example, Magnan et al. (2016) point out that decision-makers may justify adaptation interventions based on short- to medium-term benefits. They document the example of the USD 160 million Coastal Climate-Resilient Infrastructure Project in Bangladesh that aims to reduce the vulnerability of 600,000 households by upgrading physical infrastructure. Whilst this may have short-term economic benefits, unless current plans consider climate change, longer-term economic and social benefits risk being undermined as communities are encouraged to remain in areas of future high risk.

Similarly, without long-term planning that explicitly considers trade-offs, interventions that are designed to solve one problem may inadvertently create other problems. For example, Santha (2015) reports how the construction of a protective seawall in Kerala, India, meant the fishing community were no longer able to access fishing and trading grounds, which had long-term economic and social impacts. The community also felt that the seawall had created sedimentation and altered the habitat of the fishing grounds leading to long-term environmental change. Additionally, the adaptation and DRR literature highlights how the benefits of interventions are not distributed equally (Adger et al. 2005; Suckall et al. 2019). For example, gender mediates the use of cyclone shelters with women less likely to use shelters than men (Juran and Trivedi 2015), and therefore less likely to receive the benefits they provide. From the adaptation literature, it is evident that, unless competing interests are taken into account, initiatives designed to deal with climate change may create increased poverty for marginalised groups, create population displacement and create violent conflict (see Sovacool et al. 2015 for illustrative examples of these risks). Exactly how different groups will gain or lose from WCS is an area that has not been sufficiently explored by evaluators.

Finally, it is recognised that an emphasis on reporting the short-term benefits of WCS may partly be due to a lack of WCS that uses extended range forecasts in the region, although efforts are underway to create seamless forecasts that combine short-, medium- and extended-range data (Webster 2013; Manjula et al. 2022). To ensure long-term resilience to climate risks and to avoid maladaptation, evaluators could also adopt an approach that recognises long-term resilience will only be achieved by considering how information is used at different temporal and spatial scales. To this end, Singh et al. (2018) present a framework that shows how actions at one temporal scale may affect other temporal scales.

Conclusion: towards a systematic and robust approach to documenting benefits

The results that we present in this paper raise at least two important research questions. First, what are the longer-term economic implications as well as the short- and long-term social and environmental implications of the WCS currently being deployed in South Asia? Second, who are the winners and losers of WCS interventions?

To answer both questions will require increased cooperation between meteorologists, economists and social researchers to develop empirical case studies of benefits and trade-offs as they occur on the ground. Our results show that there are multiple methodological approaches used to develop such case studies. Whilst there is probably little value in creating a standardised methodology to collect data and assess benefits, a standardised methodology for reporting the benefits of WCS may be useful. If benefits are systematically reported, we can start to see patterns in what works and for whom. However, this is a complex task given that local context determines the type of benefits that are possible to achieve, as well as who those benefits reach. Similar arguments have played out in the climate adaptation field where it is acknowledged, that unlike mitigation, there is no universal unit to measure success (Dilling et al. 2015).

To overcome the conceptual and methodological difficulties associated with reporting across different geographical and social contexts, we suggest programme managers take three key points into account when evaluating benefits. First, we suggest evaluators frame evaluations around a triple bottom line approach where economic, social and environmental benefits are recorded. Using this as the starting point, evaluators should establish the local context for each of the three criteria with a specific focus on what matters to local well-being, including the community’s long-term goals for environment and society (Dilling et al. 2015).

Second, we suggest that evaluators consider how WCS can contribute to inclusive development, which is broadly defined as the aim to increase human well-being and improve environmental sustainability whilst closing the gap between rich and poor (Gupta et al. 2015). Frameworks to assess the impact of climate interventions on inclusive development already exist. For example, Suckall et al. (2019) developed a framework to identify the winners and losers of coastal adaptation based on how an intervention affected an individual’s access to resources, the allocation of the impacts associated with climate change and personal well-being. Understanding how different groups are affected by WCS will help policymakers and practices target interventions and mitigate potential risks.

Third, we reiterate the importance of engaging with end users through the entire process of WCS development. At the start, increased coproduction of WCS is more likely to improve engagement and identify poor and marginalised farmers at risk of exclusion from the benefits (Vedeld et al. 2020). Following implementation, it will be important to collect empirical data from end users themselves through ex-post evaluation that documents the actual benefits as they unfold on the ground. As part of this, baseline data may need to be collected before the WCS is implemented to make the resultant changes easier to identify (Vaughan et al. 2019). This means that evaluations will need to be built into project design from the start. Such evaluations should explicitly set out to identify, monitor and report on potential winners and losers through local level participatory research. Without empirical data on who gains and who loses, we risk delivering WCS that has high, and as yet unidentified, social, economic and environmental costs.