8.1 Introduction

We started this book looking at the disaster risk reduction environment, of which a warning system forms a part, and the governance of the warning system itself. We then proceeded to look in detail at the individual steps that form a complete warning system, starting with the decision-maker and their need for information in order to take protective decisions and actions, proceeding to how they receive the warning, how a warning is put together, how the impacts of hazards are forecast, how hazards are forecast and how the hazard-generating weather is forecast. At each of these steps, we have looked particularly at how the various professionals involved in creating that information can work in partnership to more effectively deliver the warning service.

Now it is time to put it all back together again and to look at the warning chain as a whole.

As we have seen, a successful warning draws on many sources of expertise. In particular, we have emphasised the expertise of the receiver of the warning who is the best source of information on their need at the time of receiving the warning. The expertise of the communicator is quite different from the expertise of the satellite instrument designer or the meteorologist or hydrologist, and the range of expertise needed to translate hazard into impact is extraordinarily wide. Making a successful warning depends on these sources of expertise adding to each other rather than acting independently, in competition or in conflict. We have characterised the connections that achieve this as the bridges of our warning chain and described how they may be built through partnership. At its best, this chain of partnerships transparently communicates a consistent specification of the needs of the receiver to all of the contributing experts and communicates back reliable and trustworthy information about future hazards and impacts together with their uncertainties and appropriate responses, meeting the needs of the receiver, so that trust is created and communicated back up the chain, cementing and strengthening the overall relationship. Thus, while a failure of the warning chain arises from its weakest link, its strength comes from its end-to-end-to-end integration.

Research into the performance of whole warning systems is only just beginning, and so much of this chapter is based on anecdotal evidence. In this chapter, we will:

  • Look at a long-established warning chain that is highly effective and consider what we can learn from it.

  • Consider how to evaluate the outcomes of a warning system and look at the sensitivity of those outcomes to aspects of system performance.

  • Identify some levers that can be used in optimising outcomes.

8.2 An Integrated Warning Chain

As an example of a familiar and long-established warning chain, we consider fire alarms in public buildings. The first consideration is that they exist because the government has legislated that they must and in general has defined such things as how quickly those at risk must be removed from harm. At its simplest, a fire alarm detects smoke, indicating that a fire has started. Although smoke is a hazard in its own right, it is here being used as an indirect sensor of the existence of a fire. A responder hears the alarm, knows that it is a fire alarm, evacuates the building and calls the fire service, who come and put out the fire. If the fire is small enough, a responder may also use local facilities to attack it. A visitor also recognises the alarm and follows accessible instructions or responder directions to evacuate. If it works well, no one is hurt and damage is minimal. The alarm may be communicated by a siren and/or a flashing light and/or audible instructions. It may also communicate directly with the fire and rescue service, perhaps with a manual override available to remove false alarms and tests.

For the responder, the requirement is that everyone should be evacuated safely during the short time that may be available between detection and people being trapped. This requires that regular practices are held so that they recognise the alarm and know what to do when they hear it. It may also require that the building is zoned so that different groups of people take different routes, to avoid congestion, and that the responder should know where the fire is so that proximate zones can be cleared first while avoiding any exits affected by the fire. For the visitor, the requirement is that they, personally, should evacuate safely and be inconvenienced as little as possible. For the fire service, if it is a complex building, they need to know in advance, and have trained for, entry routes, access to water, presence of hazardous materials and so on.

Taking this out of its fire context, we see that governance is essential and that it should define the desired outcome for the good of the community. Each country will define for itself what that requirement should be, though safety of life will always be fundamental. Where fire insurance is widespread, and especially if it funds the fire service, alarms may be mandated as a requirement for cover, in which case minimisation of damage will be part of the requirement. We also see the importance of preparation for responders, who should practice sufficiently frequently to be familiar with their duties. The remainder of those affected need to recognise the alarm and then either be told what to do by the alarm system, have easy access to pre-prepared instructions or be guided by responders. On the face of it, a fire alarm system has the advantage of being fixed. However, in reality, exits get blocked for repairs, and the needs of those in the building will be different each day. A best practice alarm system will direct people only to available exits and ensure responders are trained to provide assistance to those who cannot walk downstairs, for instance. We have also seen that to provide appropriate information to different receivers, the alarm system may provide tailored information.

In all of this, some key partnerships are essential for safety. The three-way relationship among the building manager, the fire service and the local responders is critical in ensuring adequate preparation. The relationships between building manager, alarm system designer and alarm sensor manufacturer determine the structure of an alarm system that will give adequate warning. The relationship between the local responders and visitors will affect the behaviour of visitors in response to the alarm. It is noteworthy that, contrary to traditional practice with natural hazard warnings, the focus here is on designing an alarm system to a specific outcome (defined in legislation), not on installing the latest sensor technology and building a warning system to use it to achieve the best outcome in safety and damage limitation.

8.3 Evaluating the Warning Chain

What makes a good warning and how can we measure it? The first question is not too difficult to answer – a good warning is one that enables negative consequences of hazards to be reduced – and a perfect warning is one that prevents all such consequences that are avoidable. Measuring the value of a warning is much more difficult, since there is no control; every event is different, affecting a different population; and the sample is small. Limited progress has been made using case studies that compare disaster events before and after the introduction of a warning system (e.g. heat wave warnings after 2003 in France – Fouillet et al. 2008). However, such studies make the assumption that each of the cases represents the whole population of events before and after the introduction, respectively, and that nothing else has happened to change people’s response to the event, despite the fact that the trauma of the event that precipitated the warning system almost certainly changed behaviour in those affected. Where events are sufficiently frequent, it is possible to compare aggregate statistics before and after introduction of a warning system, as in the HIGHWAY project to provide user-oriented warnings to fishermen on Lake Victoria (Roberts et al. 2021).

Reduction of direct impacts may be estimated by modelling what would have happened without the warning-based decision. For instance, operation of a weir or installation of temporary flood defences will protect a well-defined area from flooding of predictable depth. The cost of repairing the avoided damage can be estimated fairly reliably, at least in countries with an active flood insurance market. Similarly, for the cancellation of a sports event in extreme heat, use of epidemiological analysis can give the expected number of people who would have died or been hospitalised, within defined statistical uncertainties. It is much more difficult to make such estimates when only a fraction of people responds to the warning. For instance, a hurricane evacuation warning may reach 80% of people in an evacuation area but suppose only half of those actually evacuate and the number of evacuees is swollen by those from outside the evacuation area who also choose to evacuate. In this situation, estimating the impact of the warning is much more challenging – undoubtedly depending on the socio-economic characteristics of the affected populations. In these circumstances, it is possible to build a model using historical data and surveys to estimate response inhibitors, but it will have substantial uncertainties and be subject to change after each event.

To ground the results in reality, it is essential to gather data on how people actually respond to warnings routinely. Properly designed and sampled surveys of the at-risk populations after each warning, supported by in-depth interviews, can provide evidence for longitudinal analysis of behavioural responses. There remain issues with the small number of events, with heterogeneous populations and with external changes, especially trends and shocks in the way people behave. Such surveys of how people report their behaviour also need to be supported by evidence of how they actually behaved, and this is increasingly possible through careful analysis of social media (Anderson et al. 2016; Eyre et al. 2020). Again, analysis should be carried out routinely to maximise the sample and to enable longitudinal analysis.

Elucidation of the factors that influence response behaviours can be carried out under test conditions where people say what they would do, provided the results can be related to real-life behaviour. Increasingly, immersive gaming approaches are achieving this, but the results still need checking in real situations.

Once established, the quantitative values of these factors can be used in the models of warning chain effectiveness. However, the factors that determine effectiveness of response are needed not just for the end user but also for the intermediate actors in the warning chain: those who assess the hazard and its impact and who decide how to convert the forecast information into an actionable warning. In some cases, personal prejudices will dominate – including fear of liability, over-optimism and distrust of models – making generalisation difficult.

While it is essential to know the outcomes that result from the issue of warnings, additional information is needed to determine how to make them better. Each chapter in this book has referred to the need to evaluate that step in the warning chain – both the quality of the information going into it and the quality of the information coming out of it – and for that evaluation to be carried out in terms relevant to the decision(s) that the receivers of the warning will need to make. However, the things that are measured are very different at each stage of the warning production process.

Each discipline along the warning chain, as depicted in Fig. 1.3, has developed its own methods of evaluation, geared to optimising its own performance in meeting the needs of those who use its products. These methods are valuable, but a means of relating them to each other and to the outcome of the whole warning chain is needed. Currently, there is no accepted methodology for connecting the quality measures for each step of the warning chain that will link them to the value of the decisions taken by warning receivers. While there are mathematical tools for relating the output of a mathematical model to its inputs, these are only currently deployed for individual models, not to the warning system as a whole.

8.4 Sensitivity of the Outcome

A typical approach to investing in an early warning system is to look at where there are technical options for improvement available, for the relevant organisations to undertake independent cost-benefit analyses on each and for the upgrades to be developed and implemented in isolation. In doing this, there is inevitably double counting of benefits, as each organisation claims the benefits of increased warning effectiveness.

Ideally, we should like to target investment of limited resources where it will be most effective. In short, if a flood warning system reduces the economic loss from floods by £1 m p.a., we want to know how much of that benefit is due to the monitoring network, how much to the weather forecast, how much to the hydrological forecast, how much to the translation into economic damage and how much to the form of the warning communication. We cannot experiment with the real world, so we need to use an alternative approach. Eventually, I hope this will take the form of a systems model, which reliably reproduces the behaviour of a warning chain, and can be used to identify responses to perturbations. Such a model does not yet exist. In the interim, we consider some of the sensitivities of the warning outcomes and how they depend on aspects of the warning system.

8.4.1 Precision

Precision is a characteristic of the underlying forecast information. It can be degraded in the formulation of a warning, but it can’t be enhanced. The steps from weather to hazard to impact can either degrade or enhance precision, depending on the nature of the hazard and its impacts.

Precision can be in time, space or intensity. Precision in time may be needed for closing evacuation routes before, or letting relief in after, a disaster. Precision in space can be critical for protecting the right people and evacuating the minimum numbers. Precision in intensity may not matter for the most destructive events but may be critical for deciding whether or not to evacuate or protect from lesser ones.

Precision links to accuracy. Precision is generally of little value if it is not accurate. However, highly precise, but inaccurate, predictions may capture hazards or impacts that would be missed by a lower precision forecast system, and these may inform useful responses through upscaling in time and space.

8.4.2 Timeliness

In general, accuracy, precision and confidence in the available information get better as lead time reduces. However, the protective actions that the receiver will take require time, so there are critical lead times after which some protective actions, such as emptying a reservoir or evacuating a city, can no longer be taken. Receivers often ask for high precision, accuracy and confidence before taking an action and are then restricted in the actions that they can take. Part of an effective partnership is to share what is possible, and what might be possible with a little more investment, so that the opportunities for making a difference are maximised. This is particularly an issue for low-probability, high-impact hazards for which very short lead times are typically available, such as tornadoes. If it takes an hour for the forecast to be processed, interpreted and delivered, the window of opportunity for action has been lost. Speeding that up by 50% could be life-saving.

8.4.3 Accuracy

Accuracy is something that is a particular focus higher up the warning chain. It is a function of spatial and temporal precision and lead time. Highly inaccurate information can kill and cause damage itself, and over time it destroys trust. It is therefore important to post-process forecast information, e.g. by statistical correction, by upscaling in time and space or by subjective interpretation, to a level where accuracy can be demonstrated. Of course, such processing inevitably reduces the information content. If this makes it no longer useful for the intended action, alternative actions may have to be considered until forecast accuracy can be improved.

8.4.4 Reliability and Trust

The reliability of forecast information is an important characteristic in building trust. If it can be demonstrated to the user that a higher level of warning is consistent with a higher level of damage, that higher confidence is associated with greater likelihood of occurrence or greater proximity, the recipient is more likely to take action in the future.

Trust is, perhaps, the most difficult characteristic to invest in. Anecdotally, it is extremely important, and there is increasing documented evidence that this is so. Defining and measuring trust is difficult. As well as depending on the object of the trust, it varies enormously between communities and cultures. Trust may be intrinsic to a product or supplier, but it can also be gained by association. For instance, a respected community leader will engender trust of the community in the information they pass on. In consumer societies, a trusted consumer brand may confer trust on something it associates with, including warning information, especially if there is a perceived link.

8.4.5 Understandability

This is perhaps the most difficult characteristic of a warning message to get right. Describing the location of a hazard precisely without using names that are only recognised locally is extremely difficult. Tying road warnings to intersection identifiers or river flood warnings to local names of reaches or urban flood warnings to street names will immediately lose contact with a substantial proportion of those affected. On the other hand, overcoming those problems by the use of maps will lose those who are unable to read maps. Language is equally problematic, especially in countries with very large numbers of local languages. It is not just that the message may be missed if it is not in a person’s first language but that it may still be lost if that language is not used idiomatically.

8.4.6 Reach

The warning chain can be broken at many points. When this happens, the value of the upstream information is lost. The easiest place to break the chain is if the information fails to reach the person making the decision. When everyone watched the evening news bulletin on television or radio, it was relatively easy to reach a large audience. That is no longer the case, with separate audiences for Twitter, Facebook, TikTok, etc. In multi-cultural societies, the media choices of different cultural groups need to be taken into account. And there still remains the challenge of reaching those who lack the ability to engage with any of these media: the frail, blind, deaf and housebound.

Where the decision-maker is a professional with responsibility for taking action to protect others, the importance of reaching them with critical information is much greater. It helps if they have been placed on-call, following an earlier warning. However, they may still be at home, asleep, at the golf course or at a party. Communicating the required information to these people is likely to be more than a Tweet or Facebook post, so making sure that they can access it from a mobile device is critical. In many countries, mobile connectivity is patchy and limited, so backup options need to be available, e.g. through conventional audio telephony.

8.4.7 User-Specific Warnings

We have emphasised a warning chain driven by the receiver of the warning. Yet the warning needs to be delivered to all those at risk and to those tasked with protecting them. Providing different warnings to different groups requires additional work, but can be worthwhile where requirements clearly differ, as between professional responders and the public or between users of specialist websites, such as sports users. The danger is that the same warning, delivered by different routes, in different formats and languages, is perceived to be inconsistent. Developments in technology are beginning to make it possible to tailor warnings automatically using systems like CAP. Genuinely personalised warnings are not currently available, but it is an area that will undoubtedly develop in the future.

8.5 Optimising the Outcome

So how do we design the perfect warning system? Here we distil the key messages from this book. The list given here draws heavily on the Early Warning Checklist produced at the first conference of the International Network for Multi-Hazard Early Warning Systems (IN-MHEWS) that took place from 22 to 23 May 2017 in Cancún, Mexico (IN-MHEWS 2018), to which reference should be made for more detailed recommendations on how to carry out each step.

  1. 1.

    It is essential to understand the risks that will be faced by the community and how they are perceived by those at risk. Ocean hazards will not affect a land-locked country, but flooding may be a future risk for an arid country. The nature of the risk may also change – from direct risks to rural agriculture, to indirect risks to urban populations depending on vulnerable infrastructure. If a hazard is not considered to pose a risk to a community, then warnings will produce little response unless or until that perception changes.

  2. 2.

    With the community, identify the problem. If flooding is the biggest source of risk, is it river or surface water flooding or both? Is it frequent minor flooding or occasional major flooding? Is it damage to crops or interruption of transport? Warnings may be applied generally, but the design of the warning chain should focus on priority impacts.

  3. 3.

    With the community or organisations concerned, identify candidate mitigation actions that would be acceptable to the user and that would reduce the risk. Identify what information would be needed, by whom, when and where, to enable those actions to be undertaken.

  4. 4.

    Investigate how that information could be assembled – the areas of expertise needed, the organisations with the capability to provide services, those that are trusted to deliver, the existing working partnerships and those that have the connectivity to get the information to those who need it.

  5. 5.

    Identify the limitations to delivery – accuracy, timeliness, precision, format, etc. – and whether those limitations could be removed by investment. Where possible, use quantitative measures of capability.

  6. 6.

    Return to the user and consider whether the available capability would enable a useful and cost-effective mitigation. If there are significant benefits but the investment is unaffordable, or not cost-effective, consider whether the warnings could serve other users and if those benefits would make the service cost-effective or affordable. If not, consider whether there are alternative mitigations that might be more achievable.

  7. 7.

    Plan the selected system in detail, incorporating not just the production and delivery chain but monitoring and evaluation of each step and an assessment of how to recover when a failure occurs in any component.

  8. 8.

    Build a partnership to deliver and manage the warning system.

  9. 9.

    Train the warning producers and users, both before implementation and periodically afterwards.

  10. 10.

    Evaluate every step in the chain continually to ensure good performance. Look for opportunities to invest in further improvement, especially as new research capabilities become available.

  11. 11.

    Periodically – at least every 10 years – repeat the design process to check whether the system is still meeting priority user needs in the most cost-effective manner and to re-design if necessary.

Finally, when you have a system that is working, let all of your stakeholders know how well it is working and how effective it is at saving lives and reducing damage. That will help grow confidence and trust, build the foundations for future investment and encourage others to follow. And remember that every warning that reduces distress is a successful warning.