Background

Since 2001, there has been a dramatic increase in the use of disaster preparedness exercises among public health agencies in the United States[1, 2]. These exercises have explored a wide range of topics from foodborne toxoplasmosis outbreaks[3], chemical disasters[4], acute blood shortages[5], bioterrorism[6], and severe acute respiratory syndrome (SARS)[7]. Exercises have been designed to assess and improve a variety of capabilities such as regional disaster preparedness among rural hospitals[8], knowledge and confidence of legal authorities[9], resource allocation[10] and risk communications[11]. A large number of these exercises have been focused on the spread of infectious diseases especially the threat of pandemic influenza because the common challenges pandemic influenza shares with other types of public health emergencies[12–16].

Our knowledge of the use of exercises for public health-related disaster preparedness outside the United States is much more limited. A considerable number of these types of exercises in the United States have been published in the academic literature but few findings from exercises that have taken place outside the United States have been published. Some researchers in the United States have tried to solve this gap by publishing the findings of “virtual” internet-based, long distance exercises conducted remotely with international partners[17, 18]. Even less is known from direct experiences with in-country exercises or exercises that span multiple countries in a given region. The results of these types of exercises may be reported directly to exercise participants but often don’t make it to the scientific literature. If the results from exercises are published in any systematic way, they often get published in in-house publications for domestic audiences rather than scientific journals with a more global reach. The incentives, financial or otherwise, for researchers to turn these in-house publications into scientific papers are limited. This is a major loss to our knowledge base because countries around the world are increasingly recognizing the importance of transnational efforts to complement national efforts to detect and respond to public health threats quickly and effectively[19–23]. Exercises provide these countries with a vehicle to collaborate and test their ability to respond to these transnational threats. Exercises also provide these countries with an avenue to build relationships and trust among colleagues across sectors and across borders[24].

Methods

We developed pandemic influenza tabletop exercises that built on the “Day After” methodology developed by Millot, Molander and Wilson[25] and described elsewhere in greater detail[1]. Countries in three different geographic regions participated: Southeast Asia (Cambodia, China, Lao PDR, Myanmar, Thailand and Vietnam), the Middle East (Israel, Jordan and Palestine) and East Africa (Burundi, Kenya, Rwanda, Tanzania and Uganda). Countries that participated in the exercises were included because they were all part of sub-regional disease surveillance networks established in part through funding from the Rockefeller Foundation. Some countries not included in these networks were invited to observe the exercises. Countries ranged in their past experience with preparedness exercises with some countries having extensive past exercise experience (such as Israel, Vietnam, China and Thailand) and other countries having minimal past exercise experience (such as Cambodia, Lao PDR, Myanmar and Uganda). Exercises were developed and conducted by exercise planning teams that included external exercise development experts from the RAND Corporation as well as senior health leaders from each of the respective localities and/or countries represented in the exercise. There were three different levels of exercises: sub-national (e.g., one or more provincial areas), national (e.g., one country) and sub-regional (e.g., multiple countries from one geographic region).

All exercises were multi-sectorial in nature meaning that they involved representatives from more than one sector of government. Examples of sectors included were health, agriculture, defense and environment. Each exercise focused on three to six different broad topic areas such as surveillance and information sharing, disease control, and communications that were identified in previous exercises as important[2, 12]. Because Thailand had considerable previous experience with exercises, it designed and conducted its sub-national exercise with limited involvement from representatives at RAND. Exercise discussions focused on one topic area at a time each lasting from 30 to 90 minutes. Participants of exercises were selected by the exercise planning team and differed from exercise to exercise, but all exercises included representatives from the health sector of the locality and/or country represented. They also included senior leaders from at least one other non-health sector. In addition to participants, exercises also had “observers” who were invited to watch the exercise but did not directly engage in exercise discussions. All exercises were led by one or two exercise “facilitators” who directed the exercise discussion and probed participants for more information. In general, exercise facilitators can represent a range of disciplines from media professionals to health professionals. In these exercises the facilitators were all health officials or health researchers who were trained in the facilitation of exercises and who had extensive experience facilitating past exercises.

Exercises presented participants with a future scenario that involved an unfolding pandemic influenza crisis at different stages. They were required to respond to the scenario with the actions they would take if the scenario were actually occurring. Exercise facilitators were given discussion points and probes to keep the discussion focused and moving forward. Each section of the exercise ended with participants being asked to make concrete decisions for the topic area being discussed before moving on. The exercise concluded with a debriefing in which all participants evaluated their own response in light of what they learned during the exercise.

All exercise participants were asked to complete an evaluation form immediately after the exercise, before leaving the room. Typically, they spent about 15 minutes to respond to these questions. Six of the exercises were rated for their quality through five Likert scale questions: the overall quality of the exercise (1 = poor; 5 = excellent); the quality of the exercise discussions (1 = poor; 5 = excellent); the exercise identified important key gaps in preparedness (1 = strongly disagree; 5 = strongly agree); the exercise helped participants to better understand the roles and responsibilities of agencies and organizations responding to an influenza pandemic (1 = strongly disagree; 5 = strongly agree); and the exercise generated information that participants planned to use (1 = strongly disagree; 5 = strongly agree).

The remaining six exercises asked participants three different qualitative questions: what was the importance of the exercise; what are the most important actions that should be taken based on the exercise; and what suggestions do you have to help improve future exercises. In addition to participant evaluations, detailed After Action Reports (AARs) were developed for each exercise that summarized the exercise discussions and highlighted key aspects of each exercise. In January 2013, health leaders who were involved in the planning of the exercises from a subset of countries participated in brief semi-structured face-to-face interviews to discuss how their country followed up with the exercises and the current state of their exercise program. Health leaders included health officials working for the ministry of health in their respective countries. These health leaders were all directors of departments (such as communicable disease) within their ministry.

Results and discussion

We developed and conducted 12 exercises from August 2006 through December 2008: four sub-national exercises, five national exercises, and three sub-regional exercises (Table 1). Across all of these exercises there were a total of 558 participants and 137 observers from 14 countries. The average number of participants per exercise was 47 and the average number of observers was 11. Participants from the health sector were represented in every exercise. The most commonly represented sectors other than health were agriculture and defense. Four exercises were shorter than one full 8-hour day in length, three exercises were one full day in length, and five exercises were more than one full day in length. The average length of the exercises was 9.75 hours. All exercises covered three to six of the topic areas outlined in Table 2.

Table 1 Characteristics of exercises
Full size table
Table 2 Topic areas tested in exercises
Full size table

Table 3 highlights the participant evaluation from six exercises that used questionnaires with Likert Scale questions. Participants who completed these evaluation forms consistently rated the overall quality of the exercises as high (88-100% rating the exercise as good or excellent) with one exception (Middle East sub-regional exercise 59% rating the exercise as good or excellent). Participants also consistently rated the exercises highly for helping them to understand the roles and responsibilities of organizations and agencies responding to an influenza pandemic (91-94% rating the exercise as good or excellent in this area) with one exception (China sub-regional exercise 76% rated the exercise as good or excellent in this area). Participants differed on what they felt about the quality of the information shared in the exercises (67%-93% rated the information as good or excellent). Participants rated the exercises lowest on their ability to identify key gaps in performance (50%-73% agreeing or strongly agreeing the exercises identified important key gaps). There are a variety of factors that could explain this result. First, tabletop exercises are discussion based and do not directly test operational capabilities. This may limit the ability of tabletop exercises to concretely identify key operational gaps. Second, tabletop exercises are subjective and participants may disagree about what constitutes a key gap. Third, the limited time frame to conduct tabletop exercises and the limited number of topics that can be discussed in that time frame may inhibit the ability of these exercises to identify a significant number of key gaps. A fourth possibility is that cultural sensitivities in some of these countries may have limited participants’ comfort in identifying gaps in their government’s preparedness system. The exercises were most successful at helping participants gain knowledge that they planned to use to improve the preparedness of their organization (82%-100% agreeing or strongly agreeing that they would use what they learned from the exercise).

Table 3 Participant likert scale ratings of exercises
Full size table

Table 4 summarizes the qualitative feedback provided by participants in their evaluation forms. Three general themes came out of the participant comments on the most useful aspects of the exercises: the ability of exercises to raise awareness and understanding about public health threats, the ability of the exercises to assist in evaluating plans and identifying priorities for improvement and the ability of the exercises to build relationships and enhance preparedness and response capabilities across sectors and across countries in a geographic region. Participants also left the exercises with specific ideas about the most important follow-up actions that they should take in the near future. Specifically, participants identified better planning, improved planning coordination across sectors and countries and better training of health workers and response personnel. Finally participants provided feedback on the use of tabletop exercises for pandemic influenza preparedness. No participants stated that they felt the exercises involved too many sectors. In fact, many participants reported that they felt more sectors should be involved and that exercises should also involve more private sector partners and more partners from NGOs. Participants also felt that more could be done in the exercises to ground theoretical responses with more practical responses.

Table 4 Summary of participant qualitative feedback on exercises
Full size table

Some health leaders who were part of exercise planning teams participated in semi-structured face-to-face interviews in January 2013. Countries that reported having pre-existing exercise programs prior to participating in the exercises described here were much more likely to report conducting exercises at regular intervals over time compared to countries that did not report a pre-existing exercise program. Most countries reported modifying and using some or all of the exercise template materials that were developed for the exercises described here. However, one country that had no prior exercise experience organized and carried out numerous sub-national exercises on their own after participating in the national and sub-regional exercise. Health leaders in this country reported that participating in an exercise helped to motivate them to develop an exercise program and regularly assess different aspects of their public health preparedness. The largest barriers to continued exercising that were reported included lack of financial resources and limited support among leadership to develop and sustain an exercise program.

Conclusions

These experiences suggest that exercises can be a valuable, low-burden tool to improve emergency preparedness and response in countries around the world. They also demonstrate that countries can work together to develop and conduct successful exercises designed to improve regional preparedness to public health threats. Regular participation in exercises is associated with improved overall response to public health threats[26]. Countries that participated in sub-regional exercises together reported that these exercises improved their response to the 2009 H1N1 pandemic[24, 27]. But exercises are not perfect. Research has called into question the ability of exercises to adequately expose operational and logistical gaps[28]. This is consistent with our finding that exercise participants rated the exercises lowest for identifying key gaps. In addition, there is a lack of consensus on what makes exercises effective tools to assess public health preparedness and how the outputs of exercises such as AARs can be used to support and improve public health preparedness efforts[29]. Thus, the development of standardized evaluation methods for exercises may be an additional tool to help focus the actions to be taken as a result of the exercise and to improve future exercises. Despite these flaws, exercises show great promise as tools to build relationships, assess performance and improve collaborative planning for public health threats across multiple sectors and multiple countries over time.