1 Introduction

A resilient public health system can resist, absorb, adapt to, and recover efficiently and timely from a hazard while preserving and restoring essential services and functions through risk management (Kruk et al. 2015; Haldane et al. 2021; UNDRR 2023; Ryan, Kako, et al. 2023). This includes rapidly adjusting to situations that are dynamic, fluid, and beyond control of the system (Ryan, Kako, et al. 2023). Achieving this results in a return to stability without long-term impacts on community health, well-being, and development (Behrens et al. 2022). However, the COVID-19 (Coronavirus disease 2019) pandemic, as is often the case with many large-scale disasters, revealed the weaknesses in public health and other societal systems in many parts of the world. The difficulties encountered in managing the effects of such events have been exacerbated by longstanding social inequities, unreliable healthcare, and increased urbanization, population growth, and mobility (Rollston and Galea 2020; Assefa et al. 2022). Consistent with other disasters, the people who experienced the greatest impact were the elderly, low-income, self-employed, and people with noncommunicable diseases (NCDs) such as cardiovascular disease, diabetes, and cancer (Rollston and Galea 2020; Hassan et al. 2022; Ryan, Kako, et al. 2023).

During the pandemic, there were surges in demand for care of people with COVID-19, which made it difficult to maintain services (Sagan et al. 2022). In response, public health systems often employed two broad strategies—scaling up, which involved repurposing and redistributing capacity, and adapting or transforming service delivery by implementing alternative patient care pathways (Sagan et al. 2022). Despite these efforts, access to healthcare was impacted in various ways within and among countries. Many patients were hesitant to seek medical care out of concern for contracting the virus, even though public health measures were effective (Splinter et al. 2021; Hassan et al. 2022). Furthermore, resources were often redirected to manage COVID-19 in many countries, thus limiting access to other medical services and affecting people with NCDs and other health conditions and needs (Hogan et al. 2020; Roberton et al. 2020; Abbas 2021; Arsenault et al. 2022; Jain and Dupas 2022). Extensive quarantine measures, while effective for infectious disease control, affected quality of life, particularly for economically vulnerable populations (Ryan et al. 2020; Hassan et al. 2022). Similarly, lockdowns, which helped slow the spread of the virus while societal response capacities were being built, had diverse implications for delivery of essential services, supply chains, and mental health (Prommas et al. 2023).

Every segment of the population was affected differently during the COVID-19 pandemic. For example, school closures were implemented in many locations as a measure to slow the spread of COVID-19, but as a result, students missed out on significant education opportunities with many losing one to two years of competencies (Engzell et al. 2021). The most impacted were students living in disadvantaged households (Vlachos et al. 2021). There was also an estimated USD10 trillion loss in earnings during the pandemic (Global Preparedness Monitoring Board 2020). Many countries lacked the ability to sustain an entire nationwide response operation (Ryan et al. 2020). This experience serves as an opportunity to rejuvenate public health activities across sectors at local, state, national, and international levels to focus on leaving no one behind.

The Severe Acute Respiratory Syndrome (SARS) and Middle Eastern Respiratory Syndrome (MERS) epidemics helped many locations improve preparedness by informing institutional, legal, and physical infrastructure needs (Mohammadpour et al. 2021; Feitelson et al. 2022). This included enhancing disease surveillance, research and development systems, and improving the ability to rapidly identify, implement, and adjust precautionary measures (Algaissi et al. 2020). The locations that best weathered the pandemic applied many of these lessons to create an agile public health system, encourage people to seek early care, and balance community viability with protecting lives and livelihoods (Msemburi et al. 2022; Ryan, Kako, et al. 2023). These strategies highlight the need to consider internal capacities, managerial interventions, and external factors such as socioeconomic determinants before, during, and after an epidemic, pandemic, or disaster (Mohammadpour et al. 2021; Tambo et al. 2021).

To build capacity within and beyond public health systems, collaboration is required across sectors including industry, government, and not-for-profits such as faith-based organizations (Haldane et al. 2021). The World Health Organization (WHO) Health Emergency and Disaster Risk Management Framework (Health EDRM Framework) provides a frame for starting this trajectory by informing an all-hazards risk management approach to reducing the systemic health risks and consequences of all types of emergencies and disasters across the world (WHO 2019; Wright et al. 2020). To respond to the deficiencies of fragmented policies and programs that tended to focus only on preparedness and response to events, the Health EDRM Framework emphasizes that the entire health system and the whole-of-society must work together. This is needed to develop and implement joint actions that reduce hazards, exposures, and vulnerabilities, and build capacities for preparedness, readiness, response, and recovery.

Since all communities have their unique risk contexts, the Health EDRM Framework describes the importance of communities in leading and participating in the design, delivery, and prioritization of programs and services. There is a need to focus attention and resources on those who are disproportionately affected, such as the poor, women, children, people with disabilities, older persons, migrants, refugees and displaced persons, and people with chronic diseases (WHO 2019). The Health EDRM Framework draws on good practice from humanitarian action, epidemic preparedness and response, disaster management, and health systems strengthening. This guidance assists countries and communities in bringing coherence to the implementation of the Sendai Framework, the Sustainable Development Goals, the Paris Agreement on Climate Change, and other local, national, and international frameworks (WHO 2019).

Key to this is measuring public health system resilience and using this to inform the development of priority actions (Ryan, Kako, et al. 2023). The United Nations Office for Disaster Risk Reduction (UNDRR) Public Health System Resilience Scorecard (Scorecard) complements the Health EDRM Framework and serves as a vital tool to achieve this outcome. The Scorecard was developed with multisectoral experts and is available in multiple languages (UNDRR 2020a). Its creation was influenced by application of the Disaster Resilience Scorecard for Cities, which highlighted the need for a specific focus on public health (Ryan, Telford et al. 2023). The Scorecard is freely available online, includes contributions from some of the authors of this article (Ryan, Abrahams, and Bhatia), and was initially launched in July 2018 (Version 1.0), which was followed by an update in April 2020 (Version 2.0) (UNDRR 2020b).

To provide a path towards improving the resilience of public health systems, this study interpreted, ranked, and reached consensus on a set of prioritized strategies identified from workshops in urban settings across multiple countries where the Scorecard had been applied. Perspectives on the application of the Scorecard were also explored to guide future approaches in identifying strategies and actions for improving the resilience of public health systems. By applying this approach, this project has identified generalizable and translatable strategies to build the resilience of public health systems now and into the future.

2 Materials and Methods

The core methodology of this study hinged on a sequential modified Delphi process to develop a set of prioritized resilience strategies based on findings from workshops where the Scorecard had been applied (Hasson et al. 2000). This approach was selected to provide an opportunity to apply a consensus-based process to clarify, rank, and prioritize these strategies. The study was divided into five phases, as represented in Fig. 1. These phases were executed during a three-day workshop, and detailed information about this workshop is provided below. The approach began by exploring the 70 strategies from the 13 Scorecard workshops to create Survey 1. After participants completed Survey 1, its results informed discussion about the strategies to be included in Survey 2. Once Survey 2 was completed, the strategies were ranked and prioritized based on participant feedback. Finally, another workshop discussion was convened to establish consensus on the set of priority strategies for enhancing the resilience of public health systems.

Fig. 1
figure 1

Methodology for the Korean resilience strategies workshop, 14 to 16 March 2023 

Full size image

In the initial phase, our plan aimed to incorporate strategies formed during previous workshops where the Scorecard was applied as part of a project funded by the WHO Centre for Human Development (WHO 2023). This included eight workshops conducted in Australia (n = 1), Bangladesh (n = 2), Slovenia (n = 1), Turkey (n = 2), and the United States (n = 2). The scope for this phase was limited due to a combination of funding, project timeline, the COVID-19 pandemic, and professional networks. In an effort to maximize locations participating in this project, we discovered additional sites and opportunities. This allowed for integration of strategies from additional workshops held in Brazil (n = 3) and Chile (n = 1), which were supported by the UNDRR Regional Office for the Americas and Caribbean. Also, an additional workshop was held in Nepal (n = 1) while a facilitator from Australia was in-country. As a result, this study encompassed input from 13 previously conducted workshops, spanning eight countries and five continents with 284 participants. These meetings were conducted in locally acceptable languages from October 2021 to January 2023 (Fig. 2).

Fig. 2
figure 2

Map of previous workshop sites considered in the present study

Full size image

This modified Delphi process was selected because it allows participants to deeply explore a topic in different ways and develop generalizable strategies ready for translation into practice (Glaser and Strauss 1967; Ryan et al. 2018). Another benefit is the controlled feedback process and the variety of analysis techniques that can be used to interpret the data (Dalkey et al. 1969). These characteristics are also designed to offset shortcomings of conventional means of pooling opinions obtained from a group interaction (for example, influences of dominant individuals and group pressure for conformity) (Hsu and Sandford 2007). This article built on the analyses of Scorecard workshops conducted in Slovenia, Turkey, and the United States (Ryan, Kako, et al. 2023).

2.1 Workshop Procedure

The workshop was conducted from 14 to 16 March 2023 at the UNDRR Global Education and Training Institute (UNDRR-GETI) in Incheon, Republic of Korea. The workshop was conducted over three days. On the first day participants shared findings from their respective workshops and completed Survey 1. This included aggregated strategies from all previous workshops. This sharing of findings provided a foundation for the subsequent discussions. On day two there were discussions based on Survey 1 results to determine which strategies should be retained for Survey 2. Once these decisions were made, Survey 2 was completed to further refine the strategies. The third and final day centered on discussing the findings from Survey 2 and working towards a consensus on prioritized actions. This collaborative approach allowed for the identification of key strategies for strengthening the resilience of public health systems and communities.

It is worth noting that the recommended number of participants for a Delphi workshop is 10–15, but this workshop saw the participation of 19 individuals due to the extensive application of the Scorecard across different locations and countries (Hsu and Sandford 2007). To ensure the effectiveness of the workshop, it was led by two experienced facilitators, Ryan and Garner, who guided the discussions and ensured that valuable feedback was recorded throughout the workshop.

2.2 Participants

The participant selection process for this study adhered to purposeful and snowball sampling techniques. Purposive sampling was initially used to identify workshop participants through the UNDRR and the researchers’ networks, including those who had previously participated in the Scorecard workshops. Subsequently, participants were encouraged to suggest other individuals who might be interested in participating.

All workshop participants had previous experience in facilitating or participating in workshops where the Scorecard has been applied. However, for the two workshops conducted in Brazil (Ubá and Guidoval), the facilitators were unable to attend our workshop, but their findings were still included in the aggregated strategies. To maintain continuity and familiarity with the workshop findings in Brazil and Chile, a representative from UNDRR Americas and Caribbean was present during the workshop.

2.3 Phase 1—Survey Development

The development of Survey 1 involved a comprehensive process. The survey included questions about demographics, perspectives on the Scorecard method, general comments, and aggregated strategies from workshops where the Scorecard had been previously applied. Input for crafting the survey questions was obtained from the point of contact for each workshop. Additionally, the survey tool took into account an analysis of the Scorecard workshops in Slovenia, Turkey, and the United States (Ryan, Kako, et al. 2023). After rigorous review, removal of duplicates, and refinement, a total of 70 strategies were included in Survey 1. The strategies were listed in alphabetical order to ensure each could be considered on its own merit.

2.4 Phase 2—Survey 1

All participants in the workshop actively engaged in completing Survey 1. This survey was anonymous and served to clarify, rank, and prioritize strategies for inclusion in the subsequent survey (Survey 2). The findings from Survey 1 played an important role in informing the discussions during Phase 3 of the study, where participants deliberated on the results and decided what should be included in Survey 2. The data from Survey 1 underwent a descriptive analysis, and the rankings were analyzed by Fontenot, Ryan, Noel, and Garner immediately following the completion of the survey. Demographic data were collected along with perspectives on application of the Scorecard. This analysis along with grouping the strategies by The Ten Essentials for Making Cities Resilient (Ten Essentials) (UNDRR 2020a) was conducted post-workshop.

2.5 Phase 3—Workshop Discussion about Survey 1 Results

The results obtained from Survey 1 formed the basis for the discussions during Phase 3 of the workshop. Participants deliberated on the strategies identified for clarification or removal, aiming to refine the selection for further investigation in Survey 2. Initially, the recommended cut-off for discussion was set at 10% of the participants selecting remove, this is where a minority can reverse the opinions of a majority (Xie et al. 2011; Ryan et al. 2018). However, due to the large number of participants and time constraints, this threshold was modified to 26% (Ryan et al. 2018). Removal does not mean items are not important; rather, it indicates there are other actions needed first. Strategies that scored over 21% were also considered for discussion. The final decision on removing, clarifying, and adding a strategy was determined through a collaborative consensus-building process (Hsu and Sandford 2007). These finalized strategies were then incorporated into Survey 2.

2.6 Phase 4—Survey 2

Survey 2 was designed for participants to rank the significance of each strategy in terms of strengthening resilience. Survey 2 was anonymous and included a repetition of the demographics-related questions from Survey 1. This approach ensured that demographic information was consistently collected, allowing for comprehensive data analysis. Similar to Survey 1, participants provided their perspective on application of the Scorecard and categorized the rankings for each strategy listed in Survey 2 as “Very Low,” “Low,” “Medium,” “High,” or ‘Very High” in terms of their importance on strengthening resilience. The survey results were used to inform the final discussion of the workshop, which focused on reaching consensus about the priority strategies. Here again, the survey results were analyzed through a descriptive analysis conducted by Fontenot, Ryan, Noel, and Garner immediately following the survey. Demographic data were subsequently analyzed post-workshop along with grouping the strategies by the Ten Essentials (UNDRR 2020a).

2.7 Phase 5—Consensus on Priority Strategies

In line with the consensus approach, strategies were identified as priorities for discussion if there was a general agreement of a substantial majority (Linde et al. 2005). Initially, this agreement was set at 75% or more when considering rankings of “High” or “Very High.” However, after careful consideration of the survey results and extensive discussions with the workshop participants, the agreed-upon cut-off was adjusted to 73% (Linde et al. 2005; Ryan et al. 2018). Strategies that met or exceeded this threshold were subject to thorough discussion, and consensus was achieved. This increased the strategies considered and discussed from 19 to 22. These prioritized strategies aligned with strengthening the resilience of public health systems and reflected the specific context in which the Scorecard was applied. Subsequently, the results were reviewed, interpreted, and evaluated post-workshop by Fontenot, Ryan, Kako, Fink, and Garner. All actively participated in the workshop.

2.8 Ethics

This study was determined by the Baylor University Institutional Review Board (IRB Reference #1792629) to be exempt from review by the institutional review board. Participants were invited to attend the workshop and provided oral consent to participate.

3 Results

This section describes the workshop demographics, perspectives on the Scorecard Approach, survey results, and the prioritized strategies.

3.1 Demographics

There were 19 participants in the Incheon workshop (Table 1). This group consisted of 11 females and eight males. Participants were based in three WHO regions: the Americas (n = 12), Western Pacific (n = 4), and Europe (n = 3). The participants represented a variety of disciplines/roles. This included academia, communications, disaster risk management, environmental health, epidemiologist, education, public health, doctor, international development, linguist, nurse, president, public officer, research assistant, and vice-chancellor. Among these participants, 13 (76.5%) had directly participated in disaster response. Over half of the participants (n = 10, 52.6%) had over 21 years of work experience and 15.8% (n = 3) had 6–10 years of experience. The number of participants with 1–2 years and 16–20 years of experience was the same at 10.5% (n = 2), while 5.3% (n = 1) of participants had either 3–5 years or 11–15 years of experience.

Table 1 Participant demographics at the Incheon, Korea resilience strategies workshop
Full size table

3.2 Perspective on the Scorecard Approach

Participants were asked three questions about the Scorecard use and effectiveness in both surveys using a Likert scale of 0 (lowest) to 5 (highest). The first question focused on the Scorecard’s effectiveness in identifying gaps in the resilience of the public health system with 90.7% (81.3% Survey 1; 100% Survey 2) of the participants rating this as either a 4 (25.0% Survey 1; 35.3% Survey 2) or 5 (56.3% Survey 1; 64.7% Survey 2). Question two related to ease of use, with 68.5% (64.7% Survey 1; 72.2% Survey 2) giving scores of 4 (41.2% Survey 1; 44.4% Survey 2) or 5 (23.5% Survey 1; 27.7% Survey 2). The third question related to effectiveness of the Scorecard in unifying stakeholders across various sectors with 84.8% of participants (81.3% Survey 1; 88.3% Survey 2) giving this a 4 (25.0% Survey 1; 41.2% Survey 2) or 5 (56.3% Survey 1; 47.1% Survey 2).

For Survey 1, one participant put N/A for questions one and three, and data were missing from two surveys. For Survey 2, two respondents indicated N/A for questions one and three, one person indicated 3.5 for question one and 4.5 for question two, these scores were counted as 4 and 5, respectively, and a participant indicated N/A for question two.

The surveys included an opportunity for participants to provide general comments or suggestions. Examples included: “The needs of the ‘left out’ parts of a community, such as the economically disadvantaged, poorly housed, minority groups […] need to be part of scorecard […] these people groups often make up the most heavily affected by disaster”; “Some of the recommendations should be a part of the ICS and ESF that is followed up by PHEP and emergency management teams already”; “Develop communication materials to educate general people”; “these strategies should be extended to the national level”; “[…] other terms should be used included marginalized, underserved, vulnerable and disenfranchised”; “the tool is very useful for local planning. It would include elements of adaptation to climate change with greater emphasis on public health policy decision-making”; and “showing and exchanging information is excellent way of learning.”

3.3 Survey 1

In Survey 1, participants evaluated each of the 70 strategies and decided whether they should be kept, removed, or clarified. The scores are presented in Table 2, and have been grouped by “The Ten Essentials for Making Cities Resilient” (UNDRR 2020a) identified in the Scorecard along with the indicator/question number. Based on the responses, 39 strategies were suggested for removal by at least 26% of the participants. Consistent with the feedback, these strategies were excluded from Survey 2 without further discussion. Seven strategies had removal indications from 21 to 25% of participants. These were put forward for group discussion. A consensus threshold of 10% was established to finalize decisions on these strategies. Voting occurred via a show of hands, and the outcomes were unanimous. Consequently, 24 out of the 70 strategies (34.3%) were retained for Survey 2 without debate, and after group discussion, seven more were added, resulting in 31 strategies for Survey 2.

Table 2 Survey 1 results of the Incheon, Korea resilience strategies workshop
Full size table

3.4 Survey 2

Participants completed Survey 2 to prioritize the 31 strategies based on their importance in enhancing resilience. Of these, 11 had rankings for “High” or “Very High” that, when combined, were below 75%. However, during the analysis, it was determined that strategies with a combined “High” and “Very High” ranking of more than 73% should still be considered. As a result of this criterion, eight strategies were removed without further discussion, while three were discussed. After discussing, consensus was reached on a set of 23 strategies, as outlined in Table 3.

Table 3 Survey 2 results of the Incheon, Korea resilience strategies workshop
Full size table

Through this process all Ten Essentials were represented, except 4 and 6. The essentials with the most priority actions were 7 and 9, which indicates that these areas require the most development in terms of public health system resilience. Meanwhile, the exclusion of essentials 4 and 6 in the priority strategies demonstrates participants believe that these were the most developed in the context of resilience.

The top 10 strategies with the highest priority are listed below with the strategy reference number listed in parentheses with Table 2 for Survey 1 and Table 3 for Survey 2. Over 89% of the participants selected these as “High” or “Very High” in terms of strengthening public health system resilience.

  1. (1)

    Municipal emergency plans should consider epidemic outbreaks, pandemics, famine, water shortages, massive accidents such as plane crashes, car accidents on highways, or other disasters (reference 6 in Tables 2 and 3 in Table 3).

  2. (2)

    Planning for disaster response and recovery actions relating to public health should integrate multidisciplinary teams and include all municipal departments (reference 5 in Table 2 and 4 in Table 3).

  3. (3)

    Establish a process for continuing services for at risk populations during and after an emergency (reference 58 in Table 2 and 23 in Table 3).

  4. (4)

    Assess what funding is available to maintain public health services during and after a disaster (reference 12 in Table 2 and 6 in Table 3).

  5. (5)

    Linking high-risk populations living at home with preexisting conditions with electronic databases used for sharing information and notifications (reference 55 in Table 2 and 24 in Table 3).

  6. (6)

    Assess to what extent hospitals and emergency care centers can manage a sudden influx of patients (reference 49 in Table 2 and 20 in Table 3).

  7. (7)

    Establish local public health plans to mitigate the long-term effects of disease outbreaks, pandemics, and disasters. The plans should be integrated with national guidelines and scientific recommendations (reference 68 in Table 2 and 31 in Table 3).

  8. (8)

    Mapping the entire public health network, including pharmacies and other services beyond clinical treatment and care (reference 4 in Table 2 and 5 in Table 3).

  9. (9)

    Establish, maintain, strengthen, and restore multisector mechanisms to support local public health services in an emergency situation (reference 57 in Table 2 and 25 in Table 3).

  10. (10)

    Integrate local transport into the disaster system to supply personal protective equipment and other nonperishable items to remote hospitals, medical centers, and other allied health organizations (reference 61 in Table 2 and 26 in Table 3).

4 Limitations

Caution should be exercised when applying these findings. The prioritization of local strategies must be conducted locally through the leadership and participation of communities and local stakeholders. They will differ across communities and countries due to variations in the local risk profiles, health system architectures, capabilities, political contexts, and many other factors. Additionally, a modified Delphi process is subjective, but to mitigate bias there were two surveys and two rounds of consensus-based discussion. Therefore, all the strategies explored, ranked, and prioritized in this study provide a set of options for decision makers, practitioners, and funders.

A limitation was self-selection bias arising from the use of purposive and snowball sampling methods. For example, there were gaps in the spatial dispersal of participants. However, these sampling techniques were specifically chosen because they allow for targeted selection of individuals who possess deep expertise or firsthand experience in public health systems, disaster management, and resilience. By focusing on these informed participants, the research aimed to capture nuanced insights and complex understandings often missed in broader sampling strategies. Furthermore, although there is an inherent risk of bias, the rich data and unique perspectives derived from such samples often provide invaluable context and depth.

While the workshop set a threshold of 26% representing the participants who rated a strategy as “High” or “Very High” for examining the results from Survey 1, the widely accepted cut-off is 10%. This benchmark indicates a level at which a minority can sway the views of the majority. This cut-off was adjusted due to the number of previous workshops, participants, and time limitations. To address this constraint, participants were given the chance to discuss strategies openly. Additionally, there were opportunities to confirm and validate these strategies during workshop discussions.

The study direction was shaped by the contributions and expertise of the authors. To mitigate potential biases arising from individual perspectives, the authors represented a diverse, multinational, and interdisciplinary team. Specifically, the team of authors comprised professionals from academia, disaster risk reduction, emergency management, environmental health science, health promotion, information technology, medicine, nursing, and public health.

Strategies presented and discussed were identified through the past workshops conducted during the COVID-19 pandemic. This limitation was reduced due to the unique nature of the situation. The COVID-19 pandemic provided an opportunity to understand the impact of an ongoing event along with other hazards that may have emerged.

5 Discussion

Using the Scorecard in conjunction with a modified Delphi process is a versatile method for identifying, ranking, and prioritizing resilience strategies across urban settings in multiple countries (Ryan, Kako, et al. 2023; Ryan, Telford, et al. 2023). This method provides an opportunity to holistically understand the resilience of the public health system by exploring its complexity, subsystem interactions, and the ability to identify strengths and weaknesses affecting multiple factors (Williams and Sands 2020; Dianat et al. 2021). The interactive and participatory nature of the workshops also provided the opportunity to leverage participant knowledge to complete the Scorecard and ultimately inform future public health system preparedness strategies and processes (Ørngreen and Levinsen 2017; Sanchez-Betancourt and Vivier 2019; Shamsuddin et al. 2021; Ryan, Kako, et al. 2023).

Adaptive capacity is vital to managing risks and a wide range of potential events, especially in low-resource settings (Barasa et al. 2018; Forsgren et al. 2022). This would require better collaboration between sectors using an integrated approach and organizational learning (Forsgren et al. 2022). The Scorecard provides a mechanism for this to occur by allowing interested parties to explore key gaps, also setting the scene for public health (Assefa et al. 2021). This is where understanding the structure and functions of systems can support prediction and understanding of risks within subpopulations to identify the number of at-risk individuals in order to inform preventive and protective strategies for the risks associated with a disease outbreak, pandemic, or disaster (Dolley 2018; Assefa et al. 2021). One example of how this could occur is wastewater surveillance of diseases and wastewater-based epidemiology (Langan et al. 2022). Such data would require a multidisciplinary team working across professional boundaries to provide lead indicators of infectious diseases to public health leaders who could adapt and adjust strategies in advance of emerging outbreaks (Langan et al. 2022; Ryan et al. 2022).

The Scorecard has the potential to be applied using in-person or virtual workshops. However, we found that the Scorecard was best applied using in-person workshops. When used in virtual settings, the effectiveness of the Scorecard in encouraging discussion and feedback was compromised. This may be why 68.5% of the participants gave a score of 4 or 5 for ease of use. The Waco and Dallas workshops tested an App version of the Scorecard. It worked well as a test but not all participants could download and/or use the App. This technology could not be tested outside the United States. Consistent with another study that tested a similar App, the participants who could use this technology found it very beneficial, which demonstrates the potential for the approach (Ryan, Telford, et al. 2023). Based on this observation, an easy to use and accessible App will facilitate virtual application of the Scorecard.

The benefits of bringing the results from multiple locations together using a modified Delphi method were demonstrated during the joint workshop. Strategies 15, 34, 57, and 63 in Table 2 were identified as needing discussion from Survey 1, and without this process would not have been included in Survey 2. Consensus was reached after Survey 2 that these strategies, which became numbers 7, 11, 25, and 29 in Table 3, were priorities. After ranking these strategies using scoring from Survey 2, number 57 (Table 2) and later 25 (Table 3) came in the top 10 at number nine. This approach would support progress towards a model for more effective decision making and resource allocation based on the assessed priority needs of health systems and communities, building stronger population-based models for disaster risk management systems (Burkle et al. 2021a; Burkle et al. 2021b).

To build on this work, a validated process is needed to guide the application of the Scorecard through to the development of multiyear action plans. This would incorporate more components of the Health EDRM Framework to help ensure the inclusion of data sharing, open governance, cross-organizational, and inter-platform collaboration (Chan et al. 2022). Such plans would be designed for decision makers to consider when prioritizing policies and funding for disaster risk reduction. For example, the process could go beyond the top 10 priorities generated at our workshop and include the following steps:

  1. (1)

    Identification of Scorecard leads and workshop sites.

  2. (2)

    Application of the Scorecard and prioritization of actions during the workshop. This is scalable and could be at local or provincial/state levels.

  3. (3)

    Workshop report with priority actions presented to decision makers and/or public health leaders. The workshop participants would be provided with the opportunity to review prior to finalization.

  4. (4)

    If there were multiple applications of the Scorecard, conduct a joint workshop using a modified Delphi process to interpret, rank, and prioritize strategies identified where the Scorecard was applied.

  5. (5)

    Integration of the findings from steps 2 and 4 to develop multiyear action plans for the priorities identified. This would be tailored to local, provincial/state, and national levels.

This process would provide a pipeline for ongoing improvement of public health systems. For example, it would be aligned with the Ten Essentials, focus on local level transdisciplinary engagement, and help establish, measure, and calibrate baseline resilience, priorities, and actions overtime. Another benefit is that the Scorecard is scalable and translatable to various scenarios and settings due to the level of granularity in identifying strengths and prioritizing areas for improvement (Jones et al. 2021). Application of this approach would help public health leaders, emergency managers, and others involved in disaster risk management consider many aspects of resilience. For example, the vulnerabilities and capabilities of services, organizations, logistics, and ecosystems could be explored individually and collectively and applied to disease outbreaks, pandemics, and disasters. Routine application of the Scorecard would also help sustain gains in resilience across sectors at local, provincial/state, and national levels during a climate of rapidly shifting social, environmental, and fiscal challenges.

6 Conclusion

The Scorecard provides a versatile method for identifying, ranking, and prioritizing a set of strategies for strengthening the resilience of public health systems. This study demonstrated how this tool can be scaled and translated into various scenarios and settings. By applying this process, a range of options were identified. These included mapping the interconnectedness of health facilities and assets, and developing plans to scale and deploy nonhospital staff to complement treatment and care, as well as to expand surge capacity, improve the linking of public and private health systems to mitigate surge risks, and include multidisciplinary teams in public health activities. The integration of local transport and multisector systems is also important for prevention, preparedness, response, and recovery. Routine application of the Scorecard could help establish, measure, and calibrate resilience needs, priorities, and achievements overtime, which is increasingly important due to rapidly shifting social, environmental, and fiscal challenges. Using the method applied, a range of locally led resilience strategies can be identified and developed to prioritize current and future policies, actions, and funding to address disaster risks using a whole-of-society and all-hazards approach.