Skip to main content

A Likert Scale-Based Model for Benchmarking Operational Capacity, Organizational Resilience, and Disaster Risk Reduction

Abstract

Likert scales are a common methodological tool for data collection used in quantitative or mixed-method approaches in multiple domains. They are often employed in surveys or questionnaires, for benchmarking answers in the fields of disaster risk reduction, business continuity management, and organizational resilience. However, both scholars and practitioners may lack a simple scale of reference to assure consistency across disciplinary fields. This article introduces a simple-to-use rating tool that can be used for benchmarking responses in questionnaires, for example, for assessing disaster risk reduction, gaps in operational capacity, and organizational resilience. We aim, in particular, to support applications in contexts in which the target groups, due to cultural, social, or political reasons, may be unsuitable for in-depth analyses that use, for example, scales from 1 to 7 or from 1 to 10. This methodology is derived from the needs emerged in our recent fieldwork on interdisciplinary projects and from dialogue with the stakeholders involved. The output is a replicable scale from 0 to 3 presented in a table that includes category labels with qualitative attributes and descriptive equivalents to be used in the formulation of model answers. These include examples of levels of resilience, capacity, and gaps. They are connected to other tools that could be used for in-depth analysis. The advantage of our Likert scale-based response model is that it can be applied in a wide variety of disciplines, from social science to engineering.

Introduction

Organizations and societies are increasingly challenged by complex and systemic disaster risk, in which leaders must undertake strategic decisions in conditions of high uncertainty. In order to increase the flexibility of response and preparedness to concurrent, interacting, interconnected, and cascading events, cultural adjustments are required in both research and practice (Pescaroli and Alexander 2018). Authors such as Linkov et al. (2013) and Helbing (2013) have highlighted the need for new approaches that could help the assessment process to evolve, in order to be more suitable for applications in multiple domains and disciplines. Since then, the state-of-the-art has evolved significantly in terms of both theory and applications. A critical step was the increased prominence given to the understanding of risk and complexity as described in the Sendai Framework for Disaster Risk Reduction 2015–2030 (UNISDR 2015). This international agreement promoted further developments at the interface between academia, research, and practice. For example, the United Nations Office for Disaster Risk Reduction (UNDRR) released a series of practical guides to support the implementation of consistent measurements and datasets, including, for example, the “Words into Action” guidelines on National Risk Assessment (UNISDR 2017), and the Resilience Scorecard for Cities (UNISDR 2017). Similarly, new standards on continuity management and organizational resilience, such as the ones by the British Standards Institution, International Organization for Standardization, and National Fire Protection Association included new elements such as those that assess interdependencies and cascading effects (BSI 65000:2014; NFPA1600:1900; ISO 22316:2017; ISO 22301:2019). This has improved the principles used in benchmarking.

This is a fast-evolving field because tools such as resilience-measurement scales are the most basic source of interaction between academia, practitioners, and policymakers (Cutter and Derakhshan 2019). The academic literature proposes a range of methodologies that are available for measuring disaster risk, resilience, vulnerability, or climate change impact practices, including choices of variables and measurements, depending on the field of reference (Frazier et al. 2013; Linkov et al. 2013; Birkmann et al. 2014; Twigg 2015; Beccari 2016; Kelman et al. 2016; Gentile et al. 2019). However, the need for better standardization of indicators and reporting scales faces limitations, in particular in terms of what should be measured, how often, and by whom (Cutter and Derakhshan 2019). For example, stakeholders working on interconnected risks have suggested the need for cross-disciplinary improvements in defining operational and decision-making thresholds (Pescaroli 2018). The billion gigabits available online via channels such as social media have raised further challenges to the achievement of reliable quality standards (Kostkova et al. 2003; Alexander 2015). Moreover, end-user oriented early warning systems, multi-criteria decision support systems, and the scenario-building methods are some of the applied research fields in which updated benchmarking can make a difference and can help manage complexity (Pescaroli and Alexander 2018; Cremen and Galasso 2020).

Despite the progress that has been made, the current body of literature seems to lack a simple-to-use model that could provide benchmarking between different forms of assessment. In other words, the state-of-the-art is extensive, but it is missing a simple alternative framework of reference for cross-disciplinary assessments in those contexts in which existing scales and standards may be unsuitable, challenging, or impossible to be applied. This article started with explicitly stating the real-world need that emerged in our research on integrating organizational resilience and earthquake early warnings in Latin America. Further evidence and justification of need came from the experience of collaborating with stakeholders in Europe and Japan on operational resilience to cascading and interconnected risk. In the following sections we briefly review the methodological background and propose a rationale for a new Likert scale based on a response model for the assessment of operational gaps, organizational resilience, and disaster reduction capacity, as reported in Sect. 3. Our principal goal is to propose a simple-to-use, replicable, and adaptable tool that could be used as a benchmark across studies for anchoring information and assessing it in the field of data collection against benchmarks in a context distinguished by low levels of knowledge, training, or awareness.

General Considerations on Rating Scales, Anchoring, and Benchmarking

Research and practice need reliable data. The literature has extensively discussed the role of quantitative and qualitative methodologies for collecting and analyzing evidence, with particular attention to the replicability of the data collection process (Bryman 2016). Similarly, planning and designing a study is an essential aspect of the development of mixed-method approaches (Creswell 2014). The common ground across the disciplines is the need to anchor questionnaires and surveys to realistic points of reference in relation to the context in which the data are collected (see, for example, Alexander 2015; Ahmed et al. 2016; Hernández-Moreno and Alcántara-Ayala 2017; Pescaroli 2018; Gentile et al. 2019; Yore and Faure-Walker 2019). In the field of business continuity, a questionnaire is often the most important tool for gathering information in order to conduct a business impact analysis and assess the level of organizational resilience (BSI 2014; ISO 2019). Similarly, questionnaires are used to identify possible “performance gaps” or “capacity gaps” that could be reflected in the existence of perceived inadequate or undesirable organizational or operational states (Channon and Sammut-Bonnici 2014; UNISDR 2017).

There are two common steps to consider in this process, independent of the field or discipline. First, the questions have to be carefully thought through to assure their consistency and clarity, and to be safeguarded, as much as possible, against bias (Creswell 2014; Bryman 2016). Secondly, each answer has to be anchored to a rating scale, which can be defined as “a closed-end question whose answer alternatives are graduated or organized to measure a continuous construct, such as an attitude, opinion, intention, perception, or preference” (Peterson 2013). At this point, scholars, researchers, and practitioners have to consider a scale of reference, which may be straightforwardly to use a Likert scale (Croasmun and Ostrom 2011; Bryman 2016). The next step would involve looking at benchmarking examples such as those given by Brown (2010) and Vagias (2006). Something similar may happen in fieldwork, for example, when it is necessary to assess emergency response capacity, develop a business impact analysis, or benchmark organizational resilience at large (Alexander 2000, 2015; BSI 2014; ISO 2017).

However, in some cases, contextual limitations would suggest the need to shift the analysis to an approach that differs from the most commonly used scales. Indeed, in some fieldwork, Likert scales from 1 to 5 or 1 to 7 are simply not fit for the local cultural or social environment, or they cannot be reported in labels that use ordinary language or common idioms. Similarly, it is important to consider the qualitative and explanatory nature of the options available in the questionnaires that have to be considered complementary to the numerical values provided in the scales: different interpretation of the numbers can generate some confusion among respondents if not supported by a qualitative rationale, as for example was identified by the National electronic Library for Health project assessing the “quality” of evidence base (Kostkova et al. 2003; Wiseman et al. 2008). For instance, the process of collecting data about capacity, impacts, or preparedness in operational contexts (such as local councils) may be hindered by lack of resources, or failure to embed the process. Thus, anything more complex than what is strictly necessary might not be welcome. In such cases, the solution could be to implement scales from 1 to 3, or use binary answers such as “yes or no.” However, this approach risks oversimplification, which might compromise the outcome of the project (Croasmun and Ostrom 2011; Bryman 2016). Our experience in the field and our dialogue with stakeholders suggest the need to move to a “hybrid” approach that prioritizes clarity, replicability, and consistency across different sectors. The point of the next section is to introduce a 0–3 scale, which could be more easily accessible to respondents, while it reduces the possibility of bias in responses and assures consistency across different measurements and research domains of disaster risk reduction, operational capacities measurement, and organizational resilience.

A New Scale-Based Assessment Model

A novel Likert scale-based assessment model for measuring organizational resilience and gaps in operational and disaster risk reduction capacities is reported in Table 1. It is organized according to the following structure.

Table 1 A Likert scale-based response model for benchmarking gaps in operational capacity, organizational resilience, and disaster risk reduction capacity

Value

The numerical value chosen is from 0 to 3, with the additional inclusion of the category “don’t know” as a possible option. The value of the primary assessment criteria is reported in both the disaster resilience scorecard (UNISDR 2017), and some examples of Likert scales offered by Brown (2010) and Vagias (2006). While developing this value scale, we recognized that answers may be affected by high uncertainty and low knowledge, and responders may have the tendency to choose the middle value in the ranking. As suggested in the review by Croasmun and Ostrom (2011), the risk of a biased response could be mediated by providing a neutral response option, allowing respondents not to take an opinion if they do not have one. This risk was mediated in our scale by including the category “don’t know” as an additional choice. Finally, the numerical value is supported by a visual association with the standard “traffic light color scheme” that is also used in the dissemination of warnings in order to harmonize the meaning of the numbers for people in different backgrounds and disciplines (Kostkova et al. 2003; Wiseman et al. 2008).

Category Labels

This section provides some generic examples of descriptive attributes that could be used to develop qualitative answers and ratings that are complementary to the numerical values of the scale. This has been intended as a partial and synthetic list that is derived from the descriptive categories reported by Brown (2010), Vagias (2006), and from a review we carried out of the attributes in the categories used by UNISDR (2017) and BSI (2014). In other words, the proposed scale provides enough evidence to be taken as a reference, independent of the question that has been formulated. It does not pretend to be comprehensive.

Gap Outcome

The gap outcome in Table 1 is intended as a support for the consistency of the outcomes and results of gap analyses, that are defined in this short article as the processes of assessing and comparing one organization’s objectives and their expected outcomes to understand possible differences in the performance that has been delivered (Channon and Sammut-Bonnici 2014). This is presented in terms of “inadequate/adequate” or “undesirable/desirable” operational or organizational states (Watkins et al. 2012). First, the states are identified according to the numerical values obtained on the scale. Second, they are anchored to qualitative examples of the Likert scale such as those given by Brown (2010) and Vagias (2006). They can also be anchored to percentages reported in the resilience scorecards by UNISDR (2017). In this case, the numerical thresholds must be considered as generic reference points, and they need to be grounded in a specific context, which may either confirm or challenge their validity.

Capacity Levels

This label provides a qualitative description of the capacity levels that have been derived from the maturity model integration (CMMI) of Chrissis et al. (2003). This is integrated into the disaster resilience scorecards approach (UNISDR 2017). Capacity is intended as “the combination of all the strengths, attributes and resources available within an organization, community or society, to manage and reduce disaster risks and strengthen resilience” (UNDRR terminology, updated 2017).Footnote 1

Resilience Levels

The resilience levels have been simplified and adapted from the corresponding maturity levels reported in BS65000:2014 (BSI 2014), shifting from a 0–5 scale to a simplified 0–3 scale. In the present work, the term “resilience” is defined as “the ability of an organization to absorb and adapt in changing environment to enable it to deliver its objectives and to survive and prosper” (ISO 2017, p. 4).

Conclusion

Our scale provides a replicable, direct model for benchmarking answers, built on the real needs of stakeholders and the feedback obtained from the field. However, like other tools and methodologies that represent the current state of the art, it has some characteristics and limitations that need to be taken into account if it is to be applied effectively. It should be noted that the scale has been developed to provide a flexible assessment for contexts in which there are constraints that prevent one from conducting a wider and more complex analysis. In other words, the rationale of our approach is to maximize the reliability of answers by sacrificing, to some extent, the level of detail when anything more sophisticated is not achievable. This implies that the achievable level of accuracy of the results is lower. In this, there are two complementary considerations. In the first place, ideally, our scale should be used as a basis for developing a more complex form of analysis. In the practice of assessing resilience, this may be derived from guidelines such as those mentioned previously (BSI 2014; UNISDR 2017; NFPA 2019). In multidisciplinary research, optimal results may require broader scales, such as one in the range 1 to 7 or 1 to 10 (Croasmun and Ostrom 2011). Second, a possible alternative to compensate for the reduction in accuracy of the scale when compared to alternatives may be to plan carefully the integration of questionnaires with semistructured interviews or focus groups. This could profitably follow guidelines for mixed-method research such as those provided by Bryman (2016).

In conclusion, our approach does not pretend to be exhaustive, but it provides a practical and flexible reference method for benchmarking that can be adapted to the context in which it is used. For example, in some cases, category labels may require both language translation and changes to reflect sensitivity to cultural variations in the use of terminology. Instead of a limitation, these elements could be seen as an additional strength that could be used to promote testing and further evolution of our model, according to new experiences derived from research and practice. The result could be extended to different disciplines.

Notes

  1. https://www.preventionweb.net/terminology/view/7831.

References

  • Ahmed, B., I. Kelman, H.K. Fehr, and M. Saha. 2016. Community resilience to cyclone disasters in coastal Bangladesh. Sustainability 8(8): Article 805.

  • Alexander, D.E. 2000. Confronting catastrophe: New perspectives on natural disasters. New York, NY: Oxford University Press.

    Google Scholar 

  • Alexander, D.E. 2015. Evaluation of civil protection programmes, with a case study from Mexico. Disaster Prevention and Management 24(2): 263–283.

    Article  Google Scholar 

  • Beccari, B. 2016. A comparative analysis of disaster risk, vulnerability and resilience composite indicators. PLoS Currents. https://doi.org/10.1371/currents.dis.453df025e34b682e9737f95070f9b970.

  • Birkmann, J., S. Kienberger, and D. Alexander (eds.). 2014. Assessment of vulnerability to natural hazards: A European perspective. San Diego, CA: Elsevier.

    Google Scholar 

  • Brown, S. 2010. Likert scale examples for surveys. Ames, IA: Iowa State University. https://www.extension.iastate.edu. Accessed 1 Apr 2020.

  • Bryman, A. 2016. Social research methods. Oxford: Oxford University Press.

    Google Scholar 

  • BSI (British Standards Institutions). 2014. Guidance on organizational resilience BS 6500:2014. London: BSI Standards Limited.

    Google Scholar 

  • Channon, D., and T. Sammut-Bonnici. 2014. Gap analysis. In Wiley Encyclopedia of management, 12: 1–3, ed. C.L. Cooper. Hoboken, NJ: John Wiley & Sons.

    Google Scholar 

  • Chrissis, M.B., M. Konrad, and S. Shrum. 2003. CMMI guidelines for process integration and product improvement. Boston: Addison-Wesley Longman Publishing Co., Inc.

    Google Scholar 

  • Cremen, G., and C. Galasso. 2020. Earthquake early warning: Recent advances and perspectives. Earth-Science Reviews. https://doi.org/10.1016/j.earscirev.2020.10318.

    Article  Google Scholar 

  • Creswell, J.W. 2014. A concise introduction to mixed methods research. Thousand Oaks, CA: SAGE Publications.

    Google Scholar 

  • Croasmun, J.T., and L. Ostrom. 2011. Using Likert-type scales in the social sciences. Journal of Adult Education 40(1): 19–22.

    Google Scholar 

  • Cutter, S.L., and S. Derakhshan. 2019. Implementing disaster policy: Exploring scale and measurement schemes for disaster resilience. Journal of Homeland Security and Emergency Management 16(3). https://doi.org/10.1515/jhsem-2018-0029.

  • Frazier, T.G., C.M. Thompson, R.J. Dezzani, and D. Butsick. 2013. Spatial and temporal quantification of resilience at the community scale. Applied Geography 42: 95–107.

    Article  Google Scholar 

  • Gentile, R., C. Galasso, Y. Idris, I. Rusydy, and E. Meilianda. 2019. From rapid visual survey to multi-hazard risk prioritisation and numerical fragility of school buildings. Natural Hazards and Earth System Sciences 19(7): 1365–1386.

    Article  Google Scholar 

  • Hernández-Moreno, G., and I. Alcántara-Ayala. 2017. Landslide risk perception in Mexico: A research gate into public awareness and knowledge. Landslides 14(1): 351–371.

    Article  Google Scholar 

  • Helbing, D. 2013. Globally networked risks and how to respond. Nature 497(7447): 51–59.

    Article  Google Scholar 

  • ISO (International Organization for Standardization). 2017. ISO 22316:2017. Security and resilienceOrganizational resiliencePrinciples and attributes. Geneva: ISO.

  • ISO (International Organization for Standardization). 2019. ISO 22301:2019. Security and resilienceBusiness continuity management systemsRequirements. Geneva: ISO.

  • Kelman, I., J.C. Gaillard, J. Lewis, and J. Mercer. 2016. Learning from the history of disaster vulnerability and resilience research and practice for climate change. Natural Hazards 82(1): 129–143.

    Article  Google Scholar 

  • Kostkova, P., J. Mani-Saada, G. Madle, and J. Weinberg. 2003. Agent-based up-to-date data management in National electronic Library for communicable disease. In Applications of software agent technology in the health care domain, ed. A. Moreno, and J.L. Nealon, 105–124. Basel, Switzerland: Springer.

    Chapter  Google Scholar 

  • Linkov, I., D.A. Eisenberg, M.E. Bates, D. Chang, M. Convertino, J.H. Allen, S.E. Flynn, and T.P. Seager. 2013. Measurable resilience for actionable policy. Environmental Science & Technology 47(18): 10108–10110.

    Google Scholar 

  • NFPA (National Fire Protection Association). 2019. NFPA 1600-Standard on continuity, emergency, and crisis management. Massachusetts, USA: NFPA. https://www.nfpa.org. Accessed 10 May 2020.

  • Pescaroli, G. 2018. Perceptions of cascading risk and interconnected failures in emergency planning: Implications for operational resilience and policy making. International Journal of Disaster Risk Reduction 30: 269–280.

    Article  Google Scholar 

  • Pescaroli, G., and D. Alexander. 2018. Understanding compound, interconnected, interacting, and cascading risks: A holistic framework. Risk Analysis 38(11): 2245–2257.

    Article  Google Scholar 

  • Peterson, R.A. 2013. Constructing effective questionnaires. Thousand Oaks, CA: Sage Publications.

    Google Scholar 

  • Twigg, J. 2015. Disaster risk reduction. London: Overseas Development Institute, Humanitarian Policy Group.

    Google Scholar 

  • UNISDR (United Nations International Strategy for Disaster Reduction). 2015. Sendai framework for disaster risk reduction 20152030. Geneva: UNISDR.

    Google Scholar 

  • UNISDR (United Nations International Strategy for Disaster Reduction). 2017. Disaster resilience scorecards for citiesPreliminary level assessment. Geneva: UNISDR.

    Google Scholar 

  • Vagias, W.M. 2006. Likert-type scale response anchors. Clemson International Institute for Tourism & Research Development, Department of Parks, Recreation and Tourism Management, Clemson University, Clemson, SC 29634, USA.

  • Watkins, R., M.W. Meiers, and Y.L. Visser. 2012. Assessing needs. Washington, DC: The World Bank.

    Book  Google Scholar 

  • Wiseman, S., G. Jawaheer, P. Kostkova, and G. Madle. 2008. Specialist digital libraries—National Resource for Infection Control (NRIC)—Information overload or underload? British Journal of Infection Control 9(5): 4–9.

    Article  Google Scholar 

  • Yore, R., and J. Faure-Walker. 2019. Microinsurance for disaster recovery: Business venture or humanitarian intervention? An analysis of potential success and failure factors of microinsurance case studies. International Journal of Disaster Risk Reduction 33: 16–32.

    Article  Google Scholar 

Download references

Acknowledgements

This short article was made possible thanks to the experience matured in different projects awarded by our funders: BA/Leverhulme Small Research Grant Award 2019 supported by the United Kingdom’s Department for Business, Energy and Industrial Strategy (Grant Reference: SRG19/191797); the Earthquake Engineering Field Investigation Team Award 2019 by the Institution of Structural Engineers in the UK; the Mexican Consejo Nacional de Ciencia y Tecnología (Grant Reference: 398485); the European Union’s Horizon 2020 Research and Innovation Programme (Grant Agreement: 821046), and the TURNkey (Towards more Earthquake-resilient Urban Societies through a Multi-sensor-based Information System enabling Earthquake Forecasting, Early Warning and Rapid Response actions) Project.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Gianluca Pescaroli.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Pescaroli, G., Velazquez, O., Alcántara-Ayala, I. et al. A Likert Scale-Based Model for Benchmarking Operational Capacity, Organizational Resilience, and Disaster Risk Reduction. Int J Disaster Risk Sci 11, 404–409 (2020). https://doi.org/10.1007/s13753-020-00276-9

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13753-020-00276-9

Keywords

  • Assessment
  • Disaster risk reduction
  • Interdisciplinary
  • Likert scale
  • Questionnaires
  • Scales