Abstract
Scientific-based evaluation has become a widely accepted instrument of quality assurance for civil protection (CP) exercises. To enhance reliability and richness of evaluation data and thus quality of evaluation standards, researchers have developed a variety of various tools and methods. However, the mere accessibility of tools and methods does not necessarily improve evaluation quality. Among others, evaluation quality also hinges on competent evaluators, who are able to handle demands and challenges associated with exercise evaluation. However, research has barely focused on competence requirements for evaluators within the context of CP exercises, even though professionalisation and professionalism of evaluators have been discussed in general evaluation research. There is a need to determine whether the requirements for evaluators identified within this broader field of study can be applied to the specific realm of CP exercise evaluation. This paper aims to bridge this gap by examining the competence requirements for evaluators within the context of European CP. The study conducts a two-wave delphi survey. It solicited experts to assess the priorities of 106 competencies for exercise evaluation. The findings suggest that competence models developed in general evaluation research are proper to describe the competence requirements for exercise evaluators within the context of CP. Considering average rating scores and indicators for consent and dissent, this study concludes on the categorisation of standard requirements, context-specific requirements, and ambiguous attributes.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
For further information on the project, please visit: https://civil-protection-knowledge-network.europa.eu/projects/inegma-e2
For a detailed description of different exercise types, see, e.g. FEMA 2020, 2-6ff.
References
AES (2013) Evaluators’ professional learning competency framework
Aschbacher H, Riener, J (2020) IRONORE 2019. Project Review, Findings and Way Forward. Vienna/Laubegg
Ayoo S, Wilcox Y, LaVelle JM et al (2020) Grounding the 2018 AEA evaluator competencies in the broader context of professionalization. New Dir Eval 2020:13–30. https://doi.org/10.1002/ev.20440
Beerens RJJ, Haverhoek-Mieremet K (2021) What do practitioners expect from an evaluation report? A qualitative analysis of Dutch crisis management professionals’ expectations. Int J Emerg Serv 10:1–25. https://doi.org/10.1108/IJES-12-2019-0063
Beerens RJJ, Tehler H (2016) Scoping the field of disaster exercise evaluation - A literature overview and analysis. Int J Disaster Risk Reduct 19:413–446. https://doi.org/10.1016/J.IJDRR.2016.09.001
Beerens RJJ, Tehler H, Pelzer B (2020) How can we make disaster management evaluations more useful? An empirical study of dutch exercise evaluations. Int J Disaster Risk Sci 11:578–591. https://doi.org/10.1007/S13753-020-00286-7
Böttcher WCAHJKMMW (2016) Standards für Evaluation. Mainz
Buchanan HK-SK (2014) Evaluator competencies: the Canadian experience. Can J Program Eval 28:29–47
Caspari A, Hennen M, Scheffler D, et al (2004) Recommendations on Education and Training in Evaluation Requirement Profiles for Evaluators. Mainz
CES (2018) Competencies for Canadian Evaluation Practice. Revised version. https://evaluationcanada.ca/files/pdf/2_competencies_cdn_evaluation_practice_2018.pdf
Clinton JM, Hattie J (2021) Cognitive complexity of evaluator competencies. Eval Program Plann 89:102006. https://doi.org/10.1016/j.evalprogplan.2021.102006
Dewey JD, Montrosse BE, Schröter DC et al (2008) Evaluator competencies. Am J Eval 29:268–287. https://doi.org/10.1177/1098214008321152
EES (2020) The EES Evaluation Capabilities Framework. https://europeanevaluation.org/wp-content/uploads/2020/04/EES-EVALUATION-CAPABILITIES-FRAMEWORK.pdf
Ellebrecht N, Jenki M, Kaufmann S (2013) Inszenierte Katastrophen. Zur Genese der Übung im Bevölkerungsschutz und ihre gegenwärtigen Formen. In: Hempel L, Bartels M, Markwart T (eds.) Aufbruch ins Unversicherbare. Zum Katastrophendiskurs der Gegenwart. Bielefeld, pp 235–276.
European Commission (2021): Union Civil Protection Mechanism (UCPM) Technical Guide for UCPM Full-scale exercises. V3.0. Online available: https://ec.europa.eu/info/funding-tenders/opportunities/docs/2021-2027/ucpm/guidance/technical-guide-full-scale-exercises_ucpm_en.pdf [2023-11-08]
FEMA (2020) Homeland Security Exercise and Evaluation Program (HSEEP). https://www.fema.gov/sites/default/files/2020-04/Homeland-Security-Exercise-and-Evaluation-Program-Doctrine-2020-Revision-2-2-25.pdf
Garcia GL, Stevahn L (2020) Situational awareness and interpersonal competence as evaluator competencies. Am J Eval 41:107–124. https://doi.org/10.1177/1098214018814941
Ghere G, King JA, Stevahn L, Minnema J (2006) A professional development unit for reflecting on program evaluator competencies. Am J Eval 27:108–123. https://doi.org/10.1177/1098214005284974
Greenberg B, Voevodsky P, Gralla E (2016) A capabilities-based framework for disaster response exercise design and evaluation: findings from oil spill response exercises. J Homel Secur Emerg Manag. https://doi.org/10.1515/jhsem-2016-0034
Heinze F, Reiter S, Berg-Lupper U et al (2019) Multiple Handlungsanforderungen an Evaluierende. Zeitschrift für Evaluation 18:261–283. https://doi.org/10.31244/zfe.2019.02.05
Hense JU (2014) Selbsteinschätzung von Evaluationskompetenzen: eine explorative Studie zu Faktorenstruktur und gruppenbezogenen Unterschieden. Zeitschrift Für Evaluation 13:211–242
Karapidis A, Dworschak B (2016) Study on competence management within the union civil protection mechanism training programme. Stuttgart
King JA, Stevahn L (2015) Competencies for program evaluators in light of adaptive action: What? So What? Now What? New Dir Eval 2015:21–37. https://doi.org/10.1002/ev.20109
King JA, Stevahn L (2020) Presenting the 2018 AEA evaluator competencies. New Dir Eval 2020:49–61. https://doi.org/10.1002/ev.20435
King JA, Stevahn L, Ghere G, Minnema J (2001) Toward a taxonomy of essential evaluator competencies. Am J Eval 22:229–247
Langer A (2015) Professionalisierung in und für Evaluation. Zeitschrift Für Evaluation 14:163–194
Meyer W (2015) Professionalisierung von Evaluation: ein globaler Blick. Zeitschrift Für Evaluation 14:215–246
Niemann L, Thielsch MT (2020) Evaluation of basic trainings for rescue forces. J Homel Secur Emerg Manag. https://doi.org/10.1515/jhsem-2019-0062
Oscarsson O (2019) Boundary work to conduct business as usual: interaction at the boundary between the affected organization and emergency responders. Risk Hazards Crisis Public Policy 10:212–232. https://doi.org/10.1002/rhc3.12157
Peersman G, Rogers P (2017) Pathways to advance professionalisation within the context of the Australasian Evaluation Society. Melbourne
Perry RW (2004) Disaster exercise outcomes for professional emergency personnel and citizen volunteers. J Conting Crisis Manag 12:64–75. https://doi.org/10.1111/J.0966-0879.2004.00436.X
Pleger L, Sager F (2016) Don’t tell me cause it hurts‘—Beeinflussung von Evaluierenden in der Schweiz. Zeitschrift Für Evaluation 15:23–59
Podems D (2014) Evaluator competencies and professionalizing the field: Where are we now? Can J Program Eval 28:127–136
Sandermann M (2011) Die Bedeutung von Soft Skills für Evaluationsnutzungen. VS, Wiesbaden
Schulte N, Thielsch MT (2019) Evaluation of firefighter leadership trainings. Int J Emerg Serv 8:34–49. https://doi.org/10.1108/IJES-03-2018-0020
Simons H (2015) Towards professionalization of evaluation: voluntary evaluator peer review—is this the road to travel? Zeitschrift Für Evaluation 14:195–213
Skryabina E, Betts N, Reedy G et al (2020) Evaluation of emergence preparedness exercises: the design of a questionnaire to measure staff perceptions. Int J Emerg Manage 16:317–346
Stevahn L, King JA, Ghere G, Minnema J (2005) Establishing essential competencies for program evaluators. Am J Eval 26:43–59. https://doi.org/10.1177/1098214004273180
Stevahn L, Berger DE, Tucker SA, Rodell A (2020) Using the 2018 AEA evaluator competencies for effective program evaluation practice. New Dir Eval 2020:75–97. https://doi.org/10.1002/ev.20434
Swedish Civil Contingency Agency (2016) Handbook. Evaluation of Exercises. Sweden
Thielsch MT, Hadzihalilovic D (2020) Evaluation of fire service command unit trainings. Int J Disaster Risk Sci 11:300–315. https://doi.org/10.1007/s13753-020-00279-6
Tierney K (2012) Disaster governance. social, political, and economic dimensions. Annu Rev Environ Resour 37:341–363
UKES (2012) Framework of Evaluation Capabilities. https://www.evaluation.org.uk/app/uploads/2019/04/UK-Evaluation-Society-Framework-of-Evaluation-Capabilities.pdf
UKES (2017) Extending the Professionalisation of Evaluation. Voluntary Evaluator Peer Review (VEPR). https://www.evaluation.org.uk/app/uploads/2019/09/Voluntary-Evaluator-Peer-Review-Evaluative-Report-Annexes-Final.pdf
Vedung E (2021) The twelve-part typology of evaluation uses. Zeitschrift für Evaluation 2021:101–130. https://doi.org/10.31244/zfe.2021.01.05
Acknowledgements
The research is part of the project INEGMA-E2, which was funded by the European Union under the Union Civil Protection Mechanisms (UCPM)—Knowledge Network Partnership (UCPM-2022-KN). The project’s overall goal is to enhance the evaluation quality in the context of civil protection exercises. The information and views set out in this publication are those of the author(s) and do not necessarily reflect the official opinion of the European Union.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Pichler, S.M., Anger, I. & Norf, C. Towards a Competence Profile for Civil Protection Exercise Evaluators: Findings from the INEGMA-E2 Delphi Survey. Eur J Secur Res (2024). https://doi.org/10.1007/s41125-024-00097-7
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s41125-024-00097-7