Visually Modelling Collaborative Research into Innovative Community Disaster Resilience Practice, Strategy, and Governance
- First Online:
- 842 Downloads
In 2013 a new collaborative center was established in Wellington, New Zealand to focus on integrating resilience research with the region’s community disaster resilience strategy. An earlier study with parties to this center had indicated that researcher and practitioner groups were divided by attention to their own immediate knowledge and skills, but agreed there was a need to maximize community resilience benefits amongst a regional population. An action research workshop of researchers and practitioners used a visual logic model to focus on the pragmatic benefits of improving community resilience. The visual logic model was used to design research activities that would improve the regional community resilience strategy, which was still in an early implementation phase. Ten of 14 workshop participants were interviewed following the workshop. Statistical content analysis of interview data highlighted certain strengths of the action research process: visual monitoring and evaluation planning was a catalyst for complicated conversations between two very different groups of professionals; and researchers became more focused on practical issues as a result. Other findings suggested that in future collaborative research governance would benefit from wider cycles of strategic intelligence, enhanced research contributions, and the use of different information formats for different purposes. Different formats for different purposes should also be considered when developing and implementing large-scale disaster risk reduction policies and strategies.
KeywordsCollaborative research Community resilience Content analysis Emergency management Visual communication
The word resilience has been anecdotally described as an espirit d’temps. The popularity of resilience thinking has been recently demonstrated by a dedicated Erasmus academic network, ANDROID; the Rockefeller Foundation’s Resilient Cities initiative; and several international centers of excellence for researching disaster resilience. These centers are supported by Integrated Research on Disaster Risk (IRDR) International and the United Nations Office for Disaster Risk Reduction (UNISDR). Other large-scale attempts to analyze and deploy the often technical (Birkmann et al. 2012) and increasingly disputed (Béné et al. 2012; Huggins et al. 2015) concept of resilience have also been initiated. Such initiatives aim to use a very broad field of knowledge and expertise which, according to Birkmann et al. (2012), span social-ecological, psychological, critical infrastructure, organizational, and practical perspectives. Interventions have ranged just as widely, across objectives focused on protection to transformation, and on outcomes from stability to flexibility and change (Béné et al. 2012).
Resilience to disasters has become an expansive domain which, according to Béné et al. (2012), has become extremely difficult to summarize under a unitary definition. It can be argued that conceptual complexity, combined with a lack of consensus between different research disciplines, has seen the rise of a particularly practical perspective (Birkmann et al. 2012). This practice-based perspective of resilience appears to underpin much of the work undertaken by the International Centre of Excellence: Community Resilience (ICoE:CR) in Wellington, New Zealand. However, even within this practice-oriented approach, there have been barriers limiting the effective communication of group and discipline specific concepts. Given that much communication between researchers, practitioners, and agencies in the resilience domain is text-based and language dense, it appeared that visual representations of concepts and plans could enhance communicability and enrich individual participants’ ability to better understand and work at the research and practice interface. This article reports on research undertaken to explore potentials for using a rich visual interface to help plan, develop, and adjust practical activities within the ICoE:CR.
The Wellington ICoE:CR is one of several IRDR International Centers of Excellence. As extensions of the international IRDR program (McBean 2012; IRDR 2013), International Centres of Excellence (ICoE) are located in China, Colombia, the United States, New Zealand, and South Africa (IRDR 2014). The centers have been tasked with conducting operationally relevant research, providing technical assistance for policy and decision making, sponsoring workshops and other events, and improving member networks (IRDR 2013).
The Wellington ICoE:CR is the only IRDR International Center of Excellence that has explicitly avoided a top-down approach to research in which academic findings are pushed down for practitioner uptake. Instead, core ICoE:CR researchers and practitioners are focused on working in partnership with the Wellington Region Emergency Management Office (WREMO) and in alignment with its Community Resilience Strategy. As outlined in a recent summary of the ICoE:CR, “… ‘active’ members [of ICoE:CR] are those conducting research or practice under the Community Resilience Strategy” (JCDR 2014, p. 2). This document outlines the equivalent status of WREMO and other practitioners alongside researchers, rather than treating practitioners as passive recipients for the top-down uptake of research findings.
The ICoE:CR approach is supported by a strategic review of contemporary urban emergency management by Kapucu (2009), which concluded that effective emergency management cannot be achieved by organizations acting in isolation. Kapucu (2009) also advocated for a wider geographic approach to community resilience, as reflected in the regional approach of the ICoE:CR. However, many documented emergency management collaborations, including collaborations reviewed by Kapucu (2009), have focused on immediate emergency response. Emergency response works on a much shorter timeframe than the combination of risk reduction, readiness, and recovery collaborations that could systematically enhance community disaster resilience in the Wellington region.
The ICoE:CR has sought its “excellence” through the development of quality interactions between collaborating institutions and wider communities. The determination of that excellence relies on the careful management of commonalities and differences between diverse stakeholders. It was this notion of excellence that motivated antecedent Q-method research (Huggins et al. 2015) into strong opinions held by initial practitioner and research partners to the new ICOE:CR. This antecedent research used factor analysis to identify patterns of opinion either shared by collaborating researchers and practitioners or differentiating between them. This factor analysis detected patterns in participants’ rating of the relevance of several statements concerning the monitoring and evaluation of community resilience.
There were three key patterns that stood out from this initial research and analysis of participant viewpoints. The first was a pattern of opinions against insular, top-down decision making. This factor was consistently relevant to practitioner representatives but not researchers, with individual factor loadings from −0.39 (p < 0.05) to −0.69 (p < 0.05) amongst practitioners. The second pattern was a need for complicated analysis to inform strategic decisions. This factor was consistently shared among researchers, but not practitioners, with factor loadings from 0.31 (p < 0.05) to 0.50 (p < 0.05) amongst researchers.
Finally, both researchers and practitioners supported a pattern of opinions amounting to a need to evaluate opportunities to improve complex post-disaster outcomes at a range of societal levels. Consistently significant loadings for this factor ranged between 0.30 (p < 0.05) and 0.61 (p < 0.05) for researchers and practitioners alike. This third factor provided a particularly pragmatic, action-focused impetus for further ICoE:CR development. The first two factors identified clear differences in viewpoint. These differences would need to be acknowledged by ICoE:CR coordinators and other professionals assisting them. A popular practice—simply gathering a range of Wellington-based emergency management collaborators in the same space at the same time—seemed unlikely to overcome such deeply rooted differences.
1.1 Visual Monitoring and Evaluation Planning for the ICoE:CR
Owen et al. (2013) outlined a solution for managing this kind of emergency management dilemma, where stakeholders with very different priorities would need to collaborate despite clear differences in viewpoint. It seemed vital to document a range of understandings, without insisting that every single aspect was shared between every single stakeholder. For Owen et al. (2013), documentation that served this purpose would become “boundary objects” (p. 1) for communicating diverse understandings in a way that spanned multiple divides between diverse collaborators.
Observed strengths of the visual logic models used in VMEP were documented by Huggins and Jones (2012) and Huggins and Peace (2014). These strengths include the ability to visually communicate multiple levels of shared activities and objectives across diverse stakeholder groups, and to generate a dialogic approach to the production of indicators and evaluation questions. The outcomes theory background to VMEP outlines the concept of outcome hierarchies: “a cascading set of causes in the real world” (Duignan 2012a, p. 1). This is how VMEP suited the pragmatism shared amongst ICoE:CR researchers and practitioners, by visualizing multiple levels of shared actions and outcomes before connecting them to relevant indicators and evaluation questions.
The deployment of VMEP within the ICoE:CR also came informed by a growing body of cognitive research into how visual imagery complements other forms of communication. For example, a critical review of associated scientific literature by Tversky (2011) concluded that rich visual representations can improve understanding of action in space. Klingner et al. (2010) found that visual representations required fewer internal cognitive resources than a verbal equivalent when measured by pupil dilation during cognitive tasks. Visual representations also appear to help people engage with nonlinear concepts. For example, one experiment by Kessell and Tversky (2009) found that 60 % of participants habitually drew cycle dynamics, from part A to part B to part C to part A etcetera, along a single line. However 80 % of participants in the same experiment preferred to see these dynamics on a circular diagram.
1.2 Current Hypotheses
The main objective of the current research was to detail how and why the ICoE:CR visual logic model transformed over time. Visual logic models had played particularly pragmatic roles in prior case studies by Huggins and Jones (2012) and Huggins and Peace (2014). We therefore hypothesized that the current visual model would serve a pragmatically useful purpose for workshop participants, and that this would be evidenced by their accounts of the VMEP process.
We also wanted to show how opinion factors identified through Q-method could endure over time. It is often assumed that social science analyses, such as Q-method factor analyses, are unreliable and represent little more than a moment in time, under very particular conditions. For emphatic examples of this viewpoint, see Faigman (1989) and Kampen and Tamás (2014). The current research assumed that statistically robust factors identified through Q-method research were relatively reliable and of enduring relevance to ICoE:CR development. Our hypothesis was that the distribution of interview content related to these opinion factors would closely resemble the distribution of original Q-method factors amongst ICoE:CR groups.
Critical realism provides an important theoretical foundation for organizational research. This epistemology, or set of rules about what can be considered knowledge, states that what occurs between observable events can be even more “real” than the events themselves (Ackroyd and Fleetwood 2010). Prior critical realist research into organizational dynamics, for example by Porter (2000), has prescribed a series of methodical steps: form hypotheses; test hypotheses against empirical observations; reformulate hypotheses. The current research incorporates these steps, working from the hypotheses outlined in Sect. 1.2. Prior critical realist research into organizations, for example by Porter (2000) and Costello (2000), has depended entirely on qualitative analysis. Our current research aimed to produce much more than a persuasive narrative. Qualitative assumptions made during the current research have been therefore tested and refined through systematic statistical analyses.
2.1 Data Collection
The data collected, developed, and analyzed in the current research were compiled in two phases. The first phase involved the development of the VMEP model, and the second phase involved in-depth interviews with participants about their engagement in the VMEP process. The development of the model is described first, followed by a brief description of data collected from interviews with VMEP participants.
VMEP uses a visual logic model to plan strategically relevant monitoring and evaluation activities. According to Duignan (2013), this is achieved by: (1) drawing an initial diagram of intended outcomes and the steps required to achieve those outcomes; (2) marking the relative priority of outcomes and steps and drawing causal linkages between them; (3) identifying key performance indicators that can help gauge performance towards intended outcomes; (4) developing evaluation projects; and (5) reporting evaluation results back onto the overall diagram and revising that diagram in response to those results.
Participants in the current action research were generally unfamiliar with evaluation frameworks, so the standard VMEP process outlined above was modified in four minor ways. First, although Duignan (2012b) had advised against defining distinctions between objectives and outcomes, we were unlikely to engage ICoE:CR coordinators without a clear overarching structure. This meant that all strategy components were neatly divided into activities, objectives, and outcomes. Second, although they are not usually highlighted so explicitly during a VMEP process, overarching ethical principles were written in text above the main visual logic model. This reflected a recommendation from Huggins et al. (2015), that a strong pragmatic impetus within the ICoE:CR needed to be complemented by explicit ethical principles. According to Huggins et al. (2015), a prominent set of ethical principles would help avoid using a range of disaster-related ends to justify any given means. Third, we chose to use the term “research question” rather than “evaluation question” due to many participants’ lack of familiarity with the latter concept. The former term was more intuitive for non-practitioner representatives in particular, who were all representing research institutions. Finally, VMEP step five was not included in the scope and timeframe of the current research because indicator data would take many more months to collect.
The first three VMEP steps were completed prior to the workshop, with the Manager, WREMO Community Resilience. This involved a series of meetings where, over a period of six months, an evolving visual logic model was used to represent what the WREMO Community Resilience team was doing, show why they were doing it, and demonstrate how they were tracking progress against internally established targets. It is important to note that framework components representing higher level impacts did not survive discussions to establish this initial series of VMEP steps. Engaging a range of stakeholders during these preliminary steps could have led to a different outcome. However the main focus of pre-workshop activities was to produce a representation of the WREMO community resilience program, that WREMO management would then release for wider discussions. The workshop proper was then attended by a combination of seven representatives of research institutions and seven WREMO practitioners, all of whom were selected and invited by ICoE:CR coordinators. The workshop began with a very brief introduction to the WREMO Community Resilience Strategy, delivered by the Manager, WREMO Community Resilience. To prepare workshop attendees for active VMEP engagement, they were introduced to three preliminary pieces of information: recently developed standards from the Information and Knowledge Management for Disaster Risk Reduction group (UNISDR 2013); a summary of results from the antecedent Q-method research; and an animated Prezi™ introduction to Visual Monitoring and Evaluation Planning.
Although Fig. 2 is difficult to read in the current format, readers may note that several research questions have been added in red, to a more elaborate version of the example shown in Fig. 1. Activities are listed in boxes to the left of Fig. 2. Priority activities are indicated by a traffic light at their top right corner and these boxes are linked to other boxes showing relevant objectives towards the right of the logic model. Existing performance indicators are shown by a small yellow ruler beneath the activity or objective components that they are meant to gauge. Potential research questions are shown in italic red type and questions that were already being addressed by existing research are shown in bold red type. Each of these research questions is displayed next to the section of the logic model to which it refers. A key and a directional arrow were added to the bottom of the original version, to help provide these explanations. The original logic model also included hyperlinks that led to further details for boxes and text with small triangles in their bottom right corner.
Following the production of this model in the workshop, ten out of 14 workshop participants agreed to give semistructured, 90 min interviews about what had occurred during the VMEP process. All workshop participants’ names were replaced with pseudonyms on interview transcripts, and all minor utterances were deleted unless they were essential for interpretation. Interviewees were offered the opportunity to delete any additional text and clarify any part of their interview before the following analysis.
2.2 Data Analysis
Interview transcripts were subjected to systematic thematic analysis prior to a more statistically oriented content analysis. Relying on content analysis alone was likely to neglect richer details about “the natural everyday world of human group life” (Wilson 1985, p. 398). The thematic analysis employed was what Braun and Clarke (2006) called “theory-driven,” because our entire analysis was based upon preestablished themes derived from the antecedent Q-method research, in Huggins et al. (2015), preexisting case studies of using visual logic models by Huggins and Jones (2012) and Huggins and Peace (2014), and other literature outlined below. Besides being transparent about preconceived ideas, Marks and Yardley (2004) stated that this approach to thematic analysis allows researchers to seek contradictions to many preconceptions. A theory-driven analysis therefore made a good match with the process of reformulating critical realist hypotheses, as outlined by Porter (2000).
Systematic thematic analysis relies on the development of a coding frame in which key themes can be noted and annotated. Our initial coding frame reworked the three opinion factors from antecedent Q-method research. According to Wolf (2014), in-depth interviews extend insights that can be constrained by rushing into full, and relatively arbitrary, factor descriptions. With reference to original elements outlined in Huggins et al. (2015), the code of a Constructive Focus on WREMO Activities was derived from the opinion factor, of being “against insular, top-down decision making.” A second code, Scientist Leadership, reflected the Q factor of a “need for complicated analysis to inform strategic decisions.” Finally, the opinion factor “need to evaluate opportunities to improve complex post-disaster outcomes at a range of societal levels” gave rise to a concise code of Positive Resilience Outcomes.
Further initial codes were based on other research literature that has been largely outlined in Sect. 1 of the current article. These codes included: Professional Collaborations with reference to visual modelling case studies by Huggins and Jones (2012) and Huggins and Peace (2014); Business Marketing with reference to the role of financial rationale in the case study by Huggins and Jones (2012); Diversity with reference to diverse engagement in complex domains outlined by Rogers (2008); and Documentation to represent the boundary objects outlined by Owen et al. (2013). Relevant interview excerpts that did not fit the seven initial codes were sorted into an Other category. Like subcodes sitting beneath the initial theory-based code, codes making up the Other category were linked and spliced as thematic analysis progressed.
For the purposes of content analysis, the action research was divided into three phases: (1) before the WREMO Collaborative Research Design Workshop; (2) during the workshop; and (3) after the workshop. Determining the most salient codes for each particular phase of the action research helped keep the current analysis manageable and coherent. Analysis included as much of the original interview text as possible in order to avoid the dilemma outlined by Dey (1993), of arbitrarily excluding material that could disconfirm researchers’ preconceived assumptions. Excerpts used in the current analysis therefore incorporated over 95,000 of a total 130,514 words appearing on interview transcripts. Most of the remaining words made up the interview preamble and other interviewer utterances.
The final set of main content analysis codes achieved moderate (kappa > 0.41) to good (kappa > 0.61) interrater reliability, using standards from Landis and Koch (1977). A less sophisticated measure of percentage agreement ranged from 64 to 93 % for each final code. These percentages were much greater than 50 %, which suggested that agreement was due to more than chance alone (Stroud and de Macedo Higgins 2009). Good to moderate interrater reliability was achieved for a final set of three timing codes: Before the Workshop (kappa = 0.77); During the Workshop (kappa = 0.69); and After the Workshop (kappa = 0.45).
Main codes significantly correlated with timing phase codes
% of phase excerpts (%)
Before the workshop
Constructive focus on WREMO activities
Unique community resilience activities
Belief that WREMO approach has not been applied to disaster resilience by any other organization
Includes a drive towards a community resilience state, but often a focus on WREMO activities in and of themselves
Positive resilience outcomes
Improved coping and response
Understanding resilience as coping with and response to challenges, emergencies, and/or disasters
Knowledge, skills, and assistance via network
Belief in distributing skills, knowledge, and assistance through the volunteer network and through networked communities
Diverse groups and individuals
Considering diverse individuals
Need to consider unique individuals with diverse preferences, experiences, roles, and contexts
Considering diverse places
Highlighting unique geographic contexts
During the workshop
Positive resilience outcomes
Knowledge, skills, and assistance via a network
Belief in distributing skills, knowledge, and assistance through the volunteer network and through networked communities
Improved coping and response
Understanding resilience as coping with and response to challenges, emergencies, and/or disasters
After the workshop
Different formats for different purposes
Need for traditional Word, Excel, Pdf etc. formats for formal documents
VMEP framework as a catalyst
Examples of VMEP diagrams seen as a catalyst for progress, action, or inaction
Constructive focus on WREMO strategy
General focus on WREMO community resilience strategy
Testing the WREMO community resilience strategy
Need to test the community resilience strategy as a whole
Improving the WREMO community resilience strategy
Need to improve the WREMO community resilience strategy
As outlined earlier, three of the content analysis codes were based on patterns of opinion amongst ICoE:CR practitioners and researchers. These opinion factors had been identified in prior research by Huggins et al. (2015), using a research method called Q-method, which identified statistical patterns in participants’ ratings of the relevance of 60 different statements concerning monitoring and evaluating community resilience interventions. A statistical analysis of variance found that the only content analysis code based on these prior Q-method factors that varied significantly between practitioner and researcher groups in the current research was Scientist Leadership (F(1,8) = 12.46, p = 0.008). Researcher interviews included significantly more excerpts calling for scientific leadership (M = 38.4, SD = 11.55) than practitioner interviews (M = 15, SD = 9.3).
The main code, Constructive Focus on WREMO Activities was based on the previous Q-method factor “against informing insular, top-down decision making.” This original Q-method factor had applied almost exclusively to ICoE:CR practitioners in Huggins et al. (2015). The current, content analysis version of this factor appeared across both researchers’ and practitioners’ interview text, and was reflected in 266 out of a total 798 interview excerpts used for the current thematic and content analysis. Variance between researcher (M = 21, SD = 11.64) and practitioner (M = 32, SD = 14.86) groups did not significantly exceed within-group variance for this code (F(1,8) = 1.76, p = 0.22).
The current research primarily examines how and why the VMEP process changed over time. This first research aim is primarily addressed by breaking up the content analysis results into VMEP phases. This has enabled us to identify a number of distinct patterns amongst participants’ accounts, as outlined in Sects. 4.1, 4.2, and 4.3. This analysis showed how the visual logic model could be a catalyst for actions that would further ICoE:CR objectives. However, participants’ interviews also showed how the visual logic model would need to be converted to another format for further use. This call for traditional documentation characterizes the workshop as a transient, facilitated stage for the ICoE:CR. This is a stage that appeared to need further implementation, including the careful construction of a text-based document to represent workshop dialogue and agreements. Participants clearly outlined the way that different documents would serve different purposes. According to many of them, a more traditional format was needed to formalize individual research projects following the VMEP workshop.
The current research also reveals differences between patterns of opinions identified in antecedent research and patterns of opinions raised by the current participants. This was achieved by comparing the distribution of original opinion factors with content from interviews describing the VMEP process—regardless of whether the opinion content related to before, during, or after the workshop. Findings from this overall analysis are discussed in Sect. 4.4.
4.1 What Happened Before the VMEP Workshop
As outlined in Sect. 3 results, a Constructive Focus on WREMO Activities was distinctly related to accounts of what happened before the VMEP workshop (rho = 0.13, p < 0.01). As described above, this code related to an opinion factor from antecedent Q-method research: “against insular, top-down decision making.” The relevance of this main code to what happened before the workshop suggests that participants had been motivated to constructively engage with operational decisions being made at WREMO before the workshop began. Restricting the VMEP model towards the level of activities and objectives may have helped meet this interest once the workshop began. Content concerning Positive Resilience Outcomes (rho = 0.10, p < 0.01) was also used to account for what occurred prior to the workshop. The most prevalent subcodes for Positive Resilience Outcomes were “improved coping and response” (6 % of all phase excerpts); and “knowledge, skills, and assistance via network” (3 % of phase excerpts). This content was related to the prevalent opinion factor identified by antecedent opinion Q-method research: “need to evaluate opportunities to improve complex post-disaster outcomes at a range of societal levels.” It was not surprising to observe the ongoing relevance of this opinion factor. A strong drive within the ICoE:CR, to analyze and improve the WREMO Community Resilience Strategy was well documented and announced to many interested parties (JCDR 2014). Nonetheless, the ongoing relevance of this particularly pragmatic opinion suggests that the VMEP process was well matched with enduring motivations at the core of the ICoE:CR. VMEP was adopted to leverage strong pragmatic motivations, which were shared between two distinct groups of ICoE:CR stakeholders.
Diverse Groups and Individuals (rho = 0.07, p < 0.05) was the only main code that exclusively related to content about the lead up to the workshop. The other two main codes for this phase were also highly relevant to during and after the workshop, respectively. Participants suggested that there was a need for a more diverse array of participants at the core of ICOE:CR activities. This included suggestions for representatives from a greater number of research institutions, other agencies working with community resilience, and community members from a diverse range of cultural backgrounds. Highlighting the importance of cultural considerations in the lead up to the workshop, one researcher's interview stated: “…the idea of having to look out for people after a disaster, I don’t understand all those details but that cultural perspective…” (Alana, lines 165–167). Several participants also detailed a need to adapt the WREMO strategy for diverse groups and individuals. We assumed that these participants’ accounts align with Parkinson (2009), who has described how socially orientated programs need to systematically consider diverse stakeholder needs and viewpoints.
We’re prioritizing partnerships because partnerships happen between everybody…and we’re prioritizing participation because everyone needs to participate…and protection because everyone needs to be looked after so it’s not, it’s not because of race or previous history, it’s because people are people and so all of our principles effectively cover all people. (Bridget, lines 1126–1130)
Other interview content regarding diversity appeared to discount the value of research. An apparent focus on variability and exceptions to theoretical rules could have been constraining researchers and participants and any generalized theoretical models of resilience that they had to offer. In this way, highly prevalent subcodes, such as “considering diverse individuals” (20 % of phase excerpts) and “considering diverse places” (11 % of phase excerpts), may have represented an obstacle to generalizations from a large body of preexisting community resilience research (see for example, Birkmann et al. 2012). The subcode, “considering diverse individuals” (20 % of phase excerpts) can also be considered alongside the subcode “egalitarianism” (3 % of phase excerpts). Workshop participants may have focused on reducing the status that researcher views often receive. This egalitarian approach to diminishing researcher views may have also meant down-grading, and eventually eliminating, researchers’ focus on predictive models and downstream outcomes.
4.2 What Happened During the VMEP Workshop
There was subsequently very little discussion about downstream resilience impacts identified in surrounding academic research, such as Burton (2012). No workshop participant asked to detail downstream impacts during the workshop, even though they went on to receive an introduction to VMEP that outlined the way these components are usually represented. Likewise, no participant voiced concerns about the way that program benefits appeared unrelated to surrounding research, until this issue was specifically probed during interviews.
Participants’ comments about the workshop itself were centered around efforts to work with a visual logic model without any research questions and convert it into the version shown in Fig. 2. Positive Resilience Outcomes (rho = 0.13, p < 0.01) was the only main code that was significantly relevant to what happened during the VMEP workshop. The way that participants used content about Positive Resilience Outcomes to explain what happened before and during the workshop suggests that they found the VMEP workshop worthwhile. The VMEP workshop appears to have helped maintain the pragmatic impetus identified in antecedent research, concerning a “need to evaluate opportunities to improve complex post-disaster outcomes at a range of societal levels.”
Yeah so if there’s lots of people out there with our It’s Easy books and stuff and talking to their neighbors …. It’s like I can talk to 10 people and say, talk to…the importance of talking to networks and things, and they go to their networks and that’s great. (Kirsty, lines 457–460)
This is also how the relevance of Positive Resilience Outcomes marks a pitfall in using VMEP to support emergency management. The focus on positive, and fairly tautological, outcomes serves as a reminder of how the VMEP process does not explicitly address unintended consequences. The VMEP process itself could be seen as optimistic, because it does not involve searching for negative implications. Many of these consequences fall outside of an initial strategic planning schema, that is, the outcomes specified on a visual logic model could nonetheless be very negative. For example, the popular uptake of preparedness kits may lead to overconfidence, resulting in a lack of interest in other aspects of preparedness and mitigation activities. VMEP for emergency management may therefore need to include a more deliberate and wide-ranging search for changes in a hazard affected environment.
4.3 What was Going to Happen After the VMEP Workshop
Workshop participants were also asked about what was going to happen following the VMEP workshop. The analysis of participant accounts highlighted two main codes. The first of these main codes was a focus on Documents (rho = 0.12, p < 0.01), including the prevalent subcodes “different formats for different purposes” (12 % of phase excerpts) and “VMEP framework as a catalyst” (11 % of phase excerpts). The prevalence of “VMEP framework as a catalyst” shows how the VMEP diagram had represented a catalyst for dialogue, but was not necessarily a document for wider circulation.
Further subcodes for Documents included a “need for written text” and “different formats for different purposes.” These subcodes highlight a stated need to document workshop outcomes in a more traditional format. One participant’s interview made the relevance of both these subcodes particularly clear: “But I imagine that these will be turned into a document. And then they are more publicly, widely available…” (Alanna, lines 973–974). Here the word “document” refers to a traditional text-based document, like the original WREMO Community Resilience Strategy, which was released as a mainly text-based narrative document. This focus on an established text-based format therefore provides a context for the second most relevant theme: ‘Constructive Focus on WREMO Strategy’ (rho = 0.08, p < 0.01). Subcodes detailed in Table 1 show how many participants thought that the strategy included several assumptions, which remained to be tested and improved. This did not mean that participants supported a shift to working from another sort of document all together—hence the need to edit and redistribute the existing, text-based, version of the strategy.
As discussed in Sect. 1.1, Owen et al. (2013) outlined the need for boundary objects that link multiple organizational systems, beyond members of a single emergency management team. With this in mind, the integration of several preexisting organizations with the ICoE:CR may depend on common and largely text-based narrative documents. Text-based documents represent a strong status quo for formalizing agreements between organizations and other collaborators. Organizational change often fails when trying to eradicate all established norms and impose completely new practices (Clegg and Walsh 2004). This rationale for organizational development comes supported by blatant requests for a text-based outcome, from workshop participants and ICoE:CR coordinators alike. The visual logic model was therefore only likely to be formalized when it had been converted to a more traditional document format, such as Microsoft Office Word or Microsoft Excel formats.
This does not mean that narrative documents function well as a catalyst for creative and responsive thinking about complex dynamics. Diagram-based approaches can provide an important avenue to help us think about, and plan for, complex systems (Tversky 2011; Kessell and Tversky 2009; Huggins and Jones 2012). However it is equally important to note how text-based narratives can provide particularly in-depth descriptions. These descriptions clarify details that are relatively opaque in other forms of communication (Tversky 2011). If we were, for example, to compare the 24 pages of the WREMO (2012) Community Resilience Strategy with the Fig. 2 diagram, we would note that the diagram is brief and can be read in much less time. It also lacks a great deal of the clarifying detail available in the text document. While it has been important to develop the VMEP visual logic model (Fig. 2) as a critical engagement and process tool, this approach continues to require a supplementary, narrative explanation—especially for new collaborating parties.
4.4 Overall Analysis of Opinion Factor Distribution
The current research also aimed to test whether the distribution of Q-method opinion factors shifted during the current action research, where “distribution” refers to distribution between researcher and practitioner groups, and across interview content as a whole. As outlined in Sect. 3, the distribution of relevant interview content closely resembled the original distribution of two of the three Q-method opinion factors. In contrast, a Constructive Focus on WREMO Activities was now observed across both researcher and practitioner groups. This code was equivalent to Q-method opinion factor 1, which had not previously been consistently observed amongst ICoE:CR researchers as a group. Implications of this changing pattern of opinions are outlined in Sect. 5.
5 Conclusion: How Visual Monitoring and Evaluation Supported ICoE:CR Development
Both original hypotheses for the current research have been revised, in light of equivocal support from our analysis of the interview data. The first hypothesis was that the VMEP process would prove useful to core ICoE:CR stakeholders participating in the VMEP workshop. However, participants perceived that VMEP was more of a catalyst for wider processes than an all-encompassing system for planning ICoE:CR activities. The original hypothesis has therefore been refined, to predict that VMEP processes and outputs will be of value at certain points of developing the ICoE:CR and comparable initiatives.
The second hypothesis—that the distribution of opinions identified by Huggins et al. (2015) would remain stable—has also been revised. The distribution of one of three factors appears to have changed considerably. ICoE:CR researchers appeared to have assumed more of a focus on WREMO community resilience activities during the VMEP process than they had during the antecedent research. With this in mind, we predict that boundary objects such as visual logic models will not just accommodate different positions from diverse emergency management stakeholders. Instead, boundary objects will facilitate substantial changes to consensus at group and subgroup levels. This revised hypothesis represents an optimistic, inverted interpretation of double demotivation theory (Carr 1996), which predicts that by bridging between professions boundary objects will transform at least one of the professions involved.
5.1 Implications of Current Findings Within the ICoE:CR
VMEP outputs were eventually converted to a large 10 page spreadsheet to suit requests from participants and ICoE:CR coordinators. This spreadsheet detailed draft research questions from the visual logic model by different categories: theme; methods; other details; relevant sections of WREMO strategy; existing researchers; and a column for WREMO to show the likelihood of uptake. At the time of writing, this spreadsheet had already been requested and used by researchers wanting to work with WREMO as part of the ICoE:CR. As stated by workshop participants, in terms of “VMEP as a catalyst,” this spreadsheet would not exist if it were not for the VMEP diagram underpinning it. Likewise, the spreadsheet is still being displayed to potential ICoE:CR researchers alongside the original VMEP diagram. This approach to using multiple formats has been directly informed by the current analysis, and by follow up discussions with ICoE:CR coordinators.
A clear limitation of using visual logic models for emergency management has appeared throughout the current VMEP process. As detailed in Sect. 2.1, early stages of the VMEP process at the ICoE:CR dictated that the visual logic model layout would be simplified for sign off by busy WREMO management. Although there had been some convincing reasons to avoid a more richly layered model, the lack of higher level outcomes has now reinforced by participants’ overall focus on the operational level of WREMO activities. The resulting visual logic model and surrounding VMEP process has neglected a range of surrounding research into resilience predictions, and requires expansion as part of longer term considerations for the WREMO community resilience program.
Roorda and Nunns (2009) suggested that logic model diagrams such as visual logic models can struggle to integrate surrounding research in the same way as systems diagrams. It seems reasonable to suggest that other processes, such as researcher-led systems modelling, may help further develop collaborative initiatives such as the Wellington ICoE:CR. For the ICoE:CR, this may substantially improve on the rapid systems model developed by WREMO practitioners and outlined in WREMO (2012).
These issues reflect neglect of research-based considerations and a failure to observe unintended consequences, outlined in Sect. 4.2, which are naturally due to more than document format alone. Process itself has a very important role to play. VMEP for emergency management needs to incorporate information from much further afield than internally established key performance indicators. Relevant changes to the surrounding environment, including unintended consequences, could be just as important to respond to as the success of particular interventions. An appropriately wider search could even reflect a strategic intelligence cycle that incorporates planning and tasking, data collection, processing/exploitation, analysis, production, dissemination, and user requirements and feedback alongside internal evaluation (Krizan 1999).
5.2 Wider Implications
Attempts to integrate planning, monitoring, evaluation, and research may benefit from some form of visual logic model at many points of development. According to Huggins and Jones (2012), systems models and logic models represent a heuristic, abbreviated understanding of the world around us and are best assessed in pragmatic terms. We must ask ourselves whether the boundary object in front of us will really help us achieve what we need it to, while asking what other tools we need to deploy. To do otherwise would distance organizational development from what Huggins and Jones (2012) and Todd and Gigerenzer (2003) have referred to as “ecological rationality,” where selective and efficient understandings help make genuinely useful decisions, in particular situations. Further research into planning, monitoring, evaluating, and/or researching community disaster resilience programs will benefit from considering this established criterion.
At the time of writing concluding the current research, the new Sendai Framework for Disaster Risk Reduction 2015–2030 (UNISDR 2015) has been adopted and published as a largely linear, narrative document. Formats such as visual logic models have the potential to complement such an extended narrative document as a catalyst for relevant monitoring, evaluation, and research strategies. For example, visual logic models can use a “drill down” function to break an international scale down into regional, national, and subnational levels. This kind of rich visual and layered approach could help emergency managers meet contemporary community resilience criteria such as “collaboration and integration” and “issue and place-specific responses” (Local Government New Zealand, 2014, p. 1).
In light of the current research, processes such as VMEP and documents such as visual logic models are not simply a replacement for traditional narrative documents. The complementary role of VMEP for complex policy processes is a constructive finding, not a criticism. Experts in constructing extended linear narratives have no need to feel threatened by some new resistance to the clarity of official text-based agreements. It has become difficult to deny that VMEP represents a very different way of facilitating and documenting complicated dialogue between diverse disaster risk reduction stakeholders. But the current research also highlights the relative ease of moving into this mode of richly visual, boundary objects in dialogic, workshop contexts—before moving back to more traditional, text-based formats.
This movement between formats should certainly be considered for documenting many aspects of large-scale policies and strategies such as the Sendai Framework for Disaster Risk Reduction. Given the operational focus observed during the current research, VMEP may be particularly valuable in the implementation of such large-scale policies and strategies. Failure to consider implementation-focused innovations, such as VMEP integrated project-management proposed by Huggins and Peace (2014), could mean doing what we have always done to get what we have always got. As paraphrased from a United Nations Office for Disaster Risk Reduction review of the prior UNISDR disaster risk reduction framework (UNISDR 2014) the Hyogo Framework for Action 2005–2015, doing what we have always done to monitor and utilize core indicators is simply not good enough. As illustrated by the current research, VMEP represents a constructive challenge to this status quo. This process combines the expertise of practitioners and researchers towards improving the research-informed implementation of contemporary disaster risk reduction initiatives.
The current research was funded by the Earthquake Commission of New Zealand, The Institute of GNS Science and Massey University. Persons overseeing this funding did not play any role in the study design, collection, analysis, and interpretation of data, or in the writing and submission of this article. The research was conducted within the guidelines and procedures of the Massey University Human Ethics Committee.
- Ackroyd, S., and S. Fleetwood. 2010. Realism in contemporary organisation and management studies. In Realist perspectives on management and organisation, ed. S. Ackroyd and S. Fleetwood, 3–25. London: Routledge.Google Scholar
- Béné, C., R.G. Wood, A. Newsham, and M. Davies. 2012. Resilience: New utopia or new tyranny? Reflection about the potentials and limits of the concept of resilience in relation to vulnerability reduction programmes. Brighton: Institute of Development Studies.Google Scholar
- Birkmann, J., D. Chang Seng, T. Abeling, N. Huq, J. Wolfertz, N. Karanci, and S. Jϋlich. 2012. Systematization of different concepts, quality criteria, and indicators. Brussels: Centre for Research on the Epidemiology of Disasters.Google Scholar
- Burton, C.G. 2012. The development of metrics for community resilience to natural disasters. A thesis submitted in partial fulfilment of the requirements for the degree of doctor of philosophy. South Carolina: University of South Carolina. http://webra.cas.sc.edu/hvri/education/docs/Chris_Burton_2012.pdf. Accessed 1 Oct 2013.
- Carr, S.C. 1996. Effects of unreasonable pay discrepancies for under-and overpayment on double demotivation. Genetic Social and General Psychology Monographs 122(4): 477–494. http://www.tandfonline.com/loi/vzpm20#.VMf2wC7oO_R. Accessed 4 May 2015.
- Costello, N. 2000. Routines, strategy and change in high-technology firms. In Realist perspectives on management and organisation, ed. S. Ackroyd and S. Fleetwood, 161–181. London: Routledge.Google Scholar
- Duignan, P. 2012a. Overview of outcomes theory. www.strategicevaluation.info/ot/index.html. Accessed 15 Nov 2012.
- Duignan, P. 2012b. Stop the terminological madness now! “Outcomes”, “impact”, “results”, “goals” and the Buffalo Dung Problem. http://outcomesblog.org/2012/08/15/stop-the-terminological-madness-now-outcomes-impact-results-goals-and-the-buffalo-dung-problem/. Accessed 10 Feb 2013.
- Duignan, P. 2013. Faster monitoring and evaluation (M&E) planning. http://www.outcomescentral.org/dplan/eval.html. Accessed 15 Feb 2013.
- Faigman, D.L. 1989. To have and have not: Assessing the value of social science to the law as science and policy. Emery Law Journal 1(1): 1005–1096.Google Scholar
- IRDR (Integrated Research on Disaster Risk). 2013. Strategic plan 2013–2017. IRDR International. http://www.irdrinternational.org/wp-content/uploads/2013/04/IRDR-Strategic-Plan-2013-2017.pdf. Accessed 1 Jun 2014.
- IRDR (Integrated Research on Disaster Risk). 2014. International Centres of Excellence (ICoEs). IRDR International. http://www.irdrinternational.org/about/structure/icoes/. Accessed 1 Jun 2014.
- JCDR (Joint Centre for Disaster Research). 2014. International Centre of Excellence in Community Resilience: Wellington. Information sheet. Wellington: Joint Centre for Disaster Research.Google Scholar
- Kessell, A., and B. Tversky. 2009. Thinking about cycles: Producing sequences but preferring circles. California: Stanford University.Google Scholar
- Krizan, L. 1999. Occasional paper number six: Intelligence essentials for everyone. Washington DC: Joint Military Intelligence College.Google Scholar
- Local Government New Zealand. 2014. Managing natural hazard risk in New Zealand—Towards more resilient communities. Wellington: Local Government New Zealand.Google Scholar
- Marks, D.F., and L. Yardley. 2004. Content and thematic analysis. In Research methods for clinical and health psychology, ed. D.F. Marks and L. Yardley, 56–68. London: Sage.Google Scholar
- Porter, S. 2000. Critical realist ethnography. The case of racism and professionalism in a medical setting. In Realist perspectives on management and organisation, ed. S. Ackroyd and S. Fleetwood, 141–160. London: Routledge.Google Scholar
- Roorda, M., and H. Nunns. 2009. Lifting the lens: Developing a logic for a complicated policy. Evaluation Journal of Australasia 9(2): 24–32.Google Scholar
- Stroud, N.J., and V. de Macedo Higgins. 2009. Content analysis. In Research methods in communication, ed. S. Zhou and W.M.D. Sloan, 123–143. Northport, AL: Vision Press.Google Scholar
- Todd, P.M., and G. Gigerenzer. 2003. Bounding rationality to the world. Journal of Economic Philosophy 24(2): 143–165.Google Scholar
- UNISDR (United Nations International Strategy for Disaster Reduction). 2013. Information and knowledge management for disaster risk reduction (IKM4DRR) framework and scorecard. Geneva: Information and Knowledge Management for Disaster Risk Reduction, UNISDR.Google Scholar
- UNISDR (United Nations International Strategy for Disaster Reduction). 2014. Progress and challenges in disaster risk reduction: A contribution towards the development of policy indicators for the post-2015 framework for disaster risk reduction. http://www.unisdr.org/files/40967_40967progressandchallengesindisaste.pdf. Accessed 27 Apr 2015.
- UNISDR (United Nations International Strategy for Disaster Reduction). 2015. Sendai framework for disaster risk reduction 2015–2030. http://www.wcdrr.org/preparatory/post2015. Accessed 04 Sept 2015.
- Wolf, A. 2014. The Interview in Q-methodology: Readiness to adopt sustainable responses to climate change. Sage research methods cases. http://srmo.sagepub.com/cases. Accessed 15 Oct 2014.
- WREMO (Wellington Region Emergency Management Office). 2012. Community resilience strategy. November 2012. Wellington, New Zealand: Wellington Region Emergency Management Office.Google Scholar
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.