For a decision support framework (DSF) to enable effective decision-making in climate change adaptation, it is important that stakeholders are involved in its development, in order to ensure that it is usable and useful. More specifically, stakeholder involvement may help to ensure that the DSF better meets user needs and expectations, as well as providing legitimate, relevant and trusted information. Involving users also helps to support social learning and build a community of adaptors. This paper describes a case study in Australia of the development of a DSF, called CoastAdapt, for coastal decision makers to adapt to the impacts of climate change, in particular sea-level rise. We use the IAP2 Spectrum to outline how stakeholders were involved. We also describe the specific activities undertaken in developing the DSF, according to how they contribute to conditions of legitimacy, credibility and saliency suggested in ‘boundary’ work. We conclude with some practical suggestions for considering these attributes in development of a DSF, noting that each attribute is important and requires consideration both separately and together.
Growing understanding of sea-level rise, and its significance for coastal communities, has prompted burgeoning interest from coastal decision makers in the data, information and knowledge required to manage coastal hazards into the future. Managers of the impacts of coastal hazards include decision makers at all levels of government and from business, industry, non-government organisations (NGOs) and the community groups, and businesses or industries that depend on a coastal location or need to access coastal resources or populations. Such diversity of decision makers has focussed attention on the spectrum of knowledge needs and capacities and how decision-makers are best supported to effectively manage coastal hazards under a changing climate. Web-based decision support to address climate change includes platforms such as web portals (Sanderson et al. 2016), decision support tools and frameworks (DSFs) (Moss 2016) and online databases (Mitchell et al. 2016). These platforms enable provision of a range of data and information in flexible and accessible formats. The growing literature on DSFs for climate change adaptation acknowledges that if such tools and frameworks are to be usable and useful, and thus long-lasting, then their development requires collaboration between creators and potential users (Kopke and Dwyer 2016; Romsdahl 2011; Romsdahl and Pyke 2009).
Parallel with the development of a DSF is growth in broader concepts of scientific knowledge production that combines a plurality of knowledge types to build an integrated understanding of the system and takes account of the social and political context in which knowledge will be applied (Lemnos and Morehouse 2005; Meadow et al. 2015). The process—known as coproduction of knowledge—weaves together ‘three dimensions: interdisciplinarity, interaction with stakeholders, and production of usable science’ (Lemnos and Morehouse 2005, p.58) and involves ‘working iteratively and interactively toward collaborative learning, shared understanding of key concepts, and coevolution of common purpose and action’ (Nel et al. 2016, p.178). Cash et al. (2003), through related concepts of bridging ‘boundaries’, propose that when people are meaningfully involved in producing knowledge, they are likely to consider this knowledge as credible, salient, and legitimate and, in turn, are more likely to use this knowledge for implementation. These three attributes of credibility, saliency and legitimacy have become important benchmarks of effectiveness of the translation of science into policy or practice. However, as Heink et al. (2015) describe, they (and their inevitable trade-offs) are vague and depend on individual framing and thus require more specificity to be useful.
This paper reports on a web-based decision support and information delivery framework, CoastAdapt (www.coastadapt.com.au), developed during 2014–2017 to support coastal decision makers in Australia to adapt to climate change. The development and elements of CoastAdapt (also called here ‘the Framework’) are described by Palutikof et al. (2019a), while the consultation activities that focused on understanding and addressing user needs are described by Palutikof et al. (2019b). Here, we expand on these papers to provide further detail on the involvement of coastal decision makers in developing CoastAdapt. We use four types of feedback—an online survey, consultation workshops, advisory group feedback and participant observations by the development team—to reflect on how these activities supported the Framework to be usable. We provide a theoretical contribution through the analysis of an application of Cash et al.’s (2002) framework and an engagement tool (the IAP2 spectrum; IAP2 2007) in the context of the development of a DSF.
2 Rationale for, and development of, CoastAdapt
CoastAdapt was developed to underpin effective decision-making to support adaptation at all stages, from simple recognition of the risks through to planning, delivery, monitoring and evaluation in the coastal zone in Australia. Projections for Australia indicate a likely increase in coastal hazards associated with rising sea level, such as erosion, storm surge and inundation (McInnes et al. 2016) as well as changes in storm patterns (Christensen et al. 2013). Current estimates of sea-level rise for Australia lie in the range 6–19 cm by 2030 relative to the 1986–2005 baseline (CSIRO and BoM 2016). The Australian coast is particularly vulnerable, as the majority of the population, industry and infrastructure is located in the coastal zone: For example, the most comprehensive study to date estimated that AUD$63 billion in residential housing stock is at risk from 1.1 m rise in sea level (Department of Climate Change 2009).
Following production of two national assessments of climate change risks to coastal Australia (Department of Climate Change 2009; Department of Climate Change and Energy Efficiency 2011), in 2014, the Australian Government Department of the Environment (DoE) commissioned the National Climate Change Adaptation Research Facility (NCCARF) to ‘produce a coastal climate risk management tool to support coastal decision makers to understand future climate risks and provide practical guidance on how to manage the associated physical, social and economic risks’ (taken from the unpublished Funding Agreement).
NCCARF was selected for this project because of its role as a boundary organisation (Guston 1999), connecting science to practice and policy (Graham and Mitchell 2016). Increasingly, the concept of boundary work by such organisations has evolved from separating science from nonscience, to mediation between knowledge and action (van Kerkhoff and Lebel 2006; Clark et al. 2016). Since 2007, NCCARF has been generating, collating, synthesising and communicating knowledge to support decision makers throughout Australia as they prepare for and manage the risks of climate change (NCCARF 2014). For this project, NCCARF was given broad guidance from DoE through the funding agreement that the coastal risk management tool should be useful to coastal decision makers; however, the form and function were to be decided through an extensive consultation and co-production process with potential users. Given that the tool was required to meet the needs of a diverse range of coastal decision makers across all states and territories (and thus different geographic and jurisdictional contexts), online delivery through a DSF was considered the most appropriate mechanism.
Formally launched in May 2017, the coastal risk management tool became known as CoastAdapt and referred to as a framework rather than a tool. CoastAdapt fits Moss’s (2016, p. 146) definition of a DSF: ‘a knowledge framework that structures decision support processes of knowledge exchange to enable learning and experimentation’. CoastAdapt contains information and guidance to increase understanding of climate change and potential responses: Specifically, it includes:
Basic and advanced information and explanations of climate change and its impacts
Datasets on historical flooding, present-day coastal sensitivity to erosion, and future climate extremes, sea-level rise, and inundation for each coastal council
Tools for risk assessment, costing and planning action
Guidance on selection of adaptation actions and building support for implementation
A risk management framework supporting users through the six stages of adaptation from identifying the challenges through to monitoring and evaluation
CoastAdapt included (from April 2016 to June 2017) an interaction space to support a community of adaptors, but this area is no longer active: see discussion in Palutikof et al. (2019a).
In Australia, the key institutions for coastal management lie with state governments, yet local governments are day-to-day managers of the coast through their role in land use planning, asset management and ecosystem management. The national government’s role is the development of strategic policy for the coast (e.g. NRMC 2006), management of World Heritage sites such as the Great Barrier Reef, coastal defence and offshore dumping and shipping (Harvey et al. 2012). Other coastal decision makers include privatised utility providers, NGOs, community groups and businesses or industries that depend on a coastal location or need to access coastal resources or populations. Users of CoastAdapt are broadly defined to include managers in all of these groups, in particular local government and small businesses, which tend to have limited human and financial resources to undertake climate change adaptation. These are the key groups we sought to involve in co-production of knowledge for the Framework.
This paper reports on user involvement processes associated with CoastAdapt development, and how these aimed to contribute to CoastAdapt being usable and used through being credible, salient and legitimate. In doing this, we draw on four types of feedback as sources of data:
An online survey conducted during the scoping phase, which included an open survey online for 3 weeks in March 2015, and completed by 131 individuals (see Palutikof et al. 2019b)
Three rounds of consultation workshops, a total of 13 held in urban and regional population centres across Australia and attended by 330 individuals, conducted throughout the scoping, development and roll out phases (see Palutikof et al. 2019b)
Feedback from three project governance and advisory groups (52 individuals in total) gathered at or after each meeting or workshop through a monitoring and evaluation process managed by an independent consultant (see Sect. 4.1)
Participant observation by the eight NCCARF members of the CoastAdapt development team: a multidisciplinary team including specialists in climate adaptation, coastal science, IT and graphic design.
3 ‘Usable and useful’: building relevance through engagement
3.1 Engagement of users in decision support
The growth of DSFs across many environmental domains has been matched by increased discussion about potential users and their needs (McIntosh et al. 2007; Romsdahl and Pyke 2009; Romsdahl 2011), concerns about low take-up by target users (Valls-Donderis et al. 2014) and the need for user involvement in testing so that frameworks will be usable and used (O’Dea et al. 2007; Romsdahl 2011; Voinov and Bousquet 2010). Increasingly proposed is that developers and potential users should collaborate in DSF design and content, for three principal reasons: First, so that content meets users’ expectations, contains legitimate and relevant information, for example, at appropriate spatial and temporal scales, and can be integrated with existing organisational knowledge systems (Dilling and Lemnos 2011; Valls-Donderis et al. 2014; Meadow et al. 2015); second, so that the process of negotiating the DSF and its content builds trust in the developers and faith in the final product (Lemnos et al. 2012; Valls-Donderis et al. 2014); and finally, so that users become active rather than passive participants, helping to build the DSF community and improve practice through social learning (Jakku and Thorburn 2010; Prokopy et al. 2017). Jakku and Thorburn (2010) define social learning in the decision support context as the ability to support stakeholders to interact, share different views and experiences, learn collaboratively and make collective decisions.
Despite increasing numbers of DSFs to support climate adaptation, few examples describe how these were developed and which or how users were involved. For example, Newman et al.’s (2017) review of 77 papers of natural hazard DSFs identified only 32 papers that reported user involvement. Of these, 11 were steered by user input, 11 evolved with user input, 7 reported end user training and 3 reported using champions. Kopke and Dwyer (2016) developed an online digital atlas of the global coastline and stressed the need to ‘identify potential audiences … at the start of the development process, and to remain engaged with them throughout’. In our reading of the literature, we identified four types of user engagement in DSFs (Table 1). Our aim for CoastAdapt was to use a ‘sustained’ approach that we defined as ‘users participate through a combination of engagement activities including formal references groups, testing, and training’.
3.2 Using the IAP2 spectrum to design user engagement
In designing user involvement for CoastAdapt, we drew on the International Association for Public Participation spectrum (IAP2 2007); see NCCARF (2015a, 2015b). This framework for participation was chosen because, although generally used to describe public engagement, it is also used to describe involvement of users in DSF development (e.g. Valls-Donderis et al. 2014). In addition, it is widely adopted by Australian local governments for planning public engagement and so is familiar to many of our stakeholders (e.g. Serrao-Neuman et al. 2014).
The IAP2 spectrum outlines five levels of participation—inform, consult, involve, collaborate and empower—based on increasing levels of information, consultation and collaboration, with tools and processes suggested for each level (Fig. 1). To these, we added a pre-engagement step termed ‘information gathering’ that we defined as ‘collating existing information to plan informed engagement’. Our rationale for this step was to, first, demonstrably acquire extensive knowledge of current practice, so as to inspire confidence that the team was up to date with current issues and, second, minimise consultation fatigue (Crase et al. 2005; Brackertz and Meredyth 2009), as many of our stakeholders describe themselves as time poor (Palutikof et al. 2019a) and are frequently involved in engagement activities.
3.3 Developing an effective DSF that is salient, credible and legitimate
Boundary organisations typically use boundary objects—collaborative products such as reports, models and maps—to negotiate knowledge between the realms of science, policy and practice (White et al. 2010; Clark et al. 2016). Increasingly, DSFs have potential as boundary objects: They enable users to participate in developing a decision tool, develop a shared understanding and social learning, and yet also adapt the tool for their own context (Jakku and Thorburn 2010). To ensure CoastAdapt usability and usefulness for decision-making, the development team drew on the three attributes identified by Cash et al. (2002, 2003) as essential for the effective translation of science into action—credibility, saliency and legitimacy. Following Cash et al. (2002), credibility refers to the integrity of the information and whether it is considered as authoritative, believable, and trusted. Saliency refers to the relevance of the information to decision makers, particularly in terms of timeliness and local applicability. Legitimacy reflects fairness, particularly of the process used to produce the information and whether it takes account of appropriate values, concerns and perspectives of different actors. These three attributes are interdependent but also require trade-offs (White et al. 2010). For example, providing local information can increase the relevance (saliency) of a DSF to coastal managers working in local government. However, if that information in any way deviates from or contradicts the user’s own perception of their local environment, that will impact upon credibility and legitimacy for that user (Leitch and Robinson 2012).
For a wicked problem such as climate change, it can be difficult to establish these three attributes (Leitch 2017). For example, climate change involves various types of uncertainty, e.g. scientific, epistemological, ethical and scalar uncertainty (Leitch and Robinson 2012), that can make it difficult to maintain credibility. Climate change typically highlights the conflict between different stakeholder values (Adger et al. 2009) and so whose knowledge is included or excluded from the process to produce a DSF can build or undermine perceptions of legitimacy. Climate change impacts depend on the context (Adger and Kelly 1999; Measham et al. 2011), and without resources to undertake an equitable process (legitimacy) to develop locally relevant information, the final DSF may lack saliency, which in turn undermines credibility.
4 The phases of development and engagement activities
The pragmatic goals for CoastAdapt were to have an easily-navigable DSF containing authoritative, relevant and effectively communicated information. It was also important to build users’ capacity to understand more about climate change and adaptation, as well as to plan for and take action, and to create a community of adaptors. Achieving these goals required significant and sustained involvement of potential users in the design and development processes, which meant that the need for adequate interaction was explicitly built into the project design from the beginning (Beiers et al. 2017). Engagement to develop the Framework and its content occurred in four phases—scoping, prototype, usability, and testing and training (which are similar to the four phases of DSF development described by Valls-Donderis et al. 2014).
We now describe the engagement processes of CoastAdapt and how they contributed to usability through support of the credibility, legitimacy and saliency of the content as well as supported action and community building by coastal decision makers. Table 2 provides more detail about the information gathering and engagement activities for CoastAdapt through the three governance committees and the four National Adaptation Networks (NANs).
4.1 Phase 1: scoping
The ‘scoping’ phase began with the ‘kickoff’ meeting (Beier et al. 2017) in November 2014, bringing together a diverse range of potential users and stakeholders to provide input directly as well as to build support through their networks for engagement in DSF development. This meeting discussed the need for a coastal framework and its aims, potential form, function and content.
Next, the CoastAdapt development team sought input from a range of coastal decision makers to better understand their context, needs and aspirations for managing the coast. This occurred through a series of short- and longer-term consultation events and engagement processes, as described below.
4.1.1 Three governance/advisory committees
Beier et al. (2017) recommend establishing technical and steering committees to help guide co-production of actionable science. Three committees were set up to guide the overall NCCARF program within which CoastAdapt development formed a major part. These committees participated broadly at the IAP2 ‘involve’ level as they were engaged throughout CoastAdapt development, mostly through quarterly 1-day meetings. Committee members were selected to represent all coastal jurisdictions in Australia, and from a wide range of experience in policy, practice and/or science, in a process designed to build credibility and legitimacy of CoastAdapt and its contents. Committee members also raised awareness of the Framework among their own stakeholder networks to help develop a community of adaptors.
The three committees are described below:
The Project Review Committee (PRC) consisted of state and sector representatives to provide national oversight of CoastAdapt content. These representatives ensured inclusion of knowledge and priorities from all the coastal state-level jurisdictions. The Australian government through the Department of the Environment and Energy had observer status. The PRC provided a national perspective (legitimacy and saliency). As a governance committee, it was required to endorse key deliverables (credibility), a process that signalled consistency with state policy and was an essential step if local governments were to adopt the Framework.
The End User Reference Group (EURG), an advisory group, consisted of representatives from local government and relevant businesses (i.e. with exposure to coastal hazards, such as utility operators). Members were drawn from across Australia and were selected for their local knowledge and experience to support legitimacy and saliency.
The Technical Reference Group (TRG), also advisory, consisted of relevant experts from the bio-physical and social sciences; their expertise added credibility and saliency.
4.1.2 The National Adaptation Networks
Four NANs were established to connect researchers and research users in government, business and communities, to build and maintain capacity to adapt to climate change. Each NAN focused on a different domain: Natural Ecosystems; Settlements and Infrastructure; Social, Economic and Institutional Dimensions; and Vulnerable Communities. Although they had a role independent of CoastAdapt, their involvement and input helped ensure saliency: They contributed content relevant to their different interests and sectors and also raised awareness within their networks to help build the community of adaptors.
4.1.3 Scoping processes
Three activities in the scoping phase aimed to elicit information on key problems for coastal managers caused by sea-level rise and climate change; understand what is needed to support coastal zone management under climate change and sea-level rise; and determine how a coastal DSF might help address these challenges. These were a desktop review or State of Play report (SOP), an online survey and a series of regional consultation workshops (supplementary information Table SI-1).
The SOP (NCCARF 2015a) collated and reviewed existing science, adaptation activities and regulatory frameworks, to ensure the saliency of CoastAdapt information and grow the development team’s knowledge (credibility). The draft report was reviewed by the PRC, EURG and TRG.
An online survey solicited user views on knowledge gaps and barriers to adaptation, in order to better understand where the focus of CoastAdapt should lie. It was distributed via the NCCARF website, the NCCARF mailing list of approximately 5000 recipients, the PRC, EURG and TRG, and NANs. There were 313 completed responses (Palutikof et al. 2019b).
Eleven scoping workshops with coastal decision makers were held across Australia, with 330 participants (Palutikof et al. 2019b). To encourage attendance and lend credibility, a local champion provided advice on potential participants and suitable timing, venue, and promotion. These champions also played a key role in the workshops through presentations on local issues and the local potential for a DSF.
4.2 Phase 2: prototype
The ‘prototype’ or development phase (whereby ‘an early concept of the look and feel of the Tool is developed and tested by the stakeholders’ (Valls-Donderis et al. 2014, p.73)) established the content and aesthetics of CoastAdapt (supplementary information Table SI-2). The PRC, EURG and TRG met regularly to discuss the evolving content of CoastAdapt. Some TRG members prepared technical, in-depth content including information manuals (e.g. on community engagement, coastal engineering and coastal planning). The NANs organised several capacity building events—mainly workshops for early career researchers and practitioners—which inter alia prepared and/or reviewed draft content.
4.2.1 Tool development partners
A Tool Development Partnership (TDP) was established as an advisory group of local-scale practitioners to ensure CoastAdapt relevance and accessibility. Membership was drawn from nine coastal local governments and one agricultural peak body. Members were selected through a competitive process, and their host organisations received a small grant to compensate for their time and ensure support for their continued involvement. Through contributing their knowledge of local context, organisational and practitioner issues, challenges and opportunities, the TDP made an important contribution to CoastAdapt salience and credibility.
The TDP met for 2 days approximately every 2 months during the prototype and usability (Sect. 4.3) phases to discuss progress, review content and contribute to user experience (UX) testing. This group shared deep experience with each other and the development team—perhaps more than was the case for the more formalised PRC, EURG and TRG. This deep interaction was attributed to the lack of a formal role with no terms of reference: The lack of an externally driven agenda allowed the issues that mattered to the group to emerge. Nevertheless, participation was at the involve level for the TDP—the same as for the three governance/advisory committees and the four NANs. In post-meeting feedback gathered by the independent consultant (Coutts and Coutts 2017), TDP members were very positive about their involvement and the resulting opportunities they experienced for national-scale networking at a level rarely available to local government and small business employees.
4.2.2 Case studies
The development team sought examples of ‘adaptation in action’ from the coastal community through case studies (and shorter ‘snapshots’) of adaptation that illustrated lessons learned, effective strategies and/or potential pitfalls. TDP members each contributed a written or video case study of local adaptation within their own organisation. Other individuals, groups and organisations were also encouraged to contribute such examples through offers of prizes and recognition. The case studies—currently around 80—aimed to foster a culture of taking action and increased the credibility and relevance of CoastAdapt by providing examples of ‘real adaptation’ already underway.
4.2.3 Monitoring current topics of interest
Through the consultation workshops, users expressed the need to digitally share knowledge and experience through on-line forums to support communities of practice (see NCCARF 2015b). An online forum, known as CoastExchange, was established in response. Over its 14 months of operation, it provided opportunity for moderated discussion, and carried 67 news and feature articles prepared by a communication consultant. It also encouraged users to pose questions each month that could be selected for expert responses: 27 questions were answered by eight experts. In addition to these roles, CoastExchange provided an opportunity for the development team to monitor topics of current interest to users (increasing saliency).
4.3 Phase 3: Usability
The usability phase—the Framework being ‘improved in terms of its appearance and ease of use’ (Valls-Donderis et al. 2014, p.73)—followed launch of the beta version of CoastAdapt in August 2016 and involved review of and refinement to content and usability (supplementary information Table SI-3). In this phase, the TDP changed to a Tool Implementation Partnership (TIP) that included six additional organisations and had an increased emphasis on outreach for CoastAdapt and mentoring for users.
Review and consultation on the beta version of CoastAdapt involved further feedback workshops, conducted in 19 city and regional locations (and where possible in the same locations as the scoping workshops). At each workshop, a TIP member presented on their adaptation experiences and the uptake of CoastAdapt in their own organisation, which added credibility and saliency to the Framework and its contents. A web-based survey also elicited feedback from users. Additionally, all pages on CoastAdapt had a feedback tab enabling users to provide comments. Major changes to CoastAdapt resulting from this beta version feedback are reported by Palutikof et al. (2019b).
4.3.1 Test cases
Test cases were carried out in two phases: usability and testing (see Sect. 4.4). In the usability phase, test cases contributed to the review process and led to modifications to CoastAdapt. Groups worked with an independent facilitator to apply CoastAdapt to a specific problem over 6 weeks and provide feedback and a case study of their experience. A small budget was provided to cover involvement costs. There were seven test cases in the usability phase: five councils, an airport operator and an aquaculture company. While feedback was positive overall, there were a number of suggested improvements on Framework utility and navigability, which were implemented where time and resources permitted (Palutikof et al. 2019b). This activity provided legitimacy through testing CoastAdapt in diverse contexts and credibility through the independence of the process.
4.4 Phase 4: testing and training
A key milestone of this phase, and a contractual requirement, was the endorsement of the final Framework by the PRC prior to final release. This meant that state governments would support decisions based on the Framework, which enhanced its credibility and saliency for local governments.
Six training workshops were held in state capitals (supplementary information Table SI-4), plus one training webinar to enable remote involvement at any time. Participants worked through two exercises: first, a first pass risk assessment for an imaginary coastal asset (see Tonmoy et al. 2019); second, an exercise in stakeholder mapping, prioritisation and engagement.
Eight further test cases were carried out in five councils, one grouping of councils, one aquaculture company and a credit union. These had the primary role of further demonstrating the utility of CoastAdapt, and so building its saliency among potential users. These test cases also produced case studies published in CoastAdapt.
Finally, this phase included the formal launch of CoastAdapt in May 2017, an activity additional to those of the testing phase outlined by Valls-Donderis et al. (2014). The event aimed to promote CoastAdapt to stakeholders considered key influencers among policy makers and practitioners, as well as to acknowledge and celebrate the involvement of the many contributors to the Framework.
5 Reflections on the engagement processes
Cash et al. (2002) described how knowledge into policy requires satisfying conditions of credibility saliency and legitimacy. We now reflect on how each of these attributes was relevant in development of CoastAdapt.
5.1 Ensuring credibility of the Framework creators and content
Establishing the credibility, or scientific integrity, of the development team and CoastAdapt content was a vital ingredient for project success. For example, a key need expressed by junior council staff during consultation was for advocacy—they wanted to use CoastAdapt to convince senior managers of the need for adaptation planning—the scientific integrity of CoastAdapt information and its developers is essential if it is to be used for this purpose.
Establishing credibility began at the pre-engagement phase. An important factor was the reputation of NCCARF as a climate adaptation knowledge brokering facility hosted by Griffith University. Stakeholders indicated that the ‘brand’ of NCCARF and Griffith University encouraged them to participate early in Framework development because they understood that an established and respected research facility would have buy-in from other recognised experts. Credibility was also enhanced by the pre-engagement preparation of the SOP report (NCCARF 2015a), which assembled the current state of knowledge on coastal issues nationally. This ensured that the team’s knowledge was current, aiming to inspire stakeholder confidence in the team’s competence and minimise potential frustration or consultation burnout. In later phases, local champions and the TDP/TIP members—as trusted local practitioners—lent credibility to the team through being involved in the workshops and explaining the local relevance of CoastAdapt.
Credibility of the content was supported during writing and review. Topic experts from academic, policy and practitioner backgrounds were engaged to write material and develop data sets and tools. An extensive peer review process followed with all material reviewed by a technical expert to ensure accuracy, a practitioner to ensure relevancy, and a communication specialist to ensure clarity and consistency. While a large logistical task for a small team, this review process was considered essential for credibility and saliency. Also, for a development team, an awareness of ensuring credibility is likely to motivate team members to sustain a focus on engagement (White et al. 2010).
One issue that emerged consistently across all consultation phases was the longevity of CoastAdapt. At all workshop and events, participants expressed concern that continuing support was uncertain. Without sufficient support, it would not be possible to carry out the maintenance and up-dating that ensures the continuing credibility of the Framework. Financial support to maintain and update CoastAdapt continues to be available. This is not assured beyond the middle of 2019, although plans are in hand to ensure that a library of CoastAdapt content is maintained should the structure of the Framework be discontinued. Concerns around longevity of publicly-available datasets and resources are common across the natural and physical sciences (see, for example, Moran et al. 2009).
5.2 Ensuring saliency of the Framework through relevance and accessibility
The key to ensuring saliency was facilitating input from diverse stakeholders from a wide range of organisations, job roles, locations and experience in coastal adaptation so that CoastAdapt would be widely relevant. The development team identified that the most effective engagement mechanisms were the consultation workshops and online survey during the prototype phase; the UX testing during the prototype phase; and the TDP/TIP input during the prototype and usability phases.
The consultation workshops and online survey gathered views from coastal managers and communities across Australia, highlighting their differences (and similarities) in perceptions of climate change risks, and thus in the potential value of a DSF that could provide content to satisfy a broad range of users. Palutikof et al. (2019b); Table 1) describes these perceptions and responses in detail and how some of these views helped to shape design and content. Other views raised issues that were more challenging but important to address if CoastAdapt was to be relevant nationally. For example, internet connectivity, while not an issue for capital cities, poor internet connections in some remote communities would restrict access to the moderately data-intensive Framework. In addition, whereas CoastAdapt is a national framework seeking for national consistency, planning issues in Australia are handled at state level, leading to concerns that CoastAdapt information might not match the regulatory and legislative contexts of individual states. NCCARF worked to overcome these concerns, for example, by providing each state with the opportunity to contribute a webpage to CoastAdapt populated with relevant state-specific information.
The UX testing was vital to support CoastAdapt navigability and accessibility. Its development of profiles of typical users, or ‘personas’, helped to refine an understanding of users through the formal testing process. The development team also reported value from in-team discussion of refining these personas.
The TDP was established to help ensure the relevance and accessibility of CoastAdapt. Members’ local scale and pragmatic input, and the trust that developed through their prolonged involvement throughout CoastAdapt scoping and roll out, allowed the Partnership to provide insightful and forthright perspectives that the development team reported as invaluable. Its successor, the TIP, directly contributed to saliency through raising awareness of CoastAdapt through its networks, and as trusted independent participants in CoastAdapt consultation and promotion activities.
One activity that perhaps did not merit the expenditure of resources given its contribution to saliency was the online forum CoastExchange. This forum had 6000 sessions and 27,000 page views over its 15 months of operation (Palutikof et al. 2019a): It was reported as valuable and gaining traction by local government users (Coutts and Coutts 2017). Nevertheless, over its 14 months of operation, the volume of actual discussion on CoastExchange was low: 250 people signed up but most were ‘lurkers’ (i.e. a member of an online community who posts rarely, if ever) with few postings or comments, which is a common feature of online communities (Sun et al. 2014). However, the weekly articles provided a mechanism for awareness raising of contemporary adaptation issues, helping CoastAdapt to remain salient (and therefore supporting its legitimacy).
5.3 Ensuring legitimacy of the Framework through extensive and diverse consultation
The development team aimed to foster the legitimacy of CoastAdapt through broad geographical consultation with diverse stakeholders at NCCARF-organised workshops and in conference sessions. The online survey with around 300 responses—representative of the types of anticipated users for the Framework with respect to their location, place of work and job level—enabled remote input from anyone with an interest in coastal adaptation (Palutikof et al. 2019a, 2019b). Thirteen workshops held across seven capital cities and four regional centres (330 participants) canvassed views of attendees, including academics, state and local government officers and elected representatives, consultants and students. Where possible, consultation workshops in the scoping and usability phases were held in similar locations to enable NCCARF to report back to participants on their original feedback. Legitimacy was also facilitated through sustained engagement with the PRC, EURG, TRG TDP/TIP and NANs throughout the process.
A final ‘check’ of legitimacy (and also saliency) was through final signoff by the PRC, which signalled a commitment to CoastAdapt by state and federal agencies. Obtaining this final approval meant that the information relevant to each state or sector was relevant to that jurisdiction and therefore ‘endorsed’ as legitimate.
5.4 Using the IAP2 spectrum
Using the IAP2 spectrum helped the development team to work through the purpose of consultation and engagement activities. Most of these activities lie in the ‘inform’ to involve area of the spectrum (Fig. 1), with the PRC, EURG, TRG and TDP/TIP groups having the highest participation level (see supplementary information Tables SI-1–4). This level of engagement is common in DSF development (Valls-Donderis et al. 2014; Fig. 1). Also, while individual workshops are at the ‘consult’ level, returning to the same locations to report back on feedback and get further guidance from the same community nudged them into the involve level of higher contribution. This observation highlights the importance of considering activities as both individual and cumulative events in terms of their engagement effect.
This paper reports on the user involvement process of developing and implementing CoastAdapt, and how this process contributes to the Framework being usable through being credible, salient and legitimate. Through reflecting on the process, we can make some practical suggestions that may provide some guidance on processes for user involvement in development of a DSF.
First, establishing a strong foundation for the process is essential. For CoastAdapt, NCCARF’s existing reputation, together with the pre-engagement activities of the scoping phase—researching and writing the SOP review—were important factors in building credibility. Similarly, so did the strong foundation for the content through extensive consultation of user needs, and preparation and review by technical, practice/policy and communication experts. This strong foundation facilitated on-going support from local champions and strong willingness to participate in engagement activities.
Second, the process of stepping through engagement phases provided the opportunity to tweak activities at each new phase. For example, in the prototype phase, the TDP was very effective in supporting development of salient content so that in the usability phase, this group could be expanded to become the TIP to assist roll out of CoastAdapt and therefore support legitimacy.
Third, strong and transparent governance and advisory structures build stakeholder trust and understanding (credibility). By soliciting participation in governance and advisory groups from governments, business, technical and local scale interests, NCCARF attempted to incorporate diverse views to ensure broad legitimacy and saliency. The longer-term engagement provided through these groups complemented the shorter-term workshop activities, enabling us to better understand stakeholder needs, thus contributing to credibility and saliency.
Finally, the endorsement of CoastAdapt by state and national government was a key consideration in the Framework’s perceived legitimacy. Legitimacy reflects fairness, particularly of the process used to produce the information and whether it takes account of appropriate values, concerns and perspectives of different actors. Without this formal endorsement, local councils would not be encouraged to contribute to and use CoastAdapt.
It is well established that the attributes of credibility, saliency and legitimacy are interrelated (Cash et al. 2002), and this study confirms the finding that a focus on one attribute may support or compromise one of both of the other attributes (Cash et al. 2002; White et al. 2010; Kirchhoff et al. 2013). We found that engagement activities that built credibility also fostered saliency and legitimacy. For example, facilitating multiple rounds of workshops across Australia showed a genuine commitment to work with, and integrate, diverse knowledge and disparate views of a range of coastal stakeholders. Such intensive efforts can both build credibility as well as co-produce content that is salient.
Conversely, we found that issues that affected saliency could also undermine credibility. For example, where local knowledge conflicted with information provided in CoastAdapt, the credibility of the Framework was questioned. Similarly, for remote communities where there was little or no local scale data in CoastAdapt (reducing legitimacy), users could form the opinion that the rest of the content would not be salient, despite alternative information and ‘work arounds’ provided for these situations. This demonstrates the importance of providing information within CoastAdapt that matches the experience of users and is applicable in their local context.
As the need grows to deliver effective online DSFs to support climate change adaptation, so does the need to involve potential users in their development. It is important to document and reflect on the experience of engaging with potential users to enable sharing and learning among the DSF development and stakeholder communities, including funding agencies. This paper advances knowledge practically and theoretically by outlining our approach and documenting our reflections. The IAP2 spectrum is a well-used approach in community engagement processes and therefore has traction with many of our stakeholders. Although less used in development of decision support systems, nevertheless, we found that it helped the development team as a heuristic tool for our consultation activities as individual events and also cumulatively. It also helped us sharpen our understanding of our own role as knowledge brokers.
Cash et al.’s (2002) framework of credibility, saliency and legitimacy is well-established, and helped us consider our consultation and how activities could contribute to a useful CoastAdapt. The three attributes are each important and require consideration separately and together, because activities, when carefully implemented, can simultaneously address more than one. Furthermore, future research that probed CoastAdapt users’ perceptions of these attributes within the context of the Framework would be valuable.
Finally, it is important to consider the future of CoastAdapt. The attributes of credibility, saliency and legitimacy are dynamic, and so any future improvements to CoastAdapt will need to ensure that these attributes are considered. As the adaptation community expands and matures, what is considered to be relevant is also likely to expand. However, growth of DSF content and complexity comes with increasing challenges of accessibility which, if not addressed, will undermine credibility.
Adger WN, Kelly PM (1999) Social vulnerability to climate change and the architecture of entitlements. Mitig Adapt Strateg Glob Chang 4:253–266
Adger WN, Dessai S, Goulden M et al (2009) Are there social limits to adaptation to climate change? Clim Chang 93:335–354. https://doi.org/10.1007/s10584-008-9520-z
Beiers P, Hansen LJ, Helbrecht L, Behar D (2017) A ‘how-to guide for coproduction of actionable science. Conserv Lett 10:288–296. https://doi.org/10.1111/conl.12300
Brackertz N, Meredyth D (2009) Community consultation in Victorian local government: a case of mixing metaphors. The Australian Journal of Public Administration 68:152–166. https://doi.org/10.1111/j.1467-8500.2009.00627.x
Cash D, Clark WC, Alcock F et al (2002) Salience, credibility, legitimacy and boundaries: linking research, assessment and decision making. John F. Kennedy School of Government, Harvard University, Cambridge
Cash DW, Clark WC, Alcock F et al (2003) Knowledge systems for sustainable development. Proc Natl Acad Sci U S A 100:8086–8091. https://doi.org/10.1073/pnas.1231332100
Christensen J, Krishna KK, Aldrian E et al (2013) Climate phenomena and their relevance for future regional climate change. In: Stocker T et al (eds) Climate change 2013: the physical science basis. Contribution of Working Group I to the Fifth Assessment Report of the Intergovernmental Panel on Climate Change. Cambridge University Press, Cambridge, pp 1217–1308
Clark WC, Tomich TP, van Noordwijk M et al (2016) Boundary work for sustainable development: Natural resource management at the Consultative Group on International Agricultural Research (CGIAR). Proc Natl Acad Sci U S A 113:4615–4622. https://doi.org/10.1073/pnas.0900231108
Coutts J, Coutts R (2017) NCCARF phase 2 final evaluation report, October 2017. Available on www.nccarf.edu.au. Accessed 6 Sept 2018
Crase L, Dollery B, Wallis J (2005) Community consultation in public policy: the case of the Murray-Darling basin of Australia. Aust J Polit Sci 40:21–237. https://doi.org/10.1080/10361140500129990
CSIRO, BoM (2016) State of the climate 2016. https://www.csiro.au/en/Showcase/state-of-the-climate. Accessed 10 August 2017
Department of Climate Change (2009) Climate change risks to Australia’s coast: a first pass National Assessment. Australian Government, Canberra 168 pp
Department of Climate Change and Energy Efficiency (2011) Climate change risks to coastal buildings and infrastructure: a supplement to the first pass national assessment. Australian Government, Canberra 20 pp
Dilling L, Lemnos MC (2011) Creating usable science: opportunities and constraints for climate knowledge use and their implications for science policy. Glob Environ Chang 21:680–689. https://doi.org/10.1016/j.gloenvcha.2010.11.006
Graham A, Mitchell CL (2016) The role of boundary organizations in climate change adaptation from the perspective of municipal practitioners. Clim Chang 139:381–395. https://doi.org/10.1007/s10584-016-1799-6
Guston DH (1999) Stabilizing the boundary between US politics and science: the role of the Office of Technology Transfer as a boundary organization. Soc Stud Sci 29:87–112. https://doi.org/10.1177/030631299029001004
Harvey N, Clarke B, Pelton N, Mumford T (2012) Evolution of sustainable coastal management and coastal adaptation to climate change. In: Stocker L, Wood D (eds) Sustainable coastal management and climate adaptation. CSIRO Publishing, Melbourne, pp 75–100
Heink U, Marquard E, Heubach K et al (2015) Conceptualizing credibility, relevance and legitimacy for evaluating the effectiveness of science–policy interfaces: challenges and opportunities. Sci Public Policy 42:676–689. https://doi.org/10.1093/scipol/scu082
Holman IP, Rounsevell MD, Cojacaru G et al (2008) The concepts and development of a participatory regional integrated assessment tool. Clim Chang 90:5–30. https://doi.org/10.1007/s10584-008-9453-6
International Association for Public Participation (2007) IAP2 Public Participation Spectrum. Available from http://www.iap2.org/associations/4748/files/IAP2%20Spectrum_vertical.pdf. Accessed 10 August 2017
Jakku E, Thorburn P (2010) A conceptual framework for guiding the participatory development of agricultural decision support systems. Agric Syst 103:675–682. https://doi.org/10.1016/j.agsy.2010.08.007
Kirchhoff CJ, Lemos MC, Dessai S (2013) Actionable knowledge for environmental decision making: broadening the usability of climate science. Annu Rev Environ Resour 38:393–414. https://doi.org/10.1146/annurev-environ-022112-112828
Kopke K, Dwyer N (2016) ICAN—best practice guide to engage your coastal web atlas user community. In UNESCO IOCo (ed.), Paris, France. 35 pp. https://www.oceanbestpractices.net/handle/11329/298
Leitch AM (2017) 18 climate change: local governments adapting to the wicked problem of sea-level rise. In Sipe N, Vella K (eds.) The Routledge Handbook of Australian Urban and Regional Planning, New York USA, pp 231–253
Leitch AM, Robinson CJ (2012) Shifting sands: uncertainty and a local community response to sea level rise policy in Australia. In: Measham T, Lockie S (eds) Risk and social theory. In environmental management: from uncertainty to local action. CSIRO Publishing, Melbourne, pp 117–131
Lemnos MC, Morehouse BJ (2005) The co-production of science and policy in integrated climate assessments. Glob Environ Chang 15:57–68. https://doi.org/10.1016/j.gloenvcha.2004.09.004
Lemnos MC, Kirchhoff CJ, Ramprasad V (2012) Narrowing the climate information usability gap. Nat Clim Chang 2:789–794. https://doi.org/10.1038/NCLIMATE1614
Leskens JG, Kehl C, Tutenel T et al (2017) An interactive simulation and visualization tool for flood analysis usable for practitioners. Mitig Adapt Strateg Glob Chang 22:307–324. https://doi.org/10.1007/s11027-015-9651-2
McInnes KL, White CJ, Haigh ID et al (2016) Natural hazards in Australia: sea level and coastal extremes. Clim Chang 139:69–83. https://doi.org/10.1007/s10584-016-1647-8
McIntosh BS, Seaton RAF, Jeffrey P (2007) Tools to think with? Towards understanding the use of computer-based support tools in policy relevant research. Environ Model Softw 22:640–648. https://doi.org/10.1016/j.envsoft.2005.12.015
Meadow AM, Ferguson DB, Guido Z et al (2015) Moving toward the deliberate coproduction of climate science knowledge. Weather Clim Soc 7:179–191. https://doi.org/10.1175/WCAS-D-14-00050.1
Measham TG, Preston BL, Smith TF et al (2011) Adapting to climate change through local municipal planning: barriers and challenges. Mitig Adapt Strateg Glob Chang 16:889–909. https://doi.org/10.1007/s11027-011-9301-2
Mitchell CL, Burch SL, Driscoll PA (2016) (Mis)communicating climate change? Why online adaptation databases may fail to catalyze adaptation action. Wiley Interdiscip Rev Clim Chang 7:600–613. https://doi.org/10.1002/wcc.401
Moran MS, Hutchinson BS, Marsh SE et al (2009) Archiving and distributing three long-term interconnected geospatial data sets. IEEE Trans Geosci Remote Sens 41:59–71. https://doi.org/10.1109/TGRS.2008.2002815
Moss RH (2016) Assessing decision support systems and levels of confidence to narrow the climate information “usability gap”. Clim Chang 135:143–155. https://doi.org/10.1007/s10584-015-1549-1
Natural Resource Ministerial Council (2006) National Cooperative Approach to integrated coastal zone management: framework and implementation plan. Australian Government, Canberra
Nauta TA, Bongco AE, Santos-Borja AC (2003) Set-up of a decision support system to support sustainable development of the Laguna de Bay, Philippines. Mar Pollut Bull 47:211–218. https://doi.org/10.1016/S0025-326X(02)00407-1
NCCARF, 2014. NCCARF 2008-2013: The first five years, Gold Coast: National Climate Change Adaptation Research Facility. http://www.nccarf.edu.au/sites/default/files/research_content_downloads/NCC030-report%20FA.pdf
NCCARF (2015a) Coastal climate risk management tool: state-of-play report. Griffith University, Gold Coast 204 pp
NCCARF (2015b) Coastal climate risk management tool: analysis of end-user needs. Griffith University, Gold Coast 57 pp
Nel JL, Roux DJ, Driver A et al (2016) Knowledge coproduction and boundary work to promote implementation of conservation plans. Conserv Biol 30:176–188. https://doi.org/10.1111/cobi.12560
Newman JP, Maier HR, Riddell GA et al (2017) Review of literature on decision support systems for natural hazard risk reduction. Environ Model Softw 96:378–409. https://doi.org/10.1016/j.envsoft.2017.06.042
O’Dea L, Cummins V, Wright D, Dwyer N, Ameztoy I (2007) Report on coastal mapping and informatics trans-Atlantic workshop 1: potentials and limitations of coastal web atlases. University College Cork, Ireland 77 pp
Palutikof JP, Leitch AM, Rissik D et al (2019a) Overcoming knowledge barriers to adaptation using a decision support framework. Clim Chang. https://doi.org/10.1007/s10584-018-2177-3
Palutikof JP, Rissik D, Webb S et al (2019b) CoastAdapt: an adaptation decision support framework for Australia’s coastal managers. Clim Chang. https://doi.org/10.1007/s10584-018-2200-8
Prokopy LS, Carlton JS, Haigh T et al (2017) Useful to usable: developing usable science for agriculture. Clim Risk Manag 15:1–7. https://doi.org/10.1016/j.crm.2016.10.004
Romsdahl RJ (2011) Decision support for climate change adaptation planning in the US: why it needs a coordinated internet-based practitioners’ network. Clim Chang 106:507–536. https://doi.org/10.1007/s10584-010-9947-x
Romsdahl RJ, Pyke CR (2009) What does decision support mean to the climate change research community? Clim Chang 95:1–10. https://doi.org/10.1007/s10584-008-9538-2
Sanderson H, Hilden M, Russel D, Dessai S (2016) Database support for adaptation to climate change: an assessment of web-based portals across scales. Integr Environ Assess Manag 12:627–631. https://doi.org/10.1002/ieam.1755
Serrao-Neuman S, Harman B, Leitch AM, LowChoy D (2014) Public engagement and climate adaptation: insights from three local governments in Australia. J Environ Plan Manag 58:1196–1216. https://doi.org/10.1080/09640568.2014.920306
Sun N, Rau PP-L, Ma L (2014) Understanding lurkers in online communities: a literature review. Comput Hum Behav 38:110–117. https://doi.org/10.1016/j.chb.2014.05.022
Tonmoy FN, Wainwright D, Verdon-Kidd DC, Rissik D (2019) An investigation of coastal climate change risk assessment practice in Australia. Environ Sci Pol 80:9–20. https://doi.org/10.1016/j.envsci.2017.11.003
Torresan S, Critto A, Rizzi J et al (2016) DESYCO: a decision support system for the regional risk assessment of climate change impacts in coastal zones. Ocean Coast Manag 120:49–63. https://doi.org/10.1016/j.ocecoaman.2015.11.003
Valls-Donderis P, Ray D, Peace A et al (2014) Participatory development of decision support systems: which features of the process lead to improved uptake and better outcomes? Scand J For Res 29:71–83. https://doi.org/10.1080/02827581.2013.837950
Van Dongeren A, Ciavola P, Viavattene C et al (2014) RISC-KIT: resilience-increasing strategies for coasts-toolKIT. J Coast Res 70:366–371. https://doi.org/10.2112/SI70-062.1
van Kerkhoff L, Lebel L (2006) Linking knowledge and action for sustainable development. Annu Rev Environ Resour 31:445–477. https://doi.org/10.1146/annurev.energy.31.102405.170850
Voinov A, Bousquet F (2010) Modelling with stakeholders. Environ Model Softw 25:1268–1281. https://doi.org/10.1016/j.envsoft.2010.03.007
White DD, Wutich A, Larsen KL et al (2010) Credibility, salience, and legitimacy of boundary objects: water managers’ assessment of a simulation model in an immersive decision theatre. Sci Public Policy 37:219–232. https://doi.org/10.3152/030234210X497726
The views expressed in this paper are not necessarily the views of the Commonwealth of Australia and the Commonwealth does not accept responsibility for information or advice it contains.
CoastAdapt was funded by the Australian Government through the Department of the Environment and Energy.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
This article is part of a Special Issue on 'Decision Support Tools for Climate Change Adaptation' edited by Jean Palutikof, Roger Street and Edward Gardiner.
Electronic supplementary material
Rights and permissions
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
About this article
Cite this article
Leitch, A.M., Palutikof, J.P., Rissik, D. et al. Co-development of a climate change decision support framework through engagement with stakeholders. Climatic Change 153, 587–605 (2019). https://doi.org/10.1007/s10584-019-02401-0
- Climate change adaptation
- Coastal management
- Coproduction of knowledge
- Sea-level rise
- Coastal hazards