1 Introduction

Tree health is an important factor for New Zealand’s economic, social and cultural values. However, as a small island nation, New Zealand’s forest conservation estate and primary production sectors are at risk from invading exotic plant pests (insects and pathogens). Moreover, the scale of these biosecurity threats is escalating alongside the expansion of New Zealand’s trade and tourism industries (Goldson et al. 2015). At the same time, there is a growing recognition that effective biosecurity in this challenged future calls for people to work together in a more coordinated, collective way, using partnership-based approaches rather than command and control approaches (Hellstrom et al. 2008). Successful biosecurity is inherently a collective endeavour. This is particularly true in terms of post-border operations where there are two main aims: (i) to reduce the likelihood of harmful pests and diseases from establishing; and (ii) to reduce or contain the harm from those that have established (MPI 2016). Activities in post-border operations include monitoring and surveillance, incursion response and sustained control. Policy makers and agencies cannot address New Zealand’s biosecurity challenges in these areas without significant goodwill and collective action from Māori 1 and a range of key operational partners and associated stakeholders (including local communities).

A growing challenge for biosecurity management is to manage improved risk communication and engagement (RC&E) strategies that account for the range of different partnership and stakeholder perspectives (Enticott and Franklin 2009; Mills et al. 2011; Marzano et al. 2015). Recent research in this area highlights that agencies must step beyond a narrow technical operational focus that tends towards thinking only of RC&E as one-way delivery of information to engage more meaningfully with partners and key stakeholders and enter into dialogue based on participation, trust and understanding (Kruger 2011; Allen et al. 2014; Moser 2014). This recognises that managing an effective post-border biosecurity system—be it for surveillance, eradication or sustained control—relies on a range of activities that happen at a number of scales. Many activities are technical, but others are more about social processes (including management) and are difficult to observe or measure. Engagement and communication need to be viewed as an important part of the whole process; sharing and improving agencies’ biosecurity intentions, actions and outcomes.

However, many managers do not have tools to involve the array of stakeholders in such a meaningful way. In particular, they do not have tools to easily set out, document and communicate complex pest and disease management programme activities and their intended outcomes (Allen et al. 2017). Against this background we explore the development of a rubric as a design and assessment framework for post-border biosecurity management. Rubrics are a device, originally used within education, to articulate key elements of a task or behaviour that can be evaluated against desired outcomes or demonstration of different levels of competence. Engaging practitioners in the development of rubrics, we propose, enables people working within a complex system (e.g. surveillance or eradication) to articulate and discuss the different social, technical and management dimensions (Allen and Knight 2009). In turn, this leads to a better appreciation of the different parts and how they interact. This contributes towards skills and pathways to help agencies to take an outcomes-based approach to assess and adapt their risk communication and engagement approaches to aid future response processes.

We begin this chapter by introducing the wider biosecurity setting, and the role of risk communication and engagement within that. We then outline our action research approach and introduce rubrics as an assessment tool. We indicate how action research and rubrics can be used in tandem to encourage a group to think more widely about the complex tasks and behaviours they may be engaged in. We then use the example of surveillance systems in biosecurity as a case study. We illustrate how the rubric can be used in practice by outlining how the authorship team tested its application against the potential introduction of myrtle rust 2 (Austropuccinia psidii (G. Winter) Beenken comb. nov.) in New Zealand. We end with a discussion of the benefits and challenges from using a rubric as a thinking technology, as both a process and a product.

2 Improving Risk Communication and Engagement in an Integrated Biosecurity System

New Zealand’s biosecurity system has evolved to operate as a relatively integrated framework. As Jay and colleagues (2003) point out, the development of this system reflects New Zealand’s history as a small island nation that has experienced significant biosecurity threats and problems. Biosecurity is implemented through a risk management system that involves many participants (MPI 2016). It involves different levels of government (national and regional), different biosecurity operations (surveillance, border control and pre- and post-border control) and different biosecurity objectives (control of economically significant pests and weeds, protection of native species and ecosystems, protection of health and the like) all working with some degree of interrelationship.

The Biosecurity 2025 direction statement for New Zealand’s biosecurity system acknowledges a range of key players (MPI 2016). The Ministry for Primary Industries (MPI) is charged with overall leadership of the New Zealand biosecurity system and has a substantial operational role. At the same time, Biosecurity 2025 reminds us that an effective system will also require distributed leadership, in which other participants lead within their own parts of the system including active and general surveillance, incursion investigation and emergency response (MPI 2016). There is a wide range of other key stakeholders. These include other government agencies and Regional Councils (local government). Māori or iwi (Māori tribal groupings) are partners with the Crown through Te Tiriti o Waitangi (1840), kaitiaki (guardians) of New Zealand’s taonga (treasures) and increasingly have statutory roles in the management of natural resources. For any given pest or disease, there will also be a set of businesses and (conservation and production) land managers who have a responsibility and interest in managing risks directly related to their enterprises. Other key stakeholders include researchers (providing knowledge), and a wide set of community and other interest groups who come together to protect what they value.

The need for greater participation of stakeholders and communities in management of the environment and natural resources has become widely accepted in recent years (e.g. de Loë et al. 2009; Lockie and Aslin 2013). There are multiple rationales for this change in communication and engagement practice. It is in keeping with the democratic basis of local government internationally and in New Zealand that people should have an opportunity to take part in the decisions affecting them. Increasing stakeholder input can help ensure that the social and cultural impacts of decisions are considered (Hoppner et al. 2012), and better plans are generated (Burby 2003). There is also a realisation that scientific organisations and regulatory agencies are no longer regarded as the only source of what is to be considered in decision-making, and local and traditional knowledge needs to be recognised and considered as well (Weber et al. 2011).

Risk communication forms a key part of the biosecurity system in New Zealand and internationally, where a linear approach 3 to raising awareness of biosecurity risks is the most commonly utilised approach to increasing preparedness for newly introduced pests or diseases (Jay et al. 2003; Pegg et al. 2012; Perry 2014; Marzano et al. 2017). However, developing a closer interaction between agencies and other actors involved in these more collaborative biosecurity operations requires a different kind of understanding about risk communication and engagement. Typically, such differences from dissemination to interaction in communicating risk are described as one-way and two-way communication processes (Slovic 1986; Breakwell 2000; Frewer 2004). A growing challenge for biosecurity management is to also manage two-way risk communication and engagement strategies that account for multiple stakeholder perspectives (Mills et al. 2011).

Recent research in this area highlights that agencies must step beyond a narrow technical operational focus that tends towards thinking of communication as the one-way delivery of information to engage more meaningfully with stakeholders and take the opportunity to enter into dialogue based on participation, trust and understanding (Kruger 2011; Allen et al. 2014; Marzano et al. 2017). In this model, engagement and communication need to be viewed as an important part of the whole process; sharing and improving agencies’ biosecurity intentions, actions and outcomes. As the continuum depicted in Table 1 points out, a primary difference between communication and engagement depends on whether the intent is to have largely one-way or two-way communication.

Table 1 Seeing communication and engagement as a continuum (adapted from Morphy, n.d.)
Full size table

Our research was initiated via a New Zealand government contestable research funded programme—the Urban Biosecurity Toolkit—designed to deliver improved urban pest eradication biophysical and sociocultural technologies by looking at more targeted and socially acceptable approaches of dealing with biosecurity incursions (Scion n.d.). Two research objectives dealt with technological innovations in pesticide applications and in early detection while a third dealt with sociocultural innovation through agency-based learning. The starting point for this latter objective acknowledged that agency relations with stakeholders and communities relative to incursion response needed to be developed both during “wartime” (eradication and management) and “peacetime” (surveillance) operations. The development of the sociocultural research leading to this book chapter led to a joint MPI and research team project looking at improving risk communication and engagement in surveillance. This project enabled us to jointly reflect on the multiple elements that comprise an effective surveillance system, particularly one that involves partners and other key stakeholders reporting findings.

Our approach followed that of Mills and colleagues (2011), being a careful and considered engagement with agency professionals willing to reflect and learn about how they could create practical improvements in risk communication and stakeholder engagement. Such an engagement enabled views to be shared in a trusted environment that could critically reflect on current surveillance systems. We offered a process for engaging in a joint assessment that involved developing a rubric for identifying the elements of a surveillance system and measures of performance as a product of that engagement. We envisaged that such an integrated assessment could then be used as a device to facilitate a conversation about the performance of a surveillance system for a specific pest or disease concern, bringing in perspectives of other players or partners engaged in surveillance activities or operations.

3 Methods: Using Action Research to Co-Develop a Performance Rubric

Action research was used to guide the overall approach to learning from our case studies (Kemmis 2009; Allen et al. 2014). Action research is an approach that incorporates stakeholders as co-inquirers in processes designed to empower and change a set of circumstances in which a problem is identified. The researcher in these situations often plays a role of facilitator and collaborator rather than an expert observing and documenting phenomena (Kemmis 2012). Action research requires all those involved in the problem setting to improve their reflection and action. This approach links action, reflection, theory and practice to generate a practical solution or set of solutions (Reason and Bradbury 2008).

Our co-inquirers in the development of this performance rubric are agency staff involved in biosecurity operations. They comprise a multidisciplinary “team” of MPI scientific officers engaged in biosecurity surveillance and incursion investigations. Early discussions between the researchers and the team’s manager led to an invitation to support the team in reflecting on and enhancing the agency’s efforts in improving their surveillance systems. This recognised that an integrated system was required that linked both social and technical elements. A key idea behind this research is that one cannot be effective without the other (i.e. coming up with something that is technically very good won’t necessarily be used if people do not like it, and vice versa). It also recognised that the research team brought complementary skills to the interactions in terms of communication and engagement expertise.

A rubric is an easily applicable form of assessment that can also be thought of as a guide or an evaluation tool that lists specific criteria for assessing performance. Rubrics are most commonly used in education and offer a process for defining and describing the important components of work being assessed (Allen and Tanner 2006). They are particularly useful in helping assess complex tasks or behaviours and are typically used by teachers or trainers to assess the competencies of learners. Rubrics offer an ideal approach to assessment that can lead to greater clarity of the area of competence being developed in a learner and therefore a basis for appreciating the desired elements of competence. Our approach was to engage our co-researchers in the design of rubrics that could capture key elements of a system of surveillance that could then be used as a basis for measuring the performance of that system. Co-developing rubrics was effectively a reflective approach to identifying elements of a system in which there was desired improvement. Although the format of a rubric can vary, they all have two key components (Andrade 2000):

  • A list of criteria—or key elements that count in an activity or task; and

  • Gradations of quality—to provide an evaluative range or scale.

Co-developing rubrics helps clarify the expectations that people have for different aspects of performance by providing detailed descriptions of collectively agreed upon expectations. They not only formulate standards for key areas of accomplishment, but they can be used to make these areas clear and explicit to all those with an interest in improving performance. It is important to involve programme participants, in our case MPI biosecurity surveillance and investigation team, in developing rubrics and helping define and agree on the criteria and assessment as something they feel is achievable and within the limits of normal operations. Different people within the system can offer different perspectives of what they do in the overall system to create a more complete picture of operations. This broad involvement increases the likelihood that different evaluation efforts can provide comparable ratings of performance. It is different from a simple checklist since it also describes the gradations of quality (levels) for each dimension of the performance to be evaluated.

Rubrics are often used to assess tasks and behaviours, but many authors argue that they can serve another, more important, role as well: When used by those undertaking the task or behaviour in question as part of a formative assessment of their works in progress, rubrics can instruct as well as evaluate (Reddy and Andrade 2010). Used as part of a practitioner-centred approach to assessment, rubrics have the potential to help learners understand the targets for their learning and the standards of quality for an assigned task, as well as make dependable judgments about their own work that can inform revision and improvement.

We have combined thinking about rubrics with science and technology studies concept of a boundary object. A boundary object is described as a visual representation that connects social worlds (Henderson 1991; Franco 2013). Typically, a free hand drawing or more openly conceptualised thinking platform is used to characterise a boundary object. Such an object enables a move away from rigidity of disciplinary modes of thinking to create a wider systems perspective of a problem situation (Checkland and Poulter 2006; Allen et al. 2017). In our case, the development of a rubric as a boundary object enabled people with different views of different elements of surveillance practice to come together to discuss, challenge and reconcile different appreciations of the same general concern.

We use the example of surveillance as a case study and demonstrate how a rubric can be used to develop an improved understanding around a general surveillance system. This understanding has linked broader social, technical and organisational functions that could then be appreciated as an integrated operational system.

4 Case Study Context

Surveillance is an essential component of New Zealand’s biosecurity systems for the early detection of unwanted organisms and demonstration of freedom from pests and diseases. General surveillance is an important part of post-border pest and disease management. This type of surveillance (also known as passive surveillance and encompassing community surveillance) relies on members of the public, industry groups, plant or animal health professionals and their networks reporting suspected cases of plant or animal disease or the presence of a pest at their discretion (Hester and Garner 2012). General surveillance complements the targeted surveillance programmes managed by MPI as the lead agency for New Zealand’s biosecurity system. As Cacho and colleagues (2012) point out, general surveillance cannot be controlled directly, rather it is activated by community communication and engagement programmes—with effectiveness dependent on a range of factors including pest attributes, the people involved and the wider sociocultural context of the area. While general surveillance has enabled the detection of many exotic organisms, MPI believes that there is room for improvement in how they engage New Zealanders to maximise the benefit of these surveillance systems and the value they offer (Earl et al. 2016).

5 Developing a Rubric

A draft rubric for improving a general surveillance system was developed during two workshops. The rubric was specifically developed from the perspective of how the MPI team could improve their surveillance system. Attendees consisted of two technical leads for the “animal’’ and “plant and environment” sectors and their managers, the project manager, the project executive and two independent engagement specialists. Prior to the workshop all participants were invited to write down and share two or three elements they considered essential to a well-functioning general surveillance system. These were subsequently discussed and collated into nine key elements during the workshop. It was noticed that different people emphasised different elements, depending on their area and experience. For example, some of the participants focused on the quality of inputs and how to get greater consistency of reporting records while others were concerned with the reporting experience of citizen observers and how to tailor reporting channels to suit their needs and enable feedback on reporting. This highlighted that both social and technical components are important to the functioning of surveillance. The rubric enables both to be recognised and evaluated.

The MPI attendees were then involved in defining an evaluative range or scale that could be used to assess performance in each element. Care was taken to formulate these in an appreciative way that encourages people to improve the outcomes of each performance dimension. The scale was defined using the labels: excellent, good and emerging. The workshop participants were then asked to describe how excellence would be defined for each of these elements. This provided an initial description of performance quality, and subsequent descriptions were also developed for “good” and “emerging” quality gradations. An abbreviated summary of the final rubric designed for a general surveillance system is shown in Table 2. This is adapted from the original rubric which looked at a general surveillance system specifically from an MPI perspective. This more generic rubric shown here has been slightly modified so the elements and descriptions can be used for consideration by a wider range of stakeholders.

Table 2 An abbreviated summary of the final rubric developed by workshop participants (July–August 2016) to evaluate a general surveillance programme, modified to be applicable to a wider range of stakeholders
Full size table

The first three elements “Awareness and engagement”, “Appropriate and well-functioning networks” and “Targets at-risk locations, industries and stakeholder groups” assess stakeholder awareness, engagement and to some extent motivation as well as efforts to enhance accuracy. It is assumed that early detection will occur if all relevant stakeholders are vigilant and willing to notify. However, the group identified that within each sector there naturally exists a network of stakeholders with varying levels of expertise who already exchange information about pests and diseases. The element “Appropriate and well-functioning networks” therefore aims to enhance this network to help enable accurate notifications. The element “Timely and accurate notifications” is a technical assessment of notifications made to MPI as the lead agency for biosecurity management. The communication channel between the notifier and MPI is assessed under “Notifying channels”. To be effective, channels must be user-friendly, acceptable by the audience of potential observers and permit easy transfer of information, photos, videos and samples. The ability of MPI to respond effectively to notifications is captured specifically by “Notification data storage, retrieval and management”. The “Resourcing” element looks at funding and other capacity issues such as training and skills. “Cross- and intra- organisational connections” focus on encouraging an awareness not only of direct actors in the system, but also of the importance of linking with a range of more indirect stakeholders. These include people without a direct role—but whose interests might be affected, and a range of related skill roles within key organisations such as policy makers, information technology (IT) teams and communication units. Finally, the performance element “Monitoring, evaluation and reflection” looks to indicate and assess the regular and meaningful evaluation of the surveillance system—involving stakeholders in assessing progress in both social process and technical elements.

All those involved recognise that this assessment (Table 2) represents a first version of a rubric that can be used to illustrate and discuss the key elements of a surveillance system, acknowledge the different actors involved and gain a better understanding of how their collective work contributes and performs to achieve the broader outcomes. In particular, the rubric enables those involved in its development an opportunity to consider a range of technical and social process elements in a system rather than to try and prioritise any one over the other. This increases the possibility that a rubric can be used to measure different areas of activity that contribute to the overall performance of a system even though they are doing different things.

6 Using a Rubric: Assessing the Surveillance System for Myrtle Rust

As a subsequent exercise, we (the authors) used the example of myrtle rust as a case study to examine how a surveillance rubric can contribute to assessment of a surveillance system. When we undertook this activity, myrtle rust had not been detected in New Zealand. Since this chapter was reviewed myrtle rust has been detected in a number of regions in New Zealand. Below we provide some background to the need to protect against the introduction of myrtle rust to New Zealand. This is followed by a brief illustration of how the rubric can be used for assessment, which we ran as a participatory exercise involving the co-authors as a multidisciplinary and cross-organisational team.

6.1 Myrtle Rust Context

The causal agent of myrtle rust (Austropuccinia psidii (G. Winter) Beenken comb. nov.) is an invasive pathogen of global significance that has rapidly expanded its international distribution and host range over the past decade. The pathogen was first described from common guava (Psidium guajava (G. Winter)) in Brazil in 1884 and is believed to be native to South and Central America (Pegg et al. 2014). It was detected in Australia in 2010 and is now established along the east coast from southern New South Wales to far north Queensland (Carnegie et al. 2016). More recently that same invasive strain has been recorded in New Caledonia, Tasmania and Lord Howe Island (Pegg 2016). It was subsequently detected in New Zealand in May 2017 (although this chapter was submitted prior to this discovery).

Myrtle rust is known to have impact on young, developing tissue including infecting juvenile leaves and shoots, floral buds and/or fruit, with level of damage depending on the host (Tommerup et al. 2003; Zauza et al. 2010b). While infection can cause defoliation, twig mortality and abortion of flowers and fruits (Rayachhetry et al. 2001, citing Smith 1935), the rust affects different tissues on different species and some individual Myrtaceae plants have been found to have resistance to the damaging effects of the fungus (Zauza et al. 2010a). For some highly sensitive hosts such as rose apple (Syzygium jambos) plant mortality, including whole tree death, has been reported (Uchida and Loope 2009).

The long-term ecological implications of sustained rust outbreaks and damage are unclear for every host but some Australian experts have warned that severe damage to highly susceptible and vulnerable native species may even lead to extinction (Makinson 2016; Pegg 2016). Some of the more constructive representations of dealing with the disease include identifying and breeding plants with resistance to the disease and managing the disease through destroying infected plants before the disease spreads (Perry 2014). Measures for managing the risk of spread require very strict biosecurity practices (Pegg et al. 2012).

New Zealand Myrtaceae have been known to be potentially at threat from a biosecurity incursion of the rust for many years (Ridley et al. 2000). There is a growing acknowledgement that this will have negative economic, environmental and sociocultural impacts (Ramsfield et al. 2010; Clark 2011), including directly affecting Māori (Teulon et al. 2015). The rust is predicted to be able to survive in nearly all regions of New Zealand although warmer areas are more suitable. It poses a threat to our native myrtles such as rata (Metrosideros robusta), pohutukawa (Metrosideros excelsa), manuka (Leptospermum scoparium) and kanuka (Leptospermum ericoides), as well as eucalypt growers and the honey industry.

As Bulman (2015) notes, the MPI has been active in putting several measures in place to reduce the risk of establishment. Shortly after its discovery in Australia import requirements of whole plants and cuttings from Australia were tightened. Cut flowers and foliage of the Myrtaceae family from New South Wales, Queensland and Victoria have been prohibited from importation into New Zealand due to the risk of transmission, and in February this ban was extended to Tasmania in immediate response to the discovery there (Bulman 2015).

6.2 Using a Rubric for Assessment

We (the chapter authors) brought an interdisciplinary and cross-organisational perspective to using the rubric—taking myrtle rust as our working example. We stress that our results are only intended to be indicative and were undertaken to provide a framework to help us think about the assessment process in practice. We used an iterative and facilitated approach. We began with those of us most knowledgeable about myrtle rust beginning the process and then involved the remaining co-authors in subsequent sessions that created further discussion and filled the table out more completely (see Table 3). We also shared successive drafts of this paper which enabled everybody to see where the discussion and table had got to in each iteration, and also provided opportunities for discussions on contested areas. The only guidance we used for our contributions into our example assessment was to: (i) look at the guide provided in the general surveillance rubric; and (ii) think of an example and indicator that could be used to demonstrate performance in that general area.

Table 3 Assessment of the myrtle rust surveillance system in New Zealand (undertaken before its 2017 identification in the country)
Full size table

This initial exercise provided us with an appreciation of the utility of using a rubric to develop a discussion around the wider surveillance system. The framework proved useful in enabling different people (from our different stakeholder groupings) to add in a range of activities that they knew about, and collectively this helped everyone gain a better appreciation of the bigger picture. The approach supports an appreciative inquiry approach by asking people to think about an activity element and then to identify specific actions that they were aware of. They are also asked to provide evidence of those actions. Discussions around the validity of what constitutes evidence provide the opportunity for those involved to assess how well (or poorly) an action is being implemented. In turn, this enabled people to start their discussions about the bigger system with a more grounded understanding of what different groups were doing. As those involved repeat these assessments (and compare them), they gain an opportunity to identify and track where key activities may be reducing over time.

Our exercise served to highlight that rubrics will always need to be tailored to the context and the people involved, and be part of an ongoing process. For example, from our initial workshop we had written the first element as “awareness and motivation”. However, we found that the term motivation meant very different things to different people—and so was difficult for people to agree on the level of performance. For this exercise, we changed motivation to engagement, which seemed to work in a more complementary way with awareness. The term motivation, in hindsight, seems to be better thought of as an outcome of awareness and engagement. If new stakeholders are to be involved, they will need to have the opportunity to redefine the rubric through these types of dialogic discussions. In this way, they will often be able to add to the performance descriptors and create a richer picture of how the system is operating—bringing in the perspectives of different cultural and knowledge systems. It helped us to collectively raise our awareness of these challenges to collaboration early in the programme, and in so doing we have begun working on ways to provide for better communication across different stakeholder groups, and foster a more coordinated approach to collective action.

7 Discussion

Through this process, a number of benefits of using rubrics to help design, evaluate and improve surveillance systems began to emerge. Although rubrics are a comprehensive performance measure for use with complex systems and behaviours, they are easy to use and explain. They help multiple stakeholders make sense of how a range of different elements fit together in one system from different perspectives. This helps experts in different areas appreciate the importance of technical, social and organisational aspects—and how they link together. In turn, stronger collaborations support the range of research disciplines and end users to engage more effectively in discussions around the different areas involved. The early indicators of progress in these endeavours are supported by our reflections as a multi-author writing team who collectively cover agency, Māori and different disciplinary perspectives.

In this regard, rubrics should be seen as both a process and a product (Vogel 2012; Taplin et al. 2013). Their development involves practitioners and stakeholders in a facilitated dialogic process of analysis and reflection about the system in question. At the same time, the inquiry results in a table (or rubric) that articulates the key elements and their assessments for the project team and stakeholders. Developing a rubric should not be a one-off exercise to be used in the design (or evaluation) phase of a biosecurity initiative, but implies that those involved are entering into an ongoing process of learning and adaptive management that continues throughout the life of the initiative (Ison and Russell 2011; Cook et al. 2010).

Currently existing biosecurity programmes often fail to effectively engage their key stakeholder groups and emphasise one-way and top-down communication approaches that tend to see engagement as additional to other programme areas (Kruger et al. 2009) rather than embedded within them. The use of rubrics provides a tool that can help address this and provides a framework to guide more two-way or dialogic communication that is required to support more participatory and partnership modes. Developing the rubric helps people understand the bigger picture, and the way in which assessments are conducted invites people to explain in objective terms what is happening from their perspective, and supports an outcomes orientation.

Similarly, biosecurity programmes often lack participatory monitoring and evaluation components that could show the way to more effective engagement (Ison and Russell 2007; Kruger et al. 2009). Few biosecurity system surveillance evaluations provide any guidance or tools that help understand key stakeholder perceptions and expectations, or how to acknowledge the efforts of members of the public (Calba et al. 2015; Hester and Cacho 2017). As performance frameworks, rubrics such as that illustrated in this paper provide more informative feedback about strengths and areas in need of improvement than traditional forms of assessment do. A well-formulated rubric supports a partnership approach by helping stakeholders articulate system shortcomings in a concrete way—and provides guides to look for improvement, as well as ways in which elements are well managed. System practitioners and their partners can learn from developing and using a performance framework in a way they cannot learn from just measuring outputs or other narrow performance measures. Some newer evaluation frameworks take a more comprehensive approach which includes the need for more participatory approaches (e.g. Muellner et al. 2016), and in these cases rubrics can provide a useful tool to engage stakeholders in some of the needed conversations.

Our project took a broad view of evaluation as a starting point for helping the MPI team think about how to assess the wider surveillance system they operated within. The literature on evaluating surveillance systems is, in the main, limited to an assessment of one or two key elements in the wider system (Drewe et al. 2012). There is also a lack of consideration of the sociological aspects that may be involved for any particular setting (Calba et al. 2015). While an effective surveillance system is one that enables early detection, this effectiveness is most commonly only assessed after an incursion has occurred. There is, for example, little attention in the literature as to how we might demonstrate the presence (or lack of) appropriate surveillance capability. Measuring general surveillance during “peace time” is more difficult and is often done by measuring the quantity of notifications. However, number of notifications does not by itself provide a useful indication of vigilance across key stakeholder groups. The rubric element of intra-organisational connections (Table 2) provides an alternative point of evaluation which encourages us to look at the capacity of the networks to actively contribute to a surveillance system, and the quality of those networks to effectively detect an incursion. An evaluation framework which encompasses the multiple aspects of general surveillance was therefore helpful for those looking for appropriate performance measures that could be used and reflected on as achieving desired outcomes.

Moreover, for dynamically evolving contexts in which the effectiveness of a performance cannot be known in advance it is important to develop draft rubrics and then to periodically revise them. The context of our rubric development has been one involving different disciplinary perspectives and different organisational capacities coming together to articulate the many elements that make up a surveillance system. This gets away from a tendency to prioritise one element over another and recognises that the system works because so many elements contribute to its effective performance. We are not only involved in defining the elements of such a complex system with each other but are then able to use the development process to engage others in a broader assessment of the performance of that system. In our case, we have used the development of the rubric as a “thinking technology” where we have reflected on the process and product of rubric development. Here, we have found that the discussion (process) that goes into the development of the rubric is as important as the rubric itself once developed (product). In fact, we have found that the rubric acts as a boundary object or technology that can be used to mediate an ongoing conversation about performance (Ison and Russell 2007; Franco 2013), including discussion of what is desired—as well as discussion of different ways of achieving desired outcomes.

In these ways, rubric development can open up robust conversations about the way we see our biosecurity systems in the world and provide a space for people to offer evidence about the way these systems work. When people in a multi-stakeholder group demonstrate that they can hear the different perspectives in the group, then they are building capacity for trusting those they interact with. In this way, we create the likelihood that our diverse partners can see that they are being heard and included in the framework for system design and performance measurement. Effective risk communication ideally results from engagement with the key communities that you want to involve before, during and after emergency responses and involving them in the discussion on choices about a range of safety and wider surveillance options.

8 Final Comments

Rubrics help provide a means for reaching a shared understanding of what matters, and how to assess that in terms of what can be confidently regarded as good practice—and equally what can be agreed on as emerging practice. As Allen and Knight (2009) state, the process is neither complicated nor unduly time consuming, and benefits of collaborating are available to all participants. We have engaged with current literature and approaches to biosecurity risk communication and engagement. Through this we have recognised the need for tools that can support a range of engagement practices that can communicate complex pest and disease management programme activities and their intended outcomes. We have used a participatory action research approach to the development of rubrics as a design and assessment approach. As a tool, the development and application of a rubric can help agencies move beyond a narrow operational focus that deals with technical aspects to engage more meaningfully with partners and stakeholders and enter dialogue based on participation, trust and understanding. This can be seen to have contributed towards skills and pathways to help agencies use rubrics to assess and adapt their risk communication and engagement approaches.

Our approach sees the product of interactive processes as worthy of reflection, highlighting that processes are generative and open to review. A useful product can be operationalised but it also needs to be open to scrutiny at appropriate times (e.g. when engaging new stakeholders). A remaining challenge is to get agencies and other key stakeholder groups to see rubrics as both process and product and to move beyond a metric of evaluation to increase capacity to work more collectively. In turn, this will require operational biosecurity teams to move beyond their current focus on technical expertise to also include people with skills in surfacing other perspectives, listening and actively engaging with a range of partners.

Notes

  1. 1.

    The indigenous people of New Zealand.

  2. 2.

    Myrtle rust has been detected in New Zealand subsequent to the completion of the workshops described here and the development of the accompanying tables included in this paper.

  3. 3.

    A linear approach refers to the one-way dissemination of information or knowledge that fails to appreciate that audiences are not a “blank slate” to have ideas written on but bring their own experiences, values and judgements to understanding risk, through which new information is interpreted.