Introduction

Education and research are necessary, but insufficient, to solve sustainability challenges (Brundiers & Wiek, 2011; Wiek et al., 2012). As such, sustainability science emphasizes the need to effectively bridge the science-societal divide. Sustainability science is “a field of research dealing with the interactions between natural and social systems, and with how those interactions affect the challenge of sustainability” (Kates, 2011, p. 19,449). Ideally, sustainability science involves scientists and non-scientists engaging in transdisciplinary problem-solving research and application to catalyze the transition towards sustainability (Brundiers & Wiek, 2011; Kates et al., 2001; Lang et al., 2012; Wiek et al., 2012). As sustainability science programs and researchers are customarily based in higher education institutions (HEIs), attention has focused on partnerships with non-academic actors as a mechanism for operationalizing transdisciplinary solution-based inquiry at the science-action nexus (Bieluch et al., 2017; Brundiers & Wiek, 2011; Spangenberg, 2011; Yarime et al., 2012).

The location of sustainability science programs and researchers in HEIs presents several challenges. Sustainability science requires transdisciplinarity, thus active collaboration with a variety of societal stakeholders throughout the process, especially those outside of academia, is necessary (Bieluch et al., 2017; Brundiers & Wiek, 2011; Buckley & Michel, 2020; Dedeurwaerdere, 2013; Lang et al., 2012; Yarime, 2011; Yarime et al., 2012). Cultivating such partnerships is complicated and impacted by social, psychological, and contextual variables (Bieluch et al., 2017), with critical discussions underway about stakeholder involvement practices and ideals in relation to science (Mielke et al., 2016, 2017). Training in key sustainability competencies and pedagogical approaches for transdisciplinary research is scarce (Dedeurwaerdere, 2013; Michel, 2020; Miller et al., 2014). Moreover, institutional arrangements (e.g., structures, rewards, behavioral norms) are entrenched (Brundiers & Wiek, 2011; Dedeurwaerdere, 2013; Komiyama & Takeuchi, 2006; Miller et al., 2014; Vincent & Mulkey, 2015; Yarime et al., 2012).

The evolving community-oriented mission of HEIs affords a valuable opportunity for sustainability science (Buckley & Michel, 2020). Historically, communities were considered sources of needs or passive recipients of knowledge (Bringle et al., 1999). However, during the past two decades community engagement has become increasingly central to the core mandate of HEIs (Bawa & Munck, 2012; Fitzgerald et al., 2012; Tremblay, 2017; Yob et al., 2016). Community engagement involves “the process of working collaboratively with and through groups of people affiliated by geographic proximity, special interest, or similar situations to address issues affecting the well-being of those people” (Centers for Disease Control and Prevention [CDC], 1997, p. 9). We further focus on HEI–community partnerships, which are defined as “…dynamic relationships among diverse actors, based on mutually agreed objectives, pursued through a shared understanding of the most rational division of labour based on the respective comparative advantages of each partner” (Brinkerhoff, 2002, p. 21; see also Butcher et al., 2010). HEI–community partnerships offer important leadership for addressing environmental concerns and enabling sustainability (Brundiers & Wiek, 2011; Mosier & Ruxton, 2018; Shiel et al., 2016). Transdisciplinary partnerships, indicative of sustainability science, are particularly powerful (Bieluch et al., 2017). Close alignment of sustainability with the core mission of HEIs offers the “the potential of uniting education, research and societal contributions to form a systematic and integrated response to the sustainability crisis” (Yarime et al., 2012, p. 102; see also Brundiers & Wiek, 2011; Buckley & Michel, 2020).

Whereas HEIs have historically been concerned with accomplishments regarding research, teaching and learning (Farnell, 2020), their relationship with the community is garnering attention as an important aspect to assess performance (Brundiers & Wiek, 2011). Often referred to as the ‘third mission’ of HEIs, these partnerships are distinct from past interactions with communities and emphasize activities which contribute to societal betterment (Plummer et al., 2021; Secundo et al., 2017). As the mission of HEIs increasingly highlight their commitment to working in partnerships for societal enhancement, they need to concurrently devise corresponding mechanisms to identify, track, and gauge performance (Holton et al., 2015). In their recent work summarizing progress to date, Plummer et al., (2021, p. 1) observe that “higher education institutions are expected to account for their performance and this increasingly includes strengthening community relationships. However, assessment of Higher education institution (HEI)–Community partnerships is nascent.”

The terms assessment and evaluation are often used interchangeably in discussions of partnership performance (see Blackstock et al., 2007; Brinkerhoff, 2002). Despite being related, they are distinct and complementary. Assessment, the focus of this research, is oriented towards gathering information concerning the functioning of entities within a partnership (Astin & Antonio, 2012; Caplan et al., 2007). It uses the systematic, continuous collection of data on specified indicators to provide insight into the current performance of the partnership itself (e.g., partners’ performance, level of trust, learning opportunities; Caplan et al., 2007; Estrella & Gaventa, 1998; Onyango, 2018; Stem et al., 2005). Evaluation seeks to determine whether the partnership has achieved its’ aim; the extent of the intended effect or change attributable to the partnership (Brinkerhoff, 2002; Caplan et al., 2007). For this reason, evaluations usually occur when the partnership is nearing the end of its lifecycle. Assessment and evaluation are both imperative for partnerships. They work together to understand how a partnership is operating and what it has accomplished.

Assessing the performance of HEI–community partnerships is valuable regardless of the subject, but the importance of doing so is especially amplified in sustainability science. Sustainability science emphasizes the need to bridge the science-society divide for problem-solving through meaningful collaboration and knowledge co-creation (e.g., Clark & Dickson, 2003; Kates et al., 2001; Lang et al., 2012, 2017; Wiek et al., 2012). This imperative is aptly captured in the ideal–typical transdisciplinary research process in sustainability science as set out by Lang et al. (2012). Assessing performance is inherently oriented towards learning (Gupta et al., 2007), and learning is essential for sustainability science (Kates et al., 2001; König, 2018; Lang et al., 2012; Miller et al., 2014). As Blackstock et al., (2007, p. 727) observe, “the rationale for sustainability science reinforces the arguments for citizen and stakeholder social learning processes that should deliver increased understanding of complex systems, more durable and equitable solutions and increased capacity for active citizenship.”

Despite the imperative for bridging and the inclusion of evaluation, few sustainability science studies address these matters. One exception is the work by Lang et al. (2012) which identified continuous formative evaluation as well as evaluation of scientific and societal impacts. Also of note is the co-created knowledge phase work by Blackstock et al. (2007), who drew on three bodies of literature (participatory research, sustainability science, and evaluation of partnership processes) to undertake a summative evaluation of participatory research for sustainability. Finally, Wiek et al. (2012) investigated the promise of sustainability science by appraising five research-based projects using an analytical and evaluative framework. The projects were found to be successful in terms of problem focus and transformational research methodology while deficient in terms of stakeholder participation, action-oriented results, and substantive impacts. Although the authors were transparent about the advantages and disadvantages of deliberately selecting projects with which they are intimately familiar, the perspective of community partners was entirely absent from their evaluation.

Sustainability science would markedly benefit from routine performance assessment by all parties involved in a given initiative. As such, the primary purpose of the current study was to establish a robust, reliable, and valid instrument for assessing the performance of sustainability science partnerships. In the spirit of crossing disciplines, we start by drawing upon allied scholarship on HEI–community partnerships and highlight the recently developed HEI–Community Partnership Performance Index (HCPPI). We seek to answer two research questions in this study. First, what is the performance of three sustainability science initiatives: the Brock-Lincoln Living Lab, the Excellence in Environmental Stewardship Initiative, and Niagara Adapts? Second, is the HCPPI a heuristic tool for assessing the performance of HEI–community partnerships for sustainability science? Ultimately, performance assessment strengthens the position of sustainability science in HEIs and enhances sustainability science initiatives.

Advancements in HEI–community Partnership Assessment

The core mandate of HEIs is evolving and community engagement is becoming increasingly central (Bawa & Munck, 2012; Fitzgerald et al., 2012; Tremblay, 2017). The importance of HEI–community partnerships has emerged from the “…international convergence of interest on issues about the purposes of universities and colleges and their role in wider society” (Watson, 2007, p. 1; see also Hart & Northmore, 2011). Unsurprisingly, the environment and sustainability are among the societal issues for which the potential of HEI–community partnerships are identified (Mosier & Ruxton, 2018; Shiel et al., 2016). Sustainability science is especially relevant here as it is predicated upon the active collaboration with actors outside of academia in a transdisciplinary process (Bieluch et al., 2017; Dedeurwaerdere, 2013; Lang et al., 2012; Yarime et al., 2012). As argued by Yarime et al. (2012), it is this bringing together of education, research, and society that affords a systemic counter to the challenges of sustainability.

A second, related transformation underway is the accountability agenda for HEIs (Huisman & Currie, 2004) and the associated emphasis on performance-based metrics and funding (de Boer et al., 2015; Jongbloed et al., 2018). With the mission of HEIs expanding and emphasizing partnerships, there is an amplified need to determine appropriate corresponding mechanisms for performance assessment (Hart & Northmore, 2011; Holton et al., 2015; Plummer et al., 2021).

Measuring performance appears straightforward and involves “…a quantitative standard to systematically measure an organization’s daily operational results as they relate to its overall objectives” (Chen et al., 2009, p. 222). However, HEI–community partnerships are complex and extremely difficult to measure. For example, partnerships can take diverse forms and involve a variety of activities (Secundo et al., 2017); may occur over multiple time scales (Garlick & Langworthy, 2008); and generate various types of data (e.g., tacit, unquantifiable, informal; Secundo et al., 2017). Many HEI’s are ill-equipped and unprepared to collect critical partnership data (Holton et al., 2015), and often erroneously exclude critical data from the community entity (Hart & Northmore, 2011).

While HEI–community partnerships are diverse, formal partnership agreements are one way of fulfilling what has been referred to as the ‘third mission’ of HEIs (Secundo et al., 2017). The partnerships assessed within this study are based on formal partnership agreements. Such agreements involve written documentation which codifies parameters for partnerships between an HEI and an entity (i.e., a government agency, non-government organization, business, etc.). These formal partnerships “…represent{s} something qualitatively different from other community development strategies and from earlier forms of university outreach and technical assistance” (Rubin, 2000, p. 219).

The diversity of HEI–community partnerships presents several different avenues for their assessment. Initial progress regarding ways to measure partnerships includes self-assessment, basic checklists, and indicators, among other (e.g., Furco & Miller, 2009; Langworthy & Garlick, 2008; McClenney, 2007; Tremblay, 2017). Garlick and Langworthy (2008) observed that there has been a tendency to craft overly-elaborate measurement tools and indicator lists, thus they stressed the importance of pursuing a streamlined list of core indicators which enable both comprehensive measurement and tailoring to the particular context. Additionally, the assessment tool should connect to the higher-order objectives of the HEI, as well as meaningfully capture societal outcomes across time scales (Garlick & Langworthy, 2008). Finally, in order to obtain a more comprehensive assessment of the partnerships, it is especially important that assessment mechanisms capture both HEI and community partner perspectives (Holton et al., 2015; Wiek et al., 2012). Therefore, core indicators should also be relevant to both entities within a partnership. Beyond this preliminary progress, the development of a rigorous framework and method for monitoring and evaluation is absent in the scholarship on HEI–community partnerships (Bowen & Martens, 2006; Brinkerhoff, 2002; Holton et al., 2015; Plummer et al., 2021; Rubin, 2000; Srinivas et al., 2015).

Important inroads have recently been made in responding to the above void, offering both conceptual and methodological advancements. In drawing upon theoretical and applied experiences in the broader HEI–community partnership literature, Plummer et al. (2021) conceptualized a measure for assessing the performance of HEI–community partnerships. They focused on three salient aspects for successful partnerships—inputs, processes, and outcomes. Inputs are the contributions dedicated to the partnership by the two or more parties (e.g., financial and human resources, transparency, etc.). The process is how the partnership operates (e.g., shared decision-making, communication, etc.). Finally, outcomes include what is produced as well as the impacts for those directly involved in the partnership (e.g., publications, satisfaction with the experience, learning). Upon this basis, Plummer et al. (2021) developed a measurement mechanism for assessing the continuous performance of HEI–community partnerships. A questionnaire was sent to individuals engaged in HEI–community partnerships (including personnel from both the HEI and the community entity) and they were asked to rate their level of agreement with aspects and indicators for assessing HEI–community partnership performance. In addition, participants were asked to rank each indicator in order of usefulness/importance. The results from their study established widely agreed-upon indicators for gauging the performance of HEI–community partnerships. It also provided the foundation for developing the HCPPI, which we detail in the next section and use in this study to empirically examine the performance of three sustainability science initiatives.

Materials and Methods

Context and Description of the Sustainability Science Initiatives

Sustainability is becoming an increasingly important component of Canadian HEIs, as they have been called upon to catalyze progress towards transformation and change (Amey et al., 2020; Beveridge et al., 2015; Bieler & McKenzie, 2017; Lidstone et al., 2015). This is demonstrated by the recent inclusion of sustainability matters in higher education learning and research, institutional policy, strategic planning, and organizational goals (Beveridge et al., 2015; Bieler & McKenzie, 2017; Lidstone et al., 2015). Many Canadian HEIs have made tangible commitments to sustainability, with 40 institutions signing the Tallories Declaration—a ten-point action plan for integrating sustainability knowledge into teaching, research, operations, and outreach at HEIs (Amey et al., 2020). Additionally, 64 Canadian institutions use the Association for the Advancement of Sustainability in Higher Education’s (AASHE) Sustainability Tracking, Assessment and Rating System (STARS) for measuring their sustainability performance and guiding their sustainability practices (AASHE, 2018 in Amey et al., 2020). Finally, Canadian HEIs are prioritizing communicating and engaging with a wider community audience in order to address sustainability challenges. For example, a systematic examination of the 229 HEI websites in Canada by Plummer et al. (2021) revealed that 41% had an explicit mandate for community relationships and/or webpage.

This study empirically examines three HEI–community partnerships in Canada for sustainability science: the Brock-Lincoln Living Lab, the Excellence in Environmental Stewardship Initiative, and Niagara Adapts. These formalized partnerships were selected for several reasons. The contextual circumstances surrounding them reflect the major transitions underway in HEIs. The Brock University Institutional Strategic Plan (Brock University, 2018) set out a vision from 2018 to 2025 in which one of the four areas of strategic focus and priorities is to “enhance the life and vitality of our local region and beyond.” The Environmental Sustainability Research Centre’s (ESRC) mission, in which these partnerships are housed, advances environmental sustainability from local to global through excellence in research and education. Sustainability science anchors the functions of the ESRC and innovative partnerships are enacted to achieve their mission. The Government of Ontario announced that by 2025, 60% of HEI funding will be contingent upon 10 performance-based metrics. Four of these metrics are specifically related to economic and community impact: (1) research funding and capacity for universities, and apprenticeship related for college, (2) research funding from industry sources, (3) community/local impact of student population, and (4) institution-specific economic impact (Government of Ontario, 2020). Additionally, these partnerships align with the characteristics of sustainability science projects in the appraisal by Wiek et al. (2012) as they address a variety of topics (community wellbeing, environmental stewardship, climate change adaptation), focus on solutions, and occur in an assortment of collaborative settings. An overview of each partnership is detailed below.

Brock-Lincoln Living Lab (BL-LL)

The BL-LL partnership is a five-year agreement between Brock University and the Town of Lincoln, signed in 2017. Located in the Niagara Region, the Town of Lincoln has taken steps to prioritize community sustainability in their strategic plan, while also experiencing the largest development growth in the region. Collaborating was a strategic decision by both partners and the formalized agreement aims to address several shared goals: conducting a needs assessment to identify priorities and objectives; holding public events related to knowledge mobilization and community engagement; engaging both undergraduate and graduate students in co-operative education, research, and projects; initiating research projects by faculty; exploring and pursuing mutually beneficial opportunities of shared interest; promoting the BL-LL to develop networks nationally and internationally; and supporting Lincoln in promoting community sustainability. To date, a full action plan has been developed to guide the partnership and deliverables for several projects identified have been completed. This project has been touted by both organizations as a partnership success story.

Excellence in Environmental Stewardship Initiative (EESI)

The EESI partnership is a five-year agreement between Brock University and the Niagara Parks Commission, signed in 2018. The Niagara Parks Commission is an agency of the Government of Ontario which maintains the shoreline areas of the Niagara River, stretching from Niagara-on-the-Lake to Fort Erie. This includes several parks areas and attractions surrounding Niagara Falls, Canada. This formalized partnership agreement also aims to address several shared goals: conducting a needs assessment to identify priorities and objectives; holding public events in Niagara Parks each year to mobilize knowledge and engage visitors in environmental stewardship; engaging both graduate and undergraduate students in co-operative education, course projects, and independent research; initiating research projects by faculty; exploring and pursuing mutually beneficial opportunities of shared interest, such as grant proposals; and promoting the EESI and environmental stewardship more broadly within Niagara Parks and the Brock University community in order to develop broad networks. A highlight of this partnership is the evidence-based decisions Niagara Parks has been able to implement, in a timely manner, based on independent student and faculty research projects.

Niagara Adapts (NA)

Where the BL-LL partners with a single municipality to address community sustainability, NA, launched in 2019, is a partnership with seven municipalities in the Niagara Region focused on assisting municipal partners in the climate change adaptation planning process. The formalized NA agreement aims to cultivate a community of practice for climate change adaptation planning and build climate resilience within the region through a series of workshops, hosted by the ESRC over a two-year period, to assist municipal partners in the planning process. In addition to the workshop series, the ESRC has supported municipalities in compiling individual baseline assessment data, through large-scale surveying, to inform the allocation of resources for climate change planning and adaptation, and provided ongoing support to municipal partners through ongoing partnership communication.

Participants

Twenty participants who were actively involved in a sustainability-focused HEI–community partnership were recruited for this study. All participants were members of either BL-LL, the EESI, or NA (see above for a description of each partnership). Seven participants were Brock University faculty or staff members (1 for BL-LL, 4 for EESI, and 2 for NA), four were student participants (0 for BL-LL, 3 for EESI, and 1 for NA), and 10 were community partners from various organizations (2 for BL-LL, 2 for EESI, and 6 for NA). Participants varied with respect to the length of time that they had been involved with the partnership (between 1 and 4 years), and their level of involvement in their partnership (minimal to extensive). After providing written consent, participants were instructed to complete a 20-min questionnaire that asked them to rate the performance of their partnership across multiple sub-categories (see details below). The questionnaire was delivered online via Qualtrics, and all responses were kept strictly confidential. This study was approved by the Research Ethics Board at Brock University and conducted in accordance with Tri-Council ethical guidelines.

Data Collection: HEI–Community Partner Performance Index (HCPPI) Questionnaire

The HCPPI (Plummer et al., 2021) is a 47-item questionnaire designed to provide a comprehensive understanding of a given HEI–community partnership’s strengths and areas for improvement across three broad categories: inputs, process, and outcomes (see Online Appendix A). Participants were provided with a number of key statements regarding various aspects of the partnership (e.g., “the amount of financial resources for this partnership are adequate”), and were asked to rate their level of agreement with each statement using a 7-point Likert scale (1 = strongly disagree; 7 = strongly agree). As the nature of HEI–community partnerships can rapidly evolve over their lifespan, we asked participants to respond based on their experiences over the past 12 months.

First, participants rated 13 statements that described the inputs to the partnership. Inputs are defined as the contributions that partners dedicate to the partnership, and can be broken down into four sub-categories: financial resources, human resources, motivation for partnership, and transparency. Next, participants rated 19 statements that corresponded to the partnership process. Process refers to the manner in which the partnership operates and is broken down into five sub-categories: shared decision-making, communication, trust, mutual respect, and adaptability. Finally, participants rated 15 statements that were designed to assess the outcomes of the partnership. Outcomes refer to what is produced and the impacts to those directly involved in the HEI–community partnership. The outcomes can be broken down into three sub-categories: objective outcomes, subjective outcomes, and learning outcomes.

Analysis

After completion of the questionnaire, the participant ratings were converted to weighted scores using a ratio method (Weber & Borcherding, 1993). First, we obtained relative importance rankings for each of the 47 HCPPI statements from a Canada-wide study that was conducted during the instrument’s development (see Plummer et al., 2021). Participants in Plummer et al. (2021) received statements (also called “indicators”) grouped by sub-category, and were asked to rank the importance of all of the statements within a given sub-category for assessing the performance of HEI–community partnerships. Based on these rankings, relative weights were then assigned to each statement. With the ratio method, weights are assigned in multiples of 10, with the lowest-ranked item in a sub-category receiving a weight of 10, the next lowest-ranked item receiving a weight of 20, and so on.Footnote 1 Next, the weights are normalized by dividing the weight for each item by the sum of the weights. Finally, for each statement, the mean normalized weights were averaged across participants.

After deriving the mean normalized weights for each item in the HCPPI, we then calculated the weighted ratings for the current participants by simply multiplying each rating by its normalized weight. We also calculated an overall inputs score, process score, and outcomes score for each participant. Finally, we averaged the overall inputs, process, and outcomes score to create an overall HCPPI score.

Single-Item Partnership Success Measure

After completion of the full HCPPI, we also asked participants to rate the overall success of their partnership over the past 12 months using a 9-item Likert scale (1 = extremely unsuccessful; 9 = extremely successful). The purpose of this single-item measure was to provide us with an index of participants’ overall attitude towards the partnership so that we could later compare it to the overall HCPPI score. This comparison allows us to assess whether the HCPPI is accurately capturing each partner’s perception of the performance of the partnership (i.e., convergent validity).

Results

Overall performance

Overall, the faculty and community respondents for each of the three HEI–community partnerships provided positive ratings across all three facets of the HCPPI (see Fig. 1 for partnership-specific weighted ratings as a function of subscale). Interestingly, a series of one-way ANOVA analyses demonstrated that neither the overall HCPPI rating, nor the overall ratings for each subscale (inputs, process, outcomes), significantly differed as a function of rater (i.e., faculty, student, and community partner; all F’s < 1), thereby illustrating the ability of the HCPPI to capture the performance of a given partnership based on the perspective of multiple different stakeholders. Importantly, there was also a positive correlation (r = .481, p = .03) between the overall HCPPI score and mean scores on the single-item measure of partnership success, thereby providing evidence of convergent validity.

Fig. 1
figure 1

Mean weighted scores for the inputs, process, and outcomes subscales of the HCPPI for the Brock-Lincoln Living Lab (BL-LL), Excellence in Environmental Stewardship (EESI), and Niagara Adapts (NA) partnerships. Error bars represent the standard error for each condition mean

Full size image

Reliability of the HCPPI

Next, we examined the internal consistency reliability of both the full 47-item HCPPI and the three subscales of the HCPPI: inputs, processes, and outcomes. The Cronbach’s alpha of the full scale was .936. For the individual subscales, the Cronbach’s alpha was .813 for inputs (13 items), .900 for process (19 items), and .862 for outcomes (15 items), suggesting that the scale has excellent internal consistency reliability.

Inter-rater Reliability

Lastly, we examined the inter-rater reliability to determine whether HEI and community partners differed with respect to their performance ratings on the HCPPI (Koo and Li, 2016). Intraclass correlation estimates using a consistency two-way mixed-effects model were calculated to compare mean HEI (faculty and student) ratings to mean community partner ratings for each partnership.Footnote 2 For the BL-LL partnership (1 HEI partner, 2 community partners), the average intraclass correlation was .884 for the full scale, with a 95% confidence interval of .792 to .935. For the EESI partnership (7 HEI partners, 2 community partners), the average intraclass correlation was .993 for the full scale, with a 95% confidence interval of .988 to .996. Finally, for the NA partnership (2 HEI partners, 6 community partners), the average intraclass correlation was .978 for the full scale, with a 95% confidence interval of .960 to .988. As such, there was a high degree of consistency between HEI ratings and community partner ratings for each of the respective partnerships. Figure 2a–c depicts these differences between HEI and community partner ratings (i.e., HEI ratings minus community partner ratings) for each facet of (a) inputs, (b) process, and (c) outcomes. A positive value indicates that the HEI raters have a higher rating to a particular facet as compared to the community partner raters, whereas a negative value indicates the HEI raters provided a lower rating than did the community partners. Although the HEI and community partners numerically differ with respect to their ratings (particularly for the BL-LL partnership which had only one HEI rater), these differences are statistically non-significant.

Fig. 2
figure 2

Difference in average HEI and community partner mean ratings (HEI minus community) as a function of partnerships for a inputs, b process, and c outcomes

Full size image

In addition to examining the inter-rater reliability across partners (HEI vs community) for each partnership, we also examined the inter-rater agreement within in each rater group. For the BL-LL, the intraclass correlation for the community raters was .993 (95% confidence interval: .986 to .996).Footnote 3 For the EESI, the intraclass correlation was .991 (95% confidence interval: .985 to .995) for the HEI raters, and .949 (95% confidence interval: .908 to .972) for the community raters. Finally, for the NA partnership the intraclass correlation was .993 (95% confidence interval: .986 to .996) for the HEI raters, and .987 (95% confidence interval: .981 to .992) for the community raters. As such, the HCPPI shows excellent inter-rater reliability both between HEI and community raters, and within a given organization.

Discussion

The first goal of this study was to establish a robust tool for assessing the performance of HEI–community partnerships for sustainability science. In drawing upon recent scholarship on HEI–community partnership, we employed the HCPPI to assess performance in three sustainability science partnerships. Ultimately, the HCPPI and its subscales were found to be highly reliable, showing excellent inter-rater reliability both across and within organizations involved in each of the sustainability science initiatives. Moreover, the relationship between scores on the HCPPI and subjective perceptions of partnership success were strongly aligned. As the sample for this study was small (N = 20), it is necessary to further test the reliability and performance of the HCPPI with a larger, more diverse data set. However, these initial findings provide preliminary evidence that the instrument may be effectively able to capture the perceptions of multiple different stakeholders, thereby accounting for unique perspectives constituting the partnership and providing a comprehensive picture of performance.

The performance of the three sustainability science partnerships examined (BL-LL, EESI, NA) were each rated extremely positively by the respondents. As a composite sociometric index, the HCPPI allows us to gain insights into multiple dimensions constituting performance of HEI–community partnerships. Strong positive ratings in each were found across all three dimensions of performance: inputs, processes, and outcomes. While previous studies have not investigated these three dimensions comprehensively, substantive agreement exists with the HEI–community scholarship regarding the dynamic characteristics (i.e., process) of effective partnerships (McNall et al., 2009; Schulz et al., 2003). McNall et al. (2009) found a strong positive perception by community partners regarding group dynamics, group effectiveness, and benefits (expected and received). In another study examining partnership processes and benefits, both academics and community partners held positive perceptions regarding the perceived benefits of engaging in the partnership, including but not limited to substantial monetary and non-monetary benefits for research, creating new opportunities, enhanced skills, and so on (Buys & Bursnall, 2007). Interestingly, the participatory evaluation of sustainability in the Douglas Shire case in Australia (Blackstock et al., 2007), as well as the ‘expert’ appraisals of five diverse sustainability science initiatives (Wiek et al., 2012), reveal that lack of experience and process-oriented competencies are substantive challenges.

Gaining insights into multiple dimensions of performance is important because it gauges present circumstances and illuminates gaps from an ideal. Documenting the performance of sustainability science partnerships demonstrates alignment with the third mission and societal contributions of HEIs (Inman & Schütze, 2010; Secundo et al., 2017), specifically through taking leadership in collaboratively generating solution-oriented responses to complex and uncertain environmental challenges and enabling locally relevant sustainability (Lang et al., 2012; Mosier & Ruxton, 2018; Shiel et al., 2016; Wiek et al., 2012; Yarime et al., 2012). As Clark and Dickson (2003, p. 8059) argue, “the research community needs to complement its historic role in identifying problems of sustainability with a greater willingness to join with the development and other communities to work on practical solutions to those problems.” Such documentation simultaneously offers an evidence base for HEIs to address the increasing matter of accountability (Huisman & Currie, 2004) and performance-based allocation of resources (de Boer et al., 2015; Jongbloed et al., 2018).

As argued in the opening section of the paper, performance assessment is especially important in sustainability science. Involving individuals from both the HEI and the community in assessing performance was central to our study because sustainability science emphasizes the need to bridge the science-society divide for problem-solving through meaningful collaboration and knowledge co-creation (e.g., Clark & Dickson, 2003; Kates et al., 2001; Lang et al., 2012, 2017; Wiek et al., 2012). This imperative is aptly captured by Lang et al. (2012) and the HCPPI provides a tool by which performance can be continuously assessed and inform the ideal–typical transdisciplinary research process. Feedback from performance assessment using the HCPPI potentially provides a basis for reflection and discussion by participants, and (if appropriate) adjustments to the partnership initiatives. Social learning—arriving at shared understanding of a problem by individuals with different perspectives through interactions and engaging in iterative decision-making to navigate complexity and uncertainty (e.g., Armitage et al., 2008; Reed et al., 2010)—may be engendered by collaboratively undertaking performance assessment and incorporating the information produced into governance.

Conclusions

Sustainability science seeks to catalyze the transition to sustainability by effectively bridging the science-society divide (Kates et al., 2001; Lang et al., 2012; Wiek et al., 2012). The evolving mandate of HEIs, which emphasizes community engagement and addresses issues of importance societal concern (e.g., Bawa & Munck, 2012; Fitzgerald et al., 2012; Plummer et al., 2021; Secundo et al., 2017), provides impetus for sustainability science (Buckley & Michel, 2020). Concomitantly, attention is directed to the need for assessing HEIs performance regarding community relationships (Brundiers & Wiek, 2011; Plummer et al., 2021; Secundo et al., 2017). Departing from this nexus, we looked to the emerging scholarship on HEI–community partnerships and employed the HCPPI to examine three sustainability science partnerships: the Brock-Lincoln Living Lab, the Excellence in Environmental Stewardship Initiative, and Niagara Adapts.

The study has critical implications for HEI administrators and policymakers. Set against the emerging and intensifying accountability agenda (Huisman & Currie, 2004; Jongbloed et al., 2018), HEIs are “… expected to account for their performance and this increasingly includes strengthening community relationships” (Plummer et al., 2021, p. 1). As illustrated in this study, assessing the performance of HEI–community partnerships provides a clear signal of progress towards the strategic directions set out by the HEI, faculty and unit. Whereas there is a critical need for mechanisms to identify, track, and gauge performance of HEI–community partnerships (Holton et al., 2015), initial results suggest that the HCPPI may provide an effective, reliable, and heuristic measure of HEI–community partnership performance. In applying the HCPPI, HEI administrators can go beyond ad-hoc anecdotes and establish empirical-based evidence of partnership performance for a period of time (e.g., annual reporting) as well as capture changes over time. In keeping with the spirit of a partnership, community partner organizations participate equally in the performance review process. In addition to the overarching HEI–community performance, the HCPPI also provides community partners, as well as policymakers, with a fine-grained basis upon (i.e., input, process, outcomes) which to make decisions and direct resources.

The transition towards sustainability is the ultimate aim of sustainability science (Kates et al., 2001; Lang et al., 2012; Wiek et al., 2012). Strong performance of sustainability science initiatives is a sound first step toward this end. Our findings suggest that the HCPPI can comprehensively capture the perspective of multiple participants and provide a mechanism by which social learning may be fostered. Continuous performance assessment next needs to be linked to social and ecological consequences. Blackstock et al., (2007, p. 737) similarly observe that “formative evaluation is likely to have the same difficulties regarding responding to the participants' desire for tangible evidence of positive impact, which as discussed above, is difficult due to the lack of direct causal relationships within sustainability science.” The framing of this study provides a conceptual basis from which performance assessment may be connected to monitoring and evaluation, the latter of which addresses the attainment of aims as well as realization of intended impacts. Sustainability science partnership performance over time, as well as causal relationships with social-ecological consequences, are promising avenues for future research.