Introduction

In July 2017, representatives from urban agriculture (UA) organizations from across the Commonwealth of Massachusetts (MA)—including The Food Project, Gardening the Community, Mill City Grows, Nuestras Raices, the Urban Farming Institute of Boston, and the Worcester Regional Environmental Council—gathered to offer feedback on a proposed research project.Footnote 1 Developed by a program director at the MA Department of Public Health, a program officer from a foundation that supports UA in the Northeastern United States, and a researcher who often collaborates with UA organizations, the goal of the proposed project was to make the public health consequences of UA more visible to funders. The UA leaders at the table that morning were clear in their consensus that this project did not reflect their priorities. Rather, they asked for an alternative research project that would (1) highlight the mission(s) of UA, as understood by those “in the field”; (2) present data on how evaluation is currently being conducted by UA organizations, and; (3) explore practitioners’ perspectives on what is and is not well represented in current measurement strategies. They also requested training in policy advocacy, including detailed information about relevant federal and state policy processes and clarification regarding the limits on political activity associated with the nonprofit status of their organizations (Field notes, July 2017).

In this paper, I draw on the literature on neoliberalism, nonprofit organizations, and urban agriculture to conceptualize these two requests—for an interrogation of the limits of program evaluation and training in policy advocacy—as responses to the significant challenges faced by contemporary UA organizations, especially in regard to their mission-driven commitments to justice and equity. I develop this argument primarily by presenting an analysis of the data collected, as requested at the meeting described above, on missions and metrics in Massachusetts urban agriculture. Broadly, what emerges from this analysis is a portrait of how UA practitioners—i.e., those who lead organizations, fundraise, implement programs, design evaluations, build gardens and farms, organize communities, advocate for racial justice and health equity, and grow food in MA cities—navigate what scholars have described as an “inherent contradiction” in contemporary urban agriculture: its simultaneous orientation to “neoliberal” (social service) and “radical” (social justice) agendas (McClintock 2014). Specifically, I contend that UA practitioners’ critiques of extant practices of program evaluation, visions of alternatives, and desire to build capacity for policy advocacy point to their interest in acting as a collective movement for social justice, rather than primarily as discrete social service organizations. This analysis focuses especially on the complex role of metrics, which not only entrench neoliberalism in UA organizations but also may provide a mechanism for challenging its assumptions and advancing the radical project of food justice.

Background

Neoliberalism is “predicated upon the belief that the maximization of social good requires locating all human action in the domain of the market” (Munshi and Willse 2017, p. xiv). In the United States, neoliberal governance has included both the “rolling back” of social safety net programs (e.g., the “fiscal reforms” of the 1970s) and the “rolling out” of social and economic structures that support capitalist accumulation (e.g., the dismantling of the social safety net in the 1980s) (McClintock 2014, p. 155). The brunt of these reforms has been disproportionately born by Black, Indigenous, and people of color (BIPOC), even as the racialized (and gendered) burdens of neoliberalism are obscured by discourses that blame individuals for the consequences of changes in public policy (Munshi and Willse 2017, p. xiv).

Nonprofit organizations are central to neoliberal governance, especially “as the state has disavowed its responsibility for the health and well-being of its population, non-profit industries have grown to assume this role, providing essential social and health services” (Munshi and Willse 2017, p. xv; see also Wolch 1990). Consequently, the nonprofit sector has expanded, both in response to the dire “service void” created by the roll back of state programs and requirements that organizations be fully incorporated (e.g., as nonprofits) to be eligible for the government grants upon which they now depend to provide urgently needed services (Gilmore 2017, p. 45). Together, these dynamics have transformed many advocacy and social movement organizations, as they have shifted their focus to service provision and become constrained in their ability to challenge “the root causes of the exploitation and violence” experienced by those they serve (Kivel 2017, p. 129).

Alongside grants from government agencies, many nonprofits rely extensively on philanthropic funding. While foundation funding supports the provision of essential health and social services—including food (Fisher 2017)—to people who have been “abandoned” by the state (Gilmore 2017, p. 44), there are several ways that it also may impede meaningful social change. First, philanthropies may exclude or marginalize more politically progressive organizations from funding opportunities, whether in how they craft and/or distribute solicitations or how they make awards (Faber and McCarthy 2005). Alternatively, philanthropies may “capture” organizations by “leverag[ing] their financial resources to apply pressure and influence the decision-making process of…organizations” (Francis 2019, p. 276). Foundation funding may also lead to the colonization of organizations, as when “distracted and bogged down by professional management and partnership requirements, short-term foundation funded programs replace the day-to-day engagement required to organize people in movement building” (Kohl-Arenas 2016, p. 9). At the same time, “the constant chase after money (grant renewals) creates distrust and competition between…organizations and stifles opportunities for unified action as they arise” (Kohl-Arenas 2016, p. 80). In the professionalized “business culture” imposed by such requirements (Perez 2017, p. 92), “leaders and organizers become institutional professionals accountable to foundations and not to the people they claim to represent or serve” (Kohl-Arenas 2016, p. 9).

Accountability: audits and outcome evaluation

For many nonprofit organizations, accountability to funders takes the form of audit practices—including performance measurement, benchmarks, rankings, ratings, certification systems, etc.—which also advance the “transition toward new forms of governance/governmentality under neoliberalism” (Campbell 2013, p. 178). Such “rituals of verification” (Power 1997) “disseminate the market model to economic and noneconomic domains of activity, (re)configuring human beings, organizations, and states as market actors” (Mennicken and Espeland 2019, p. 234). Audit practices, and the quantitative metrics that they both rely upon and generate, thereby “create new incentives and power dynamics” for “organizational behavior, policy, and strategy” (Espeland and Sauder 2016, pp. 3, 8–9).

Metrics also shape what is valued in a field through the process of reactivity, which refers to the “idea that people change their behavior in reaction to being evaluated, observed, or measured” (Espeland and Sauder 2007, p. 1). This happens when individual and organizational actors “seek to perform well according to the logic of the measure in order to establish legitimacy and obtain resources” (Barman 2016, p. 12).Footnote 2 For example, nonprofit organizations may shift their focus “from strategies for radical change to charts and tables that demonstrate how successfully the work has satisfied foundation-determined benchmarks” (Perez 2017, p. 93). In this way, metrics have the power to “to restructure the social spaces they depict” (Espeland and Sauder 2016, p. 22).

In the past three decades, the audit practice of quantitative outcome measurement has become a ubiquitous feature of program evaluation in nonprofit organizations. The outcome measure is meant to provide a “quantitative assessment of an organization’s effectiveness” (Barman 2016, p. 39). As an organizational practice, outcome measurement emerged in the early 1990s, when federal agencies and nonprofit funders (e.g., the United Way) decided that they needed to gather performance data from the health and social service organizations that they were supporting, in order to demonstrate the effectiveness of programs and “communicate their legitimacy to external critics” (Barman 2016, pp. 53–54).Footnote 3 In contrast to measuring program outputs—e.g., the number of goods delivered or the number of individuals who attend a program—outcome measurement requires the tracking of “changes in conditions, behavior, or attitudes that indicate progress toward achievement of the mission and objectives of the program” (Hatry 1999, in Barman 2016, p. 39).

Outcome measurement wrought two major changes for nonprofit organizations. First, in contrast to prior program evaluation practices, in which an external evaluation professional gathered retrospective outcomes data on a one-time basis, now staff members must repeatedly gather outcomes data on an ongoing basis for performance management, budgeting, and to demonstrate accountability (Barman 2016, p. 58). Second, outcome measurement transformed the process of causal attribution, which “no longer needed to be proven by the use of an experimental design (i.e., with the use of randomized control trials), but rather could be inferred from the specification of a program’s logic model” (Barman 2016, p. 58). A program’s logic model is based in its “theory of change,” which posits a cause and effect claim about how specific activities will create the desired outcomes in beneficiaries: “It requires the identification of the assumptions that underlie a program, including the specification of the social problem to be solved, the presumed cause of the problem, and how the program intervention is understood to correct the problem” (Barman 2016, p. 59). Within contemporary nonprofit organizations, accountability mechanisms such as logic models and outcomes measurement “shape people’s ideas of what the task before them is” (Krause 2014, p. 76), often resulting in “a valuation of short-term goals over long-term purposes” (Muller 2018, p. 20).

Studies of varied components of the food system—including food banks (Fisher 2017), farmers markets (Mino et al. 2018), and food policy councils (Webb et al. 1998)—suggest that the “high levels of reporting and accountability” central to program evaluation are not only time consuming and “tedious” (Mino et al. 2018, p. 828), but also limit organizations’ “‘bandwidth’ to attempt projects of a more preventative nature…such as policy advocacy” (Fisher 2017, p. 67). What little has been written specifically about program evaluation in UA suggests its rhetoric, practices, and products reinforce neoliberalism, both by individualizing program outputs and undermining organizational capacity for transformative social change. For example, in her study of youth gardening programs, Cairns proposes that the “rhetoric of effects…mobilizes particular conceptions of children (as outputs) and social change (as occurring through individual transformation)” and thereby obscures “the need for collective action and structural reform in a deeply inequitable and unsustainable food system” (2018, pp. 517–519). In their study of urban agriculture in New York City, Reynolds and Cohen identify funders’ emphasis on quantifiable outcomes as a “constraint on urban agriculture activists who are focused on advancing social justice” (2016, p. 107).

Urban agriculture

As scholars have observed, “while justice is open to multiple interpretations, food justice activism in the US has placed racial equity and racial justice (rooted in civil rights and environmental justice struggles) at the heart of its praxis” (Coulson and Milbourne 2021, p. 43–44). Many food justice organizations in the US grew out of environmental justice organizations, bringing with them a foundational focus on redressing environmental racism (Sbicca 2018; Shostak 2021). UA practitioners also take inspiration, insight, and strategies from the Black agricultural cooperatives of the rural South (White 2018) and the Black Panther Party’s (BPP) “survival programs” which sought to meet the urgent health and safety needs of Black urban communities “that had long been neglected by the state” (Hassberg 2020, p. 87). While different in their geographic locations and organizational forms, both Black farmers’ cooperatives and the BPP’s food distribution programs offer examples of “strategies for collective agency and community resilience” and resistance (White 2018, p. 8), that include meeting “the basic human needs of oppressed people” (Hassberg 2020, p. 92). This dual focus—on organizing for social justice, on the one hand, and providing essential health and social services, on the other—is similarly a defining characteristic of many contemporary UA organizations (McClintock 2014). Now, as then, “providing food is not the revolution, but a vehicle through which revolution is made possible” (Hassberg 2020, p. 97–98).Footnote 4

The narratives of UA practitioners in Massachusetts highlight their commitment to addressing America’s long history of systemic racism and its many contemporary health, social, and economic consequences; in deeply historicized accounts, they emphasize, for example, the intergenerational trauma and loss wrought by slavery, share-cropping, and dispossession in the rural South, and by racialized urban social processes, including redlining, white flight, and arson, in the North (Shostak 2021). When UA practitioners in MA call attention to the lack of access to healthy food in BIPOC neighborhoods, they tend to reject the relatively ahistorical and apolitical concept of “food deserts,” speaking rather of a socially and historically constructed system of “food apartheid” which has had devastating health consequences over generations (Field notes, December 2018).Footnote 5 At the time of this study, all of the UA organizations in MA sought to serve BIPOC communities, although only half had BIPOC leadership.

From this perspective, UA in Massachusetts, and beyond, is about more than growing food; it is a means of reclaiming cultural knowledge and practices and reestablishing “right” relationships to the land and to each other—all of which have been disrupted, appropriated, and undermined by colonialism, slavery, and other forms of racialized exploitation (Penniman 2018). Related, UA offers a way to bring communities together to envision and create new systems that will not only meet basic human needs but also “challenge white supremacy and political and economic exploitation” (White 2018, p. 142). As a practitioner from Western Massachusetts stated at the MA Urban Farming Conference, “our agricultural system is built on stolen land and stolen labor. If you aren’t willing to have this conversation, you aren’t working for justice, even if you’re growing food…” (Field notes, March 2018).

Nonetheless, even UA organizations with deep commitments to racial justice and equity may contribute, however inadvertently, to neoliberal social processes (McClintock 2014). For example, UA may lead to the revalorization of unused land (Sbicca 2019) and urban “greening” is strongly associated with gentrification (Alkon and Cadji 2018). As a consequence, recent scholarship highlights the importance of “radical allyship” in which “food justice activists develop strong relationships with long-term community members, and together, they work to avoid the latter’s displacement” (Alkon et al. 2019, p. 798).

Given that it both has “radical antecedents and revolutionary possibilities” and sometimes plays a role in “neoliberal urban restructuring” (McClintock 2014, p. 157), contemporary urban agriculture is an intriguing, if complex, case for thinking through the relationships between nonprofit organizations, their funders, and movements for justice and equity. As I describe below, UA practitioners in Massachusetts appear to be moving towards collective action, after years of operating, more often, as relatively separate and distinct nonprofit organizations. At the same time, this analysis suggests that some of the same mechanisms at work in the colonization (and subsequent deradicalization) of social movement organizations—specifically, the evaluation and reporting requirements associated with grant funding—also pose challenges to nonprofits seeking to come together as a social movement. The following analysis explores how UA practitioners navigate these complexities, by focusing, per their request, on program evaluation.

Data and methods

The data presented here come from the Missions & Metrics Project (2017–2018), a qualitative exploratory study which included formal interviews with representatives from nonprofit UA organizations (n = 6), and with individuals who have worked with multiple UA organizations, typically as consultants, on their evaluation plans and procedures (n = 3).Footnote 6 The following analysis also draws on extensive field notes from the meetings in which this research project was collaboratively designed (2017), sessions on evaluation held at the New Entry Sustainable Farming Project's Community Food Systems Conference (December 2017), and the MA Urban Farming Conference (March 2018). Lastly, the analysis draws on notes from a meeting of ~ 30 UA practitioners who were invited to offer comments on a draft report of the findings of the Missions & Metrics project; this review was followed by a training session on policy advocacy (December 2018). Consequently, while individuals from only ~ 75% of the nonprofit UA organizations in MA (of which I am aware) appear in the interview sample, I believe that nearly all were represented at the events recorded in my field notes.

Both the interview transcripts and all of the field notes were coded thematically, in Atlas.ti, using the principles of constructivist Grounded Theory, a systematic method of qualitative data analysis that is ideal for inductive (i.e., rather than hypothesis testing) research (Charmaz 2006). Grounded theory begins with “open coding,” in which codes are developed from the data, rather than a predetermined coding structure defined by the researcher’s theoretical commitments (Charmaz 2006). In this study, the iterative and comparative coding process at the heart of Grounded Theory supported inquiry both into the lived experiences of those interviewed and “how things have been organized so that speakers have the experiences that they speak about as they do” (Kim and Campbell 2013, p. 187). I organize the following section according to the central themes that emerged from this analysis, which include (1) UA organizations’ commitments to working towards food justice and, related, confronting the systems of oppression that create inequities in access to healthy food; (2) the multiple challenges of quantitative program evaluation; (3) practitioners’ visions of alternative strategies for telling the stories of their work that would center their commitments to justice and equity and support collective action for transformative social change.

Results

Missions

The formal mission statements of UA organizations in Massachusetts highlight both the ambition and the diversity of their goals (Table 1).

Table 1 Examples of UA organizations’ mission statements, Massachusetts
Full size table

As a consequence, UA organizations are often running multiple, interlocking programs of service to their communities. These include growing food, training new farmers, building raised beds for community gardens, building and running school garden programs, organizing farmers’ markets, providing job training and employment opportunities for youth, teaching cooking classes, doing anti-racist trainings, cleaning up abandoned lots and remediating soil, and developing community land trusts. Even so, UA practitioners emphasize that their organizations strive to achieve much more than the sum of these programmatic parts.

All of the participants in this study expressed an explicit commitment to food justice. While their definitions of food justice vary, they share a focus on increasing access to healthy food and improving community health, especially in BIPOC neighborhoods affected by food apartheid, challenging racism in the food system, advancing environmental justice and sustainability, and bringing people together to create a more just and equitable future. Practitioners clearly see food (and food systems transformation) “as a catalyst” for broader forms of social change, including addressing “fundamental challenges” like racism (Field notes, January 2018). That is, they seek not only to improve access to healthy food in their communities, but also to address the larger systems that underly food (in)justice: “The problem is racism, nativism, age-ism, sexism, unequal access to resources…” (Field notes, December 2017).

UA practitioners are well aware that the inequities and injustices that they seek to address have deep roots, with histories embedded in generations of policy and practice: “If you understand how this system was built historically…from when the Europeans first arrived to this continent; every policy along the way…has benefited some people and not others” (Interview 06). From this perspective, policy advocacy also is an essential aspect of the work of UA:

This is social justice…lifting up the voices of people who are not being heard…To me…the most important form of social justice…is having policy systems align with what people are saying they need (Interview 01, emphasis added).

At the same time, practitioners observe that their “equity work,” including policy advocacy and community organizing, can be more difficult to fund than their service work, such as growing and distributing food (Field notes, March 2018). Movement towards equity and justice, as well, is perceived as more challenging to measure and is less often included in program evaluations.

Metrics

Broadly, UA practitioners see evaluation as a means of holding their organizations accountable to their missions by ensuring that their programs are making “real specific changes” (Interview 06). Especially in the context of multifaceted and often quickly growing organizations, practitioners observe that is important to stay “really rooted and held by our strategies and our models and our metrics” (Interview 02). UA practitioners report spending significant time and energy developing the logic models that articulate how their activities are expected to generate specific results. Many organizations develop these models—which I often saw prominently displayed on their walls—through “strong participatory processes” with youth program participants and/or community members (Interview 03).Footnote 7

As part of their evaluation strategies, UA organizations deploy a wide variety of quantitative measurement techniques. For example, using both point of sale information and surveys, UA organizations track pounds of food grown, value of food sold and donated, numbers of new and continued markets, attendees at farmers’ markets and educational programs, volunteers at their farm and garden sites, and acres of land put into production. Less often, they use qualitative methods, such as focus groups and in-depth interviews, to gather stories from program participants. They hope to learn from these measures whether their programs are creating “the change that we want to see” (Interview 02).

Despite such hopes, UA practitioners offered a myriad of critiques of program evaluation. As described in detail below, these include not only the practical challenges of ongoing quantitative data collection and analysis, but deep concerns about how reporting requirements shape their organizations.

Practical challenges

Because nonprofit UA organizations depend on state and philanthropic funding for a significant proportion of their operating budgets, each has multiple reporting obligations to a wide variety of private and public granting agencies. A few grantors use a common reporting framework, however, there are no standardized metrics for evaluating UA. Therefore, most organizations are constantly gathering varied data for a multitude of reports, with due dates arrayed across the year. There was broad consensus across organizations that evaluation demands significant resources, including dedicated staff time, mechanisms for data collection and analysis (from paper surveys to electronic dashboards), and processes for translating the data.

The burden of doing multiple, ongoing evaluations is exacerbated by the seasonality of growing food in New England. As one respondent observed, it’s “especially hard during the summer season” for UA organizations to “build in the time” for evaluation (Field notes, January 2018). Additionally, practitioners noted that the reporting requirements of grantors often orient to the fiscal year, which begins on 1 July. This means that they are required to report to funders in the middle of the growing season and while they are running their most intensive youth and volunteer programs. Therefore, practitioners believe that grantors are getting a “smaller picture of the work” than they would if evaluation was oriented to the agricultural calendar (Interview 02). Practitioners also identify a variety of challenges pertaining to the specific data they are asked to collect. Of particular concern are reporting requirements that are “not trauma informed” and/or may “interfere with people’s experiences” of programs, such as farmers markets, where they intend to not only provide a needed service, but also build community (Field Notes, December 2018).

Practitioners also expressed frustration at the lack of standard metrics to assess UA programs, which complicates even seemingly straight forward quantitative measures, such as food donations. For example, when UA organizations donate to mission driven organizations, such as subsidized community supported agriculture programs and food banks, this is typically tracked in pounds of food donated. When they donate to farmers markets and businesses, however, this is tracked in dollars, i.e., the estimated monetary value of the food. UA practitioners find it “hard to translate” across these measures in reports to grantors, as “donating 25% of the pounds of food we’ve grown isn’t the same as donating 25% of the monetary value of the food we’ve grown” (Field notes, January 2018). Moreover, there can be multiple metrics for calculating identical outputs. For example, while some funders estimate that each pound of vegetables donated creates 3 servings of food, others count each pound of vegetables donated as 4 servings. UA organizations are therefore using different formulae even for the same outputs, depending on to whom they are reporting. As “everybody asks slightly different questions,” this creates a lot of extra work (Interview 03).

Reactivity: when metrics shape action(s)

Alongside concerns about the burdens of collecting, analyzing, and translating evaluation data, UA practitioners express concern that the grant-making requirements of their funders, including reporting requirements, “dictate what we can do” (Field notes, January 2018). While UA organizations seek to maintain the integrity of their programs, because they are “very live or die by grants,” practitioners also have to “shape the programs” to “the priorities” of the “so many funders” on which they depend (Interview 02). Practitioners identify this as an ongoing challenge: “is your organization aligned with what they're trying to do? Often times, that doesn't match” (Interview 06).Footnote 8

Moreover, to be competitive in their grant applications, “every year you have to increase your numbers” and often times “programs have to change to reflect these metrics” (Field notes, July 2017). Practitioners are concerned that these market-based imperatives can adversely affect program participants. For example, when a grant requires that a certain number of pounds of vegetables be donated to food access or hunger relief, the relative weight of different kinds of vegetables might drive crop planning and/or donation practices. As practitioners recounted, this can lead to “mismatches” between meeting grant goals and providing culturally appropriate foods:

…beets weigh more than lettuce. So, if a grant requires a certain number of pounds of donations, that may shape what is donated. It’s difficult, then, when there is a mismatch between weight imperatives and what people want to eat, like if they have cultural preferences for specific crops. It’s hard when the food that is lighter, that weighs less, is more appropriate or desired (Field notes, January 2018).

They are also concerned that the imperative to do more each year—in order to be competitive in their grant applications—distracts them from addressing the “root causes” of inequities in their communities (Field notes, December 2018).

Related, while practitioners understand the “political pressures” that lead funders—especially state agencies—to ask them to report “dollars and cents,” they contend that this obscures what they perceive to be their “real strengths,” including the ways that they advance social change to meet the needs of their communities (Field notes, July 2017). In Springfield, a practitioner noted that that while they can track the value or pounds of food grown, they “don’t have a good way of talking about what is changed, what is different” in a neighborhood, when “suddenly, people are able to access fresh, locally grown, healthy food every day of the week…” (Field notes, January 2018). In Boston, a practitioner expressed frustration that the overall “reason for the work”—including “creating space for community members to articulate” their policy priorities and “to tell us…what something better looks like”—is never included in their organization’s grant applications “because it’s not measurable” (Field notes, January 2018). Conversely, practitioners express appreciation when program officers “give us a lot of room to be ourselves…[and] to say, ‘This is what we do, and this is what [data] we're collecting. And this is how it's creating a more just community food system’” (Interview 02).

These data don’t tell the whole story

Perhaps most of all, UA practitioners express concern that these data aren’t “telling the whole story” of their organizations’ work, as they understand it (Interview 02). The limits of quantitative metrics are a particular concern. “Funders typically want quantitative data,” observed a practitioner, “and there’s so much happening that we therefore don’t have a way to report” (Field notes, January 2018). Related, practitioners are concerned that their commitments to justice and equity are obscured by current practices of program evaluation.

There are multiple dimensions of practitioners’ concerns about what is left out of—and therefore made less visible—in current practices of program evaluation. To begin, they assert that even “very dense” measures still aren’t the same as “explaining…how the program impacts the world” (Interview 03). For example, practitioners point out that they are asked to do evaluation of their individual programs rather than assessing the contributions of their organizations as a whole: “we're being asked to evaluate, in these very narrow bands” (Interview 01). This assumes that each program within an organization is having independent effects, however, UA organizations hope that the effects of their programs are interactive and mutually reinforcing. Lastly, even organizations with relatively sophisticated dashboards for data collection find that “it’s really hard to see” the overall effects of their work in “simply quantitative” metrics: “It's like…this is total amount of foods that you've [grown] … This is [the] number of gardens…There's not a lot of depth to what kind of real change that's creating” (Interview 02). UA practitioners therefore are interested in “holistic approaches” to evaluation that ask broadly “how is the world different?” because of the work that they are doing (Interview 01) and allow for a “broader vision” that “goes beyond most funders’ quantitative outcomes” (Field notes, January 2018).

What lies beyond most funders’ quantitative outcomes includes what UA practitioners call “the intangibles,” by which they refer to “all the things” that are “not market related” (Field notes, January 2018). Practitioners perceive non-market outcomes as undervalued and underreported, which they attribute, at least in part, to the limitations of required measurement strategies. As well, practitioners are concerned that quantitative metrics focused on market-related outputs will systematically undervalue what it means for their communities to be moving toward greater racial justice and equity. For example, as a practitioner observed, simply tallying increases in acres of land under production does not convey the “impact of owning land—not because vegetables can be grown, but because of the reversal of hundreds of years of racism in a community that historically was not able to own land” (Field notes, July 2017). “How do we measure justice?” asked another (Interview 02).

Practitioners also suggest that the data generated by quantitative program evaluation is less impactful in policy advocacy and therefore potentially less relevant to their social change agendas. For example, they note that while stories are discounted in quantitative program evaluation, they are “what moves the legislators” (Field notes, December 2018) and have proven “powerful in…the advocacy process” (Interview 03). Consequently, as described below, when practitioners imagine alternatives to current practices of program evaluation, they seek to center their social justice and policy advocacy goals.

Envisioning alternatives

Urban agriculture practitioners are deeply interested in alternative and “holistic” forms of evaluation (Field notes, 2017). Specifically, they propose interventions that highlight their social justice commitments and could support UA organizations in working together, i.e., as a social movement.

Standardizing measures and defining collective impacts

While acknowledging that “there’s a long way to go,” UA practitioners express interest in identifying “best practices” for evaluation and then “moving towards standardization” (Field Notes, 2017). Echoing an earlier generation of food system advocates who proposed that “a ‘menu’ of approaches and indicators could be developed for projects with similar goals and strategies” (Webb et al. 1998), many indicated that it would be practically helpful to have a set of shared metrics. Shared metrics would not address practitioners’ concerns about what gets left out in quantitative program evaluation, nor would this strategy necessarily highlight UA organizations’ social justice missions. However, it could reduce the burden of program evaluation, including the current need to do multiple calculations for the same outcomes, as required by different funders.

Moreover, practitioners are interested in the potential of shared metrics for telling a collective story about UA, which is not visible in their individual program evaluations. That is, they hope that shared metrics will make clear “the bigger picture” of their collective impact (Field notes, March 2018). They imagine that if UA organizations were “asking questions in the same way…so that data could be aggregated across” organizations and cities (Interview 03), it tell a more complete story about UA in Massachusetts (Field notes, March 2018).

Related, practitioners conceptualize measuring what they do collectively as a step towards becoming a “collective campaign” or a “movement,” rather than a loose network of discrete organizations (Field notes, December 2018). They also see risks in this strategy, especially insofar as they may continue to compete for the same grants. At the meeting at which practitioners reviewed and discussed a draft report from this project, a Boston based urban farmer asked her peers directly, “Are we really ready to do this? How much are we willing to collaborate to manifest all these ideas?” (Field notes, December 2018). In response, practitioners agreed to work together to identify the data that they would be willing to share and to “craft metrics collectively.” This process is ongoing, shaped by awareness of the historic limitations of collective impact models in regard to equity (Hoey et al. 2017) and attuned to possibility that, without intentional and ongoing communication and collaboration, it runs the risk of replicating the very challenges it seeks to address.

Valuing justice and equity

Additionally, practitioners hope that by sharing data they will be able to identify “simple indicators that indicate larger systems change” (Interview 01). Aware that measurements “make things real” (Field notes, March 2018), they are interested in developing “interim measures” to assess how their work supports communities in “moving towards health and justice” (Field notes, July 2017) and addressing the “root causes” of inequities (Field notes, December 2018).Footnote 9

Interim measures are especially important given the longstanding structural inequities that UA practitioners seek to transform. As one practitioner observed, “the work that we do is based on a historical system of inequality. It took hundreds of years to get here. …[it] will take our whole lives to do this work” (Field notes, July 2017). Therefore, UA practitioners want to identify “the things that change” that indicate movement towards long-term social transformation (Interview 06). They suggest that these could include, for example, “recruiting and empowering leaders of color,” “addressing the racism throughout the food system,” increasing the sovereignty and “civic muscle” of BIPOC communities, and reducing racialized inequities in health (Field notes, July 2017 and December 2018; Interview 04).

Practitioners also seek to develop strategies that allow them to advance their policy advocacy efforts. Even within the constraints posed by their nonprofit status, they see this as an essential part of their work, as “when you keep pushing, you realize that you're not going to be able to make the changes that you’re trying to make by just doing these programmatic…things” (Interview 01). They observe, as well, that UA practitioners “do civic engagement and activism beyond urban ag” and want to lift up the activism emerging from their farms and gardens (Field notes, December 2018). For example, urban gardeners in Springfield “rallied together” against the proposed siting of a power plant as a “threat to children’s health”; this campaign gained urgency not only from Springfield’s extraordinarily high prevalence of asthma (and high rates of associated emergency room visits), but also from gardeners’ recognition of the disproportionate burden of asthma in BIPOC and low-income communities (Field notes, December 2018). At the same time, practitioners recognize that making advocacy a central focus of their work will require new kinds of conversations and relationships with funders.

Transforming relationships with funders

Across the Commonwealth, the leaders of UA organizations expressed interest in more interactive and engaged relationships with the public and private funders of their work. On the whole, practitioners believe that foundations “have the best intentions” and “believe that we should live in a better world” (Interview 06). Therefore, they see themselves as potential collaborators with funders that are interested in “honest conversations about how to fund this work in a way that makes sense for organizations and communities” (Field notes, July 2017).

For example, practitioners asked if “funders would be interested in a fundamentally different process?” and a “more equally powered relationship” in which “funders that are interested in community and in health… sit with community groups…in a space that is about co-creation” (Field notes, July 2017). This approach would “make UA practitioners the experts in the room” and recognize that

Those of us who are practitioners of this work, have dirt under our fingernails, [are] running these experiments [and] taking the beating when they don’t work. We are generating next ideas for work to move forward…we [could] have huge effects on the thinking that goes into how resources get distributed (Field notes, July 2017).

In such conversations, UA practitioners hope to challenge the neoliberal assumption that underpins both philanthropic funding and program evaluation, that is, “you give me money and I change how somebody else behaves…” (Field notes, December 2017). They state clearly that the predominantly BIPOC youth and community members who participate in their programs should not be seen as “objects of the work,” rather they are “agents of the work” of creating more just and equitable communities: “we’re not doing it for them, we're doing it with them” (Field notes, December 2017).

Related, UA practitioners report a growing awareness of “how much the dominant white culture is imbedded in foundations—their role and culture” (Field notes, December 2018). “As a movement” observed a practitioner, “more land and more food are not necessarily our end direction, but that’s where funders take us.” Therefore, she suggested, it is imperative that UA organizations come together to define their collective goals and avoid “the trap” of having them defined, rather, by grant writing for foundations. A program officer responded by encouraging practitioners to see themselves as having a “teaching role” about “how the world is changing” and how philanthropy must change in response (Field notes, December 2018). While this suggestion does not take into account the power imbalances between foundations and nonprofit UA organizations, it does highlight their relationships as an important locus for change.

Discussion

Powerful empirical analyses of the effects of state and philanthropic funding on movements for social and economic justice demonstrate that social movement organizations tend to be deradicalized, over time, through their relationships with government and/or philanthropic funders (Ferguson 2013; Francis 2019; Kohl-Arenas 2016). This paper suggests that some of the same mechanisms that have been identified as at the center of deradicalization processes also pose barriers to UA organizations as they consider collaborating, as a movement, to advance their commitments to justice and equity. Specifically, this analysis highlights the role of program evaluation in impeding the emergence of more collective and radical agendas in contemporary UA, suggesting that required measurement and reporting requirements are among the mechanisms through which the potentially radical project of food justice is coopted by neoliberal logics.

UA practitioners’ concerns about the practical challenges of evaluation—i.e., the demands made on staff time and other resources—are consistent with previous research that identifies “the required reporting and paperwork” of state and philanthropic funders as one way that organizational time and energy gets diverted from policy advocacy and movement building (Kohl-Arenas 2016, p. 41). Beyond the burden of these requirements, UA practitioners highlight the ways that program evaluation limits their ability to tell “the whole story” of their work, obfuscates their foundational commitments to justice and equity, and reifies their identities as discrete organizations, rather than a collective movement that aims to transform inequitable systems through multiple strategies, including systems change.

When asked about their visions for alternatives to current evaluative practices, UA practitioners propose interventions that they hope would address these challenges. For example, standardizing metrics could reduce the amount of time that staff have to spend on calculations for reporting requirements. Practitioners also raised the question of whether shared metrics would allow them to better convey the collective impact of UA in Massachusetts,Footnote 10 which they see as important to highlighting their collective contributions to making the “systemic changes” that are central to their missions. At the same time, they emphasize the value of gathering qualitative data, such as stories, especially in the context of policy advocacy (Lincoln et al. 2003; Shapiro et al. 2021).

More broadly, UA practitioners propose interventions to lift up the social justice and racial equity commitments of their organizations. First, they point to the importance of “interim measures” that could be used to assess progress towards their long-term social justice goals, such as improving health equity, creating sustainable local food systems, and challenging systems of racialized oppression. Second, and related, UA practitioners raise questions about whether it is possible for funders to recognize the expertise not only of practitioners, but also the members of the communities whom they see as partners in their work (Field notes, July 2017).

This may be a particularly propitious time for UA organizations to engage with funders in regard to issues of evaluation, equity, and justice. Both the Black Lives Matter movement and the COVID-19 pandemic have called attention to long-standing inequities in the food system and their devastating consequences for BIPOC communities (Alkon et al. 2020; Pirtle 2020), heightening funders awareness of the importance of centering racial equity in their work. A recent report identified evaluation as one of four “key barriers” that drive racial disparities in philanthropy; it recommended that funders be willing to learn from community-based organizations, especially those led by people of color, as “funders often lack understanding of culturally relevant approaches, leading them to over rely on specific forms of evaluation and strategies with which they are familiar” (Dorsey et al. 2021, pp. 12–13, emphasis added). Moreover, the report suggested that funders “actively build knowledge of, connection to, and mutual trust with communities most impacted by the social change issues you seek to address” (Dorsey et al. 2021, p. 18). While such recommendations fall short of demanding that philanthropies confront white supremacy, they should create opportunities for explicit consideration of how reporting requirements may have silenced and/or disciplined the voices of BIPOC leaders of UA organizations and communities.

The crises of this historical moment have also mobilized UA organizations in new ways. For example, in Spring 2020, UA practitioners in Massachusetts came together in a coalition to discuss how they have changed their operations to respond to COVID-19 safety concerns and to meet the soaring need for healthy food in their communities.Footnote 11 The UA Coalition continues to meet regularly to share resources and to identify opportunities for collective action, including policy advocacy. As well, it has served as a forum for discussing the history and limitations of program evaluation and advancing collaborative work-in-progress to design alternatives.Footnote 12

To be sure, this analysis is limited by its focus on one set of organizations, in one state, and during a relatively short period of time. Nonetheless, this work highlights UA practitioners’ important critiques of extant practices of program evaluation, as well as their visions of alternative practices. Together, their perspectives point to the ways that extant evaluation practices entrench neoliberal logics in UA organizations, while posing barriers to their mission-driven commitments to social justice, community health, and racial equity. They also highlight the intriguing possibility that metrics could be used to bring organizations together in collective action to advance their shared goals. To the extent that they succeed in working together with each other, and possibly with funders, to move beyond the neoliberal assumptions underlying extant practices of program evaluation, UA organizations may well reorient their field more decisively towards the radical project of food justice. As well, they may provide a model for other nonprofit organizations seeking to move towards collective action for transformative social change.