1 Introduction

Urban agriculture is defined as an “agriculture located in the city or on its outskirts, whose products are (mostly) destined for the city and for which there is an alternative between agricultural and non-agricultural use of resources, leading to competition but also to complementarities” (Moustier et al. 1999). Urban agriculture has been growing rapidly in countries of the Global North in recent years, a trend illustrated by the emergence of different types of projects with diverse purposes pertaining to the sustainability of cities (Orsini et al. 2020). Among them, professional intra-urban agriculture (PIUA) projects include rooftop farms and urban farms as described by the typology of urban agriculture published by (Opitz et al. 2016a). Conversely, PIUA excludes agricultural holdings as they are located in peri-urban areas, but also community, allotments, and backyard gardens, as they are not professional activities. Urban agriculture is considered professional when it commercializes agricultural products, goods, or services (Saint-Ges 2021). PIUA involves soil-based agriculture, hydroponics, raised bed farming, and rooftop farming, with both outdoor and indoor farming activities. These projects can pursue productive, environmental, social, or educational objectives and sometimes combine several growing techniques and several objectives (Clerino and Fargue-Lelièvre 2020). Thus, some projects focus on a productive objective while others combine productive and educational or cultural and social objectives, as illustrated in Fig. 1.

Fig. 1
figure 1

The diversity of professional intra-urban agriculture projects in France: a production-oriented rooftop farm in Paris, using aeroponic systems; b soil-based farm with a cultural and social focus in Saint-Denis; c raised bed farming with a productive and educational focus on a rooftop in Saint-Denis.

Full size image

To date, the trend of PIUA projects in the Global North is towards growth (Orsini et al. 2020). The exact number of PIUA projects developed in France is difficult to track as the sector is rapidly growing. However, the French professional network of urban farmers recorded 109 members and 952 cultivated sites in June 2022 (AFAUP 2022), which confirms the significance of this sector.

A range of sponsors supports the development of these projects by providing land or funding to project leaders, to develop their PIUA project, while experts advise sponsors and project leaders on projects’ development or selection. These stakeholders (sponsors, experts, and project leaders) might be public, private, or civil society organizations. Sponsors include local authorities, social landlords, urban planners, banks, and foundations. The project leaders may be urban farmers, civil-society organizations, specialized consulting firms, real estate developers, local authorities, or architects. The experts include research organizations, consulting firms, and public institutions such as the Chambers of Agriculture. Some sponsors can also be project leaders, and project leaders may in some cases be called upon as experts. Project leaders, sponsors, and experts are concerned with assessing ex-ante the sustainability of the PIUA projects, to guide projects’ elaboration, identify their strengths and weaknesses or to compare project proposals and select the winner of a call for proposals. In all these situations, stakeholders need to assess the sustainability of PIUA projects. In the literature, sustainable agriculture has been defined as a “form of agriculture that is economically viable, environmentally friendly, and socially fair. It contributes both to the overall sustainability of the local area in which it is practised, and to the achievement of global sustainable development objectives” (Zahm et al. 2015). However, from the conceptualization to the operationalization of PIUA, it is complicated to know whether the development of such forms of agriculture is sustainable.

Several assessment tools or frameworks have been developed to assess projects’ sustainability. Some assessment methods are designed to measure impacts in an ex-post situation, once the project has already been implemented or even completed. This is the case of the OECD method (OECD 2019), which proposes assessment criteria such as effectiveness and efficiency, estimated based on the achievement of objectives. Such criteria cannot be used ex-ante, at the proposal stage of a project, as they rely on field measurements and observations unavailable before project implementation. Also, the ability to reach objectives set upstream depends not only on the resources allocated to the project which are known at the proposal stage but also on external factors that are difficult to predict (Samset and Christensen 2017), such as involvement of inhabitants in the case of PIUA. The assessment methods proposed for agricultural development projects (World Bank 2006; Delarue and Cochet 2013) are also only suitable for ex-post use: these methods compare the impacts of projects to the scenario that would have prevailed without them, and are difficult to apply ex-ante as they rely on indicators measured when the project is implemented.

Other assessment tools are designed to evaluate ex-ante project proposals. However, many of them are specifically designed for industries and investment in new technologies (Tran and Daim 2008), which do not concern the majority of PIUA projects. They tend to focus on economic criteria, with little consideration for the social and educational dimensions which are important for many PIUA projects. Some decision-making methods can be used in agriculture but are applied at the plot scale rather than farm level, such as the MASC method (Sadok et al. 2009).

Some multi-criteria assessment methods are designed to assess the sustainability of farms, but the literature has pointed out that these tools are unsuitable for multifunctional farms, as they focus on agricultural activities and production and fail to take into account non-agricultural activities (Barbier and Lopez-Ridaura 2010), whereas PIUA projects include educational and social activities that are not strictly productive (Orsini et al. 2020). Also, they include criteria that are not applicable ex-ante such as soil cover index (Migliorini et al. 2018), pesticide use (Meul et al. 2008), or phosphorus and potassium use (Roesch et al. 2017).

Some assessment methods have been designed specifically for urban agriculture but mainly focus on evaluating the environmental impacts of urban agriculture or the ecosystem services provided (Langemeyer et al. 2015; Lin et al. 2015; Petit-Boix and Apul 2018; Wang and Pryor 2019), without considering social and economic aspects. For instance, Life Cycle Assessment has been applied to urban agriculture but only focuses on environmental impacts and resources used by urban farms (Dorr et al. 2021). A multi-criteria assessment method has been developed to assess urban socio-ecological systems but is applied at city scale and mainly focus on ecological and socio-economic flows exchanged among industries and the environment (Galychyn et al. 2022). Studies on the assessment of all dimensions of sustainability focus on evaluating benefits (Altman et al. 2014; Teitel-Payne et al. 2016; Mackenzie and Davies 2019) and on a farm’s contribution to urban sustainability (Tapia et al. 2021), but not the sustainability of the farm itself, when the internal sustainability of PIUA projects is an important assessment topic for sponsors and project leaders (Clerino and Fargue-Lelièvre 2020). A list of indicators has been published by FAO to assess the sustainability of urban agriculture, but it mainly targets urban agriculture in Global South countries where food security is a major goal for urban farmers. In this context, sustainability indicators linked to pedagogical aspects or improvement of urban life quality for instance are not included (FAO 2014).

Since the overall sustainability of PIUA projects cannot be assessed using established tools or methods, the stakeholders implement their own practices to do so. Several studies have shown that grassroots actors are a source of innovation by building new knowledge and practices (Leitgeb et al. 2011; Dolinska and d’Aquino 2016; Tambo and Wuenscher 2017) that can be disseminated and benefit to other stakeholders (Wu and Zhang 2013; Gupta et al. 2019; Salembier et al. 2021). We thus posit that PIUA stakeholders have developed innovative practices to assess the sustainability of their projects. Sanyé-Mengual et al. (2018), studying the conceptualization of sustainability from the urban agriculture stakeholders’ perspective, identified sustainability elements belonging to the three classic sustainability dimensions (environmental, social and economic). Nevertheless, they did not explore the details of the assessment practices, criteria or indicators used. Our study is aimed at investigating the stakeholders’ assessment practices, to identify their specific features concerning sustainability assessment of PIUA. We first focus on the sustainability dimensions considered, with a special interest for those that differ from the classic triptych “environmental, social and economic dimensions”. Then, we study the nature of criteria and indicators used and the way stakeholders use them.

2 Material and method

To explore stakeholders’ practices regarding the sustainability assessment of PIUA projects, we performed a “diagnosis of uses” (Cerf et al. 2012). A diagnosis of uses is an approach designed by ergonomists and agronomists, aiming at identifying issues faced by stakeholders when they implement a specific activity and the way they use diverse tools to cope with these issues. It is then a preliminary stage of the design of a new and more efficient tool. The diagnosis of uses relies on data collection from various potential users of the new tool. In our case, the diagnosis of uses applies to the sustainability assessment of PIUA projects: it is aimed at highlighting the diversity of criteria and indicators used by different stakeholders, in order to enrich the design of an assessment tool adapted to the diversity of stakeholders’ working situations. This diagnosis covered 19 case studies in which PIUA projects were evaluated. We selected case studies covering the wide range of possibilities concerning the type of stakeholders involved in the evaluation process, the type of projects, and the project selection processes. First, a census of PIUA projects in France was realized based on internet research, including press articles, public calls for projects, and consultation of the website of the French professional network of urban farmers (AFAUP). Internet research was completed by discussion with PIUA stakeholders (experts, sponsors, and project leaders). Among the identified projects, we selected 19 case studies with a snowball sampling. We started with a case study and carried on with other case studies involving different types of stakeholders, different selection processes, or different characteristics, in order to cover a diversity of case studies. Among the 19 case studies, 7 relied on implemented projects (which were around 2 years old when the diagnosis was performed) and 12 were at the project proposal stage. Some (12 among 19) deal with a single project which has been evaluated by sponsors when they decided to finance it or when project leaders were elaborating the project proposal. In other case studies (7 among 19), sponsors and experts assess sustainability of several projects, when calls for proposals cover different projects’ sites. Among the 19 case studies, three cover the whole French territory, thirteen the Ile-de-France Region, where most of the French PIUA projects are implemented, one the Pays de la Loire region, one the Haut-de-France region, and one the Centre-Val de Loire region. The details of the 19 case studies are presented in Table 1.

Table 1 Description of the 19 case studies, according to project location, whether it concerns one or several projects, the location of the crops, the cultivation techniques, its main activities, the projects’ selection process, and the source of data.
Full size table

In order to capture the diversity of assessment practices, the sample of case studies was selected to represent:

  1. 1.

    The range of stakeholders involved in PIUA projects: we interviewed local authorities, a public company, a public bank, urban planners, a private foundation, a food retailer, and social housing services. The project leaders also vary. We interviewed a property developer, a civil society organization, an urban agriculture company, and architectural firms. Finally, we met different experts mobilized by the sponsors for their expertise: consulting firms, companies specialized in urban agriculture, and a public regional authority for food and agriculture.

  2. 2.

    The diversity of agronomic characteristics of PIUA projects: PIUA projects vary depending on their location and their cropping system. Of the 19 case studied, the location of the crops was known at the time of the assessment in 12 cases: two of the cases provided for rooftop cultivation, four for ground-based crops, three for both rooftop and ground based crops, and three for indoor farming. The cropping systems were unknown at the time of the project assessment in 8 cases, soil-based in two cases, raised beds in five cases, hydroponics in one case, and combined different cultivation supports in three cases.

  3. 3.

    The diversity of project selection processes: sponsors can adopt a variety of processes to select a PIUA project. Of the 19 cases we studied, three used calls for expression of interest, three used calls for applications, eight used calls for proposals, two used mutual agreement processes, and two used requests for funding. Finally, one of the processes involved a closed competition. The different processes entail different levels of expectation from sponsors regarding the project proposals submitted by project leaders. With calls for expression of interest, applicants are free to propose a wide range of PIUA projects; the project proposal does not necessarily have to be a final version. Calls for applications are geared towards selecting a project to develop a particular space; here again, a wide range of PIUA projects are possible. Calls for projects generally target more precise needs than calls for expression of interest and calls for applications: the project proposal must fit a specified framework. Closed competition allows public actors to select PIUA projects based on expectations that are very well defined upstream. All these processes involve competitive project proposal bidding. Two processes allow for selection without competition: mutual agreement processes, where the sponsor and the project leader agree on the PIUA project together, and requests for funding, where the project leader submits a project to a sponsor, who decides to support it or not, without comparing it to other project proposals.

We studied the 19 cases by analysing three types of data:

  1. 1.

    Semi-structured interviews with a range of sponsors, experts, and PIUA project leaders: a total of 22 interviews were conducted. Some interviews covered different case studies, when a sponsor, expert, or project leader was involved in different case studies. For 18 of the 19 case studies, at least one interview was held: one interview for 11 of the cases, two interviews for four of the cases, and three interviews in three cases, when a diversity of stakeholders was involved. Interviews were held during a live meeting or through telephone for two of them. During the interviews, questions were asked about the history of the PIUA projects, their objectives, and characteristics, the stakeholders involved in the selection processes, and how the project proposals were evaluated, by who and based on which criteria or indicators. All interviews were recorded and summarized.

  2. 2.

    Official documents relating to selection processes: these are specifications and regulations for selection procedures that are published and publicly available. We studied a total of 11 official documents, which provided data for 14 of the 19 case studies. Some documents gave information for two case studies related to the same call for proposals. We extracted from these documents’ information about the project’s objectives, some of their characteristics, but also about the selection processes, how they were put in place, the stakeholders involved, and some of the criteria and indicators used to assess project proposals and select awardees.

  3. 3.

    Project proposal analysis frameworks supporting the selection of winning proposals as part of calls for projects, when they could be retrieved: we were able to access two analytical frameworks, which provided information on two of the 19 case studies. These frameworks, which do not always exist, are confidential and difficult to access. They reflect the internal discussions of a selection committee and are generally not shared outside this committee. We extracted, from these documents, criteria and indicators used to compare project proposals during selection processes.

The study of the interviews recordings and documents desk review allow us to identified items used by stakeholders to assess PIUA projects, which we considered as criteria. Criteria are variables that help approach the sustainability and serve as a basis for formulating assessments (Lairez et al. 2016), such as Local consumption and affordability of products. We systematically recorded in an Excel database every criterion and classified them according to sustainability themes and sub-themes, to ease their presentation and understanding. Themes and sub-themes encompass several criteria, such as Contribution to global sustainability and Contribution to access to quality local food. Some themes and sub-themes have been directly mentioned in documents or during interviews. For some criteria, stakeholders specified the way they estimated it. We identified these means of criteria estimation as indicators. Indicators are quantitative or qualitative variables used to estimate criteria (Lairez et al. 2016), such as Share of the production sold locally or Local sale of the production. We obtained 10 different themes, 67 criteria, and 138 indicators.

We analysed the diversity of the different themes, criteria, and indicators used but also their frequency of use (occurrences) among the 19 case studies, knowing that different case studies may use the same criteria or indicators.

A statistical analysis based on a Hierarchical Clustering (HC) was also carried out to categorize the case studies according to the number of themes, criteria, and indicators they applied (number of themes, criteria and indicators used by each case study). HC is a statistical analysis allowing the identification of clusters within a dataset. It provides a dendrogram, also called cluster tree, which classifies clusters of data with similar profiles in a hierarchical manner. Our dendrogram represents case studies according to their Euclidian distance, which corresponds to their level of dissimilarity. The smaller the Euclidean distances between two case studies, the more similar the profiles of these case studies are. The HC was performed using XLSTAT software, using Ward criteria and an automatic truncation based on inertia. It allowed us to identify three groups of case studies.

3 Results and discussion

3.1 Diversity of assessment structure among case studies

We observed a wide disparity in the number of themes, criteria, and indicators used per case study as presented in Fig. 2. Some case studies assess a small amount of sustainability themes (with a minimum of three themes assessed for CS7, CS8, CS9 and CS17), whereas CS14 and CS15 consider up to 10 sustainability themes. An average of 15 criteria were used per case, with a minimum of 4 criteria for CS9 and a maximum of 33 for CS6. While some projects were evaluated based on a very small number of themes and criteria, others were analysed in great depth, suggesting that the evaluators’ expectations can vary widely. Finally, for some case studies, we identified a large number of indicators (with a maximum of 30 for CS6), whereas no indicators were identified for CS19, suggesting that indicators may be implicit or confidential.

Fig. 2
figure 2

Number of different themes, criteria, and indicators used by each case study (CS). For each case study, the city and crop location are specified.

Full size image

For instance for CS7 corresponding to a farm run by a local association in a school, sustainability assessment was based on (1) the theme Coherence and technical robustness assessed by criteria Respect and personal fulfilment of employees and Sustainability of contracts for the staff; (2) the theme Contribution to local sustainability assessed by criteria Fostering of neighbourhood life, Suitable activities proposed on the farm for vulnerable populations, and Job creation; and (3) the theme Contribution to global sustainability assessed by criteria Preservation of biodiversity, Hosting of school groups, and Organization of workshops. Three indicators were identified for CS7: Creation of jobs with permanent contracts (used to estimate two different criteria: Sustainability of contracts for the staff and Job creation), Hosting of pupils during school time, after school and during vacations (used to assess the criterion Hosting of school groups), and Conducting workshops on nature with a science teacher (used to assess the criterion Organization of workshops).

3.2 Specific features of the sustainability dimensions and themes for PIUA projects

We identified 10 different themes of sustainability, and gathered those under four dimensions of sustainability (Fig. 3).

Fig. 3
figure 3

Occurrences of criteria used by the 19 case studies according to their related sustainability dimension (in dark blue) and sustainability theme (in pale blue). The number of occurrences for each dimension equals the sum of occurrences of the themes below.

Full size image

3.2.1 Nature of the sustainability dimensions and themes assessed by stakeholders

The first dimension encompasses themes pertaining to the external sustainability of a project. This concept of external sustainability applied to urban agriculture was defined by Aubry et al. (2012), based upon the territorial sustainability of agriculture that refers to the contribution of agriculture to the sustainable development of its territory. In an urban context, external sustainability is linked to the multifunctionality of PIUA and gathers the services provided by PIUA projects to the city. In our study, external sustainability relies on themes such as (i) project’s contribution to sustainability at a local level (city, neighbourhood), (ii) at a global scale (issues relevant at the country or world scale, such as biodiversity or heritage preservation), and (ii) project’s contribution to the sponsor’s sustainability (positive contribution to its image or to its economic added value for instance).

The second dimension comprises themes and criteria relating to the internal sustainability of a project. In agriculture, internal sustainability can be defined as the internal goals that a farmer wants to achieve (Zahm et al. 2018). In an urban context, internal sustainability relies on different themes such as the project’s technical coherence, its economic robustness, and the management of regulatory aspects which generally address safety or trading standards in the Global North (respecting the regulations in force, knowledge of the necessary authorizations).

The stakeholders interviewed and the documents analysed pointed out the importance of a third dimension of sustainability assessment: the innovative nature of a project. Numerous interviews and documents highlighted that new issues are emerging around urban agriculture, such as limited and non-traditional access to land (i.e. rooftop or underground farms and precarious lease), use of urban soils and alternative growing media (i.e. soil pollution management and use of coffee ground as substrate), the specific legal and political environment, and the functions which are not strictly productive or the involvement of non-traditional farmers (Pfeiffer et al. 2015). All these distinct features encourage the development of innovative practices, particularly important to adapt to specific urban challenges (Schans et al. 2014). Innovative projects offer levers to overcome these challenges and ensure their sustainability, but novelties applied in urban agriculture also enhance the social, ecological, and economic impacts of practicing agriculture within urban areas, holding the potential to contribute to sustainability (Opitz et al. 2016b). To evaluate the innovative nature of a project, our results suggested to rely on two elements: (1) project’s originality, whether the project involves an innovation, by introducing new concepts (such as new technology or new form of organization) or by representing a novelty when a similar project has not been seen before, and (2) project’s participation in the evolution of knowledge, by generating new knowledge through experimentation or by disseminating new knowledge through workshops and trainings. This configuration echoes the Concept-Knowledge design theory or CK theory (Hatchuel and Weil 2009). This theory considers innovative design as the result of an expansion of both concepts and knowledge. The two spaces evolve together during the design process, as formulating a new concept leads to requesting new knowledge, which can lead in the formulation of another new concept.

The fourth and final dimension of sustainability identified in interviews and documents encompasses themes and criteria for assessing the credibility of a project leader, i.e., evaluating the robustness of the project’s governance and the adequacy of the project leader and partners’ profiles (references, training, and motivations). These criteria serve to ascertain whether the project leader is able to ensure the implementation of the project and the achievement of its objectives (set in the project proposal). Such criteria are considered as particularly relevant by stakeholders interviewed, as many project leaders do not come from the farming world, and a lack of training in agriculture is perceived as a risk for the sustainability of projects, as described by Sanyé-Mengual et al. (2018).

3.2.2 Frequency of assessment of the sustainability dimensions

Figure 3 highlights that most criteria used by the 19 case studies refer to external and internal sustainability (229 occurrences of criteria assessing external and internal sustainability). Case studies use more criteria associated with external sustainability than with internal sustainability (141 occurrences for criteria assessing external sustainability versus 88 occurrences for criteria assessing internal sustainability), pointing that in PIUA projects considerable attention is paid to the project’s contribution to sustainability at broader levels, and especially at local level, on the scale of the neighbourhood, city or region in which the farm is based.

The criteria pertaining to the credibility of the project leader or the innovative nature of the project are less used than the ones related to internal and external sustainability (28 occurrences for criteria related to the credibility of the project leader and 27 occurrences for criteria related to the innovative nature of the project) but are far from anecdotal in the assessment of PIUA project sustainability. Few previous studies attest to interest in integrating innovation: one study highlights that innovation is an important dimension for defining sustainable urban agriculture (Sanyé-Mengual et al. 2019), and the IDEA tool displays a sustainability objective based on the production and sharing of knowledge to assess sustainability of rural farms (Zahm et al. 2018). However, none of them include the credibility of the project leader, whereas Chopin et al., (2021) stressed that governance aspects and the characteristics of the project leader ought to be included in the sustainability analysis of farming projects.

3.3 Diversity of criteria for evaluating the sustainability of PIUA projects

The four dimensions of sustainability are organized into themes covering 67 different criteria. Some themes are divided in sub-themes for ease of reference, within the external and internal sustainability dimensions. As presented in Tables 2, 3, 4, and 5, we analysed the nature of the 67 different criteria and observed the occurrence of each criterion among the 19 case studies to highlight which criteria are mostly used by PIUA stakeholders.

Table 2 Criteria related to external sustainability organized per dimensions and themes and their occurrence among the 19 case studies.
Full size table
Table 3 Criteria related to internal sustainability organized per dimensions and themes and their occurrence among the 19 case studies.
Full size table
Table 4 Criteria related to the innovative nature of the project organized per dimensions and themes and their occurrence among the 19 case studies.
Full size table
Table 5 Criteria related to the credibility of the project leader organized per dimensions and themes and their occurrence among the 19 case studies.
Full size table

3.3.1 Nature of the sustainability criteria used by stakeholders

The external sustainability dimension is the richest, with 3 themes and 11 sub-themes covering 35 different criteria (Table 2). The internal sustainability dimension comprises 3 themes, 6 sub-themes, and 22 different criteria (Table 3), the innovative dimension comprises 2 themes and 5 different criteria (Table 4), and the dimension of the project leader’s credibility also counts 2 themes and 5 different criteria (Table 5).

First, we observed that some criteria are similar to the criteria found in existing assessment tools. For instance, the criteria Monitoring and limiting of resource consumption and Preservation of biodiversity are similar to the criteria Use of inputs and Biodiversity found in MOTIF tool (Meul et al. 2008). The criterion Limiting soil and water pollution and the theme Contribution to heritage preservation are similar to the indicators Reduce impact on human health and ecosystems and Preservation of local heritage found in IDEA tool (Zahm et al. 2018). Then, criteria related to the consistency and economic robustness such as Aid and Subsidies obtained or expected or Turnover and expected results are also similar to criteria found in IDEA tool.

We also note original criteria compared to those generally found in methods for evaluating the sustainability of rural farms (Grenz et al. 2009; Schader et al. 2016; López-Ridaura et al. 2002; Meul et al. 2008; FAO 2014).

Some criteria assess the risk, for the farm, of being refused by the neighbourhood: Limitation of disturbances for neighbours, Aesthetic quality of the farm, and Take-up of the project by locals. The stakeholders interviewed explained for instance that bad smells or noise due to agricultural activities can increase the risk that neighbours rejecting the project; conversely, involving inhabitants in the farm’s activities can decrease this risk. They also mentioned the importance of the aesthetic quality of the farm, referring to inhabitants’ perception of the farm. For instance some stakeholders may consider that plastic greenhouses are not aesthetically pleasing and do not participate to the beauty of the city landscape. Previous studies identified the risk of the farm being refused by the neighbourhood (Desrousseaux and Stahl 2014; Specht et al. 2016), but no corresponding criteria were included in existing assessment methods.

We identified other criteria specific to the risks linked to an urban location such as Capacity to move to another location, Land tenure compatible with urban agriculture, and Adaptation of the project to a rooftop location. For instance, in CS18, land is provided under a short-time lease before the start of a construction project, so the ability of the farm to move to another location is an important criterion. In the case of a rooftop location like in CS5, specific attention is made to the safety measures put in place or the bearing capacity of the roof. These criteria allow to assess how the project will mitigate the risks related to precarious or unsuitable nature of the land available to set up agricultural activities that were pointed out by Sanyé-Mengual et al. 2018.

Conversely, other criteria highlight benefits specific to intra-urban settings, such as a Contribution to the attractiveness of the neighbourhood, Improvement of locals’ living environment, or Fostering of neighbourhood life and the capacity to provide ecosystem services, such as Contribution to stormwater abatement or Contribution to the reduction of the urban heat island effect. Contribution to the attractiveness of the neighbourhood is a criterion concerning deprived neighbourhoods, suffering from a bad image, where the implementation of a PIUA projects may attract investors and instill a new dynamic. These neighbourhoods are particularly targeted by urban renewal policies. Moreover, by avoiding long transportation time and by selling perishable products shortly after harvest, intra-urban agriculture makes fresh fruits and vegetables available to city dwellers, like in CS11 where the production is located within a supermarket. We identified the criterion Freshness and nutritional quality of produce to assess the benefits of growing food as close to the consumer as possible, benefits that were pointed out in the literature (Opitz et al. 2016a).

The integration of the theme Contribution to the inclusion of vulnerable populations and the related criteria confirms that intra-urban agriculture is a real support to develop activities with social benefits that might be integrated to the primary goals of the farm and not only considered as co-benefits of the production activity. This theme includes several criteria, such as Creation of vocational rehabilitation jobs, which indicates that the vulnerable people in this case are populations unemployed for a long time. Another criterion included is Accessibility for people with reduced mobility. In this case, vulnerable populations refer to people with reduced mobility. The last criterion included in this theme is Suitable activities proposed on the farm targeting disabled people, students in school dropout situations or precarious populations. This is the case for instance in CS1 where the farm is co-designed with a local association which provides shelter to homeless people in order to include them in the farm activities.

The ecosystem of actors specific to PIUA also justified the addition of specific criteria. Integration of the project into the sponsor’s strategy underlines that PIUA projects are also guided by objectives specific to the stakeholders supporting their development. Contributing to the sponsors’ strategy can justify why a sponsor allocates resources to the project (subsidies or land) and is also relevant when evaluators need to justify project selection to their hierarchy. Compliance with urban agriculture regulations is also a criterion linked to the specific nature of PIUA stakeholders. Sponsors and project leaders are sometimes unfamiliar with these regulations, most often derived from classical agricultural regulations.

Our study suggests to consider innovation as a sustainability dimension relying on five criteria related to the evolution of knowledge (Ability to generate new knowledge, Ability to disseminate new knowledge, and Replicability of the project) and to the originality of the project (Project involving an innovation and Novelty of the project). These criteria are consistent with some criteria proposed by Le Masson et al., (2010) to assess an innovative design process: by evaluating the Ability to generate new knowledge and the Ability to disseminate new knowledge, we assess the Value of knowledge generated by the projects, and by evaluating the Replicability of the project, we assess the Robustness of the concepts and knowledge generated. These criteria are particularly important in PIUA as the sector is fairly young. Indeed, PIUA often requires the implementation of new technologies about which very little is known (Specht et al. 2013; Orsini et al. 2020). Consequently, existing projects are actively involved in creating and capitalizing on knowledge on various issues linked to urban agriculture. An existing tool includes a sustainability criterion related to the ability of the farm to adopt a new technology (López-Ridaura et al. 2002), which does not consider social innovation or capacity to create and share knowledge. Another existing tool includes a criterion related to the participation in innovation networks (Zahm et al. 2018), which does not consider the originality of a project or its ability to replicate.

Finally, we identified the criterion Complementarity with the rural farms of the area, which reflects the sponsors’ desire to develop spatial and functional complementarities between rural and urban agricultures, such as selling rural farm’s products on an urban farm to offer a more diverse range of products to the consumer. Similarly, a previous study identified the need to consider the complementarity of urban and rural agriculture in land-use planning (Valente et al., (2014)).

3.3.2 Frequency of use of the sustainability criteria by stakeholders

Looking at the occurrences of criteria in Table 2, we can see that the themes mostly assessed among those related to local sustainability are Connection with local actors (8 case studies on 19), Improvement of locals’ living environment (used by 7 case studies on 19), Aesthetic quality of the farm (7 case studies on 19), and Local consumption and affordability of products (7 case studies on 19). The regular use of these criteria emphasizes the links between a farm and its surroundings, highlighting that PIUA takes place on an ultra-local level, on the scale of a city or even a neighbourhood. Regarding the contribution to global sustainability, the most used criteria are Monitoring and limiting of resource consumption (used by 8 case studies on 19), Preservation of biodiversity (6 case studies on 19), and Recycling and waste recovery (6 case studies on 19), highlighting that a PIUA project must consider, for numerous stakeholders, environmental issues. The last criterion of the external sustainability dimension which is mostly used is Integration of the project in the sponsor’s strategy, used by 9 case studies on 19, which confirms that PIUA projects are part of overall strategies, territorial or even national, supported by private companies, local authorities, or national public stakeholders.

For internal sustainability (Table 3), the most used criteria are Amount of investments compared to financing capacity (used by 12 case studies on 19) and Robustness of the marketing plan (11 case studies on 19). The use of these economic criteria by most of the case studies confirms the importance of economic viability and robustness of the financing plan for PIUA projects, no matter what the crop location or cultivation techniques are, as the criterion Amount of investments and financing capacity is used among others by CS3 (crops ground-based and cultivation soil-based), CS13 (crops located on rooftop and ground-based, cultivation with hydroponics and raised-beds), and CS14 (crops located indoor, cultivating in raised-beds). The realistic nature of the technical proposals, evaluated by 7 case studies on 19 with the criterion Realistic cropping systems and yields, confirms that evaluators wish to anticipate the risks that high investments represent, especially as urban farmers are often not coming from the agricultural sector and might lack agricultural skills.

Within the dimension related to the innovative nature of the project (Table 4), the criterion Project involving an innovation is used by 11 case studies on 19, confirming the important link between PIUA and innovation, again no matter what the crop location or cultivation techniques are, as the criterion is used when crops are located on rooftops (CS5), indoor (CS6, CS14), or ground-based (CS18), and when the cultivation techniques are hydroponics (CS11), soil-based (CS12), or raised beds (CS6, CSS14, and CS18) .

Within the dimension related to the credibility of the project leader (Table 5), the most used criterion is Quality and consistency of references of the project leader’s team and partners, used by 11 case studies, which can balance the risk induced by innovation. The project might implement a new cultivation technique or involve a social innovation which represents a risk if few feedbacks of similar projects are available. However, relevant and consistent references of the project leader might mitigate this risk.

3.4 The qualitative nature of the sustainability indicators identified

Our analysis identified 138 different indicators used by at least one of the 19 case studies. We identified indicators within documents and interviews’ verbatim as a way to estimate sustainability criteria. For instance, during an interview related to CS18, a stakeholder explained that to ensure the involvement of inhabitants in the PIUA project they “preferred to test things through workshops [...] with the city’s non-profit organizations [...] to see whether there were any advantages for the inhabitants”. In this case, we considered that testing farm activities through workshops was a way to assess the criterion Take-up of the project by locals and formulated the qualitative indicator Implement workshops to test farm’s activities, which can be answered by “Yes” or “No”. Some indicators are used by different case studies (such as Farm site open to the public or Response to a political will) even most of them are unique and used by only one case study. Some indicators are used by a case study to assess different criteria (such as the indicator Creation of jobs with permanent contracts used by CS7 to assess both criteria Sustainability of contracts for the staff and Job creation).

Table 6 presents several indicators, their related criteria, their nature, and the data they are extracted from. The sample of indicators has been selected as they are related to different sustainability dimensions and criteria and exemplify the qualitative and quantitative nature of indicators encountered. A complete list of the 138 indicators and related criteria is provided in Supplementary Materials.

Table 6 Overview of the diversity of sustainability indicators identified.
Full size table

Of the 138 indicators, only 31 are quantitative indicators whereas 107 are qualitative. Quantitative indicators thus account for just 22% of the indicators recorded.

Table 7 analyses how qualitative and quantitative indicators are spread between the different sustainability dimensions and themes, to estimate which kind of indicators are the most used to assess which theme of sustainability. Note that some indicators are used by different case studies or used by a same case study to assess different themes or criteria: therefore, the number of occurrences—184—is higher than the number of indicators—138.

Table 7 Occurrences of indicators used by the 19 case studies according to their related sustainability dimension and theme.
Full size table

Table 7 shows 25 occurrences of the theme Consistency and economic robustness among the 34 occurrences of quantitative indicators, underlining that quantitative indicators are mostly used to assess economic criteria. However, 27 occurrences of qualitative indicators are also recorded to assess this theme, confirming that qualitative indicators represent an alternative to assess economic criteria, such as Diversification of the sources of income or Farmer paid as city agent. In addition, qualitative indicators are used to assess a wide diversity of theme, unlike quantitative ones, such as Contribution to global sustainability (31 occurrences), Contribution to local sustainability (25 occurrences), or Coherence and technical robustness (22 occurrences).

In practice, PIUA stakeholders thus tend to use more qualitative than quantitative indicators to assess the sustainability of projects. Qualitative indicators, especially those reported as “present/absent” or “yes/no” that do not include thresholds, are easier to establish and to articulate. It is therefore unsurprising that in the absence of a consensual assessment method proposed by scientists, PIUA stakeholders have themselves developed qualitative indicators.

This trend is not systematically observed in the literature, or at least to a lesser extent. Only 25% of the indicators used in the Five Borough Farm tool are qualitative (Altman et al. 2014), against 55% in the FADEAR tool (FADEAR 2013) and 62% in the IDEA method (Zahm et al. 2018). However, our results support the conclusions of some studies which stress that qualitative indicators are essential for evaluating sustainability, alongside quantitative indicators, as they allow for better inclusion of stakeholders’ values and practices impacting their capability to implement sustainability (Scerri and James 2010). Likewise, they align with the finding that qualitative methods are in the majority for the assessment of the socio-cultural benefits of urban agriculture (Ilieva et al. 2021).

3.5 Identification of three groups of case studies according to assessment practices

The last stage of analysis focused on the links between case studies and themes, criteria, and indicators used for assessment. We analysed whether certain assessment situations led to the use of specific themes, criteria, and indicators. However, it was not possible to search for links between sustainability criteria and types of stakeholders, as there is overlap between them. For instance, when projects are selected by a sponsor, this one is often advised by experts, and it is thus not possible to differentiate the criteria proposed by the sponsors from those emerging from the experts. Similarly, during selection, criteria also emerge from project leaders (through project proposals), and again, it is not possible to know whether the criteria used to select the project were anticipated by the sponsor or inspired by project leaders. Thus, we decided to focus on the links between case studies and criteria.

A Hierarchical Clustering divided the case studies into three groups, maximising the inertia between them, according to Ward criteria. The results are presented as a dendrogram in Fig. 4, where case studies are classified according to their Euclidian distance (or dissimilarity). Each group includes case studies with similar trends in the number of themes, criteria, and indicators used. Group 1 includes five case studies (CS3, CS5, CS6, CS14, and CS15), Group 2 gathers nine case studies (CS1, CS2, CS4, CS7, CS9, CS11, CS12, CS17, and CS19), and Group 3 includes five case studies (CS8, CS10, CS13, CS16, and CS18).

Fig. 4
figure 4

Dendrogram of the Hierarchical Clustering identifying three groups of case studies.

Full size image

The characteristics of the groups are presented in Tables 8 and 9.

Table 8 Features of the three groups of case studies identified by a Hierarchical Clustering, according to the average number of criteria and indicators used.
Full size table
Table 9 Features of the three groups of case studies identified by a Hierarchical Clustering, according to the average occurrences of criteria per case study.
Full size table

Group 1 includes 5 case studies, using the highest number of themes, criteria, and indicators to assess their projects. This group uses in average 29 criteria to assess sustainability, underlining that the evaluators of Group 1 have a precise idea of the kind of project they expect. Indeed, within this group, the selection processes are calls for projects and closed competition, processes used when evaluators already know precisely the PIUA project they want to implement. Group 1 uses the most indicators (17 in average per case study), several criteria related to Coherence and technical robustness (5 in average), and is the only group to use criteria related to Management of regulatory aspects, suggesting that the evaluators have advanced knowledge about technical and legal related issues. This is confirmed as all the case studies of Group 1 call in external expertise in urban agriculture, either during the selection process, or both before and during the selection process.

The second group includes 9 case studies, using the lowest average number of criteria and indicators (7 criteria used in average per case study and 3 indicators). Among the 9 case studies, 7 did not call any external expertise in urban agriculture, which can explain the low number of criteria and indicators used. In this group, 4 case studies ended up in several projects and 5 in a unique project; and the group includes the 2 case studies where projects are implemented through a mutual agreement between sponsors and project leader. Therefore, the low number of criteria might not only be linked to the lack of expertise in PIUA but can be a deliberate strategy: some case studies might use few criteria as they concern different sites of implementation or as they will create criteria during the process of project elaboration. Case studies of Group 2 use most of their criteria to assess external sustainability and more precisely Contribution to local sustainability (2.9 occurrences representing 39% of the occurrences for this group, Table 9) and Contribution to global sustainability (1.1 occurrence representing 15% of the occurrences for this group), underlining that these topics are of primary concern for evaluators even when the project is not defined, and may be their first motivation to implement a PIUA project.

Group 3 includes 5 case studies, using an intermediary number of criteria and indicators compared to Group 1 and Group 2 (average of 14 criteria and 12 indicators used by case studies). Four of the five case studies of Group 3 involved external expertise in PIUA either during or before and during the selection process of projects, explaining why this group uses more criteria and indicators than Group 2. Case studies of Group 3 also implemented selection procedures such as call for expression for interest and calls for applications, which are selection processes that allow a wide variety of project proposals and are generally launched when the project idea is not totally mature. This can explain why the number of criteria and indicators used is lower than in Group 1. The case studies of Group 3 focus on Consistency and economic robustness (average of 3.8 criteria representing 27% of the occurrences) suggesting that economic aspects are a major concern for evaluators.

The three groups use different kinds and amounts of criteria and indicators to assess PIUA projects. A first hypothesis to explain it could be the co-evolution of the project and the assessment process: the process of assessing PIUA projects is not fixed in time, but evolves alongside the project. The way in which the sustainability of a PIUA project is assessed changes as the project progresses, adapting to the evolution of the project. The clearer the project idea is, the more accurate and specific the criteria to assess it can be. This is the result of a dialogue between stakeholders involved (sponsors, project leaders and experts), and represents a process of mutual learning between them. Thanks to discussions, they refine the characteristics of the project itself, the way it is perceived as sustainable, and of the criteria to assess it. In our study, Group 2 could represent the first stage of evolution, based on project ideas and few criteria, then Group 3 the second stage of evolution, when the project idea is a bit more mature and criteria more diverse, and Group 1 the third stage where project proposals are defined in detail and criteria to assess them precisely designed. The need to adapt the assessment method to the project cycle has already been highlighted in the context of the assessment of the innovation’s social impact (Molecke and Pache 2019). McConville and Mihelcic (2007) have also developed a matrix to assess water and sanitation project sustainability combining sustainability factors and project life-cycle stages. However, no existing method to assess farm sustainability suggests different sets of criteria according to the project development stage (Grenz et al. 2009; FAO 2014; Schader et al. 2016; Zahm et al. 2018), whereas the need to develop a temporal dynamic assessment of farm sustainability has been described (Chopin et al. 2021).

A second hypothesis to explain the disparities between the three groups regarding the number and type of criteria used could be that evaluators adapt the design of criteria to their use. In some situations, a large number of criteria and indicators is needed to explain why a project should be selected, when an evaluator needs to convince a supervisor or a local politician. A diversity of criteria can also be needed when the design of a project is the result of a collective process involving stakeholders who may have different objectives. Both situations are represented in Group 1 where the case studies involve local authorities as sponsors with different partners and external expertise, suggesting that the decisions to design and select PIUA projects involved various stakeholders and needed to be well justified as it involved public investments. In other cases, few criteria and indicators are needed, as the evaluators do not want to design the project in detail but want to allow another stakeholder to do it without too many constraints and to foster innovation to receive original project proposals. A small number of criteria can also be formulated when the decision to select a project does not need to be thoroughly argued or when project criteria will be the result of a collective learning: this is the case when a project is selected by a single actor or based on mutual agreement between different stakeholders. Both situations are present in Group 2 where external expertise was rarely called upon and mutual agreement used, highlighting more individual and intuitive decisions than in Group 1, which needs less justification.

Our study is part of a project aiming at designing a specific tool to assess the sustainability of PIUA. As discussed by Cerf et al. (2012), when a new tool is designed, acknowledging diversity among its application implies the need for flexibility into the tool. In our case, the three groups of case studies identified confirmed a diversity of assessment practices (using more or less criteria and indicators, focusing on different sustainability themes), and suggest a diversity of uses for the assessment tool to be designed. Flexibility means that the assessment tool will provide information relevant for a diversity of decision contexts (for different stages of project development) and operating methods (whether the assessment is made collectively or not, to support the design of a project, justify its selection, assess its potential impacts, etc.). Many studies pointed out the low level of use of decision support tools due to the gap between the way designers elaborated the tool and the way users make decisions (Díez and McIntosh 2009; McCown 2002; McIntosh et al. 2007); therefore, our study enriches the understanding of users’ assessment practices that should be taken into account for the design of a flexible tool to assess sustainability of PIUA projects.

4 Conclusion

The rapid development of PIUA projects is generating the need to assess their sustainability. Stakeholders such as sponsors, project leaders, and PIUA experts have developed their own assessment practices. This study examined these practices with a view to shedding light on the specific features of the assessment of PIUA projects’ sustainability as implemented by the stakeholders involved. The analysis of 19 case studies allowed us to identify four dimensions of sustainability used to assess PIUA projects: external sustainability, internal sustainability, credibility of the project leader, and innovative nature of the project—the last two of which are original in the context of sustainability assessment in urban agriculture. We also identified 67 assessment criteria, some of these being particularly original, compared to the classical methods of assessment of agriculture: for instance, a project’s contribution to the appeal of its neighbourhood, its complementarity with the rural farms in the area, or the freshness of its produce. Finally, we showed that assessment practices differ among case studies by identifying three groups of case studies, some using a large number of criteria and indicators, other only a few, and focusing on different sustainability themes. Our work identified two hypotheses to explain this diversity of practices, namely, an evolution of assessment practices over time and a variety of assessment situations. The sustainability assessment practices of PIUA stakeholders are proving to be a source of innovation, informing the sustainability assessment of urban agriculture. Our study could be the first step to the design of a tool to assess sustainability of PIUA projects. Our conclusions confirm the need for a flexible tool where criteria and indicators used can vary according to the project stage and the assessment situation. We are aware that the set of criteria and indicators identified based on 19 case studies is not comprehensive; however, it can be a basis to design a tool for assessing the sustainability of PIUA projects, which will be completed by supplementary criteria identified by local stakeholders as relevant for their specific situation. In this perspective, the assessment tool should be very flexible, both to sort the relevant criteria and to complete the list of criteria, matching the diversity of stakeholders’ practices and expectations.