Indicators of Social Innovation for Cities’ Action Plans Evaluation

Abstract

For each of the ten categories of the social innovation component of an action plan (SIAP), a set of indicators is developed which can be utilized by the public administration to monitor implementation and outcome of social innovation actions at urban level.

For each of the ten categories of the social innovation component of an action plan (SIAP), a set of indicators is developed which can be utilized by the public administration to monitor implementation and outcome of social innovation actions at urban level.

In this chapter, a comprehensive catalogue of over one thousand indicators is provided, according to the ten social innovation categories described in this chapter and cluster according to the five criteria of effectiveness, efficiency, sustainability, replicability, scalability explained in the next section. Indicators are derived from scientific literature, existing projects or, when not available from existing sources, developed by the authors.

3.1 Impact Measurement and Evaluation Approach

In general terms the evaluation will take place at the level of the cities’ action plan, and at the level of initiatives stemming from the single categories of the action plan. The evaluation approach is based on indicators, which build on the intervention logics in Chap. 2 and are integrated with indicators extracted from existing frameworks. The evaluation criteria for the plan are effectiveness, efficiency, relevance, replicability, and scalability (European Union, 2021). These criteria are applied to each city but tailored according to the respective objectives of each city.

• The effectiveness criterion refers to the capability of the plan to reach its intermediate and strategic objectives. The evaluation considers the quality of the plan proposed solutions, its community engagement, how the technical solution interacts within existing technical and dataset ecosystems, as well as how effectively it has improved urban sustainability;

• The efficiency criterion aims to evaluate whether the plan outputs and outcomes were achieved at a reasonable cost. The evaluation considers the efforts, time and budget provided by stakeholders as well as the capability of the project to obtain the same results with lower expenditure;

• The relevance criterion aims to evaluate if the objective of the plan intervention is adequate to respond to the needs of the stakeholders. The evaluation considers the profiles of the stakeholders in terms of needs, perceived benefits, and participation, as well as the methodological and technical design of the project;

• The replicability criterion refers to the ability of the plan to be reproduced in similar policy contexts. The evaluation considers technical, financial, skills and governance requirements to reuse the NCZ social innovation plan;

• The scalability criterion refers to the potential of the plan to be extended to other policy contexts.

Figure 3.1 outlines the impact assessment metric development process. Firstly, overall project objectives and plan-specific objectives are defined and then evaluation criteria are established. Next, general evaluation questions are created, followed by specific evaluation questions. These questions are translated into indicators that will measure the project’s and plan’s achievements and success. Finally, the process includes consideration of the necessary sources where the indicator data is gathered from.

Fig. 3.1

Impact assessment metric development process

At the overall level of the city’s action plan, all the evaluation criteria are considered. For each of the ten categories, the focus is on the criteria of effectiveness and efficiency. Finally, input, output and outcome indicators are developed, which will serve as a basis for the elaboration of the indicators related to the evaluation criteria:

• Input indicators: resources available within cities’ budget, programming, and accounting documents, relate to resources allocated to each specific intervention.

• Output indicators: represent the immediate result of interventions and data about their progresses and are reported in monitoring documents of each intervention.

• Intermediate outcomes: distinguished between direct and indirect benefits that citizens or target groups can gain from the interventions. They are structured according to the dimensions of specific impact that have been identified as relevant.

• Long term outcomes: allow the estimation of the contribution that those interventions have in terms of systemic broader impact in a time horizon of 5 years.

The evaluation of the plan excludes the use of the Social return on investment (SROI): although SROI is an internationally recognized performance management method, utilized by social enterprises to demonstrate the social, economic and environmental value they create, the method is not free of challenges for social enterprises and social innovation initiatives (Arvidson et al., 2010; Millar & Hall, 2013) and it is focused on assessing impact in economic terms, shifting the focus from the necessary systemic changes aimed for in the NZC project. While knowledge of the SROI performance measurement tool can be useful for social innovators and public officials, this performance assessment method is suggested only as an optional tool in evaluating the single initiatives stemming from the plan.

3.2 Existing Indicators

Indicators are sourced from extant publications and catalogues of indicators related to social innovation from scientific literature and research projects (Table 3.1):

Table 3.1 Sources of indicators

RESINDEX

The Regional Social Innovation Index (RESINDEX) Model (Unceta et al., 2016), is the result of a research project funded by Innobasque, the Basque Innovation Agency and comparing the potential capacity to the realized social innovation capacity. It comprises a series of indicators grouped in three indexes: (1) capacity for potential innovation—composed of (1a) capacity for knowledge, (1b) capacity for earning, (1c) capacity for socialization, (1d) capacity for development, (1e) capacity for Association; (2) realized capacity of social orientation index—composed of (2a) knowledge acquisition, (2b) development of social projects, (2c) impact of social projects, (2d) governance, and (3) realized capacity of social innovation index—composed of (3a) knowledge acquisition, (3b) development of innovative social projects, (3c) impact of innovative social projects and (3d) governance.

SIMRA

A comprehensive evaluation framework for evaluating social innovation has been developed by Secco et al. (2019a) and applied to a variety of contexts, from forest-dependent rural communities (Secco et al., 2019a), to social farming, community energy, food cooperatives. The framework is the backbone of the EU-funded project SIMRA (Social Innovation for Marginalized Rural Areas) and has been utilized for the assessment of social innovations across Europe. It was developed based on a literature review of over hundreds of existing frameworks (Secco et al., 2019a) with the aim of developing a method and categories for evaluating social innovations. The resulting SIMRA framework builds in particular on the approach of the Theory of Change, detailing the causal mechanisms that led to changes, which is the base of any evaluation approach. The comprehensive SIMRA framework (Secco et al., 2017) includes an analysis of the context, and this takes into account 9 main elements: (1) the trigger (that is, individual and collective needs), (2) the perceived context at international, national, regional and local level, (3) the agents (ideas, values, willingness, reflexivity, capacity for change) which influence the context and the (4) preparatory actions for collective benefit, which in turns affect the (5a) reconfiguring of the system. The (5b) reconfigured systems (new networks, new government arrangements and new attitudes), lead to (6) project activities with specific procedures and practices. Such social innovation activities produce (7) outputs in the form of identifiable products and service, which in turns produce (8) outcomes and impacts (positive or negative) on economic, social, environmental and governance/institutional aspects. Finally, (9) the learning processes provide feedback loops and multiplier effects, to inform the context and the social innovation activities. In practical terms these nine key aspects are assessed with a mixed quantitative–qualitative methodology (Secco et al., 2017) and a combination of expert and participatory-based evaluations (Secco et al., 2019a).

EU POLIS

The project EU POLIS is an EU-funded project aimed at developing an “Integrated NBS-based Urban Planning Methodology for Enhancing the Health and Well-Being of Citizens” (Zafeiropoulos et al., 2021). As part of the project, indicators are defined to assess the baseline status/challenges of the demonstration cities in five living categories: urban, social, environmental, economic, health and wellbeing (Bozovic et al., 2021).

Evaluating the Impact of Nature-Based Solutions

Finally, the Evaluating the Impact of Nature-based Solutions—Appendix of Methods (Dumitru & Wendling, 2021) published by the EU Directorate-General for Research and Innovation, is a 1077 page long catalogue of indicators related to Nature-Based Solutions, democratic participation, health and other outcome and impact measures, mostly based on EU-funded projects and scientific literature.

3.3 Evaluation Questions and Indicators in the General Case

Firstly, the evaluation questions and indicators for the general case are produced by the research team and mapped from existing frameworks, for all the five criteria (effectiveness, efficiency, sustainability, replicability, scalability). Table 3.2 shows the input/output/outcome indicators elaborated by the research team.

Table 3.2 Input/output/outcome indicators (own elaboration) in the general cases

Table 3.3 depicts the evaluation questions and indicators of Effectiveness (own elaboration).

Table 3.3 Evaluation questions and indicators of Effectiveness (own elaboration) in the general cases

Table 3.4 depicts the evaluation questions and indicators of Efficiency (own elaboration).

Table 3.4 Evaluation questions and indicators of Efficiency (own elaboration) in the general cases

Table 3.5 depicts the evaluation questions and indicators of Relevance (own elaboration).

Table 3.5 Evaluation questions and indicators of Relevance (own elaboration) in the general cases

Table 3.6 depicts the evaluation questions and indicators of Replicability (own elaboration).

Table 3.6 Evaluation questions and indicators of Replicability (own elaboration) in the general cases

Table 3.7 depicts the evaluation questions and indicators of Scalability (own elaboration).

Table 3.7 Evaluation questions and indicators of Scalability (own elaboration) in the general cases

3.4 Evaluation Questions and Indicators for Category 1: Social Innovation Capacity Building of Public Officials and Policy Makers

Table 3.8 shows the input/output/outcome indicators elaborated by the research team.

Table 3.8 Input/output/outcome indicators (own elaboration) for category 1: Social Innovation capacity building of public officials

Table 3.9 depicts the evaluation questions and indicators of Effectiveness (own elaboration).

Table 3.9 Evaluation questions and indicators of Effectiveness (own elaboration) for category 1: Social Innovation capacity building of public officials

Table 3.10 depicts the evaluation questions and indicators of Efficiency (own elaboration).

Table 3.10 Evaluation questions and indicators of Efficiency (own elaboration) for category 1: Social Innovation capacity building of public officials

Table 3.11 depicts the indicators from existing frameworks mapped to the category, and mostly related to effectiveness/impact.

Table 3.11 Indicators from existing frameworks for category 1: Social Innovation capacity building of public officials

3.5 Evaluation Questions and Indicators for Category 2: Social Innovation Skills of Citizens and Urban Stakeholders

Table 3.12 shows the input/output/outcome indicators elaborated by the research team.

Table 3.12 Input/output/outcome indicators (own elaboration) for category 2: Social Innovation skills of citizens and urban stakeholders

Table 3.13 depicts the evaluation questions and indicators of Effectiveness (own elaboration).

Table 3.13 Evaluation questions and indicators of Effectiveness (own elaboration) for category 2: Social Innovation skills of citizens and urban stakeholders

Table 3.14 depicts the evaluation questions and indicators of Efficiency (own elaboration).

Table 3.14 Evaluation questions and indicators of Efficiency (own elaboration) for category 2: Social Innovation skills of citizens and urban stakeholders

Table 3.15 depicts the indicators from existing frameworks mapped to the category, and mostly related to effectiveness/impact.

Table 3.15 Indicators from existing frameworks for category 2: Social Innovation skills of citizens and urban stakeholders

3.6 Evaluation Questions and Indicators for Category 3: Co-design of Policies with Social Innovators and Urban Stakeholders

Table 3.16 shows the input/output/outcome indicators elaborated by the research team.

Table 3.16 Input/output/outcome indicators (own elaboration) for category 3: Co-design of policies with social innovators and urban stakeholders

Table 3.17 depicts the evaluation questions and indicators of Effectiveness (own elaboration).

Table 3.17 Evaluation questions and indicators of Effectiveness (own elaboration) for category 3: Co-design of policies with social innovators and urban stakeholders

Table 3.18 depicts the evaluation questions and indicators of Efficiency (own elaboration).

Table 3.18 Evaluation questions and indicators of Efficiency (own elaboration) for category 3: Co-design of policies with social innovators and urban stakeholders

Table 3.19 depicts the indicators from existing frameworks mapped to the category, and mostly related to effectiveness/impact.

Table 3.19 Indicators from existing frameworks for category 3: Co-design of policies with social innovators and urban stakeholders

3.7 Evaluation Questions and Indicators for Category 4: Co-creation of Social Innovation Initiatives with Citizens and Stakeholders

Table 3.20 shows the input/output/outcome indicators elaborated by the research team.

Table 3.20 Input/output/outcome indicators (own elaboration) for category 4: Co-creation of Social Innovation initiatives with citizens and stakeholders

Table 3.21 depicts the evaluation questions and indicators of Effectiveness (own elaboration).

Table 3.21 Evaluation questions and indicators of Effectiveness (own elaboration) for category 4: Co-creation of Social Innovation initiatives with citizens and stakeholders

Table 3.22 depicts the evaluation questions and indicators of Efficiency (own elaboration).

Table 3.22 Evaluation questions and indicators of Efficiency (own elaboration) for category 4: Co-creation of Social Innovation initiatives with citizens and stakeholders

Table 3.23 depicts the indicators from existing frameworks mapped to the category, and mostly related to effectiveness/impact.

Table 3.23 Indicators from existing frameworks for category 4: Co-creation of Social Innovation initiatives with citizens and stakeholders

3.8 Evaluation Questions and Indicators for Category 5: Funding/Supporting Community-Led Initiatives and Small-Scale Pilots/Experimentations

Table 3.24 shows the input/output/outcome indicators elaborated by the research team.

Table 3.24 Input/output/outcome indicators (own elaboration) for category 5: Funding/supporting community-led initiatives and small-scale pilots/experimentations

Table 3.25 depicts the evaluation questions and indicators of Effectiveness (own elaboration).

Table 3.25 Evaluation Questions and Indicators of Effectiveness (own elaboration) for category 5: Funding/supporting community-led initiatives and small-scale pilots/experimentations

Table 3.26 depicts the evaluation questions and indicators of Efficiency (own elaboration).

Table 3.26 Evaluation questions and indicators of Efficiency (own elaboration) for category 5: Funding/supporting community-led initiatives and small-scale pilots/experimentations

Table 3.27 depicts the indicators from existing frameworks mapped to the category, and mostly related to effectiveness/impact.

Table 3.27 Indicators from existing frameworks for category 5: Funding/supporting community-led initiatives and small-scale pilots/experimentations

3.9 Evaluation Questions and Indicators for Category 6: Enabling Social Innovation/Entrepreneurship Initiatives Scale-Up Beyond Pilot Projects

Table 3.28 shows the input/output/outcome indicators elaborated by the research team.

Table 3.28 Input/output/outcome Indicators (own elaboration) for category 6: Enabling Social Innovation/entrepreneurship initiatives scale-up beyond pilot projects

Table 3.29 depicts the evaluation questions and indicators of Effectiveness (own elaboration).

Table 3.29 Evaluation questions and indicators of effectiveness (own elaboration) for category 6: Enabling Social Innovation/entrepreneurship initiatives scale-up beyond pilot projects

Table 3.30 depicts the evaluation questions and indicators of Efficiency (own elaboration).

Table 3.30 Evaluation questions and indicators of efficiency (own elaboration) for category 6: Enabling Social Innovation/entrepreneurship initiatives scale-up beyond pilot projects

Table 3.31 depicts the indicators from existing frameworks mapped to the category, and mostly related to effectiveness/impact.

Table 3.31 Indicators from existing frameworks for category 6: Enabling Social Innovation/entrepreneurship initiatives scale-up beyond pilot projects

3.10 Evaluation Questions and Indicators for Category 7: Testing and Prototyping New Funding Mechanisms

Table 3.32 shows the input/output/outcome indicators elaborated by the research team.

Table 3.32 Input/output/outcome indicators (own elaboration) for category 7: Testing and prototyping new funding mechanisms

Table 3.33 depicts the evaluation questions and indicators of Effectiveness (own elaboration).

Table 3.33 Evaluation questions and indicators of effectiveness (own elaboration) for category 7: Testing and prototyping new funding mechanisms

Table 3.34 depicts the evaluation questions and indicators of Efficiency (own elaboration).

Table 3.34 Evaluation questions and indicators of Efficiency (own elaboration) for category 7: Testing and prototyping new funding mechanisms

Table 3.35 depicts the indicators from existing frameworks mapped to the category, and mostly related to effectiveness/impact.

Table 3.35 Indicators from existing frameworks for category 7: Testing and prototyping new funding mechanisms

3.11 Evaluation Questions and Indicators for Category 8: Public Procurement of Social Innovation Services for Sustainability

Table 3.36 shows the input/output/outcome indicators elaborated by the research team.

Table 3.36 Input/Output/Outcome Indicators (own elaboration) for category 8: Public procurement of Social Innovation services for sustainability

Table 3.37 depicts the evaluation questions and indicators of Effectiveness (own elaboration).

Table 3.37 Evaluation questions and indicators of Effectiveness (own elaboration) for category 8: Public procurement of Social Innovation services for sustainability

Table 3.38 depicts the evaluation questions and indicators of Efficiency (own elaboration).

Table 3.38 Ten evaluation questions and indicators of Efficiency (own elaboration) for category 8: Public procurement of Social Innovation services for sustainability

Table 3.39 depicts the indicators from existing frameworks mapped to the category, and mostly related to effectiveness/impact.

Table 3.39 Indicators from existing frameworks for category 8: Public procurement of Social Innovation services for sustainability

3.12 Evaluation Questions and Indicators for Category 9: Urban Planning for Social Innovation

Table 3.40 shows the input/output/outcome indicators elaborated by the research team.

Table 3.40 Input/output/outcome indicators (own elaboration) for category 9: Urban planning for Social Innovation

Table 3.41 depicts the evaluation questions and indicators of Effectiveness (own elaboration).

Table 3.41 Evaluation questions and indicators of effectiveness (own elaboration) for category 9: Urban planning for Social Innovation

Table 3.42 depicts the evaluation questions and indicators of Efficiency (own elaboration).

Table 3.42 Evaluation questions and indicators of Efficiency (own elaboration) for category 9: Urban planning for Social Innovation

Table 3.43 depicts the indicators from existing frameworks mapped to the category, and mostly related to effectiveness/impact.

Table 3.43 Indicators from existing frameworks for category 9: Urban planning for Social Innovation

3.13 Evaluation Questions and Indicators for Category 10: Resource Circularity

Table 3.44 shows the input/output/outcome indicators elaborated by the research team.

Table 3.44 Input/output/outcome indicators (own elaboration) for category 10: Resource circularity

Table 3.45 depicts the evaluation questions and indicators of Effectiveness (own elaboration).

Table 3.45 Evaluation questions and indicators of Effectiveness (own elaboration) for category 10: Resource circularity

Table 3.46 depicts the evaluation questions and indicators of Efficiency (own elaboration).

Table 3.46 Evaluation questions and indicators of Efficiency (own elaboration) for category 10: Resource circularity

Table 3.47 depicts the indicators from existing frameworks mapped to the category, and mostly related to effectiveness/impact.

Table 3.47 Indicators from existing frameworks for category 10: Resource circularity

3.14 General Indicators

This final subsection depicts a series of evaluation indicators for initiatives stemming in general from the plan and its categories of implementation (Table 3.48).

Table 3.48 General evaluation for initiatives

Finally, Table 3.49 provides a summary of the number of indicators per category, per criteria (effectiveness—EFFE, efficiency EFFI, sustainability—SU, replicability—RE, scalability—SC), ad distinguishing if own (O) or mapped (M) from an existing framework (Table 3.49).

Table 3.49 Summary table on the number of indicators