In the previous chapters, over one thousand indicators for assessing the social innovation component of cities action plans have been presented, according to defined categories. The practical and theoretical implications of such catalogue of indicators are discussed, firstly providing concrete steps and checklists for deploying indicators in city by publica administrators, policy makers and transition teams members. Secondly, a set of 40 process indicators is provided to equip cities with questions for progressive evaluation and reflexive learning. Finally, theoretical implications and future directions are discussed.

4.1 Operationalization of the Impact Measurement Tool

The following table presents a checklist of the steps that city administrators have to follow in order to apply the indicators to their local case.

 

WHAT

WHO

HOW

WHEN

Step 1

Stakeholders’ engagement

Staff of the public administration and of social innovation organizations at policy/strategic/operational level and representatives of relevant stakeholders in the area of reference of the intervention

Mapping of the stakeholders according to their stake and interest to collaborate as well as the role/availability. Involvement of the stakeholders in the decision process through dedicated communication channels and periodical meetings/workshops/focus groups or interviews

Steps 1, 2 and 3 are crucial for ex-ante evaluation to support conducting the situational analysis (e.g., SWOT Analysis) (see Step 3) as well as the estimation of expected outputs/outcomes/impacts to be achieved by the intervention (see Step 5) However, these activities should be conducted periodically during the execution of the social innovation interventions to support the in-itinere evaluation and at the end of the intervention as part of the ex-post evaluation, in order to support assessment of progresses and impacts achieved by the interventions

Step 2

Appraisal of mission, strategy and objectives of the public administration and of the social innovators

Structured discussion (e.g., workshops/focus groups) between staff members and engaged stakeholders

Step 3

Context analysis and baseline definition

Staff of the public administration and social innovation organizations at policy/strategic/operational level, and, if required, with the support of external experts and/or local researchers, as well as involving experts and representatives of relevant stakeholders for integrating external perspectives and suggestions

Collection of quantitative data based on primary and secondary sources on the environmental and socio-economic context of reference for the analysis complemented by qualitative situational analysis

Step 4

Appraisal of the strategic and operational objectives of the intervention/s to be implemented/monitored/assessed

Analysis and selection of the intervention/s on the basis of: context analysis trends; relevance in comparison with other interventions; external factors and policy goals at intermediary and context level

Steps 4, and 5 are crucial for ex-ante evaluation to support designing the strategy and operational intervention/s to be implemented/monitored/assessed and associate relevant indicators for monitoring and evaluation as well as estimate expected target value for each indicator in relation to the objectives of the social innovation intervention/s

However, these activities should be conducted periodically during the execution of the intervention to support the in-itinere evaluation and at the end of the intervention as part of the ex-post evaluation, in order to support assessment of progresses and impacts achieved by the social innovation intervention

Step 5

Selection and quantification of the Input, Output, Outcome, Impact indicators

Staff of the public administration and social innovation organizations at policy/strategic/operational level, and, if required, with the support of external experts and/or local researchers

The selection of indicators can be done combining: (1) The selection of suitable indicators from the system of social innovation indicators; (2) The definition of more specific additional indicators more representative of the characteristics of the intervention/s to be implemented/monitored/assessed

Step 6

Definition of a data gathering strategy (monitoring) and methodology for (impact) evaluation

Define practical steps and procedures for monitoring and evaluation (i.e., data gathering and evaluation methodologies to be used). This depends on availability of data, resources and expertise (internal/external) to conduct the monitoring and evaluation

Step 6 and 7 should be defined during the ex-ante evaluation and updated if needed during the execution of the social innovation intervention/s to support in-itinere and ex-post evaluations

Step 7

Development of questionnaires for data gathering of selected indicators (including, if required a specific set of questionnaires for impact evaluation)

Staff of the public administration and social innovation organizations at policy/strategic/operational level, and, if required, with the support of external experts and/or local researchers

Questionnaires should be able to capture relevant indicators defined according to the system of measurement indicators or additional more appropriate indicators identified, and they should be adapted to the intervention/s that are under observation and the indicators that have been decided to measure in Step 5

Step 8

Data gathering based on the strategy defined and indicators selected

Input and output data: Staff of the public administration and social innovation organizations at operational level

Outcome and Impact data: Staff of public administration and social innovation organizations at operational and strategic level, and, if required, with the support of external experts and/or local researchers

Various techniques for data gathering can be applied:

(1) Input data are mainly related to list of beneficiaries of the intervention/s and (human, material, financial, technological, etc.) resources allocated. They are usually available to the organisation managing the intervention/s

(2) Output data are mainly related to the immediate results of the social innovation intervention/s (e.g., participants that completed a training course, etc.). They are usually available to the organisation managing the intervention/s

(3) Outcome data are mainly related to information gathered in the short/mid-term after the end of the intervention (e.g., six months to one year). They are usually collected in a systematic way administering questionnaires (e.g., on-line questionnaires, emails, CATI, direct phone calls or papers). (4) Impact data are mainly related to information gathered in the mid-long term after the end of the intervention (usually more than 1 year). They are usually collected through ad hoc surveys on a sample of beneficiaries (through e.g., on-line questionnaires or CATI)

Step 8 and 9 are conducted during the execution of the social innovation intervention/s to support in-itinere and ex-post evaluations

However, once data are available the results of these activities are highly beneficial during ex-ante evaluation to define or update benchmarks for constructing the baselines and estimate target objectives/indicators for the next programming period

Step 9

Analysis of the data gathered and quantification/estimation of measurement indicators

Staff of the public administration and social innovation organizations at policy/strategic/operational level, and, if required, with the support of external experts and/or local researchers

Typical activities include: (1) Data cleaning to remove possible inconsistencies in the answers collected in the previous steps; (2) Statistical elaborations to associate raw data collected to the system of measurement indicators identified in Step 5 and quantify/estimate the indicators on input, output, outcome and impact

Step 10

Measurement of the degree of efficiency, effectiveness and sustainability of the intervention/s in order to assess the degree of achievement of the specific dimensions of impact on sustainability associated to the system of measurement indicators

Aggregated measurement of indicators gathered and calculation of the ratios according to the evaluation criteria as follows: (1) Efficiency: calculating the OUTPUT/INPUT ratio from the measurement indicators. (2) Effectiveness: calculating the OUTCOME/OUTPUT ratio from the measurement indicators. (3) SUSTAINABILITY: calculating the IMPACT/OUTPUT ratio from the measurement indicators

Step 10 is normally conducted at the end of the social innovation intervention/s to support ex-post evaluation. If data are gathered in a structured manner since the beginning of the intervention it can also be used to support in-itinere evaluation and, once data are made available the results of this measurement can be used for ex-ante evaluation to define or update benchmarks for constructing the baselines and estimate target objectives/indicators for the next programming period

4.2 Process Indicators and Questions for Reflexive Learning

The large number of indicators identified and described in the previous chapter provides a comprehensive catalogue of indicators that cities can select based on their specific needs and aims. Based on the work developed in the NetZeroCities project, experts’ opinion and cities’ feedback, questions for progress evaluation are developed and included to provide cities with reflexive questions for qualitative in-depth evaluations of progresses, related to specific actions, explained in detail in related publications and project deliverables (Bresciani et al., 2023a, b).

To support cities in the selection of indicators, a set of core indicators is presented in Table 4.1: the structured list can be utilized by transition teams and public administrators to monitor progresses of their efforts to leverage social innovation for supporting climate neutrality and it comprise both quantitative and qualitative answers, related to actions as defined in Bresciani et al. (2023b).

Table 4.1 Selection of qualitative and quantitative indicators of social innovation
Full size table

4.3 Scientific Implications: Responding to the Need for Assessing People-Based Solutions for Sustainability and Climate Neutrality

The focus on reflexive learning and quali-quantitative assessment is rooted in scientific evidence. In the scholarly article “Why sustainable development requires societal innovation and cannot be achieved without this”, Diepenmaat et al. (2020) review multi-disciplinary perspectives related to societal innovation for sustainable development. The authors propose the need to acknowledge that “actors require each other in realizing their own needs and wishes and may help each other in this respect. Contextual aspects are embedded through the improvement perspectives (Diepenmaat et al., 2020). Their work presents a co-evolutionary understanding of innovation-based transformations, which is based on an iterative relationship between innovations, improvement perspectives and socio-economic transformations (Diepenmaat et al., 2020, p. 3). They specifically frame societal innovation as systemic type of innovation which requires design thinking and system building. The focus on design thinking finds justification in the ability of the method in facilitating the identification of configurations that are suitable for diverse types of actors. They base their argument on the work of Ceschin and Gaziulusoy (2016) who highlight how the focus of design has broadened to include socio-technical system innovation. Such socio-technical transitions are required for deep decarbonization (Geels et al., 2017) and entails a shift in the way the transition to carbon neutrality is framed and communicated to the broader public (Rosenbloom et al., 2016). Terstriep et al. (2020) provided a framework for favourable social innovation ecosystems, while Engelbrecht (2018) outlined how the nexus between social innovation and perceived wellbeing can be assessed. According to Unceta et al., (2020, p. 908), social innovation “measurement and socioeconomic impact have been for a long time a required and challenging area of research inside SI studies, acknowledged by the research community, policymakers, social investment funds, practitioners, social entrepreneurs and social innovators themselves. However, there is still a lack of consensus on what are the major and determining methodological tools and indicators involved in its measurement and impact assessment. Despite this difficult task, there are three approaches that can be identified in the academic field which seek to build a system of indicators for SI measurement: ‘the individualistic approach’, ‘the organizational approach’ and ‘the regional/national approach’ (Unceta et al., 2016).”

The structured catalogue of indicators provided in this book offers a first answer to these calls to action for the context of carbon neutrality.

4.4 Future Developments and Applications

In this book we have provided indicators and assessment methods of the social innovation categories of cities’ action plan. In order to assess the impact of social innovation, it is necessary to measure which activities lead to which outputs (direct result), outcomes (intermediate results) and impacts (long-term results). The developed methodology focuses on measuring the effectiveness, efficiency, relevance, replicability, and scalability of the social intervention in the future pilots devising ten categories of interventions ad produced a set of intervention logics and indicators for the general case and for each related category. Next, key indicators have been selected and adapted to cities’ needs of progressive evaluation and reflexive learning: such selected core indicators are part of the NetZeroCities comprehensive indicators set.Footnote 1

Over one thousand indicators have been proposed and clustered in this volume. This broad set of indicators can be utilized by cities to select indicators relevant for their specific local needs, according to their readiness level in terms of sustainability and social innovation. The core indicators are currently being utilized in the NetZeroCities project and refined through testing with cities, adapting them according to the feedback provided by the co-design of impact assessment framework with each pilot city of the project. From this pragmatic application, several important theoretical implications can be derived: the developed social innovation evaluation framework provides scholars with a solid and comprehensive assessment methodology. Results of assessments based on the framework, through the indicators, can support scholars testing hypotheses and theory of change models, in addition to providing evidence-based design guidelines for selecting people-centred solutions to reach climate neutrality in cities.

Policy makers, civil servants, public administrators, service designer and relevant stakeholders can be informed by the results of applying the social innovation assessment framework.