Keywords

1 Introduction

Sustainability has become a global concern (Keivani 2010) for urban development, and the sustainable development goals (SDGs) defined by United Nations represent the framework to be followed as a combination of three global problems: ecological, economic, and social issues. The SDGs define new guidelines to be included in a holistic framework and offer quantitative goals to reach a global model of sustainable development (Sachs 2012). In this regard, cities have a fundamental role in the advancement of SDGs (OECD 2016), and European local administrations are beginning to establish local goals to design new strategies and policies that follow SDGs. Monitoring the SDGs is an important challenge and a strategic opportunity for stakeholders and beneficiaries involved in the Agenda 2030 at all levels (Sachs 2012), and this requires a method to measure SDGs for defining and eventually achieving them (Lu et al. 2015; Schmidt et al. 2015). In this context, data analytics for evaluation and policy-making is the key priorities for the future of smart and sustainable cities (Kamp et al. 2003), and public administration, at the national, regional, and local level, has acquired a new commitment towards horizontal strategies and performances called Open Government (Lu et al. 2015). Public administration generates and collects a vast quantity of data in many domains (Janssen et al. 2012), and data supports policy-makers in addressing complex problems (Arzberger et al. 2004); therefore, indicators are being increasingly recognized as a useful tool in policy analysis and public communication (Melchiorri and Siragusa 2018).

Over recent years various organizations have put in great efforts to establish a standard or a methodology of indicators to measure SDGs at the national and local level. Sustainable development solutions Network (SDSN) has developed the SDG Index (Biggeri et al. 2019; Sachs et al. 2017) to monitor progress across a diverse set of goals with multiple targets and indicators and to track overall progress. Several similar studies have been undertaken at the national level by various chapters of SDSN, and the first report that measured SDGs in cities in an European context was the SDG Spanish Index Report (Sánchez de Madariaga et al. 2018). The application of the tool is based on the SDGs framework following, as a starting point, the indicators defined in the SDG Spanish Index Report (Sánchez de Madariaga et al. 2018). The report assesses 100 Spanish cities with more than 80,000 inhabitants and all the provincial capitals, according to 85 indicators regarding the 17 Sustainable Development Goals (SDGs) defined by United Nations in 2015, and it is the primary official reference to measure SDGs at the local level in Spain.

The main objective of this research is to describe in detail the methods and results of the application of the city assessment tool, called the Smart&City SDG evaluation tool, for measuring the impact of public policies in the socioeconomic and environmental structure of a city in the framework of the SDGs. This research shows the methodology used to build the tool based on urban indicators to define valid metrics to evaluate the strategic plan of an urban area.

The city selected to apply the tool was Alcobendas, Spain (with 120.000 inhabitants), in the framework of the evaluation of its strategic plan “Diseña 2020.” It is intended to obtain a final report, as objective as possible, that reflects the results achieved of the strategies of the public policies carried out. In addition, Alcobendas is compared with European cities with similar characteristics that enable positioning the city in the achievement of each of its strategic goals, incorporating the current trends, and the SDGs.

The goal is to give the administration a quantitative result of its strategy and to obtain results to scale and apply the tool to the Spanish and European city context to have a standard tool to evaluate the real impact on of the cities’ strategies in the SDGs.

2 Method

The main procedure used for the construction of the tool is based upon the methods applied by SDSN (Sachs et al. 2017), by the Organization for Economic Cooperation and Development (OECD 2008) and by Sánchez de Madariaga et al. (2018) in the SDG Spanish Index Report. The first step in designing the tool is the selection of indicators followed by the SDGs alignment, standardization, normalization, and aggregation. The authors have carried out a quantitative analysis using time series from 2012 to 2018. With the selected set of indicators, and adapting it to the case of study, the tool is able to quantify the impact of the policies in the city and is also able to support the decision-making processes of the administration. A time-series data evolution is used to monitor and benchmark the effects of the applied policies at the local level and on the SDGs. The tool was developed using a set of Spanish and European cities with similar characteristics to those of Alcobendas, the city that was the object of the study (Fig. 1).

Fig. 1
figure 1

Construction Scheme of the Smart&City SDGs evaluation tool case of application: the strategic plan of Alcobendas

Full size image

A strategic plan is a set of policies and/or projects that determine the general development of an urban area (Gubaidullina et al. 2018) and that, due to its transversal and general nature, must be analyzed by pragmatically considering the 17 SDGs. At present, the planning is mainly oriented towards adapting the strategy of the cities to the changing times, reorienting its activity towards a qualitative, intelligent and sustainable growth, and reorienting institutions towards a model of open government and collaborative administration. In this context of change, the Alcobendas City Council defined a new vision for the city with which municipal actions would be aligned. The choice of this city is due to three main reasons: It is the city with the greatest availability of data in Spain; it is the first city in Spain that developed a long-term strategic plan untied to the duration of the legislature; and the city had the need to objectively evaluate the results of the strategies applied during the last eight years. The assessment of the strategic plan was carried out through the analysis and comparison of values of various urban indicators and their contrast with:

  • the degree of achievement of the actions and objectives of the plan

  • the degree of compliance regarding the SDGs

  • a comparison with similar cities (“twin” cities).

Quantitative analyses were carried out among the various levels of disaggregation of approaches (Sisto et al. 2018). The objective was to identify which sets of urban areas or domains are relevant and what level of association should be reached with the indicators. Based on this analysis, paths and orientations were designed to be followed through five major strategic goals for the development of the city:

  • Goal 1. Promotion of the city, economic development, innovation, education, and employment

  • Goal 2. Sustainable development: smart growth

  • Goal 3. Good governance, open, and responsible management

  • Goal 4. Social responsibility

  • Goal 5. Quality of life: culture, leisure, and sport.

The plan collects in detail the five strategic goals, 16 general objectives, 38 projects, and 206 actions. To measure the degree of fulfillment of the objectives and, therefore, of the strategic goals, a series of monitoring indicators was defined by means of which the advances in the implementation of the agreed strategies and its effects on the city have been measured.

2.1 Data Availability and Selection of Indicators

The first step of the procedure is an exploratory analysis of the available data at the local level and the selection of the known and official databases (local, regional, national, European). The starting point of the dataset is the one defined in the SDG Index Spanish report (Sánchez de Madariaga et al. 2018), and due to its special relevance, the following reference documents and databases were reviewed to complete the indicators:

  • Global indicator framework for the SDG and targets of the 2030 Agenda.

  • ISO 37120. Sustainable development communities. Indicators for city services and quality of life.

  • CASBEE for CITIES 2015. Environmental Performance Assessment Tool for Municipalities.

  • ITU-T Y.4903/L.1603. KPI for smart sustainable cities to assess the achievement of SDG.

  • ICR iCityRate. Classifica delle città intelligenti italiane. Citta e Comunita Sostenibili.

The databases used to obtain data at the local level for the municipality of Alcobendas and the others cities selected (see Sect. 2.5) were: Statistics Portal of Alcobendas (Observatorio de Alcobendas), Statistics Portal of Community of Madrid (ALMUDENA), National Statistics Institute of Spain (INE) and Eurostat.

2.2 Characteristics of the Indicators

Once the urban domains have been identified and based on the experience and knowledge of the official databases available, indicators are screened by criteria such as quality, availability, and reliability. For the sake of future replicability of the evaluation, the compliance criteria of five key characteristics of the SMART scheme have been used: specific, measurable, attainable, realistic, and timely (Sisto et al. 2018). According to the reliability criteria, only official and/or prestigious sources were considered, with open data, grouped and consolidated. As far as possible, databases of international data according to replicability criteria have been prioritized. Likewise, the selection criteria of indicators were adapted to the official indicators proposed by the Inter-agency and Expert Group on SDG Indicators.

In turn, within the available indicators, three types are distinguished according to the approximation they intend to measure:

  • Control indicator: It measures the execution of projects and politics; it is a subjective measure defined by the local administration.

  • Effort indicator: It measures resources aimed to reach a particular goal (such as budget, number of policies); it is used to measure whether the resources allocated to the project have actually been implemented.

  • Outcome indicator: It measures a magnitude directly related to the quality of life of citizens, (gdp, pm10, employment, etc.).

The tool mainly encompasses outcome indicators aligned with the SDGs. To the extent possible, outcome indicators should always be chosen for the evaluation of the elements of the plan because the effort is assessed according to the achievement percentages of the various projects. The outcome indicators reflect the true impact of the policies on the well-being of citizens, although there is a risk of confusing the real effect with the situation or the actions of other agents. Therefore, it is key to introduce comparisons between the subject of study and similar entities.

On the other hand, the value of the indicators is highly dependent on the contextual aspects. The composition of indicators and the creation of indexes is the technique that makes possible the mitigation of this effect and the monitoring of the evolution of an indicator among the various comparison elements. For the composition of indicators, a “matrix” with long-term context variables and more short-term evolution variables is established. The purpose of this diagnosis is to have continuity in the evaluation phase.

2.3 Alignment with SDGs

The next most relevant indicators for urban ecosystems have been identified from a sustainable development point of view. Indicators have been grouped according to the concepts or keywords described in each SDG; the most frequent and redundant have been preselected. From this first preliminary selection of indicators, consultations with experts from each sector have been made to validate and/or discard some of them; the associated databases have been identified and each indicator was related to one or more specific target of the SDGs. Some goals have not been considered due to one or more of the following reasons:

  • the goal has already been reached;

  • the goal is focused on the regional or national level;

  • the goal is not susceptible to urban scale measurement.

As a result of the previous process, a definitive list of quantitative indicators (see the Appendix), considered valid only data at the municipal level was associated with the appropriate goals.

2.4 Comparability

The comparability is key to the objectiveness of the project because it will enables evaluating the real situation regarding its objectives and expectations. This means that, as far as possible, the proposal of indicators is put into context, on one hand, with the values of the entire environment and the city, and, on the other hand, with the values of similar cities to Alcobendas. The comparability is established in two dimensions; on the one hand, official public indicators must be calculable beyond the city of application itself. Therefore, the desirability of opting for open, public, and official databases is reiterated; on the other hand, benchmarking cities that, due to their similar characteristics, make possible a constructive comparison between them should be selected. Comparison cities were chosen at three geographical levels: regional, national, and European. The selection criteria, defined in consultation with sectorial experts, of these cities were:

  • Population similar to Alcobendas—between 110,000 and 130,000 inhabitants

  • Urban typology—cities in a metropolitan area

  • Unemployment rate close to the country structural unemployment

  • Gross domestic product per capita.

The cities selected were the following: Bruges (Belgium), Saint-Denis and Argenteuil (both, France), Bremerhaven, Offenbach, Recklinghausen (all, Germany), Zoemeter (the Netherlands), Bergamo and Monza (both, Italy), Pozuelo de Alarcón, Majadahonda, Las Rozas de Madrid, San Sebastián de Los Reyes, Rivas-Vaciamadrid, Sant Cugat del Vallès, Santa Coloma de Gramanet, Reus, Barakaldo and Getxo (all, Spain), Winterhur (Switzerland), and Bolton and Blackburn (the UK).

2.5 Normalization Procedure

After the identification of appropriate indicators, the next step was the normalization of data. The collected values for the indicator are composed of different units of measurement, which makes it difficult to proceed to the aggregation of data and the obtention of a total score. To compare indicators with different metrics within the objectives and projects groupings defined in the plan, a normalization of the values obtained was carried out, based on the definition of maximums and minimums for each metric according to the relative objectives to the current state. The standardization method applied is an escalation to a value in scale 0–1 using the Min-Max method (Sachs et al. 2017). This technique normalizes the indicators to attain an identical range (0–1) by subtracting the indicator value from the minimum value and dividing the difference by the range of the indicator values (Sachs et al. 2017; OECD 2008). The formula for the Min-Max normalization method is:

$$X^{\prime } = \frac{{X - X_{\rm{min}} }}{{X_{\rm{max}} - X_{\rm{min}} }}$$

where X′ = normalized value of the indicator (0–1), X = value of the indicator, Xmin = the minimum value of the indicator, and Xmax = the maximum value of the indicator.

Each indicator has been normalized using a scale 0–1, 1 being the best possible score and 0 the worst; but this equation is only applicable to values with a positive direction (the higher, the better). As the purpose of the normalization is to create a uniform set of values for the indicators in which all have a positive direction, and, to apply the equation also to the values with a negative direction (the lower, the better), a further normalization method was employed (1 − X). At least, to differentiate the results between negative and positive, the minimum value (0) was transformed in −100 and the positive value (1) was transformed in +100.

2.6 Definition of Maximum and Minimum and Aggregation

The standardization of the obtained values was carried out, based on the definition of maximums and minimums for each metric. To define the optimal value, the following criteria were sequentially used: global goals as defined from recognized reports; national or regional goals, calculated from the values of the comparison cities (SDSN methodology) (Sachs et al. 2017); and local goals. For most indicators, the worst value has been defined as the 2.5 percentile of the distribution of the cities selected following the methods established in the SDSN methodology (Sachs et al. 2017). This standardization enables building an index per project by adding all its associated indicators and assessing its growth with an annual average of the normalized values of each indicator and the difference between the 2014 value and the 2018 value (Fig. 2).

Fig. 2
figure 2

Interpretation of the visual tool

Full size image

Equal weights were given to each indicator with regard to its values in each aggregated index. Equal weighting is a viable alternative due to the fact that all the indicators involved are priority policy concerns. Thus, in light of the measurement of the projects, all indicators are equally considered and none is more important than the others. In terms of the aggregation method, the linear additive technique was adapted. This involves adding all the normalized values arithmetically:

$${\text{Score}} = \sqrt {\frac{1}{n}\sum\limits_{i - 1}^{n} {X_{i} } }$$

where score = aggregated result of the subindices analyzed (total score, strategic goals, objectives, projects), X = value of the subindices and n = number of subindices.

The aggregation of several motor projects for the evaluation of a general goal is done in the same way (Fig. 3), calculating the mean of the resulting values per year of each project. More detailed information about the upper and lower limits of each indicator are shown in the appendix.

Fig. 3
figure 3

Aggregation of values at various levels

Full size image

3 Results

The results obtained from the analysis are summarized in the following time graph.

The actions that make up the Design Alcobendas 2020 Plan have so far been 83% executed, a very high percentage for such a plan. The 64 selected urban indicators reflect a positive evolution in all the strategic goals. Since the implementation of the strategic plan in 2014, the city has improved by +23 points (in a scale from −100 to +100) regarding the selected urban indicators. This positive evolution occurs in all strategic goals and is consistent with the compliance percentages achieved, according to the internal evaluations of its follow-up. In particular, strategic goal 1 (Eje 1 in Fig. 4) has improved by +36 points, strategic goal 2 has improved by +16 points, strategic goal 3 has improved by +12 points, strategic goal 4 has improved by +20 points and, finally, strategic goal 5 has improved by +23 points. In comparison with the rest of the cities, Alcobendas has also experienced a positive evolution above average. The most outstanding evaluation results are presented next.

Fig. 4
figure 4

Historical evolution of the results of the strategic goals of the plan (Eje)

Full size image

4 Discussion

Strategic goal 1 highlights Project 1, “Employment Promotion,” with a positive evolution of 48 points (Fig. 5). This is because the unemployment rate (7.1% in 2018) follows a trend of improvement well above the average of the Community of Madrid (11%) and that of Spain (14%).

Fig. 5
figure 5

Time series results of the evolution of the unemployment rate in Alcobendas

Full size image

In strategic goal 2, Project 14 citizen security stands out with a positive evolution of 53 points. The crime rate has continued its downward trend of improvement throughout the duration of the plan. However, a similar trend is observed in all the cities analyzed in the Madrid region (Fig. 6). This may point to the fact that some supramunicipal policy has caused the increase of citizen security.

Fig. 6
figure 6

Time series results of the evolution of the crime rate in Alcobendas

Full size image

Strategic goal 3 highlights Project 15 transparency, open government, and administration, whose level of excellence and demand has been preserved throughout the development of the plan. In this case, it should be noted that although Alcobendas is the best city in this respect (basically it cannot improve), it does not score positively, and the value is 0 (Fig. 7). This serves to highlight that it is necessary to maintain the level of excellence in this respect, but maybe the city council should also devote attention to other issues or problems in future strategic plans.

Fig. 7
figure 7

Time series results of the evolution of the transparency index

Full size image

It can be reasonably deduced that the designed strategy has been largely implemented and that it has had, so far, a positive impact on the public policies that were pursued. In view of future strategic plans and upcoming evaluation cycles, it is recommended:

  • To realize a previous alignment of projects to the SDGs in their various applications in different contexts. This will allow comparability and result orientation with an international projection.

  • To specify the actions in quantifiable objectives to be achieved based on a previous diagnosis. Thus, a more precise compliance can be monitored and the relevant corrective measures can be evaluated.

  • To carry out a study of the analysis of results periodically during the implementation of the plan.

5 Discussion

The advantages of the tool are linked to the possibility of obtaining an objective picture of the strategic areas that work better or need more attention in the development of new policies. The tool shows and quantifies immediately which areas have performed better and which strategies need adjustments. The structure of the tool enables an in-depth analysis by using the scores attributed to each sub-index, which serves to better identify each element. The selected indicators were adequate as they accurately reflect the factors that most influenced the quality of life and the SDG achievement in urban areas. In addition, since the results are expressed in numeric values, the tool significantly reduces the subjectivity in the evaluation of the plan and enables comparability of the various topics.

Another strength is the time series used. It is essential to monitor the scores over time to show the tendencies, weaknesses, and opportunities that the public administration should consider when designing its policies and planning its actions.

This research also aims to explore and study the limits founded. The authors decided to apply the same weight to all indicators, but some topics should play more important roles in the urban context. The tool partially solves this problem, favoring the use of internationally recognized indicators in the measurement of each topic without using other specific variables that could distort the results. However, the introduction of a different weight for each topic, based on recognized criteria, is needed. Another important challenge is the limited availability of data: In some case, the indicators with available and comparable data were not the best for the measurement of policies or a topic. In this regard, much progress is needed in the availability of data and the homogenization of databases.

6 Conclusion

The value proposition of this tool consists of:

  • Measuring the impact of the strategy deployed in the territory where it is applied.

  • Being able to translate the global impact into a single aggregate indicator for stakeholders.

  • Measuring the situation of the city compared with other similar cities in achieving its goals in strategic areas.

  • Comparing the situation of the city against the SDGs of the 2030 Agenda.

  • Identifying best practices within a strategy and serving as an input for the next planning cycles.

It should be noticed that, thanks to the quantitative comparable results and the objective approach, the Smart&City SDGs evaluation tool does not reflect SDGs and the achievement of local goals per se, but rather achievements in the identified indicators in comparison with other similar cities. Although the values do not necessarily imply that a city is smart or sustainable or not, they can help assess the performance of a city and how it fared in comparison with the achievement of other cities. This helps quantify the impact of the adopted policies, identify best practices within its strategy, and improve future policies for the next strategic plan. Furthermore, the subindices help identify the weaknesses and strengths of the city, which provides insights on how to improve their performance. The visualization of the urban economic, environmental, and social impacts goes beyond the support of the decision-making processes and can also be used as a political argument to promote and foster long-term planning (Waas et al. 2014) and urban sustainable development in European cities. Finally, the selection of urban indicators is proposed as a part of a standardization and scalable process which can be replicated in other cities.