Evaluating the Design of Integrated Urban Development Strategies: Evaluability, Plan Quality and Planning Learning Processes

Abstract

Policy design is a growing area of study in policy studies due to its importance in ensuring good implementation and impact. A ‘good design’ ensures good implementation processes and the proposed policy outcomes. Nevertheless, this issue has received little attention in urban initiatives promoted by the EU, at least through the analysis of local policy portfolios from a comparative perspective. This chapter applies the CUPPA approach to analyse the quality of local strategies design from a comparative perspective applying the comparative urban policy portfolio approach. The chapter establishes quality dimensions of local plans for each phase of the policy cycle (from diagnosis to evaluation); and two broad dimensions to analyse local plans’ evaluability: the practical dimension (information needed to evaluate local plans) and the analytical dimension (the coherence between problems, objectives and planned actions). The analysis of URBAN and URBANA Initiatives shows a medium level of quality of local plans design, although very low regarding evaluation. The comparison between URBAN and URBANA local projects shows learning processes regarding the design of governance processes and mechanisms in local strategies, a better definition of instruments designed to ensure coordination among policy sector departments in local governments, between local and supra-municipal authorities, and between public and societal actors. Therefore, portfolio design analysis shows a growing trend towards multi-level governance in urban initiatives, but it also indicates evaluation is a dimension that needs improvements (low levels of quality and no improvements between programmes analysed).

Introduction

The New Urban Agenda promoted by the United Nations indicates sustainable urban development strategies must pay particular attention to their design to ensure predictability and coherence in urban development (Naciones Unidas, 2017). The Urban Agenda for the European Union expresses similar concerns (European Commission, 2014). Previously, in its proposal to evaluate the Structural Funds, the European Union stressed the importance of design evaluation, the so-called ex-ante evaluation, to ensure adequate implementation and success of the actions to be developed under this framework (European Commission, 2013).

Interest in the design of public policies is also present in the academic sphere, and there is even renewed interest in this regard (Howlett & Lejano, 2013). It focuses not only on the analysis of decision-making processes leading to public policy formulation but also on evaluability, whether the design of a public policy is such that it can be evaluated (OECD, 2010; Trevisan & Walser, 2014). Specifically, this approach has been developed by the plan quality evaluation approach to study urban policies and their specific projects (Lyles & Stevens, 2014). Similarities between both perspectives provide a framework to analyse urban integrated strategies design quality as an evaluability exercise (Rodríguez-García and Navarro-Yáñez, 2022).

In this chapter, we will adopt this perspective to analyse the design quality of the urban development projects encompassed by the URBAN and URBANA Initiatives; in other words, if they have been designed in a way that it is possible to evaluate them. In doing so, we are pursuing two objectives. On the one hand, this chapter tries to provide policy evidence about the quality levels of project design and, therefore, information to assess their evaluability. And on the other, it provides policy evidence about the existence of a learning process in the policy design of integrated urban strategies as an expected add value derived from the urban dimensions of EU cohesion policy.

The Quality of Integrated Urban Policies Design: Comparative Tools to Assess Evaluability

One perspective that might be appropriate for studying the evaluability of urban development projects is the public policy cycle (Dunn, 2011). This approach establishes a set of phases in the planning process that should lead to the design of the project. These phases mainly concern two major aspects or dimensions. On the one hand, the strategy proposed by the project: the diagnosis or starting situation, the objectives or desirable situation once the project has been developed and, finally, the planning of policy actions that will make it possible to achieve this. On the other hand, procedural aspects relating to the governance processes of the project (to ensure coordination and participation among concerned agents) and those relating to monitoring and evaluation (the extent to which the evaluative strategy of the project is planned in the project design).

This perspective implies understanding planning as a decision-making process aimed at achieving a desired situation taking into account the starting situation and the internal and external factors (positive and negative) that could influence the achievement of objectives. Pursued outcomes are, therefore, established, along with what should be done, how, and when (and in what sequential order). It is a set of time- and space-specific strategies formulated in terms of measurable objectives with regard to cost and results. It also involves understanding evaluation from a holistic and integral perspective that takes place in every phase of the public policy life cycle. As well as encompassing the results and effects, the evaluation also means the analysis of design, implementation, governance processes and participatory channels (Guerrero-Mayo et al., 2022). Therefore, evaluation is not the last phase of the policy cycle, but instead must be implemented from the beginning of the planning process (Guyadeen & Seasons, 2018; Rossi & Freeman, 1993).

From this perspective, the essential elements to be considered when studying the evaluability of public policy design and, therefore, the dimensions to be analysed are as follows: (1) Diagnosis: analysing the set of problems subject to intervention, identifying their causes and effects, differentiating between the normative (‘should be’) and the positive (‘is’), what is actually happening; (2) Objectives: the results to be achieved, derived from the problems identified, as well as the relationship between them; (3) Action strategy: the actions to be developed in order to achieve the objectives established, as well as the relationships between them and/or with others developed in the same territory; (4) Governance and participation: instruments to establish the processes of collaboration and participation of the different stakeholders involved in the design and development of public policy; and finally, (5) Implementation and evaluation: mechanisms to ensure the development of the actions as designed, as well as the outcomes achieved with them in respect of the objectives established. These dimensions are similar to those proposed by the plan quality evaluation perspective. This approach tries to compare urban plan design (the results of the planning process, generally in a document) with normative principles defining a ‘good plan’. In this case, two broad dimensions or principles are differentiated, including specific principles: direction-setting principles (fact bases, objectives, actions) and action-setting principles (implementation, monitoring, participation) (Lyles & Stevens, 2014). Complementariness between these two approaches means the analysis of local plan quality supposes an exercise of evaluability. In our case, about the design of sustainable and integrated urban projects (Rodríguez-García & Navarro, 2022).

In the framework of the CUPPA approach, 17 items have been defined to measure specific aspects of the five dimensions mentioned above (see Table 3.1). Each item measures whether the project design is close—or not—to an ‘ideal situation’ defined according to the literature on policy evaluability, plan quality evaluation and urban policies evaluation. Reliability and dimensions validity tests show these items, and summative scales based on them, are valid instruments to analyse the quality of policy design of urban policies and integrated urban initiatives in particular (Navarro-Yáñez et al., 2020). Therefore, based on the public policy cycle perspective, this evaluative system can measure the design quality of the five core dimensions (as a summative index of their items) and a global level of quality for the project in order to evaluate a single project or conduct comparative analyses.

Table 3.1 Project quality as evaluability assessment: dimensions and indicators

Some of the items also allow for the study of the practical and analytical dimensions of evaluability (Davies & Payne, 2015), and, more specifically, readiness and internal consistency as two quality dimensions of urban plan design proposed by Rodríguez-García and Navarro-Yáñez (2022). Practical evaluability seeks to ascertain whether the main dimensions of the design are defined in such a way that they can be understood and analysed. Readiness encompasses the clarity and specificity of challenges identified in the diagnosis, the definition of objectives pursued, the policy actions established to achieve them, mechanisms to ensure governance and participation, implementation management and evaluation. Analytical evaluability seeks to ascertain whether there is an internal logic that adequately links the objectives with the problems that justify them and the actions. Internal consistency means the project design shows the correspondence between established goals and needs (goals are based on identified needs or challenges) and between goals and policy actions (these are adequate to achieve goals allowing for causal attribution between policy actions and outcomes).

By crossing these two quality dimensions, it would be possible to establish four evaluation scenarios or spaces, which would give an account of the kind of evaluation that could be carried out based on project design (Rodríguez-García & Navarro-Yáñez, 2022). The space of ‘analytical evaluation’ means the project design shows a high level of readiness and internal consistency. Therefore, it is possible to know the results obtained (because they are well defined) and whether these are the product of implementation (due to adequate internal consistency). The ‘results-oriented evaluation’ combines high readiness and low internal consistency levels. Therefore, it is possible to know what has been done and what has been achieved, but it would not be possible to reconstruct the explanatory logic linking goals with challenges and policy actions. In the ‘process evaluation’ space, however, it is possible to reconstruct the logic that connects problems, objectives and actions, but it is not possible to analyse the results achieved because those elements have not been well defined (a low level of readiness). Finally, the ‘social-political evaluation’ scenario combines low readiness and internal consistency levels. It is challenging to analyse processes and results (due to their lousy definition) and whether outcomes are a consequence of the plan established to achieve them (due to the lack of internal consistency). In this case, evaluation should be based on the participation and evaluative statements of public officials, staff and stakeholders involved.

In sum, based on the items proposed to measure plan quality from a policy evaluability perspective, different analyses could be done regarding a single project or develop a comparative study. The five dimensions mentioned above could help analyse the evaluability of integrated urban projects applying the policy cycle perspective or the plan quality evaluation approach. Based on the proposal of the two plan quality evaluation dimensions, readiness and internal consistency can help assess two central aspects of project design quality. And finally, by combining these two dimensions, different evaluative scenarios could inform the evaluation it can do according to the policy design of urban integrated projects.

The Quality of Integrated Urban Initiatives: URBAN and URBANA Initiatives

To analyse the quality levels of the URBAN and URBANA projects, we have computed a summative index for each dimension according to the public policy cycle approach. To facilitate the interpretation of the results, we have transformed the original five-point scale (1–5) into a 0–1 scale. We have also computed readiness and internal consistency indexes to measure the two quality dimensions of urban plan design mentioned above. The readiness index is the average of items measuring the quality of the definition of challenges, objectives and policy actions (the first items in these three dimensions, see Table 3.1). The internal consistency index is the average of the two internal coherence indexes (the second item in objective and policy actions dimensions in Table 3.1). Finally, evaluation scenarios are defined by crossing readiness and internal consistency indexes (the four scenarios are delimitated according to values below and above the theoretical mean of these indexes, value 0.5).

Analysis of the 64 projects studied (22 from the URBAN and 42 from the URBANA) shows that most indicators have means below the midpoint of the scale (Table 3.2). The average of the global scale is equal to 0.37 points. The objectives dimension yields a similar score, with diagnosis and policy actions scoring slightly higher (around 0.5) and the design of the processes to ensure governance somewhat lower (average equal to 0.3). However, the monitoring and evaluation dimensions show the lowest level (score equal to 0.2). These results are typical in the literature on public policy and the quality of urban plans (Jun, 2014; Rodríguez-García & Navarro-Yáñez, 2022).

Table 3.2 The design quality of URBAN and URBANA projects (Means on scales 0–1 [standard deviations])

Regarding the two quality dimensions of urban plan design, readiness and internal consistency indexes show average scores slightly higher and slightly lower than the middle point of the scale, respectively (0,55 and 0,44 points). Therefore, challenges, objectives and policy actions are better defined than the relations between them, the internal logic establishing an appropriate link between challenges and objectives, and objectives with policy actions. In fact, the absence of internal logic in the European Union’s Structural Funds is highlighted as an aspect that makes them difficult to assess (Gaffey, 2013).

The comparison between URBAN and URBANA projects shows the scores are somewhat higher for the latter in specific dimensions. However, effect sizes show slight and not statically significant differences (Fig. 3.1). The global quality index is favourable for URBANA but is very small (g = 0,236). This difference is mainly due to the governance dimension, where there are significant differences between the two programmes. The effect size value indicates that around 77% of URBANA Initiative projects achieve higher levels than the average of URBAN Initiative projects in this dimension (g = 0.751). Differences also exist regarding the policy actions dimension, albeit very small and not statistically significant (g = 0.267). In contrast, URBAN projects present higher levels for the evaluation dimensions and the internal consistency index. However, the differences are minimal and not statistically significant (g equal to −0.176 and −0.138, respectively).

Fig. 3.1

Project design quality: differences between URBAN and URBANA projects (Effect size [Hedges’ g] and confidence interval [CI90%])

Table 3.3. shows the distribution of the projects into the four evaluative spaces defined previously. Around 36% are situated in the space for socio-political evaluation and 44% in the analytical evaluation scenario. Just over 10% of projects are located in the results-oriented evaluation scenario, and only 6% are in the space that would allow for processes (implementation) evaluation but without adequately evaluating the results obtained. Thus, most projects are located in more different evaluative scenarios: the one that would enable to assess of whether the logic of the intervention produces the expected results (analytical evaluation) and the one that would be based on the view of stakeholders (or who participate in the evaluation process). Differences between URBAN and URBANA projects show a slight improvement in the second case (a higher percentage in the results and analytical evaluation scenarios). Above all, these results indicate projects are usually designed with high-quality criteria in all dimensions or present a low level of quality in all of them. Projects in different evaluation spaces have different quality levels in the global index: an average oaf 0.45 for those in the socio-political evaluation space and 0.66 among those in the analytical evaluation space. This result could be explained because the global scale includes indicators defining evaluation spaces (Fig. 3.2).

Table 3.3 Evaluative scenarios and evaluation planning (Percentage over total cases in each evaluative scenario)

Fig. 3.2

Evaluative scenarios in URBAN and URBANA projects (Percentage over total cases [URBAN = 22, URBANA = 42])

However, these differences also exist if we analyse more specific aspects of evaluation planning in project design not covered by the previous measurements (items and scales). We have examined whether projects present the following situations: an evaluation plan exists for the project as a whole, a specific team will carry out the evaluation, socio-economic agents will participate in it and mechanisms or processes exist to facilitate public participation in monitoring and evaluation processes. For all these issues, which show the extent to which evaluation is planned in projects and the role given to different stakeholders in this process, the values are much higher for projects situated in the analytical evaluation scenario than for projects located within the socio-political evaluation scenario (Table 3.3). Moreover, even where evaluation depends more on participation (socio-political evaluation), the number of projects including mechanisms to ensure this process is lower than in the analytical evaluation scenario. In this case, a good definition of the pursued outcomes and the process for achieving them comes with detailed evaluation planning in project design. Therefore, in this case, evaluation is understood as a crucial element of project design and its posterior implementation, not only as a final task.

Integrating Evaluation in Planning Sustainable and Integrated Urban Development Strategies

At the start of this book, we noted the widespread importance of the urban integrated strategy proposed by the EU for urban policies in Spain through the URBAN and URBANA Initiatives, as well as a growing community of agents around this issue. And that this strategy is also applied in the current EDUSI Initiative and the Spanish Urban Agenda. Has this experience generated any learning in design planning processes? Is the extension and recurrence of practice reflected in a higher quality of designs? These were the main research questions proposed for this chapter.

Analyses provide mixed policy evidence about these questions. Regarding our first research question, results show project design has medium or low-quality levels, especially the evaluation dimension (see vertical axis in Fig. 3.3). And most of the project designs analysed are not located in the analytical evaluation scenario. Regarding our second research question, there do not seem to be significant learning effects (see horizontal axis in Fig. 3.3). The projects developed under the URBANA Initiative only present slightly higher quality levels than those developed under the URBAN Initiative.

Fig. 3.3

On add values of the urban dimension of the cohesion policy in Spain: quality levels and planning learning effects in integrated project design

Moreover, no significant learning effect exists for the three central elements of the projects (diagnosis, objectives and actions), and especially for the monitoring and evaluation dimension (in this case, URBAN projects have higher quality levels than URBANA projects). Only evident learning effects exist in the case of the governance dimension. This effect could be explained by the growing importance of this aspect since the 1990s, following the emergence of the White Paper on European Governance and the idea of multi-level governance surrounding the EU cohesion policy. Moreover, the consensus created around this issue in the ‘Urban Acquis’ derived from the URBAN Initiative and the following programme documentation on European urban development initiatives. Multi-level governance is explicitly listed as one of the key elements to ensure a good design of an integrated urban strategy, in addition to agenda integratedness and the participation of local socio-economic stakeholders (Fioretti et al., 2020; Urban-Future, 2005).

Increasing concern on multi-level governance could be the more evident add value of the EU proposal for urban policies, as the quality level of this dimension show against other project design dimensions. On the contrary, evaluation is and should be a big concern for the urban integrated strategy promoted by the EU. Its quality levels are very low, and learning effects do not exist, making it very difficult to apply an analytical evaluation to urban initiatives designed and implemented to know the impact of implemented strategies; or at least, to reduce the ‘attribution gap’ common in urban policies evaluation (Guyadeen & Seasons, 2018). The policy design of local integrated strategies should incorporate evaluation as an essential task to be included from the beginning of the planning process, not only as a collection of outcomes to show implementation levels of objectives proposed at the end of project implementation.

This chapter provides specific policy evidence about the added value of cohesion policy on urban policies, at least as improvements in the design of sustainable and integrated urban strategies. We have proposed and applied a validated instrument to measure the quality of project design as a perspective to assess its evaluability. It can be used, together with other evidence or procedures, to analyse this issue retrospectively (as we have done here). It could also be a helpful tool for staff in charge of projects and stakeholders participating in this process during the planning process. In sum, it is a management tool to plan (ex-ante) and improve the quality of integrated urban development initiatives launched by the EU by practitioners and to carry out comparative analyses within and between national and regional programmes or programming periods.