Keywords

Introduction

As indicated in the introductory chapter (Chapter 1), sustainable and integrated urban development strategies promoted in the EU should have two central add values: on the one hand, demonstrative effects regarding improvements in the design and implementation of urban policies; on the other hand, to improve the cohesion of the territorial areas where they are applied, specifically urban areas (neighbourhoods) in the case at hand. However, whereas the first objective has been the subject of analysis, studies on the second objective are almost non-existent.

Over the coming chapters, some evidence on this issue will be presented. These chapters try to provide policy evidence about the impact of integrated urban development strategies on different aspects of quality of life and the scale of territories targeted by programmes. Previously, in the first section of this chapter, we will briefly review the strategies commonly used to analyse the effects of these types of initiatives and their scope. In the second section, we will present the general approach we have used to analyse the impact of the URBAN and URBANA initiatives as examples of integrated urban strategies promoted by the EU.

Analysing the Effects of Integrated Urban Development Strategies on Socio-Spatial Cohesion: Main Methodological Strategies

The analyses of the effects of integrated urban development programmes promoted by the EU on socio-spatial cohesion have mainly utilised two strategies: intensive—in-deep—analysis of specific projects or the analysis of programme perspective. The first approach involves analysing projects as case studies or comparing cases to yield conclusions regarding demonstrative effects (lessons learned) for other cases or applications. These studies provide detailed evidence about project design and, particularly, the implementation process and results (outcomes) according to the objectives set. Their most relevant contribution has been demonstrating what integrated strategy development entails and the elements that facilitate or inhibit its implementation. This approach is applied, for example, in the URBACT networks, the Urban Development Network, and specific studies carried out in Spain (De Gregorio, 2015).

The second perspective has focussed on extensive analysis of the implementation and outcomes in projects applied under a programme—or a programming period. Two main strategies are used for this purpose: the study of implementation and objectives achievement indicators or the consultation processes with experts and stakeholders involved. Regarding the first strategy, implementation analysis has focussed mainly on the level of spending execution or the study of some established indicators of objectives achievement. Good examples of this perspective are the ex-post evaluations made about URBAN initiatives and the urban initiatives implemented in the 2007–2013 programming period (European Commission, 2003, 2010, 2016). However, in their final reports (‘closing reports’), not all projects state the value achieved for the indicators, making it difficult to analyse the programmes as a whole. Moreover, given the bottom-up logic of their design, in many cases, the establishment of achievement levels might not be very realistic, as evidenced by the ex-post evaluation of initiatives developed between 2007 and 2013, including the URBANA Initiative (European Commission, 2016).

For example, in the case of Spain, the ‘closing evaluative reports’ for the URBAN projects provide fairly detailed information, with specific indicators that are quite similar between projects, as well as measurements of their expected and eventually achieved levels. This detail, however, does not appear in the closing reports of the URBANA Initiative. Using the information provided by the former, we have calculated the ‘performance’ level of each policy action included in local projects, comparing the value assigned as the objective to be achieved with the value actually achieved. Of the set of interventions analysed, slightly less than half obtained values greater than 100% and almost half of these values were greater than 150%.Footnote 1

The strategy based on consulting experts and stakeholders often provides information about the strategy developed, as well as factors that have facilitated—or inhibited—its implementation and the achievement of objectives. Thus, for example, in the ex-post evaluation of the URBAN I Initiative, the experts indicated that, in Spain, 100% of the projects had positive effects on the neighbourhood’s urban environment or socio-economic conditions (European Commission, 2003). This report also includes several ideas about factors enhancing or eroding the programme implementation and its success. The ex-post evaluation for the 2007–2013 period also includes a survey among stakeholders providing information about strategies and their success.

In some cases, evaluation based on experts and stakeholders is done through specific assessment protocols in order to provide a comparative assessment. In this regard, the EU has promoted the Territorial Impact Assessment approach. This approach is based on a participatory process of experts and stakeholders to provide a ‘theory’ about the potential territorial effects of EU policies (programmes, initiatives, regulations,…). This ‘theory’ is applied to existing statistical data to catch the territorial heterogeneity before policy implementation and provide a set of potential outcomes according to the ‘theory’ elaborated by experts and stakeholders. The main idea is to show the potential heterogeneous impacts of EU initiatives across European territories (regions, cities, …) This approach has not been applied to the territorial targets of EU urban initiatives (cities or neighbourhoods).Footnote 2 Nevertheless, some exercises have used this approach to check the potential effects of the Territorial Agenda 2030 on cities or the effects of the COVID-19 pandemic (Dallhammer et al., 2018; Gaugitsch et al., 2020).

Overall, both intensive case studies and the programme analysis perspective have shown the value of EU-integrated urban strategy for urban policy formulation and implementation. However, these approaches do not provide policy evidence of impact because these perspectives only analyse implemented projects without comparisons with other urban areas where EU urban integrated strategies are not implemented (Table 7.1).

Table 7.1 Analysing the effects of EU integrated urban development strategies on socio-spatial cohesion: main perspectives
Full size table

This lack of comparison makes it difficult to provide evidence about their impacts because other factors could also explain outcomes and changes in areas targeted by integral strategies. For instance, changes observed in targeted areas can be similar to more general trajectories of change in the urban area in which projects are implemented (the city, the region,..); improvements observed could be caused by sectoral policies that are applied simultaneously across municipalities or neighbourhoods, or observed outcomes could depend on the initial situation in targeted urban areas before project implementation, as the TIA approach sustains.

Thus, these strategies face two methodological challenges: limits to generalisation (or external validity) and limits to explanatory capacity (or internal validity). First is the external validity challenge when dealing with unique case studies, comparative cases without an explicit sample framework explaining controlled factors included in the analyses or the programme establishing specific selection criteria for targeted territories. These criteria mean a ‘policy selection bias’; selected urban areas do not represent all the urban reality in Europe, a state member or a region, only those types targeted by the policy frame. For instance, the focus on vulnerable neighbourhoods in big cities in early EU urban initiatives in the 1990s provided relevant information and results that could be not valid for small and medium-sized cities. Therefore, observed outcomes, changes, or improvements in socio-spatial cohesion will be valid for other cases and contexts. Second is the internal validity challenge due to their ability to show that programmes and projects generate the desired changes in targeted territories, since they do not ‘isolate’ (control) the effect of other potential explanatory factors. Are we sure integrated strategies are the explanatory factor of socio-spatial cohesion improvements in targeted territories?

In this regard, the commonly used strategy to try to overcome these challenges is to establish ‘controlled comparisons’ between places where the programme is implemented and places without this implementation that were ‘similar’ to them before programme implementation. Then, policy impact is analysed as the differences between their respective trajectories of change between pre- and post-implementation periods (Rossi, 1999). This strategy involves comparing urban areas that, firstly, are differentiated by whether they have been the object of intervention or not, and secondly, are similar in aspects that might explain the change between the moments before and after implementation. This approach faces at least three challenges: delimiting the territorial spaces to be compared, defining which conditions make the areas comparable in terms of their ‘starting conditions’ before implementation, and establishing a methodological strategy to ensure this, to ensure this similarity before implementation.

The first of these challenges requires projects to define in some detail the urban area in which it is applied. Albeit unevenly, the URBAN and URBANA projects analysed include some description of the urban area subject to intervention, either in the form of an illustration, a list of affected streets, or a map to identify the census sections it comprises. However, other urban areas where projects have not been implemented should be defined in the same cities where projects have been implemented. These should be similar to the previous ones before project implementation. This issue relates to the next challenge.

The second challenge concerns the need to establish how the urban areas should be similar, in other words, the ‘starting conditions’ that should be controlled. At least two approaches can be considered in this regard: a theoretical approach or a policy-applied approach. The first approach means similarity is established according to prior knowledge about causes explaining changes in urban areas that could be included or not in the policy theory of the programme. The second approach is based on the policy theory established, implicitly or explicitly, by the programme in its policy frame, regardless of other theoretical knowledge about urban change not included in the programme's policy. The policy frame sets goals, implementation preferences, and criteria for territorial targets. These eligibility criteria could mean targeted territories could have specific and differentiated traits from other urban areas.

The programmes analysed here define their territorial targets as areas with high levels of socio-spatial vulnerability within the framework of their respective urban contexts (cities), establishing a series of problems that must be evidenced in the diagnosis of the projects to show their eligibility. These criteria respond to the logic of their policy frame based on the idea of integrated urban regeneration promoted by the EU since the 1990s. The programmes aim to reduce processes of socio-spatial segregation by bringing these urban spaces closer to the general dynamics of their cities. To this end, they ‘add’ a specific integrated policy strategy to the existing sectoral policies actions in the urban area where they act. The main assumption is that these targeted actions will close the gaps between target areas and other urban areas. Therefore, to be successful, programmes must select areas with high socio-spatial vulnerability appropriate to this policy frame (Rae, 2011; Thomson, 2008). Thus, the impact analysis must compare urban areas chosen by the projects with similar areas according to the eligibility criteria established by the programme, the ‘starting conditions’, which, accordingly, justify their implementation and potential effects in selected urban areas. This ‘applied policy approach’ is the strategy we will use here.

The third challenge concerns the methodological strategy used to establish and ensure the similarity in the starting conditions among urban areas that will be compared. We have used propensity score matching, as other evaluative exercises have done for similar area-based initiatives in other regions (Bondonio & Greenbaum, 2007; Ploegmakers & Beckers, 2015) and to analyse the impact of Cohesion Policy at the regional level (Dall'Erba & Fang, 2017). This method allows us to select similar urban areas, in terms of their likelihood of being chosen as the territorial target of the programmes, to the areas eventually chosen to implement the projects, as we will explain below.

Impact Analysis Using Controlled Comparisons: A Quasi-Experimental Research Design Approach

Based on the criteria indicated in the previous section, this section presents the general strategy used to analyse the impact of EU-driven integrated urban development programmes in Spain between 1994 and 2013 (the URBAN and URBANA initiatives). We will consider only URBAN I projects, as URBAN II was implemented in only ten cities, which is too small a sample for controlled comparison analyses. We will detail the strategy used in the context of the three challenges mentioned above.

Observation Units: ‘Homogeneous Urban Areas’

The urban area in which each project was developed has been delimitated as groupings of census sections based on the illustration, description, or map included in each project documentation. For the rest of the areas in each of the cities that participated in the URBAN I and URBANA Initiatives, homogeneous urban areas (HUAs) have been defined by applying four criteria. Firstly, they were similar to the selected areas in their resident population, an eligibility criterion included in URBAN but not in URBANA projects. Secondly, given the approach adopted in their policy frame, they were internally homogeneous regarding socio-spatial vulnerability. Thirdly, they did not include major urban discontinuities due to large avenues, train rails, roads, etc. And fourthly, they would allow adequate estimates based on the 2011 Population Census samples. As regards the second criterion, we used a synthetic indicator of socio-spatial vulnerability validated at the census tract level in 1991 and 2001, which includes four sub-indicators: housing in poor conditions, unemployment rate, uneducated population, and percentage of unskilled workers (Fernández et al., 2018).

The HUAs were defined for 2001, and the same territorial areas were delimited for 1991 and 2011. Therefore, these are similar territorial urban areas spaces enabling their comparison over time. In the case of the URBAN I programme, 542 urban areas were defined for a set of 26 cities, since in three of the cities that implemented this programme was no geo-referencing of their census sections for the year 1991 (Langreo, Castellón, and Avilés). For the URBANA programme, 576 areas in 43 cities were identified (in the case of Coslada there is no documentation on its project).Footnote 3

The targeted areas defined by the projects are extensive. Therefore, the delimitated homogeneous urban areas are also vast. This large size might dilute evidence of the impact of projects. First, the policy actions could be carried out in specific spaces of the selected urban area as a whole, and, therefore, they affect such spaces or only some of the area’s residents to a greater extent. Second, some heterogeneity could exist between different spaces included in targeted areas (in their urban fabric, social composition, or other specific aspects), meaning that the whole area is not exactly equal in terms of its starting conditions. However, our study is not about the change in urban areas, but about the impact of the programme. We have studied the design and development of the projects, trying to stick as closely as possible to the urban areas as defined in them. That is to say, our analysis is about the change promoted by integrated strategies promoted by the EU in their targeted territories, not about general processes of urban change.

Reducing the ‘Selection Bias’ of the Programmes: Propensity Score Matching for Experimental and Control Areas Selection

As indicated above, the urban areas selected by the projects should present high levels of socio-spatial vulnerability according to the eligibility criteria established by the programmes. These mainly include urban fabric degradation, bad environmental quality, low levels of economic activity, and high levels of social exclusion, especially unemployment; although URBAN II also includes other aspects regarding educational achievement or specific problems linked to intercultural relations and security in neighbourhoods (European Commission, 1994, 2000; Ministry of Public Administration, 2007; Ministry of Economy and Finance, 2007). Except for environmental quality, for which there is no information at the census section level for either 1991 or 2001, we have elaborated specific indicators to measure these eligibility conditions: population density, percentage of housing in good condition, the density of establishments that develop economic activity (per thousand inhabitants), and the rate of employment. In the URBANA Initiative, we have also considered the percentage of the population without any formal education title since problems related to educational achievement and the level of academic training of its residents are explicitly included in the eligibility criteria.

The comparison between selected and non-selected areas shows that the former are more vulnerable, especially regarding house conditions and unemployment (Table 7.2). Although the differences are not statistically significant, they also have higher demographic density and business density, which would not correspond to the established criteria. This might be explained because selected areas included historical centres of cities which concentrate on economic activities—because of their centrality in the urban space—although they present a considerable level of socio-spatial vulnerability according to urban environment conditions or social composition (Fernández Salinas, 1994).

These differences would show that the selection of territorial targets within the programmes was adequate since they are urban areas with high levels of socio-spatial vulnerability.Footnote 4 However, this implies the existence of a ‘selection bias’ that would affect the impact analysis of jointly analysing all these urban areas (selected and unselected). In other words, we would be comparing areas based on different starting conditions making it difficult to establish whether changes were due to the implementation of a project or to these initial differences.

Table 7.2 Eligibility criteria for the territorial targets of the programmes: differences between selected and unselected areas in the URBAN and URBANA initiatives (Mean [standard deviation])
Full size table

We applied the propensity score matching method to ensure comparisons between urban areas with similar starting conditions. This method ‘matches’ selected areas with non-selected urban areas according to their similarity in terms of the eligibility criteria established by the programmes. First, a logistic regression model is used to estimate the probability that an urban area would have been selected. The dependent variable is whether an urban area has been the object of intervention or not (selected vs. non-selected UHAs), and the indicators that measure the eligibility criteria of each programme are included as covariables. Second, urban areas with similar propensity scores (similar odds of being chosen) are ‘matched’. There are different matching methods. We have applied the nearest-neighbourhood method matching each selected area by the programme with the closest one according to the propensity score. In addition, we applied more specifications. First, the experimental and control groups only include urban areas in the ‘common support’ area (overlap between the propensity score distributions of selected and non-selected HUAs). Second, up to five non-selected areas have been chosen for each area included in the experimental group. Although this specification might be less effective in reducing selection bias, it improves the accuracy of impact estimates (Ming & Rosenbaum, 2000). Finally, we have applied three callipers: levels of standard deviation values equal to 0.0, 0.1, and 0.2 of the logit function of the propensity score have been set.

To validate the propensity score models, we analysed the balance in the distribution of the propensity score and covariables within matched urban areas, including areas selected by the programme, which would constitute our experimental group, and those not selected by the programme, which would make up our control group. We compared the standardised differences between experimental and control groups to ensure they do not exceed 25% and that the overall RMI L1 indicator is small after matching. In addition, we replicated the logistic regression model to verify that its fit-level (Pseudo-R2) is lower in the model that includes matched areas than in the original (including all urban areas). These analyses show all models are well balanced, both globally and for covariables, with a lower fit for the regression model with the paired areas (Table 7.3)Footnote 5.

Table 7.3 Experimental and control groups: validation of propensity score matching models
Full size table

Analysis of the ‘Average’ Impact of Programmes and Potential Heterogeneous Effects: Controlled Comparisons of Urban Trajectories of Change and Other Possible Specifications

To analyse the impact of the URBAN and URBANA Initiatives, we will compare the trajectories of change in the experimental and control groups between pre and post-implementation periods. The difference between their trajectories will inform us of the impact of the programmes. To perform this analysis, we have used existing secondary data sources. This will limit the analysis we will be able to carry out, but it will also show that, within these limits, it is possible to analyse the impact of urban integrated strategies promoted by the EU.

The following chapters provide policy evidence in this regard. They should also be taken as exercises that seek to show different strategies for making controlled comparisons from existing data sources to analyse the impact of EU-integrated strategies. The first chapter applies repeat measures analysis between pre- and post-intervention times based on ecological information (secondary data aggregated at the neighbourhood—urban area—level) to analyse URBAN policy impacts regarding some of their objectives (Chapter 8). The second study trends of change in experimental and control areas by analysing individual cross-sectional data for periods before and after URBANA Initiative implementation (Chapter 9). The third applies repeat measures models with ecological measurements at three points in time to study the short-time and long-time impacts of the URBAN Initiative on the cultural amenities in neighbourhoods (Chapter 10). And finally, we will contextualise the trajectories of change in experimental areas within the urban context of their respective cities applying a comparative case analysis (Chapter 11). In all cases where we analyse the impact based on propensity score matching, we will present the results with the models that use a calliper width equal to 0.2, as this will provide the best results for the estimates (Austin, 2010; Yongji et al., 2013).

As regards the general policy theory behind sustainable and integrated strategies implemented by URBAN and URBANA Initiatives (Chapter 6), two main dimensions or areas of policy impact could be defined; on the one hand, the contextual dimension of targeted urban areas, therefore, improvements that suppose a better opportunity structure for residents. For instance, the urban fabric (better or more houses, public spaces, basic urban services,..) urban mobility opportunities, increasing economic activity and employment opportunities, access to more or new services, environmental quality, low levels of social conflict, crime, … On the other hand, improvements in the ‘individual’ living conditions of residents and their households (socio-economic situation, employment, educational attainment, health status, participation in neighbourhood activities and participatory processes,…). The following chapters will analyse some of these aspects. All these issues could not be examined due to limitations in secondary data disposable and used in the subsequent analyses.Footnote 6 Nevertheless, the analyses done include some of the main goals of the EU cohesion policy and its urban integrated strategies (for instance, habitability, economic activity, employment, or educational attainment).

These analyses will show the ‘average impact’ as differences in the trajectory of change between similar (targeted and non-targeted) urban areas. The results will provide evidence for the following question: what would have happened if the project had not been implemented? This situation is represented by the trajectory of change in the control areas (counterfactual conditional). Comparison with the trajectory of the experimental areas will make it possible to show the impact of the programmes as a whole; in other words, ‘what happens when the project has been implemented’. Therefore, the added value promoted by the urban dimension of the EU cohesion policy through integrated urban development strategies.

This assumes that the analyses will not consider the possible existence of ‘composition effects’ derived from potential diversity between projects, urban areas, or within them not controlled by matching. That is to say, the possible existence of policy heterogeneous effects beyond programme eligibility criteria. For example, according to the specific strategy of each project (its theory), these might include the nature and intensity of residential mobility phenomena or the generational change in neighbourhood residents during the projects implementation period, or specific differences in the starting conditions not included in the eligibility criteria of the programmes (e.g. historic centres vs. peripheral neighbourhoods, or, even, because of the intensity of socio-spatial vulnerability processes). These phenomena can produce opposite effects that, on the whole, blur the evidence of programme impact. Therefore, these elements could also be included in controlled comparisons. Hence, in addition to the selection bias of each programme (matching through propensity score matching), each of the following chapters will add other specific control elements in order to show how possible composition effects could be considered using existing data sources, which are not otherwise considered in the documentation relating to the programmes (and the theory that may arise from them). In other words, we will use propensity score matching based on programmes eligibility criteria (a ‘policy approach’ to ensure similarity between experimental and control groups) plus some theoretical ideas about change in urban areas (a more ‘theoretical approach’) to try to pay attention to potential heterogeneous effects of programmes. Table 7.4 presents the research questions analysed.

Table 7.4 Analysing added value II (better places): research question, analytical and observational units
Full size table

Thus, the following chapters provide evidence on the ‘average’ impact of the URBAN and URBANA Initiatives, that is, an initial approach that considers the ‘general policy theory’ of change implied by these programmes. This brings evidence to an area of study, the impact of EU-integrated urban strategies, for which there is not a lot of previous analysis. The following chapters also provide examples of methods that can be used to analyse the impact of integrated urban strategies using data sources that are usually available to the researcher community and agents involved in the programmes or their specific projects. More detailed access to these and other sources, or the production of data specifically oriented towards the analysis of their impact, might refine the results presented below, so they are able to specify, to an even greater degree, ‘controlled comparisons’ that, for example, derive from the specification of the theories behind these programmes and their projects.