Introduction

Evaluation has been a critical endeavor, and integral to the operations of the WHO European Healthy Cities Network (WHO-EHCN) since its very inception. The European regional office of the World Health Organisation in Copenhagen started these efforts in the mid-1980s. Inspired by North American efforts (scholarly activism by Professor Leonard Duhl from Berkeley, California, and activist of public health by Dr. Trevor Hancock from Toronto1), the idea was proposed to cities at meetings in Lisbon and Dusseldorf. The express intent of this effort was to pilot at the local level both the emerging health promotion agenda (cf. the Ottawa Charter) and the strategy for achieving European Targets for the Health for all by the Year 2000. Clearly, successful piloting requires a process of assessment.

This assessment was not initially formalized in rigorously developed theory-based evaluation schemes. The sheer enthusiasm expressed by mayors, councillors, health promotion and public health workers and community representatives in the early days of the movement overwhelmed the small WHO secretariat. This enthusiasm in itself was, legitimately, considered significant validation of the power of the Healthy Cities perspectives to put health on local agendas.

The rapid growth of the initial group of cities in the second half of the 1980s clearly presented serious challenges, both to the Healthy City leadership at WHO, their governing bodies (traditionally WHO's business is with its member states, not directly with lower levels of government and governance!), and the evolving network of cities. The formal planning horizon for WHO is no more than a few years, and many within the movement argued for an extended period of investment in European Healthy Cities. Such a perspective was reinforced by the great success that ‘Healthy Cities’ soon started to have in other parts of the world, particularly in Oceania, Canada and Central America. A formal continuation in Europe depended on a formal assessment of what was and what was to be.

It was in particular Ron Draper, a Canadian who had already been instrumental in creating the conditions for the development and adoption of the Ottawa Charter, who took the lead at the European regional office of WHO in providing what we now would probably call a ‘realist evaluation’ of what became Phase I of the Healthy Cities project. Tirelessly, he spoke to Healthy Cities operators (activists, politicians and professionals) and assessed their performance against their potential. Apart from an internal discussion paper,2 the most prominent outcomes of these processes were the publication of the ‘Twenty Steps’ guidelines and ‘A Project Becomes a Movement’ full-colour coffee table book. Both of these are, 25 years after publication, still very popular and in fact on the most-downloaded list of the WHO Healthy Cities publications website.3

The project was continued in a second phase, and more formal evaluation became a serious issue in a stricter system of designation and requirements for delivery by cities of outcomes, outputs and processes.4 The phase was furthermore characterised by the production of a multitude of publications on how to measure health, how to produce city health profiles and how to assess progress in and for Healthy Cities. It seems that the technical support through WHO, and its mobilization of international expertise for the benefit of local health, contributed significantly to the growing maturity of the ‘early adopters’ in the first 10 years of the European movement.5

An initial guide to assessing Healthy Cities suggests that the important questions for Healthy Cities are process questions6; in an adaptation of this framework for the third phase of the European project, questions were further refined and adapted to reflect processes, changes as well as results. This happened in the monitoring, assessment, reporting and impact assessment framework.7 Processes, and not particular quantitatively framed products, remain the important critical drivers of what are currently thousands of cities in Europe, either in direct association with WHO through its designation processes or indirectly through the thriving network of National Networks for Healthy Cities.8,9

Four interrelated components of successful development of Healthy Cities are referred by Tsouros in a companion article10 in this supplement and earlier in the development of the WHO-EHCN.11 These core components are: political commitment; vision and strategy; and institutional change and networking. However, it will also be clear that the unique historical, cultural, economic and population context for each (associated) member of the Healthy Cities movement will create unique opportunities and barriers to work with these processes toward the achievement of the overarching Healthy Cities ideal: to maintain in a sustainable way the prominence of health and its determinants on social and political agendas of local government areas12 and to ensure governance and accountability mechanisms that enable responsive adaptation to the continuously changing environment.13 Such an observation must lead to some theoretical and methodological inferences.

Theoretically, it is clear that there is a profound need for what can be called ‘hybrid theorizing’. Particularly in the highly complex urban health environment complementary or overlapping theoretical perspectives may yield the added value that more traditional views cannot attain.14 An example of such an approach has been described by Greenhalgh and Stones.15 The set of values that drives Healthy Cities, formed and formalized historically, causally and normatively, exemplifies the multidisciplinary nature of the endeavour and adds urgency to a need to establish research frameworks that would logically formulate pertinent hybrid theories for their study. Apart from the much needed more descriptive studies (addressing questions of ‘what is there?’), hybrid theories would allow us to also ask ‘how does it work?’ questions.16

Elsewhere, I have argued that it is critical for the evaluation of Healthy Cities (and the argument applies for all health action) to establish an optimal fit between the nature of the problem, the appropriateness of the theoretical and conceptual frameworks that drive an understanding of (solutions for those) problems, the methodological considerations that would yield most meaningful insights and operational methods to generate apt data that can be processed through the most fitting tools of analysis.17 Although the argument will not be repeated here, it remains important to observe that the generation of meaningful and applicable information on the success of urban health action should not be driven by efforts to profess a belief in ‘gold standards’ for scholarly research (such as the randomized controlled trial, which in itself is a highly appropriate inquiry design only for the study of phenomena that can be fully manipulated and controlled by the researchers). ‘Health’ is a relative state18 and highly contextual, driven by individuals, groups and communities that constantly live and create health in a complex, changing and adaptive social and physical environment. The research of and for health should be profoundly responsive to its nature19 and cannot be addressed by a ‘gold standard’ alone, not in the least because some methods and methodologies do not allow for the direct engagement and action from ‘those affected’ (that is, communities, families, but also policy makers and other—private—actors).20

A Theoretical and Methodological Appraisal

My purpose here is to situate, both theoretically and methodologically, the further evolution of evaluation efforts that have been undertaken by a group of senior researchers for the assessment of Phase IV of the European Healthy Cities Network reported in this special issue of the Journal of Urban Health. 21 Lawrence continues the argument about the nature of health development in cities which he has eloquently analyzed in a number of earlier publications.22

Lipp et al. ask questions around the engagement of cities in partnership development and maintenance. The level addressed in their piece is of a higher aggregate, drawing on insights that intersectoral action (within government sectors, but particularly beyond these into private, non-governmental and quasi-governmental (QUANGOs) arenas) will enable cities to address health and its determinants more widely. The assumption here is that more inclusive, broader and sustained partnerships will yield opportunities for health development that the public sector on its own cannot generate. This assumption is grounded in an emerging theoretical and conceptual—hybrid—framework that stipulates characteristics of successful health promotion partnerships.23,24 The subsequent methodology, though not predetermined on the basis of conscientious research considerations, has a strong flavour of a descriptive responsive realist evaluation.

Webster and Sanderson address an aspect of Healthy Cities that has been integral to the project since its inception: the city health profile as a critical tool of health information for all stakeholders in the Healthy City. The development and regular update of these city health profiles, it is assumed, is a higher level precondition for policymaking and intervention development. The authors describe how the set of indicators has been adapted to suit local data needs and to enable further implementation feasibility. The analytical approach to this descriptive study resembles a realist meta-analysis.

The DECiPHEr team (Whitfield, Machaczek and Green) deploys a realist synthesis approach to estimating the health impacts of distal determinants (and their changes) on city health. DECiPHEr utilizes the ‘hybrid theory’ approach in that it takes the tenets of realist evaluation (that evaluation efforts need to take context formally into account) and integrates these with an explicit conceptual perspective on proximal and distal determinants of health.7 The realist synthesis approach then allows for specific and pertinent guidance for city administrations to choose interventions that would have the greatest impact, even in a cost–benefit analysis.

Dooris and Heritage look at the nature of community participation and empowerment, not in terms of their effectiveness (which have been demonstrated by, for instance, Laverack25 and Wallerstein26) but in terms of the qualitative presence of elements of Davidson's wheel of participation.27 Again, their question addresses a higher level complexity: engagement by Healthy Cities in participation and empowerment endeavors. Davidson's work is unequivocally a non-hierarchical conceptual frame of reference that allows for enhanced understanding of the simultaneous approaches a city administration can unfold in engaging its people in increasing their control over what determines their health. Dooris and Heritage explicitly chose this theoretical approach to counter the critique of WHO's adoption of empowerment and participation principles as ‘atheoretical and depoliticized pragmatism’. Such a choice is, again, typically inherent to a realist evaluation approach which would consider it important to appreciate the ‘a priori’ political context of policy programs. However, these authors recommend that a full-scale realist evaluation effort necessarily requires face-to-face engagement between partners in the inquiry effort. This has not been part of the mixed-method approach chosen, including a qualitative content analysis of responses to questionnaires, annual report template submissions and case study descriptions. Face-to-face elicitation and validation of data and information would have enhanced the quality of the empowerment and participation assessments.

In Ritsatakis' article, the question that is addressed is whether European Healthy Cities have placed equity and the wider social determinants of health on their political and social agendas. The question asked here steps beyond the moral, ethical or philosophical (on effectiveness or otherwise) as these have already been addressed adequately elsewhere.28 But connecting the health equity and social determinants agendas with the broader Healthy Cities approaches and value system would, it is taken, lead to better capacity to develop equitable population health characteristics. Ritsatakis was dependent on the three data sources mentioned above (questionnaire, annual reports and case studies) and her qualitative analytical approach leans toward hermeneutics. Particularly in the area of equity (with its strong moral and ethical associations), such a perspective seems highly relevant.

Another procedural question is asked in the paper by Ison. Health impact assessment (HIA) is a method used to assess the potential effects of a policy, program or project on the health of a population. It has been defined as a critical tool for building Healthy City policies and plans. There is an increasing evidence base that HIA is an effective tool across the full range of social, health and equity issues29, so Ison does not need address the question whether HIA is effective. Ison's study, in connection with the other tenets of the European project, assumes that the effectiveness of Healthy Cities ways of working will improve through qualitatively and quantitatively better action on the part of all stakeholders involved: again, this is a complex multi-level approach. The author chose a comparative multi-method review of the data that were generated through the WHO evaluation effort and found significant increases in adoption of HIA in the group of cities that signed up to the pilot effort, but some levelling off of interest among other cities. Her recommendation that a staged, tactical introduction of the method in urban environments would yield significant gains necessarily has consequences for the future development of theory and methodology that would guide HIA diffusion.

Green and Faskunger, in two essays on specific aspects of health (ageing and active living, respectively), look at systems and processes that enhance the inclusion of these areas in urban health programmes. In both cases, the premise is that cities are supremely equipped to deal with systems and infrastructure development for ageing populations and active living. Both adopt rigorous approaches to data and methodological triangulation which fits with a strong qualitative methodology.30

Barton and Grant also deploy triangulation procedures when they retrace the re-integration of public health and urban planning (once inseparable children of the same parent31) in the healthy urban planning approach. The logical conceptual framework that has driven the emergence of healthy urban planning now dictates a complex, multi-level and intersectoral endeavor that intricately links with other aspects of Healthy Cities. They find that healthy urban planning is both qualitative as well as quantitative on the increase in all European Healthy Cities. In their argument, they acknowledge that healthy urban planning is both complementary and overlapping with many of the other components of Phase IV Healthy Cities parameters. Although this increases complexity and diversity in theoretical and methodological perspectives, they demonstrate that credible and responsive data for the further development of sound urban planning in Healthy Cities can and should be generated. The one issue they identify within such an approach is of a ‘realist’ nature: as there is a considerable response bias, face-to-face data collection and validation should enhance the immediacy of evaluation results to all Healthy Cities.

Finally, Heritage and Green take a look at possibly the highest level of influence on health in European Healthy Cities: the networking amongst the 87 designated cities and their own networking within national and regional European entities (totalling some 1,300 cities). Again, the merit and value of networking per se are uncontested because they have been established earlier32,33 and the review addresses the complex and reciprocal pathways and functionalities that make up such networks. This study stands out as it has adopted a rigorous conceptual and theoretical framework that established the ‘inquisitive gaze’. They assume (in line with the original pragmatism that guided the European developments) that networks of Healthy Cities, at different levels and interactive manners, are epistemic communities. Such communities, networks of academics and professionals who share a number of principles and beliefs about their policy environment across settings, would endeavor to grow for improved action. In transnational policy and organizational research, the model has acquired great validity, recently in the reaffirmation of the value of citizen participation in primary health care.34 Heritage and Green, and their initial research collaborator Janss Lafond, have crafted a profound analysis that should inspire future generations of urban health scholars.

Discussion

Common Theoretical Assumptions

All authors share at least three theoretical assumptions about the generation of population health, and specifically, urban health within the healthy city value system: First, health is determined by distal, structural factors (Marmot35 referred to them as ‘the causes of the causes’) as described in this volume by Lawrence.36 These factors, then, are embedded in complex reciprocal chains of causality with the more proximal determinants.7,36 Second, the actors most likely to effectively impact on those complex arrays of determinants of health are municipalities, communities and other local partners that all have a legitimate institutional role in protecting and promoting health. This assumption is implicit in most analyses in the contributions in this volume, but very explicitly addressed by Lipp et al.37 and Whitfield et al.35 Third, an integral part of the conceptual perspective adopted by all authors is that Healthy Cities projects have a key role in generating the critical success components for city health development as outlined by Tsouros and by De Leeuw.7

Interventions and Impact

As soon as the complex and volatile field of determinants of urban health is acknowledged, combined with the strong value base of European Healthy Cities (including principles of equity, sustainability and participation), it would be clear that research design in this field is a quagmire of methodological challenges. Elsewhere, we have already argued that the nature of the problem and its conceptualization should drive conscientious selection of methods.15 Thus, for the particular issue and its conceptualization, the strongest possible methodology should be selected. Considering the wicked nature of Healthy Cities' research5, this means that the gold standard of health research (the randomized controlled trial) is untenable and conceptually inappropriate. Thus, when critics of Healthy Cities focus on weak methodology, they do not seem to have appreciated the most appropriate approach for evaluating both the health outcomes of specific projects and programs and the wider impacts of the whole city systems.

A recent review of appropriate methodologies to evaluate impact15 asserts that in such complex environments, four types of questions should be asked38: (a) To what extent can a specific (net) impact be attributed to the intervention? (b) Did the intervention make a difference? (c) How has the intervention made a difference? and (d) Will the intervention work elsewhere? These impact questions align more appropriately with the observation that Healthy Cities' interventions are more effectively evaluated by a realist approach. This acknowledges the importance of context and multiple, simultaneous interventions and outcomes. Reflecting the WHO-EHCN approach2,4,7, authors in this volume have focused on the city processes and preconditions for improved health outcomes. Some authors, e.g., Barton and Grant39, have utilised the ART time series. This approach records critical success processes put in place and maps whether conditions have enhanced or eroded the opportunities to deliver anticipated outcomes and impacts. All authors have utilized the general evaluation questionnaire (GEQ). This may not have been the most rigorous method under the circumstances. The GEQ specifically asks cities to self-assess performance both on the processes which should have endured and enhanced over Phases II–IV and also the four specific themes of Phase IV. No author identifies a measurable health impact, though this may be inferred from a realist synthesis of the evidence.35 For example, the city of Helsingborg has pursued healthy urban planning and invested in cycling infrastructure. As a likely consequence, utility cycling accounts for 26 % of all commutes. Scientific evidence shows that such exercise reduces the risk factors of cardiovascular disease.

Weaknesses and Opportunities

The evaluation of Phase V of the WHO-EHCN is a great opportunity to address the substantial weaknesses in previous evaluations. This hinges on issues of causality and attribution to Healthy Cities interventions (‘did they make a difference?’). A more rigorous realist conceptual framework (uniform and unified, negotiated and integrated) would encourage evaluators to cut through the complexity of cities and identify more precisely the outcomes of specific interventions and their contribution to wider health impacts. Indeed, after four previous evaluations spanning more than two decades, this is imperative if the WHO-EHCN is to maintain credibility in a world of public health still dominated by orthodox, narrow research paradigms. This approach would require more resources than available for the evaluation of previous phases. These would strengthen research capability at the core of the operation. Equally important, modest investment would capture the intelligence/evidence continuously generated by network cities, but currently dissipated.

The variability of theory-driven and methodological quality impacts on the overall quality of the research on this current phase of the project; this phenomenon is best illustrated Liebig's law, after Justus Baron von Liebig who determined—for essential nutrients of flora—that a barrel can only be as full as its lowest stave determines40 (Fig. 1).

Figure 1
figure 1

Von Liebig's barrel—quality of Healthy Cities evaluation.

Full size image

Following this notion, we believe that the overall quality of Healthy Cities' research is not only determined by limited quality research, but also the quantity of the material, and its nature, that is poured into the barrel. Moreover, possibly some of the wealth of information that is generated in the ‘real life laboratory’ that Healthy Cities goes unexploited and spills over unused when we do not apply appropriate research frameworks. Approval of the new European Health 2020 strategy suggests that there is a renewed determination by WHO to support this effort.41 As academics, we must contribute to forging effective partnerships towards these strategic objectives.42