A participative procedure to select indicators of policies for sustainable urban mobility. Outcomes of a national test

Background The assessment of policies for sustainable urban mobility features two basic characteristics: incommensurability and strong uncertainty. This is why multiple dimensions of evaluation and a structured room for collective deliberation and learning should be considered. Methods A participative procedure is used to select a core set of performance indicators of policies for sustainable urban mobility. Citizen participation and stakeholder involvement are obtained through a national sample survey and a deliberative multi-criteria analysis, respectively. This procedure is applied to the Italian case. Results Citizens are more oriented towards reducing private transport costs, air pollution and traffic accidents; stakeholders are more in favour of improving car-free accessibility and reducing the consumption of land and public space generated by urban mobility. The resulting core sets of indicators are highly sensitive to the threshold chosen for the selection. Using a lower cut-off threshold, four performance indicators are shared between the two sets: ‘CO2 from transport’, ‘Quantity and quality of public transport’, ‘PMx, COVNM, NOx, CO from transport’, ‘Death and injuries from traffic accident’; using a higher cut-off threshold the two sets feature no intersection. Concluding remarks Further testing at a local scale is needed in order to explicitly consider context-specific objectives, indicators and data; stronger interactions among experts, citizens and stakeholders are needed in order to avoid the generation of equivocal results.


Introduction
Sustainable urban mobility is a well established environmental issue, not only in local plans, but also in international guidelines [1][2][3][4] and national legislations: this is the case of, among others, the French and Italian laws on Urban Mobility Plans, and the last generation of UK Local Transport Plans. Many large State programs are oriented towards sustainable urban mobility too: the Canadian ecoMOBIL-ITY program and the Indian Sustainable Urban Transport Project are just two recent examples of an ever increasing list. 1 Policies for Sustainable Urban Mobility (PSUM) share two basic characteristics with all other environmental policies. The first one is their intrinsic incommensurability. Due to the co-existence of different objectives, criteria and values, the environmental, social and economic dimensions cannot in fact be compared using a common unit of measurement as suggested by standard techniques based on monetary evaluations (e.g. external costs and cost-benefit analysis). Other techniques that are able to use different metrics and explicitly take into account multiple dimensions of sustainability (e.g. indicator systems and multi-criteria analysis) are best suited for the purpose [5]. The second basic characteristic common to all environmental policies (and-among others-by PSUM) is the presence of strong uncertainty: the probabilities of future changes are not known ex-ante, nor is the set of possible changes. In these cases individuals and society feature bounded rationality [6]; hence policies can no longer be based on neutral values and given preferences, and they must make room for deliberation and learning. As argued by Vatn [7, p. 2211] "We move from aggregating individual measures or bids to reasoning over, and potentially agreeing on common priorities". All these considerations led to the diffusion of participative procedures to establish environmental policies, which usually consist of a deliberation arena to involve citizens and stakeholders, combined with a structuring technique (usually a simplified multi-criteria) to achieve two equally important goals: easing mutual understanding and reaching final recommendations [8].
Similar considerations in favour of involving the people in a multi-dimensional approach to policy design, implementation and appraisal have been recently proposed by authoritative researchers with specific reference to sustainable transport [9][10][11]. More specific calls for the integration of stakeholders into multi-criteria procedures in the context of policies for sustainable transport were recently added to the debate; as clearly stated in the research report of the COST Action 356 "(…) the largest potential for MCDA [Multi-Criteria Decision Analysis] in decision making on sustainable development appears to lie in a combination of MCDA algorithms with participatory techniques and in their better integration into specific transport decision making contexts." [12, p. 271]. The research we present here shares these theoretical and practical concerns and builds a participative procedure to select performance indicators for PSUM. 2 Such a procedure was implemented at a national scale, involving Italian experts, citizens and stakeholders.
The paper is structured as follows. In Section 2 the methodology of the participative procedure is analysed. Outcomes of the procedure are shown and discussed in Sections 3 and 4, respectively. The last section concludes.
2 The participative procedure: methodology Table 1 shows the structure of the proposed procedure, which integrates top-down (expert-led) steps and bottomup (participated) steps [14]. In Step 1, starting from a framework based on the dimensions of urban sustainability and the objectives of PSUM, we selected a first set of performance indicators. In Step 2 this framework was evaluated by citizens and by stakeholders: citizens' opinion about dimensions and objectives were collected through a national survey (Step 2a); stakeholders were involved in a "stakeholder dialogue analysis"-i.e. a participated multi-criteria analysis-in which the dimensions were used as assessment criteria and the objectives as issues to be evaluated (Step 2b). In Step 3, the results of Step 2 were used to rank the initial set of PSUM performance indicators and to select the most relevant among them.

Step 1: a conceptual framework
In the first step of the procedure we further developed a framework which was already used by Isfort 3 as a conceptual tool to select performance indicators of PSUM [15], and which was based on the dimensions of urban sustainability and the objectives of PSUM. The resulting conceptual framework follows the Thematic Indicator Development [16], which is explicitly conceived to manage sustainability policy issues, instead of the more diffused-but less policy oriented-Driving forces-Pressure-States-Impacts-Response approach [17].
Starting from a literature survey on indicators [18] 4 and some workshops and seminars with experts, the three standard dimensions of social, environmental and economic sustainability were articulated into thirteen PSUM objectives, each of which was linked to one (single or composite) performance indicator (see Table 2). The social dimension of sustainability was split in two sub-dimensions: accessibility and liveability. Accessibility was then broken down into four objectives, considering that it depends on more than just transport factors, and that it can be operationalised in several ways [19]. The first objective refers to the ease with which urban services can be used without moving; the others explicitly take the different modes of urban transport into consideration. Then, we explicitly considered how urban liveability is affected by some negative effects of urban mobility: the erosion of public space caused by parked and circulating motorized vehicles, the generation of noise and air pollution, traffic accidents. The environmental dimension of sustainability was translated into three more standard objectives of PSUM: reducing greenhouse-gasses, waste and land consumption generated by mobility. Finally, the economic dimension of urban sustainability is pursued by reducing public and private transport costs. This top-down approach generated a core set of PSUM indicators that meets two of the main criteria of indicator selection: exhaustiveness (every objective of PSUM has its specific indicator) and efficiency (no redundant indicator is considered). Though every indicator of the set is derived from a single objective, some indicators have implications for multiple objectives and most of them have implications for multiple dimensions. It must be stressed that the detailed and operational specification of the indicators goes beyond the objectives of the present study; this is why some indicators are only broadly defined.

2.2
Step 2a: the national sample survey Through the quarterly Isfort's "Audimob" national survey on passenger mobility, a representative sample 5 of the Italian population was asked to evaluate both the generic four dimensions of urban sustainability and the above list of 13 specific objectives. The sample was composed of 3,600 people, reproducing the main structural features of the Italian population aged 14-80 years: sex, age, professional status, region and size of the municipality of residence. The national survey employed the CATI (Computer Assisted Telephone Interviewing) technique, this is why respondents were not directly asked to rank the whole set of objectives and sustainability dimensions, nor were they asked to react to questions based on a Likert scale (both are difficult tasks for a phone interview), but they had to prioritize the importance of dimensions and objectives according to the following qualitative scale: 'Priority'; 'Important but not priority'; 'Useful but not urgent'. In order to avoid any averaging or aggregation of qualitative answers, only the percentage of respondents who rated a given dimension/objective as a "priority" was used to rank citizens' opinion.

2.3
Step 2b: the "stakeholder dialogue analysis" The "stakeholder dialogue analysis" is a participative multicriteria technique that is successfully used to assist stakeholders in discussing a general political issue and in reaching a common position on it [20]. Because of budget constraints, we opted for a simplified "dialogue" 6 that can be summarized as follows. First, we selected relevant Italian stakeholders among the following categories: national and local institutions; associations of consumers/users, environmentalists, workers and companies; political parties. 7 Then, we asked (by e-mail or by fax) their representatives to individually weight the dimensions of urban sustainability mentioned above. Finally, we invited all stakeholders to attend a 1-day meeting during which a multi-criteria scheme was used to rank the objectives of PSUM. Two sub-groups were created to collectively score all PSUM objectives against one dimension of urban sustainability at a time; stakeholders used the following scores: 1 0 useful, but non urgent; 2 0 relevant, but not a priority; 4 0 a priority. These evaluations generated four Reducing noise generated by mobility % of population exposed to harmful noise Reducing air pollutants generated by mobility Main air pollutants from transport: PM X , COVNM, NO X , CO Increasing transport safety Deaths and injuries from traffic accidents Environmental sustainability Reducing greenhouse-gasses generated by mobility CO 2 from transport Reducing waste generated by mobility Waste from transport Reducing land consumption generated by mobility Land occupied by transport infrastructure Economic sustainability Reducing mobility costs: public transport Household expenditures for public transport Reducing mobility costs: private transport Household expenditures for private transport 5 Confidence interval: 1.65; confidence level: 95%. 6 A standard stakeholder dialogue analysis consists of four meetings aimed at setting and using a multi-criteria scheme. 7 For the detailed list of participating stakeholders, see Appendix B. scores for each objective of PSUM, which were then aggregated by using the average weights assigned to dimensions by stakeholders. Two sensitivity tests were carried out to check the robustness of the results of the "dialogue": in the first one, weighted average scores were calculated by using the weights assigned to dimensions by citizens (instead of those assigned by stakeholders); in the second test we lowered from 4 to 3 the score associated to those objective which was considered as "a priority" by stakeholders. In both cases we did not register any change in the final ranking of PSUM objectives.

Step 3: selection of performance indicators
Citizens' and stakeholders' evaluations on objectives of PSUM have been used to select the most relevant PSUM indicators among those ensuing from the first step of the procedure. The two selection criteria we used are very simple: 1) the higher the position of an objective in the ranking, the higher the relevance of the indicator associated with that objective; 2) a threshold value is arbitrarily set to cut off the less relevant indicators of PSUM. It must be stressed that using a cut-off threshold value is more correct than selecting the first X indicators: only in the first case the evaluations of citizens and stakeholders determine which and how many indicators are selected. 8 Moreover, the threshold value can be lowered (raised) if more (less) resources are available to finance the data collection and processing activities needed to use the selected PSUM indicators. Obviously, there is no objective rule to set the threshold value, but the higher the difference between the value of the last of the selected indicators and the value of the first of the non-selected indicators, the lower the arbitrariness of the choice.
3 The participative procedure: results

Citizens' opinion
According to citizens' opinion, environmental sustainability and liveability are the most relevant issues (see Tables 3 and  4): reducing greenhouse-gasses, air pollutants, waste and accidents from transport are considered as priority by more than 58% of respondents. The objectives related to the dimension of accessibility rank low (see Table 4), except for the objective of easing public transport (that ranks 6th). Economic sustainability stands in a middle position, which is the average of the 2nd and 8th positions reached by the objective of reducing private and public transport costs, respectively.
The results of the national survey were also analysed with reference to the size of cities where citizens live and to their preferred mode of transport. 9 The ranking of two important objectives is correlated to urban size: easing public transport ranks 4th according to the opinion of citizens living in big cities, while it ranks 7th in medium cities and metropolitan belts, and 9th in small cities; reducing air pollutants ranks 1st in big cities, 2nd in metropolitan belts, 3rd in medium cities and 4th in small cities. Instead, the relevance of the issue of reducing private transport cost is inversely correlated to urban size: it ranks 1st in small cities and 5th in big cities.
A relevant difference between user groups concerns the issue of accessibility: easing non-motorised mobility ranks 2nd for cyclists (9th and 10th for public transport and car users, respectively); easing public transport ranks 2nd for public transport users (6th and 10th for car users and cyclists, respectively). Unexpectedly, easing private transport stays at the bottom of the ranking for all user groups. 10

Stakeholders' appraisal
Easing non motorized mobility and public transport are the two objectives of PSUM that achieved the maximum weighted score among stakeholders (see Table 5). Because of both low weights and very low scores, the two objectives of reducing private and public transport costs lie on the opposite side of the rankings. A low weighted score is also reached by the objective of easing private motorized mobility. Other objectives connected to the dimensions of environmental sustainability and liveability scored high in the ranking; while reducing noise and waste generated by transport are perceived as less relevant objectives of PSUM. 11 Data were also analysed with reference to other demographic and social variables (sex, age, region of residence, education, profession, etc.), but those results were not significantly different from those emerging from the whole sample. 10 For a more detailed analysis of segmented data, see [18]. 11 See Appendix C for detailed results.

The selection of indicators
In order to select PSUM performance indicators, a threshold % of 48 was applied to citizens' opinion and a threshold score of 2.85 was applied to stakeholders' appraisal.
The eight indicators selected on the basis of citizens' opinion cover all dimensions of urban sustainability, of these three refer to the dimension of liveability (air pollutants, accidents and noise) and only one (public transport) refers to the dimension of accessibility (see again Table 4). Land consumption is the highest ranking indicator not selected by citizens. It must be noted that three more indicators would be cut off if we used a slightly higher threshold (50% instead of 48%).
Eight indicators came out of stakeholders' evaluation too (see again Table 5), but none of them refers to the economic dimension of urban sustainability. Three indicators of accessibility were selected, two of which (easing public and nonmotorised transport) reached the maximum score. Noise ranks first among the indicators not selected by stakeholders. The number of selected indicators is halved if one uses a slightly higher threshold (3.0 instead of 2.8).

Contribution to the relevant literature
The methodology we implemented to select PSUM indicators is based on two main tools: 1) a conceptual framework of dimensions and objectives of PSUM, which is used to select a first core set of performance indicators; 2) the participation of citizens and stakeholders, in order to rank the selected indicators and-depending on the available resources-cut off the less relevant ones. The used methodology explicitly refers to the theoretical and practical work of two research groups, led by Mark Reed and Jacquelin Burgess, respectively [14,20].
As far as we know, no reference can be found in the literature to the application of such an integrated methodology to the issue of sustainable urban mobility: many studies explicitly consider the dimensions of environmental, social and economic sustainability, some of them refer to objectives of policies for sustainable transport, but only two [21,22] are based on the participation of stakeholders (and not of citizens) in order to select a limited number of indicators from a much longer initial list. 12 We may then conclude that the main contribution of this study to the relevant literature is the integration of the participation of both citizens and stakeholders into the selection of PSUM indicators. The remaining part of this section discusses separately the three steps of the implemented procedure and its results.

Discussion of step 1 (a conceptual framework)
Though the definition of the initial conceptual framework is not the objective of our research (nor its added value), it must be emphasized that the starting reference to an already partially developed framework constitutes the main limitation of this study. We integrated the first version of the framework on the basis of a review of the relevant literature and meetings with 12 See again Appendix A for more details. other experts, but we remain aware that it needs further improvement. Inter alia: the objectives of urban density and social inclusion should be explicitly considered; all indicators-especially in the case of practical applications-should be better specified in order to check the availability of data and associate quantitative targets to objectives. At the same time, we remain convinced of our choice of referring to generic objectives and not to specific actions; otherwise the acceptability of a policy, instead of its relevance, is likely to be assessed.

Discussion of step 2 (ranking of PSUM objectives)
First of all, it must be said that we had to repeat the national survey after interviewers reported that respondents found some questions hard to understand. This is why we needed to clarify, for example, the difference between greenhouse gasses and local air pollutants, and between land consumption and the occupation of urban space; moreover, we had to add some practical examples to explain what we meant by 'waste generated by mobility'. It remains that some respondents could have responded superficially, without signalling to the interviewer that they did not completely understand the question. 13 This is a problem which could be managed only by direct contact with citizens: ex-ante, through focus groups aimed at reaching a shared terminology; ex-post, through structured arena, where citizens could deliberate on the results emerging from the survey. There is one striking result emerging from the national survey: the objective of 'reducing private transport costs' ranks 2nd, which seems not so much consistent with the other high ranking objectives ('reducing greenhouse gases', 'reducing air pollutants', 'increasing safety', etc.). Actually, one should consider that most citizens are car drivers, who aspire at a more sustainable urban environment and-at the same time-are budget conscious; such an interpretation is confirmed by data segmentation: the need of reducing private costs ranks 5th in big cities (were the % of car drivers is lower) and 6th for public transport users. In more general terms, the analysis of the results of the national survey by urban size and transport mode shows that opinions on objectives of PSUM depend on the size of the city where citizens live and on their actual mobility behaviour.
Stakeholders' appraisal was more consistent: the objectives of easing private mobility and reducing its costs have been positioned at the bottom of the ranking; moreover, stakeholders did not explicitly refer to specific situations (even if one should suppose they generally refer to metropolitan contexts). Most important, though stakeholders were assigned the burdensome task of carrying out 52 assessments (13 objectives against four criteria), deliberation was easier than expected: the assignment of reaching shared evaluations pushed the mutual understanding between involved parties and the arrangement of all interests at stake. During the "dialogue", stakeholders acted as 13 This point was raised by an anonymous referee. Reducing noise generated by mobility 2.14 9°% of population exposed to harmful noise Reducing waste generated by mobility 2.12 10°Waste from transport Easing private motorized mobility 1.26 11°Congestion Reducing public transport costs 0.62 12°Households expenditures for public transport Reducing private transport costs 0.62 12°Households expenditures for private transport a 1 0 useful, but non urgent; 2 0 relevant, but not a priority; 4 0 a priority experts too; that is, they suggested some integrations to the proposed framework: in particular they asked for the explicit consideration of the issue of density 14 and a greater articulation of the dimension of economic sustainability.

Discussion of step 3 (selection of PSUM indicators)
The two selections of PSUM indicators-based on citizens' opinion and stakeholders' appraisal, respectively-depend on the level of the thresholds which are used to cut off the less relevant indicators (see Table 6). When lower thresholds are used, only four performance indicators are shared between the two resulting sets: 'CO2 from transport', 'Quantity and quality of public transport', 'PMx, COVNM, NOx, CO from transport', 'Deaths and injuries from traffic accidents' (and only one indicator is cut off from both lists: 'congestion'). When higher thresholds are used, the two sets show no intersection. There is no immediate explanation for the divergence between the two selections. We can only stress one relevant difference in the composition of the selecting groups: on one side, car users are almost 85% of the sample, which implies that their opinion strongly influences the results of the national survey; on the other side, only one stakeholder (out of 13) directly represented the interests related to the car. Moreoveras already stressed-all representative of the involved stakeholders live in a metropolitan context where the car is less used, and alternatives to the car are more diffused. Finally, it must be said that we had planned for stakeholders to know the evaluations of citizens before starting their "dialogue", with the purpose of reducing the risk of generating equivocal results, but this was not possible because the results of the national survey were not available when the stakeholder dialogue started (a delay caused by the already mentioned repetition of the national survey).

Conclusions and further research
In the research project presented here, the direct participation of citizens and stakeholders was used in order to manage the high level of uncertainty and incommensurability featured by environmental issues. A national survey and a "stakeholder dialogue analysis" (based on a multi-criteria technique) were integrated into an original procedure aimed at selecting a core set of performance indicators of policies for sustainable urban transport. The procedure proved valid in showing that citizens' opinion is context-and behaviourspecific, and generates a selection of indicators which diverges from that emerging from stakeholders' appraisal.
Most insights which result from this first national test stress the need of more widely shared basic concepts and less equivocal results, thus asking for a closer interaction between experts, citizens and stakeholders and for a more contextbased approach to objectives, indicators and data. This is why a more articulated procedure is now proposed for future testing at a local scale (see Table 7). First of all, a new 'step 2' is added at the beginning of the procedure to check if the starting conceptual framework is exhaustive and widely shared, that is, if it covers all relevant issues connected to the sustainability of urban mobility, especially those which are context-specific. This new step would also be helpful for finding a widely understood terminology to be used in the national survey. Then, a new 'step 3' should help in making the basic framework more operational: a context-specific indicator should be associated to every objective; data should be provided about its current status and realistic target; the threshold used for the selection of indicators should be based on the cost of data collecting and processing. Finally, it must be ensured that the procedure is able to effectively manage the evaluations of citizens and stakeholders, even when they are divergent. Though this is an issue that needs deeper understanding, we think that assigning greater importance to citizens' opinion could be a solution. Stakeholders should know the evaluations of citizens before starting their meetings, and citizens may have "the last word" about the selection of indicators: a final "joint workshop" [23] or a "citizens' jury" [24] could close the participative procedure with the ambitious goal of generating serviceable results. Moreover, a direct contact with experts could help citizens to avoid misunderstandings and superficial responses.
Acknowledgements The research presented here was promoted and funded by Isfort-Istituto Superiore di Formazione e Ricerca per i Trasporti, an Italian private research body specializing in the transport sector (http://www.isfort.it/). The research was partially co-funded by the University of Sassari (Italy). Authors thank Eleonora Pieralice of Isfort for data processing and three anonymous referees for their helpful comments.
Open Access This article is distributed under the terms of the Creative Commons Attribution License which permits any use, distribution and reproduction in any medium, provided the original author(s) and source are credited.
Appendix A Table 7 A participative procedure to select performance indicators of sustainable urban mobility policies (PSUM): the revised version (added steps in italics) Step  Integrated approach based on land-use transport models, spatial disaggregation of the data, economic/social evaluations, and multi-criteria analysis Appendix B