Skip to main content

A Framework for Post-Project Evaluation of Multicriteria Decision Aiding Processes from the Stakeholders’ Perspective: Design and Application

Abstract

Numerous multicriteria decision aiding (MCDA) methods have been developed over the last decades and are now applied in various domains, sometimes using facilitated group workshops to create models. These models are all designed to improve decision processes. However, the lack of follow-up and post-project evaluations limit the understanding of how the participants experienced the group workshops and how the results were subsequently used within the organization. This is in contrast with the public participation research field, where a rich literature was developed for a posteriori evaluation of projects. Based on this literature, our research proposes a framework to evaluate, ex-post, MCDA projects. In order to illustrate this framework, we apply it to an MCDA project in Quebec City where a spatial decision support system to prioritize the redesign of streets as Complete Streets was built. Individual interviews were conducted with the Quebec City professionals that currently use, were leaders of the project, or have participated in the development of the decision support system. This research has identified that the need for change of practices within the workplace, communication problems, and the requirement for multidisciplinary work were at the root of the various challenges encountered during the workshops. Based on our experience, we propose some lessons learned and potential solutions that can enhance the body of literature in MCDA.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Availability of data and material

To respect the consent form that all participants signed, the individual results from the participant interviews such as the transcripts are not available, and participants’ quotes were anonymized in this paper.

Notes

  1. 1.

    The term artifacts (Simon 1996) will be used in this paper as a global term that refers to the set of new objects (models, tools, or decision support systems) that are created and designed in an MCDA or OR project (Dresch et al. 2014). In more particular cases, the specific term such as “the model” or “the tool” will be used.

References

  1. Ackermann F, Yearworth M, White L (2018) Micro-processes in Group Decision and Negotiation: practices and routines for supporting decision making. Group Decis Negot 27:709–713. https://doi.org/10.1007/s10726-018-9590-x

    Article  Google Scholar 

  2. Ackoff RL (1962) Some unsolved problems in problem solving. Oper Res Q 13:12. https://doi.org/10.2307/3007575

    Article  Google Scholar 

  3. Ackoff RL (1979a) The future of operational research is past. J Oper Res Soc 30:93. https://doi.org/10.2307/3009290

    Article  Google Scholar 

  4. Ackoff RL (1979b) Resurrecting the future of operational research. J Oper Res Soc 30:189. https://doi.org/10.2307/3009600

    Article  Google Scholar 

  5. Andersen DF, Richardson GP (1997) Scripts for group model building. Syst Dyn Rev 13:107–129. https://doi.org/10.1002/(SICI)1099-1727(199722)13:2%3c107::AID-SDR120%3e3.0.CO;2-7

    Article  Google Scholar 

  6. Bana e Costa CA, De Corte J-M, Vansnick J-C (2005) M-MACBETH. Version 2.4.0. BANA Consulting

  7. Bana e Costa CA, De Corte J-M, Vansnick J-C (2016) On the Mathematical Foundations of MACBETH. In: Greco S, Ehrgott M, Figueira JR (eds) Multiple Criteria Decision Analysis. Springer, New York, NY, pp 421–463

    Chapter  Google Scholar 

  8. Banville C, Landry M, Martel J-M, Boulaire C (1998) A stakeholder approach to MCDA. Syst Res Behav Sci 15:15–32. https://doi.org/10.1002/(SICI)1099-1743(199801/02)15:1%3c15::AID-SRES179%3e3.0.CO;2-B

    Article  Google Scholar 

  9. Barcus A, Montibeller G (2008) Supporting the allocation of software development work in distributed teams with multi-criteria decision analysis. Omega 36:464–475. https://doi.org/10.1016/j.omega.2006.04.013

    Article  Google Scholar 

  10. Becker KH (2016) An outlook on behavioural OR—three tasks, three pitfalls, one definition. Eur J Oper Res 249:806–815. https://doi.org/10.1016/j.ejor.2015.09.055

    Article  Google Scholar 

  11. Behzadian M, Kazemzadeh RB, Albadvi A, Aghdasi M (2010) PROMETHEE: a comprehensive literature review on methodologies and applications. Eur J Oper Res 200:198–215. https://doi.org/10.1016/j.ejor.2009.01.021

    Article  Google Scholar 

  12. Bouyssou D, Perny P, Pirlot M et al (1993) A manifesto for the new MCDA era. J Multi-Criteria Decis Anal 2:125–127. https://doi.org/10.1002/mcda.4020020302

    Article  Google Scholar 

  13. Braune I, Pinkwart A, Reeg M (2009) Application of multi-criteria analysis for the evaluation of sustainable energy systems - A review of recent literature. In: Proceedings of the 5th Conference on Sustainable Development of Energy, Water and Environment Systems. Dubrovnik, Croatia, p 13

  14. Brown G, Chin SYW (2013) Assessing the effectiveness of public participation in neighbourhood planning. Plan Pract Res 28:563–588. https://doi.org/10.1080/02697459.2013.820037

    Article  Google Scholar 

  15. Browne D, Ryan L (2011) Comparative analysis of evaluation techniques for transport policies. Environ Impact Assess Rev 31:226–233. https://doi.org/10.1016/j.eiar.2010.11.001

    Article  Google Scholar 

  16. Chakhar S, Mousseau V (2017) Multicriteria spatial decision support systems. In: Shekhar S, Xiong H, Zhou X (eds) Encyclopedia of GIS. Springer International Publishing, Cham, pp 1404–1411

    Chapter  Google Scholar 

  17. Chess C (2000) Evaluating Environmental Public Participation: Methodological Questions. J Environ Plan Manag 43:769–784. https://doi.org/10.1080/09640560020001674

    Article  Google Scholar 

  18. Choi BCK, Pak AWP (2005) A catalog of biases in questionnaires. Prev Chronic Dis 2:13

    Google Scholar 

  19. Churchman CW (1970) Operations research as a profession. Manag Sci 17:B-37-B-53. https://doi.org/10.1287/mnsc.17.2.B37

  20. Communauté métropolitaine de Québec (2013) Plan métropolitain d’aménagement et de développement du territoire de la Communauté métropolitaine de Québec

  21. CORS (2019) Practice Prize Competition. In: CORS. https://www.cors.ca/?q=content/practice-prize-competition. Accessed 26 Feb 2020

  22. Damart S (2010) A cognitive mapping approach to organizing the participation of multiple actors in a problem structuring process. Group Decis Negot 19:505–526. https://doi.org/10.1007/s10726-008-9141-y

    Article  Google Scholar 

  23. de Gooyert V, Rouwette E, van Kranenburg H, Freeman E (2017) Reviewing the role of stakeholders in operational research: a stakeholder theory perspective. Eur J Oper Res 262:402–410. https://doi.org/10.1016/j.ejor.2017.03.079

    Article  Google Scholar 

  24. Decision Analysis Society (2020) DAS Practice Award. In: INFORMS. https://www.informs.org/Recognizing-Excellence/Community-Prizes/Decision-Analysis-Society/DAS-Practice-Award. Accessed 26 Feb 2020

  25. D’Este G (2009) Capturing different viewpoints in multi-criteria analysis. In: Proceedings of the 32th Australasian Transport Research Forum. p 11p (session Wed 3A)

  26. Dresch A, Lacerda DP, Antunes Jr. JAV (2014) Design science—the science of the artificial. In: Design science research. Springer, Cham, New York

  27. Eden C (1995) On evaluating the performance of ‘wide-band’ GDSS’s. Eur J Oper Res 81:302–311. https://doi.org/10.1016/0377-2217(93)E0241-O

    Article  Google Scholar 

  28. Eden C, Ackermann F (1996) “Horses for courses”: a stakeholder approach to the evaluation of GDSSs. Group Decis Negot 5:501–519

    Article  Google Scholar 

  29. ESRI (2014) ArcMap. Version 10.2.2. ESRI

  30. Ferreira F, Santos S, Rodrigues P (2011) Adding value to bank branch performance evaluation using cognitive maps and MCDA: a case study. J Oper Res Soc 62:1320–1333

    Article  Google Scholar 

  31. Franco LA, Greiffenhagen C (2018) Making OR practice visible: Using ethnomethodology to analyse facilitated modelling workshops. Eur J Oper Res 265:673–684. https://doi.org/10.1016/j.ejor.2017.08.016

    Article  Google Scholar 

  32. Franco LA, Hämäläinen RP (2016) Behavioural operational research: returning to the roots of the OR profession. Eur J Oper Res 249:791–795. https://doi.org/10.1016/j.ejor.2015.10.034

    Article  Google Scholar 

  33. Franco LA, Montibeller G (2010) Facilitated modelling in operational research. Eur J Oper Res 205:489–500. https://doi.org/10.1016/j.ejor.2009.09.030

    Article  Google Scholar 

  34. Franco LA, Nielsen MF (2018) Examining group facilitation in situ: the use of formulations in facilitation practice. Group Decis Negot. https://doi.org/10.1007/s10726-018-9577-7

    Article  Google Scholar 

  35. Fung A (2006) Varieties of participation in complex governance. Public Adm Rev 66:66–75. https://doi.org/10.1111/j.1540-6210.2006.00667.x

    Article  Google Scholar 

  36. Geldermann J, Bertsch V, Treitz M et al (2009) Multi-criteria decision support and evaluation of strategies for nuclear remediation management. Omega 37:238–251. https://doi.org/10.1016/j.omega.2006.11.006

    Article  Google Scholar 

  37. Genard J-L, Pirlot M (2002) Multi-criteria decision-aid in a philosophical perspective. In: Bouyssou D, Jacquet-Lagrèze E, Perny P et al (eds) Aiding decisions with multiple criteria: essays in honor of Bernard Roy. Springer, US, pp 89–117

    Chapter  Google Scholar 

  38. Govindan K, Jepsen MB (2016) ELECTRE: acomprehensive literature review on methodologies and applications. Eur J Oper Res 250:1–29. https://doi.org/10.1016/j.ejor.2015.07.019

    Article  Google Scholar 

  39. Greene R, Luther JE, Devillers R, Eddy B (2010) An approach to GIS-based multiple criteria decision analysis that integrates exploration and evaluation phases: Case study in a forest-dominated landscape. For Ecol Manag 260:2102–2114. https://doi.org/10.1016/j.foreco.2010.08.052

    Article  Google Scholar 

  40. Henao F, Franco LA (2016) Unpacking multimethodology: impacts of a community development intervention. Eur J Oper Res 253:681–696. https://doi.org/10.1016/j.ejor.2016.02.044

    Article  Google Scholar 

  41. Hess P (2009) Avenues or arterials: the struggle to change street building practices in Toronto, Canada. J Urban Des 14:1–28. https://doi.org/10.1080/13574800802451049

    Article  Google Scholar 

  42. Hovmand PS, Andersen DF, Rouwette E et al (2012) Group model-building ‘Scripts’ as a collaborative planning tool: scripts as a collaborative planning tool. Syst Res Behav Sci 29:179–193. https://doi.org/10.1002/sres.2105

    Article  Google Scholar 

  43. Jones NA, Perez P, Measham TG et al (2009) Evaluating participatory modeling: developing a framework for cross-case analysis. Environ Manage 44:1180–1195. https://doi.org/10.1007/s00267-009-9391-8

    Article  Google Scholar 

  44. Keeney RL (1996) Value-focused thinking: identifying decision opportunities and creating alternatives. Eur J Oper Res 92:537–549. https://doi.org/10.1016/0377-2217(96)00004-5

    Article  Google Scholar 

  45. Keren G, de Bruin WB (2005) On the Assessment of Decision Quality: Considerations Regarding Utility, Conflict and Accountability. In: Hardman D, Macchi L (eds) Thinking: Psychological Perspectives on Reasoning, Judgment and Decision Making. John Wiley & Sons, Ltd, Chichester, UK, pp 347–363

  46. Kunc M, Malpass J, White L (eds) (2016) Behavioral operational research: theory, methodology and practice. Palgrave Macmillan, London

    Google Scholar 

  47. Landry M, Banville C, Oral M (1996) Model legitimisation in operational research. Eur J Oper Res 92:443–457. https://doi.org/10.1016/0377-2217(96)00003-3

    Article  Google Scholar 

  48. Leleur S (2017) Green Decision Making: How systemic Planning can support Strategic Decision Making for Sustainable Transport Development. Technical University of Denmark, Kongens Lingsby, Denmark

    Google Scholar 

  49. Lienert J, Koller M, Konrad J et al (2011) Multiple-criteria decision analysis reveals high stakeholder preference to remove pharmaceuticals from hospital wastewater. Environ Sci Technol 45:3848–3857. https://doi.org/10.1021/es1031294

    Article  Google Scholar 

  50. Malczewski J (2006) GIS-based multicriteria decision analysis: a survey of the literature. Int J Geogr Inf Sci 20:703–726. https://doi.org/10.1080/13658810600661508

    Article  Google Scholar 

  51. Marleau Donais F, Abi-Zeid I, Lavoie R (2017a) Building a shared model for multi-criteria group decision making. In: Schoop M, Kilgour M (eds) Group decision and negotiation. A Socio-Technical Perspective. Springer, Cham, pp 175–186

  52. Marleau Donais F, Abi-Zeid I, Lavoie R (2017b) A loose-coupling integration of the MACBETH approach in ArcGIS. In: Linden I, Mareschal B, Liu S et al (eds) Proceedings of the 2017 international conference on decision support system technology. EWG-DSS, Namur, Belgique, pp 125–131

  53. Marleau Donais F, Abi-Zeid I, Waygood EOD, Lavoie R (2019) Assessing and ranking the potential of a street to be redesigned as a complete street: a multi-criteria decision aiding approach. Transp Res Part Policy Pract 124:1–19. https://doi.org/10.1016/j.tra.2019.02.006

  54. Marttunen M, Hämäläinen RP (2008) The decision analysis interview approach in the collaborative management of a large regulated water course. Environ Manage 42:1026–1042. https://doi.org/10.1007/s00267-008-9200-9

    Article  Google Scholar 

  55. Marttunen M, Lienert J, Belton V (2017) Structuring problems for Multi-Criteria Decision Analysis in practice: a literature review of method combinations. Eur J Oper Res. https://doi.org/10.1016/j.ejor.2017.04.041

    Article  Google Scholar 

  56. Matheson JE (2013) Decision analysis in practice. In: Gass SI, Fu MC (eds) Encyclopedia of operations research and management science, 3rd edn. Springer, Boston

  57. McCann B (2013) Completing our streets: the transition to safe and inclusive transportation networks. Island Press, Washington

    Book  Google Scholar 

  58. McCardle-Keurentjes M, Rouwette EAJA (2018) Asking questions: a Sine Qua non of facilitation in decision support? Group Decis Negot 27:757–788. https://doi.org/10.1007/s10726-018-9573-y

    Article  Google Scholar 

  59. McCartt AT, Rohrbaugh J (1995) Managerial openness to change and the introduction of GDSS: explaining initial success and failure in decision conferencing. Organ Sci 6:569–584. https://doi.org/10.1287/orsc.6.5.569

    Article  Google Scholar 

  60. Midgley G, Cavana RY, Brocklesby J et al (2013) Towards a new framework for evaluating systemic problem structuring methods. Eur J Oper Res 229:143–154. https://doi.org/10.1016/j.ejor.2013.01.047

    Article  Google Scholar 

  61. Mingers J, Brocklesby J (1997) Multimethodology: Towards a framework for mixing methodologies. Omega 25:489–509. https://doi.org/10.1016/S0305-0483(97)00018-2

    Article  Google Scholar 

  62. Montibeller G (2005) From (and To) a New Generation of Multi-Criteria Decision Analysts: An Introduction to the Field and a Personal View on its Future. In: Keynote Papers of YOR 14 Conference. Birmingham, p 14

  63. Munda G (2008) Social multi-criteria evaluation for a sustainable economy. Springer, New York

    Book  Google Scholar 

  64. Mustajoki J, Hämäläinen RP, Marttunen M (2004) Participatory multicriteria decision analysis with Web-HIPRE: a case of lake regulation policy. Environ Model Softw 19:537–547. https://doi.org/10.1016/j.envsoft.2003.07.002

    Article  Google Scholar 

  65. Paillé P, Mucchielli A (2016a) L’analyse qualitative en sciences humaines et sociales, 4e edn. Armand Colin, Paris, France

  66. Paillé P, Mucchielli A (2016b) L’analyse thématique. In: L’analyse qualitative en sciences humaines et sociales, 4e edn. Armand Colin, Paris, France, pp 235–317

  67. Papamichail KN, Alves G, French S et al (2007) Facilitation practices in decision workshops. J Oper Res Soc 58:614–632. https://doi.org/10.1057/palgrave.jors.2602373

    Article  Google Scholar 

  68. Paschetta E, Tsoukiàs A (2000) A real-world MCDA application: evaluating software. J Multi-Criteria Decis Anal 9:205–226. https://doi.org/10.1002/1099-1360(200009)9:5%3c205::AID-MCDA277%3e3.0.CO;2-O

    Article  Google Scholar 

  69. Phillips LD (2007) Decision conferencing. In: von Winterfeldt D (ed) Edwards W, Miles RFJr. Advances in Decision Analysis. Cambridge University Press, Cambridge, pp 375–399

    Google Scholar 

  70. Phillips LD (2011) Group dynamics processes for improved decision making. In: Wiley Encyclopedia of Operations Research and Management Science. John Wiley & Sons, Inc., Hoboken, NJ, USA

  71. Phillips LD, Phillips MC (1993) Facilitated work groups: theory and practice. J Oper Res Soc 44:533–549. https://doi.org/10.1038/sj/jors/0440601

    Article  Google Scholar 

  72. QSR International (2019) N-Vivo. Version 12. QSR International. URL https://www.qsrinternational.com/nvivo/nvivo-products/nvivo-12-plus

  73. Quinet E, Meunier D (2012) CBA § Darwin: the case of transport infrastructure in France. Proc Soc Behav Sci 48:3051–3064. https://doi.org/10.1016/j.sbspro.2012.06.1272

    Article  Google Scholar 

  74. Rouwette EAJA, Vennix JAM, Felling AJA (2009) On evaluating the performance of problem structuring methods: an attempt at formulating a conceptual model. Group Decis Negot 18:567–587. https://doi.org/10.1007/s10726-007-9100-z

    Article  Google Scholar 

  75. Rouwette EAJA, Vennix JAM, van Mullekom T (2002) Group model building effectiveness: a review of assessment studies. Syst Dyn Rev 18:5–45. https://doi.org/10.1002/sdr.229

    Article  Google Scholar 

  76. Rowe G, Frewer LJ (2000) Public participation methods: a framework for evaluation. Sci Technol Hum Values 25:3–29. https://doi.org/10.1177/016224390002500101

    Article  Google Scholar 

  77. Rowe G, Frewer LJ (2004) Evaluating public-participation exercises: a research agenda Sci Technol Hum Values 29:512–556. https://doi.org/10.1177/0162243903259197

    Article  Google Scholar 

  78. Salo A, Hämäläinen RP (2010) Multicriteria decision analysis in group decision processes. In: Handbook of group decision and negotiation. Springer, pp 269–283

  79. Schuwirth N, Reichert P, Lienert J (2012) Methodological aspects of multi-criteria decision analysis for policy support: a case study on pharmaceutical removal from hospital wastewater. Eur J Oper Res 220:472–483. https://doi.org/10.1016/j.ejor.2012.01.055

    Article  Google Scholar 

  80. Scott RJ, Cavana RY, Cameron D (2013) Evaluating immediate and long-term impacts of qualitative group model building workshops on participants’ mental models: Group Model Building and Participants’ Mental Models. Syst Dyn Rev 29:216–236. https://doi.org/10.1002/sdr.1505

    Article  Google Scholar 

  81. Simon HA (1996) The sciences of the artificial, 3rd edn. MIT Press, Cambridge

    Google Scholar 

  82. Smart Growth America, Coalition NCS (2018) The Best Complete Streets Initiatives of 2017. Smart Growth America, Washington, D.C.

    Google Scholar 

  83. Statistique Canada (2017) Québec, V [Subdivision de recensement], Québec et Québec, TÉ [Division de recensement], Québec (tableau). Profil du recensement. http://www12.statcan.gc.ca/census-recensement/2016/dp-pd/prof/details/page.cfm?Lang=F&Geo1=CSD&Code1=2423027&Geo2=CD&Code2=2423&Data=Count&SearchText=quebec&SearchType=Begins&SearchPR=01&B1=All&TABID=1. Accessed 17 May 2017

  84. Tavella E, Papadopoulos T (2015a) Expert and novice facilitated modelling: a case of a Viable System Model workshop in a local food network. J Oper Res Soc 66:247–264. https://doi.org/10.1057/jors.2013.187

    Article  Google Scholar 

  85. Tavella E, Papadopoulos T (2015b) Novice facilitators and the use of scripts for managing facilitated modelling workshops. J Oper Res Soc 66:1967–1988. https://doi.org/10.1057/jors.2015.7

    Article  Google Scholar 

  86. Venable J, Pries-Heje J, Baskerville R (2016) FEDS: a framework for evaluation in design science research. Eur J Inf Syst 25:77–89. https://doi.org/10.1057/ejis.2014.36

    Article  Google Scholar 

  87. Ville de Québec (2017) Rues conviviales: Une approche intégrée de conception de rues pour améliorer la qualité de vie urbaine

  88. White L (2006) Evaluating problem-structuring methods: developing an approach to show the value and effectiveness of PSMs. J Oper Res Soc 57:842–855. https://doi.org/10.1057/palgrave.jors.2602149

    Article  Google Scholar 

  89. Yates JF, Veinott ES, Patalano AL (2003) Hard decisions, bad decisions: on decision quality and decision aiding. In: Schneider SL, Shanteau J (eds) Emerging perspectives on judgment and decision research. Cambridge University Press, Cambridge, pp 13–63

    Chapter  Google Scholar 

Download references

Acknowledgements

The authors wish to thank all the interviewees from Ville de Québec for their input and valuable time.

Funding

Financial support was provided for this project through scholarships by the Social Sciences and Humanities Research Council of Canada (Grant 752–2017-2758), the Fonds de Recherche du Québec—Nature et Technologies (Grant 207792), the Institute Hydro-Québec in Environment, Development and Society and the Transportation Association of Canada.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Francis Marleau Donais.

Ethics declarations

Ethical approval

Ethics and research committee of Laval University (no 2018-275/ 17-12-2018).

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Marleau Donais, F., Abi-Zeid, I., Waygood, E.O.D. et al. A Framework for Post-Project Evaluation of Multicriteria Decision Aiding Processes from the Stakeholders’ Perspective: Design and Application. Group Decis Negot 30, 1161–1191 (2021). https://doi.org/10.1007/s10726-021-09753-y

Download citation

Keywords

  • Complete Street
  • Multicriteria analysis
  • Case study
  • Facilitation
  • Ex-post analysis
  • Lessons learned