A comparison of preference elicitation methods for multi-criteria design decisions about resilient and sustainable buildings

  • Sahar Mirzaee
  • David FannonEmail author
  • Matthias Ruth


Designing sustainable and resilient buildings is a complex process involving multiple actors and numerous, often conflicting, criteria to describe the social, environmental, and economic consequences of design decisions. Multi-Criteria Decision Making (MCDM) is a well-established branch of systematic decision science that may be used to narrow the choice set and balance decision criteria and objectives. MCDM methods rely on eliciting decision makers’ (DM) criteria preferences through surveys—a critical, but often time-consuming and expensive process. This study compares two survey methods—Simple Multi-Attribute Rating Technique and Potentially All Pairwise RanKings of all possible Alternatives—used to capture DM’s preferences for subjective criteria relevant to the design of resilient and sustainable buildings. Significant differences were found between the results of the two methods, indicating that building design criteria are evaluated differently in different contexts. However, most DMs reported that both surveys accurately captured their preferences, complicating conclusions about DM’s actual preferences through single surveys. The design professionals surveyed also report lower cognitive load for the direct weighting method. Although this study does not report a preferred survey method, the findings suggest that great care must be taken when eliciting DM preferences, and when attempting to apply MCDM to building design decisions.


Multi-Criteria Decision Making (MCDM) SMART PAPRIKA Decision-making cognitive load Criteria preference techniques 



This research is based on work supported by the National Science Foundation grant number CMMI #1455450. The authors wish to thank Dr. Mehrdad Sasani, Dr. Matthew Eckelman, Matthew Joyner, Robert Phillips, and Luke Troup for their iterative and constructive feedback.


  1. Alexander D (2006) Globalization of disaster: trends, problems and dilemmas. J Int Aff 59(2):1–22Google Scholar
  2. Alexander DE (2013) Resilience and disaster risk reduction: an etymological journey. Nat Hazard Earth Syst Sci 13(11):2707–2716Google Scholar
  3. Aloysius JA, Davis FD, Wilson DD, Taylor A, Ross, Kottemann JE (2006) User acceptance of multi-criteria decision support systems: the impact of preference elicitation techniques. Eur J Oper Res 169(1):273–285Google Scholar
  4. Antonenko PD, Niederhauser DS (2010) The influence of leads on cognitive load and learning in a hypertext environment. Comput Hum Behav 26(2):140–150Google Scholar
  5. Ayres P (2006) Using subjective measures to detect variations of intrinsic cognitive load within problems. Learn Instr 16(5):389–400Google Scholar
  6. Balcomb JD, Curtner A (2000) Multi-criteria decision-making process for buildings. In: Energy conversion engineering conference and exhibit, 2000. (IECEC) 35th Intersociety (Vol 1, pp 528–535). IEEEGoogle Scholar
  7. Brans J-P (1986) L’élaboration d’instruments d’aide à la décision. Nadeau, Raymond et Maurice Landry, 183–213Google Scholar
  8. Buchanan JT, Daellenbach HG (1987) A comparative evaluation of interactive solution methods for multiple objective decision models. Eur J Oper Res 29(3):353–359Google Scholar
  9. Burroughs S (2017) Development of a tool for assessing commercial building resilience. Proc Eng 180:1034–1043Google Scholar
  10. Carlsson C, Korhonen P (1986) A parametric approach to fuzzy linear programming. Fuzzy Set Syst 20(1):17–30Google Scholar
  11. Cegan JC, Filion AM, Keisler JM, Linkov I (2017) Trends and applications of multi-criteria decision analysis in environmental sciences: literature review. Environ Syst Decis 37(2):123–133Google Scholar
  12. Chang C-C, Yang F-Y (2010) Exploring the cognitive loads of high-school students as they learn concepts in web-based environments. Comput Educ 55(2):673–680Google Scholar
  13. Chen F, Zhou J, Wang Y, Yu K, Arshad SZ, Khawaji A, Conway D (2016) Robust multimodal cognitive load measurement. Berlin: Springer, AhmadGoogle Scholar
  14. De Jong J, Den Hartog D (2010) Measuring innovative work behaviour. Creat Innov Manag 19(1):23–36Google Scholar
  15. Deierlein G, Krawinkler H, Ma X, Eatherton M, Hajjar J, Takeuchi T et al (2011) Earthquake resilient steel braced frames with controlled rocking and energy dissipating fuses. Steel Constr 4(3):171–175Google Scholar
  16. Edwards W (1977) How to use multiattribute utility measurement for social decisionmaking. IEEE Trans Syst Man Cybern 7(5):326–340Google Scholar
  17. Edwards W, Barron FH (1994) SMARTS and SMARTER: improved simple methods for multiattribute utility measurement. Organ Behav Hum Decis Process 60(3):306–325Google Scholar
  18. Elkington J (2013) Enter the triple bottom line. In: The triple bottom line. Routledge, London, pp 23–38Google Scholar
  19. Flager F, Welle B, Bansal P, Soremekun G, Haymaker J (2009) Multidisciplinary process integration and design optimization of a classroom building. J Inform Technol Constr 14(38):595–612Google Scholar
  20. Formisano A, Mazzolani FM (2015) On the selection by MCDM methods of the optimal system for seismic retrofitting and vertical addition of existing buildings. Comput Struct 159:1–13Google Scholar
  21. Gandibleux X (ed) (2006) Multiple criteria optimization: state of the art annotated bibliographic surveys, vol 52. Springer, BerlinGoogle Scholar
  22. Gimino AE (2002) Factors that influence students’ investment of mental effort in academic tasks: a validation and exploratory studyGoogle Scholar
  23. Glavič P, Lukman R (2007) Review of sustainability terms and their definitions. J Clean Prod 15(18):1875–1885Google Scholar
  24. Gopher D, Braune R (1984) On the psychophysics of workload: why bother with subjective measures? Hum Factors 26(5):519–532Google Scholar
  25. Gunderson LH (2000) Ecological resilience: in theory and application. Annu Rev Ecol Syst 31(1):425–439Google Scholar
  26. Hansen P, Ombler F (2008) Decision support system and method: Google PatentsGoogle Scholar
  27. Harkouss F, Biwole PH, Fardoun F (2018) Passive designs and renewable energy systems optimization of a net zero energy building in Embrun/France. Paper presented at the IOP conference series: earth and environmental scienceGoogle Scholar
  28. Holling CS (1996) Engineering resilience versus ecological resilience. Eng Ecol Constraints 31(1996):32Google Scholar
  29. Hsieh T-Y, Lu S-T, Tzeng G-H (2004) Fuzzy MCDM approach for planning and design tenders selection in public office buildings. Int J Project Manag 22(7):573–584Google Scholar
  30. Invidiata A, Lavagna M, Ghisi E (2018) Selecting design strategies using multi-criteria decision making to improve the sustainability of buildings. Build Environ 139:58–68Google Scholar
  31. Jin Z, Eikevik TM, Nekså P, Hafner A (2016) Investigation on CO2 hybrid ground-coupled heat pumping system under warm climate. Int J Refrig 62:145–152Google Scholar
  32. Kalyuga S, Sweller J (2005) Rapid dynamic assessment of expertise to improve the efficiency of adaptive e-learning. Educ Technol Res Dev 53(3):83–93Google Scholar
  33. Kamal MM (2012) Shared services: lessons from private sector for public sector domain. J Enterp Inform Manag 25(5):431–440Google Scholar
  34. Kolokotsa D, Pouliezos A, Stavrakakis G, Lazos C (2009) Predictive control techniques for energy and indoor environmental quality management in buildings. Build Environ 44(9):1850–1863Google Scholar
  35. Laboy M, Fannon D (2016) Resilience theory and praxis: a critical framework for architecture. Enquiry 13(1):39–52Google Scholar
  36. Larichev OI, Nikiforov AD (1987) Analytical survey of procedures for solving multicriteria mathematical programming problems (MMPP) toward interactive and intelligent decision support systems. Springer, Berlin. pp 95–104Google Scholar
  37. Lee CD, Goldman SR, Levine S, Magliano J (2016) Epistemic cognition in literary reasoning. Handbook of epistemic cognition, 165–183Google Scholar
  38. Lin FR, Yaffe K, Xia J, Xue QL, Harris TB, Purchase-Helzner E, Health ABC Study Group F et al (2013) Hearing loss and cognitive decline in older adults. JAMA Intern Med 173(4), 293–299Google Scholar
  39. Linkov I, Bridges T, Creutzig F, Decker J, Fox-Lent C, Kröger W, Nathwani J et al (2014) Changing the resilience paradigm. Nat Clim Chang 4(6):407Google Scholar
  40. Linstone HA, Turoff M (2011) Delphi: a brief look backward and forward. Technol Forecast Soc Chang 78(9):1712–1719Google Scholar
  41. Lützkendorf T, Lorenz DP (2006) Using an integrated performance approach in building assessment tools. Build Res Inform 34(4):334–356Google Scholar
  42. Magelssen M, Supphellen M, Nortvedt P, Materstvedt LJ (2016) Attitudes towards assisted dying are influenced by question wording and order: a survey experiment. BMC Med Ethics 17(1):24Google Scholar
  43. Maggino F, Ruviglioni E (2009) Obtaining weights: from objective to subjective approaches in view of more participative methods in the construction of composite indicators. In: Proceedings NTTS: new techniques and technologies for statistics, pp 37–46Google Scholar
  44. Majumder M (2015) Multi criteria decision making. In: Impact of urbanization on water shortage in face of climatic aberrations. Springer, Berlin, pp 35–47Google Scholar
  45. Martino JP, Harold A, Turoff M (1976) The Delphi method: techniques and applications: Linstone, Harold A., and Murray Turoff. Advanced Book Program. Addison-Wesley, Advanced Book Program, ReadingGoogle Scholar
  46. Mirzaee S, Ruth M, Fannon D (2018) Reconciling diverse perspective of decision makers on resilience and sustainability. In: Advanced introduction to ecological economics. Edward Elgar, CheltenhamGoogle Scholar
  47. Moser CA, Kalton G (2017) Survey methods in social investigation. Routledge, LondonGoogle Scholar
  48. Mota P, Campos AR, Neves-Silva R (2013) First look at MCDM: choosing a decision method. Adv Smart Syst Res 3(2):25–30Google Scholar
  49. Narasimhan R, Vickery SK (1988) An experimental evaluation of articulation of preferences in multiple criterion decision-making (MCDM) methods. Decis Sci 19(4):880–888Google Scholar
  50. Noseworthy T, De Coster C, Naden R (2009) Priority-setting tools for improving access to medical specialists. Poster presentation. Paper presented at the 6th health technology assessment international annual meeting, SingaporeGoogle Scholar
  51. Önüt S, Kara SS, Işik E (2009) Long term supplier selection using a combined fuzzy MCDM approach: a case study for a telecommunication company. Expert Syst Appl 36(2):3887–3895Google Scholar
  52. Paas FGWC, Van Merriënboer JJG (1994) Variability of worked examples and transfer of geometrical problemsolving skills: a cognitive-load approach. J Educ Psychol 86(1):122Google Scholar
  53. Paas F, Renkl A, Sweller J (2003) Cognitive load theory and instructional design: recent developments. Educ Psychol 38(1):1–4Google Scholar
  54. Pater LR, Cristea SL (2016) Systemic definitions of sustainability, durability and longevity. Proc Soc Behav Sci 221:362–371Google Scholar
  55. Phillips R, Troup L, Fannon D, Eckelman MJ (2017) Do resilient and sustainable design strategies conflict in commercial buildings? A critical analysis of existing resilient building frameworks and their sustainability implications. Energy Build 146:295–311Google Scholar
  56. Pill J (1971) The Delphi method: substance, context, a critique and an annotated bibliography. Socioecon Plan Sci 5(1):57–71Google Scholar
  57. Polatidis H, Haralambopoulos D (2002) Normative aspects of renewable energy planning; a participatory multi-criteria approach. The International Society for Ecological Economics. Ecological economics working paper seriesGoogle Scholar
  58. Pomerol J-C, Barba-Romero S (2012) Multicriterion decision in management: principles and practice, vol 25. Springer, BerlinGoogle Scholar
  59. Powell C (2003) The Delphi technique: myths and realities. J Adv Nurs 41(4):376–382Google Scholar
  60. Roy B, Bouyssou D (1986) Comparison of two decision-aid models applied to a nuclear power plant siting example. Eur J Oper Res 25(2):200–215Google Scholar
  61. Ruzgys A, Volvačiovas R, Ignatavičius Č, Turskis Z (2014) Integrated evaluation of external wall insulation in residential buildings using SWARA-TODIM MCDM method. J Civ Eng Manag 20(1):103–110Google Scholar
  62. Saaty TL (1980) The analytic hierarchy process: planning, priority setting, resources allocation. McGraw, New York. p 281Google Scholar
  63. San Cristóbal JR (2011) Multi-criteria decision-making in the selection of a renewable energy project in spain: the Vikor method. Renew Energy 36(2):498–502Google Scholar
  64. Scharfenberg F-J, Bogner FX (2013) Teaching gene technology in an outreach lab: students’ assigned cognitive load clusters and the clusters’ relationships to learner characteristics, laboratory variables, and cognitive achievement. Res Sci Edu 43(1):141–161Google Scholar
  65. Scheffer M, Hosper SH, Meijer ML, Moss B, Jeppesen E (1993) Alternative equilibria in shallow lakes. Trends Ecol Evol 8(8):275–279Google Scholar
  66. Schön DA (1983) The reflective practicioner: how professionals think in action, vol 1. Basic books, New YorkGoogle Scholar
  67. Triantaphyllou E (2000) Multi-criteria decision making methods. In: Multi-criteria decision making methods: a comparative study. Springer, Boston, pp 5–21Google Scholar
  68. Tzeng GH, Chen TY, Wang JC (1998) A weight-assessing method with habitual domains. Eur J Oper Res 110(2):342–367Google Scholar
  69. Walgrave S, Wouters R, Ketelaars P (2016) Response problems in the protest survey design: evidence from fifty-one protest events in seven countries. Mobilization 21(1):83–104Google Scholar
  70. Wallenius J (1975) Comparative evaluation of some interactive approaches to multicriterion optimization. Manag Sci 21(12):1387–1396Google Scholar
  71. Wang T-C, Lee H-D (2009) Developing a fuzzy TOPSIS approach based on subjective weights and objective weights. Expert Syst Appl 36(5):8980–8985Google Scholar
  72. Woudenberg F (1991) An evaluation of Delphi. Technol Forecast Soc Chang 40(2):131–150Google Scholar
  73. Yang AL, Huang GH, Qin XS, Fan YR (2012) Evaluation of remedial options for a benzene-contaminated site through a simulation-based fuzzy-MCDA approach. J Hazard Mater 213:421–433Google Scholar
  74. Yoon K (1987) A reconciliation among discrete compromise solutions. J Oper Res Soc 38(3):277–286Google Scholar
  75. Zimmermann H-J (1990) Decision making in ill-structured environments and with multiple criteria. In: Readings in multiple criteria decision aid. Springer, Berlin, pp 119–151Google Scholar
  76. Zuo J, Zhao Z-Y (2014) Green building research-current status and future agenda: a review. Renew Sustain Energy Rev 30:271–281Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.School of Public Policy and Urban AffairsNortheastern UniversityBostonUSA
  2. 2.School of ArchitectureNortheastern UniversityBostonUSA
  3. 3.Department of Civil and Environmental EngineeringNortheastern UniversityBostonUSA
  4. 4.Department of EconomicsUniversity of AlbertaEdmontonCanada

Personalised recommendations