Integrating Managerial Preferences into the Qualitative Multi-Criteria Evaluation of Team Members

  • Ann BarcombEmail author
  • Nicolas Jullien
  • Patrick Meyer
  • Alexandru-Liviu Olteanu
Part of the International Series in Operations Research & Management Science book series (ISOR, volume 274)


Managers can find it challenging to assess team members consistently and fairly. The ideal composition of qualities possessed by good team members depends on the organization, the team, and the manager. To enable managers to elucidate the qualities they require, we make use of an innovative methodology. This methodology is based on a multi-criteria decision aiding process, starting with the identification and definition of the dimensions that will be used to evaluate team members, then the inference of the manager’s preferences through a multi-step protocol combining multiple types of preference models, and finally extracting a set of rules that can support the manager in his/her tasks. We illustrate this methodology in the case of free/libre/open-source software development teams, where we were able to elicit the characteristics of a good, acceptable, or bad contributor based on multiple managers’ perspectives. We additionally provide an example on how to reproduce this experiment using the MCDA package for the R statistical environment.



This work was supported, in part, by Science Foundation Ireland grants 10/CE/I1855 and 13/RC/2094 to Lero—the Irish Software Research Centre (

We would like to thank the six FLOSS community managers who participated in this research and Pr. Mathieu Simonnet, who provided input on the modeling of psychological traits.


  1. Allen, M. S., Greenlees, I., & Jones, M. (2013). Personality in sport: A comprehensive review. International Review of Sport and Exercise Psychology, 6(1), 184–208.CrossRefGoogle Scholar
  2. Barcomb, A., Grottke, M., Stauffert, J.-P., Riehle, D., & Jahn, S. (2015). How developers acquire FLOSS skills. In Proceedings of the 11th international conference on open source systems (OSS 2015). Berlin: Springer.CrossRefGoogle Scholar
  3. Bassellier, G., & Benbasat, I. (2004). Business competence of information technology professionals: Conceptual development and influence on IT-business partnerships. MIS Quarterly, 28, 673–694.CrossRefGoogle Scholar
  4. Beecham, S. (2014). Motivating software engineers working in virtual teams across the globe. In Software project management in a changing world (pp. 247–273). Berlin: Springer.Google Scholar
  5. Beecham, S., & Noll, J. (2015). What motivates software engineers working in global software development? In P. Abrahamsson, L. Corral, M. Oivo, & B. Russo (Eds.), International conference on product-focused software process improvement (pp. 193–209). Berlin: Springer.CrossRefGoogle Scholar
  6. Bigaret, S., Hodgett, R., Meyer, P., Mironova, T., & Olteanu, A.-L. (2017). Supporting the multi-criteria decision aiding process: R and the MCDA package. EURO Journal on Decision Processes, 5(1–4), 169–194.CrossRefGoogle Scholar
  7. Bouyssou, D., & Marchant, T. (2007). An axiomatic approach to noncompensatory sorting methods in MCDM, I: The case of two categories. European Journal of Operational Research, 178(1), 217–245.CrossRefGoogle Scholar
  8. Bouyssou, D., & Marchant, T. (2007). An axiomatic approach to noncompensatory sorting methods in MCDM, II: More than two categories. European Journal of Operational Research, 178(1), 246–276.CrossRefGoogle Scholar
  9. Bouyssou, D., Marchant, T., Pirlot, M., Tsoukiàs, A., & Vincke, P. (2006). Evaluation and decision models with multiple criteria: Stepping stones for the analyst (vol. 86, 1st ed.). Boston: International Series in Operations Research and Management Science.Google Scholar
  10. Capraro, M., & Riehle, D. (2017). Inner source definition, benefits, and challenges. ACM Computing Surveys (CSUR), 49(4), 67.Google Scholar
  11. Carillo, K. D. A., & Marsan, J. (2016). “The dose makes the poison”—exploring the toxicity phenomenon in online communities. In International conference on information systems ICIS.Google Scholar
  12. Castilla, E. J., & Benard, S. (2010). The paradox of meritocracy in organizations. Administrative Science Quarterly, 55(4), 543–676.CrossRefGoogle Scholar
  13. Chen, S.-J., & Lin, L. (2004) Modeling team member characteristics for the formation of a multifunctional team in concurrent engineering. IEEE Transactions on Engineering Management, 51(2), 111–124.CrossRefGoogle Scholar
  14. David, P., Ghosh, R. A., Glott, R., González-Barahona, J. M. Federico Heinz, & Shapiro, J. (2007). Free/libre and open source software: Worldwide impact study. FLOSS World D31: Track 1 International Report.Google Scholar
  15. Deshpande, S., & Richardson, I. (2009) Management at the outsourcing destination-global software development in India. In Fourth IEEE International Conference on Global Software Engineering, 2009. ICGSE 2009 (pp. 217–225). Piscataway: IEEE.Google Scholar
  16. Dias, L. C., & Clímaco, J. N. (1999). On computing ELECTRE’s credibility indices under partial information. Journal of Multi-Criteria Decision Analysis, 8(2), 74–92.CrossRefGoogle Scholar
  17. Dias, L. C., & Clímaco, J. N. (2000). ELECTRE TRI for groups with imprecise information on parameter values. Group Decision and Negotiation, 9(5), 355–377.CrossRefGoogle Scholar
  18. Dias, L. C., Mousseau, V., Figueira, J., & Clímaco, J. N. (2002). An aggregation/disaggregation approach to obtain robust conclusions with ELECTRE TRI. European Journal of Operational Research, 138(2), 332–348.CrossRefGoogle Scholar
  19. Driskell, J. E., Goodwin, G. F., Salas, E., & O’Shea, P. G. (2006). What makes a good team player? Personality and team effectiveness. Group Dynamics: Theory, Research, and Practice, 10(4), 249.CrossRefGoogle Scholar
  20. Fitzgerald, B. (2006). The transformation of open source software. MIS Quarterly, 30(3), 587–598.CrossRefGoogle Scholar
  21. Gilson, L. L., Maynard, M. T., Young, N. C. J., Vartiainen, M., & Hakonen, M. (2015). Virtual teams research: 10 years, 10 themes, and 10 opportunities. Journal of Management, 41(5), 1313–1337.CrossRefGoogle Scholar
  22. Gosling, S. D., Rentfrow, P. J., & Swann, W. B. Jr. (2003). A very brief measure of the big-five personality domains. Journal of Research in Personality, 37(6), 504–528.CrossRefGoogle Scholar
  23. Grant, R. M. (1996). Toward a knowledge-based theory of the firm. Strategic Management Journal, 17, 109–122.CrossRefGoogle Scholar
  24. Guimera, R., Uzzi, B., Spiro, J., & Amaral, L. A. N. (2005). Team assembly mechanisms determine collaboration network structure and team performance. Science, 308(5722), 697–702.CrossRefGoogle Scholar
  25. Hess, C., & Ostrom, E. (2006). Introduction: An overview of the knowledge commons. In C. Hess & E. Ostrom (Eds.), Understanding knowledge as a commons. From theory to practice (pp. 3–26). Cambridge: MIT Press.CrossRefGoogle Scholar
  26. IEEE Spectrum. (2016). The 2016 top programming languages. Available from: Google Scholar
  27. Ihaka, R., & Gentleman, R. (1996). R: A language for data analysis and graphics. Journal of Computational and Graphical Statistics, 5(3), 299–314.Google Scholar
  28. Introduction to R. (2018) . Accessed March 15 2018.
  29. Jarvenpaa, S. L., Knoll, K., & Leidner, D. E. (1998). Is anybody out there? Antecedents of trust in global virtual teams. Journal of Management Information Systems, 14(4), 29–64.CrossRefGoogle Scholar
  30. Katsikopoulos, K. V., Durbach, I. N., & Stewart, T. J. (2017). When should we use simple decision models? A synthesis of various research strands. Omega, 81, 17–25.CrossRefGoogle Scholar
  31. Kayworth, T. R., & Leidner, D. E. (2002). Leadership effectiveness in global virtual teams. Journal of Management Information Systems, 18(3), 7–40.CrossRefGoogle Scholar
  32. Keeney, R. L., & Raiffa, H. (1976). Decisions with multiple objectives: Preferences and value tradeoffs. New York: Wiley.Google Scholar
  33. Kimmelmann, N. (2013). Career in open source? Relevant competencies for successful open source developers. IT–Information Technology, 55(5), 204–212.Google Scholar
  34. Kogut, B., & Zander, U. (1992). Knowledge of the firm, combinative capabilities, and the replication of technology. Organization Science, 3(3), 383–397.CrossRefGoogle Scholar
  35. Kudaravalli, S., Faraj, S., & Johnson, S. L. (2017). A configural approach to coordinating expertise in software development teams. MIS Quarterly, 41(1), 43–64.CrossRefGoogle Scholar
  36. Leroy, A., Mousseau, V.., & Pirlot, M. (2011). Learning the parameters of a multiple criteria sorting method. In R. Brafman, F. Roberts, & A. Tsoukiàs (Eds.), Algorithmic decision theory (vol. 6992, pp. 219–233). Berlin: Springer.CrossRefGoogle Scholar
  37. Locander, W.B., Albert Napier, H., & Scamell, R. W. (1979). A team approach to managing the development of a decision support system. MIS Quarterly, 3, 53–63.CrossRefGoogle Scholar
  38. Malinowski, J., Weitzel, T., & Keim, T. (2008). Decision support for team staffing: An automated relational recommendation approach. Decision Support Systems, 45(3), 429–447.CrossRefGoogle Scholar
  39. Mc Crae, R.R., & Costa, P. T. (1989). Reinterpreting the Myers-Briggs type indicator from the perspective of the five-factor model of personality. Journal of Personality, 57(1), 17–40.CrossRefGoogle Scholar
  40. Meyer, P., & Olteanu, A.-L. (2017). Integrating large positive and negative performance differences into multicriteria majority-rule sorting models. Computers & Operations Research, 81, 216–230.CrossRefGoogle Scholar
  41. Miller, G. A. (1956). The magical number seven, plus or minus two: Some limits on our capacity for processing information. The Psychological Review, 63(2), 81–97.CrossRefGoogle Scholar
  42. Moreno-León, J., Robles, G., & Román-González, M. (2016). Examining the relationship between socialization and improved software development skills in the scratch code learning environment. Journal of Universal Computer Science, 22(12), 1533–1557.Google Scholar
  43. Mousseau, V., Dias, L. C., & Figueira, J. (2006). Dealing with inconsistent judgments in multiple criteria sorting models. 4OR, 4(3), 145–158.CrossRefGoogle Scholar
  44. Mousseau, V., Dias, L. C., Figueira, J., Gomes, C., & Clímaco, J. N. (2003) Resolving inconsistencies among constraints on the parameters of an MCDA model. European Journal of Operational Research, 147(1), 72–93.CrossRefGoogle Scholar
  45. Mousseau, V., Figueira, J.., & Naux, J. P. (2001). Using assignment examples to infer weights for ELECTRE TRI method: Some experimental results. European Journal of Operational Research, 130(2), 263–275.CrossRefGoogle Scholar
  46. Mousseau, V., & Słowiński, R. (1998). Inferring an ELECTRE TRI model from assignment examples. Journal of Global Optimization, 12(2), 157–174.CrossRefGoogle Scholar
  47. Myers, I. B., McCaulley, M. H., & Most, R. (1985). Manual: A guide to the development and use of the Myers-Briggs Type Indicator (vol. 1985). Palo Alto: Consulting Psychologists Press.Google Scholar
  48. Myers, M. D. (1997). Qualitative research in information systems. Management Information Systems Quarterly, 21(2), 241–242.CrossRefGoogle Scholar
  49. Nafus, D. (2012). ‘Patches don’t have gender’: What is not open in open source software. New Media & Society, 14(4), 669–683.CrossRefGoogle Scholar
  50. Ngo The, A., & Mousseau, V. (2002). Using assignment examples to infer category limits for the ELECTRE TRI method. Journal of Multi-Criteria Decision Analysis, 11(1), 29–43.CrossRefGoogle Scholar
  51. Noll, J., Beecham, S., Richardson, I., & Canna, C. N. (2016). A global teaming model for global software development governance: A case study. In 2016 IEEE 11th International Conference on Global Software Engineering (ICGSE) (pp. 179–188). Piscataway: IEEE.CrossRefGoogle Scholar
  52. Olteanu, A.-L., & Meyer, P. (2014). Inferring the parameters of a majority rule sorting model with vetoes on large datasets. In DA2PL 2014: From Multicriteria Decision Aid to Preference Learning (pp. 87–94).Google Scholar
  53. Rammstedt, B., & John, O. P. (2007). Measuring personality in one minute or less: A 10-item short version of the big five inventory in English and German. Journal of Research in Personality, 41(1), 203–212 (2007)CrossRefGoogle Scholar
  54. Reagle, J. (2012). “Free as in sexist?” Free culture and the gender gap. First Monday, 18(1), 2012.Google Scholar
  55. Roy, B. (1991). The outranking approach and the foundations of ELECTRE methods. Theory and Decision, 31, 49–73.CrossRefGoogle Scholar
  56. Roy, B. (1996). Multicriteria methodology for decision aiding. Dordrecht: Kluwer Academic.CrossRefGoogle Scholar
  57. Ryan, A. M., Wiechmann, D., & Hemingway, M. (2003). Designing and implementing global staffing systems: Part II - best practices. Human Resource Management, 42(1), 85–94.CrossRefGoogle Scholar
  58. Sarker, S., Ahuja, M., Sarker, S., & Kirkeby, S. (2011). The role of communication and trust in global virtual teams: A social network perspective. Journal of Management Information Systems, 28(1), 273–310.CrossRefGoogle Scholar
  59. Scacchi, W. (2007). Free/open source software development: Recent research results and methods. Advances in Computers, 69, 243–295.CrossRefGoogle Scholar
  60. Simon, H. A. (1976). Administrative behavior; a study of decision-making processes in administrative organization-3. New York: The Free Press.Google Scholar
  61. Sobrie, O., Mousseau, V., & Pirlot, M. (2013). Learning a majority rule model from large sets of assignment examples. In International Conference on Algorithmic Decision Theory ADT. Lecture notes in computer science (vol. 8176, pp. 336–350). Berlin: Springer.CrossRefGoogle Scholar
  62. Stewart, G. L. (2006). A meta-analytic review of relationships between team design features and team performance. Journal of Management, 32(1), 29–55.CrossRefGoogle Scholar
  63. Stol, K.-J., Babar, M. A., Avgeriou, P., & Fitzgerald, B. (2011). A comparative study of challenges in integrating open source software and inner source software. Information and Software Technology, 53(12), 1319–1336.CrossRefGoogle Scholar
  64. Stol, K.-J., & Fitzgerald, B. (2014). Two’s company, three’s a crowd: A case study of crowdsourcing software development. In Proceedings of the 36th International Conference on Software Engineering, ICSE 2014 (pp. 187–198). New York: ACM.CrossRefGoogle Scholar
  65. Teece, D. J., Pisano, G., & Shuen, A. () Dynamic capabilities and strategic management. Strategic Management Journal, 18(7), 509–533.CrossRefGoogle Scholar
  66. Tsoukiàs, A. (2007). On the concept of decision aiding process: An operational perspective. Annals of Operations Research, 154(1), 3–27.CrossRefGoogle Scholar
  67. Turner, J. R., & Müller, R. (2003). On the nature of the project as a temporary organization. International Journal of Project Management, 21(1), 1–8.CrossRefGoogle Scholar
  68. Varni, G., Volpe, G., & Camurri, A. (2010). A system for real-time multimodal analysis of nonverbal affective social interaction in user-centric media. IEEE Transactions on Multimedia, 12(6), 576–590.CrossRefGoogle Scholar
  69. Venables, B., Smith, D., Gentleman, R., & Ihaka, R. (1998). Notes on R: A programming environment for data analysis and graphics. Auckland: University of Auckland.Google Scholar
  70. Wageman, R. (2001). How leaders foster self-managing team effectiveness: Design choices versus hands-on coaching. Organization Science, 12(5), 559–577.Google Scholar
  71. Wingreen, S. C., Blanton, J. E., Newton, S. K., & Domino, M. (2005). Assessing the IT training and development climate: An application of the Q-methodology. In Proceedings of the 2005 ACM SIGMIS CPR Conference on Computer Personnel Research (pp. 12–23). New York: ACM.CrossRefGoogle Scholar
  72. Wuchty, S., Jones, B., & Uzzi, B. (2007). The increasing dominance of teams in the production of knowledge. Science, 316, 1036–1039.CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Ann Barcomb
    • 1
    Email author
  • Nicolas Jullien
    • 2
  • Patrick Meyer
    • 3
  • Alexandru-Liviu Olteanu
    • 4
  1. 1.Lero - The Irish Software Engineering Research CentreUniversity of LimerickLimerickIreland
  2. 2.IMT Atlantique, LEGO-M@rsouinUniversity of Bretagne LoireBrestFrance
  3. 3.IMT Atlantique, Lab-STICCUniversity of Bretagne LoireBrestFrance
  4. 4.Lab-STICC, CNRSUniversité de Bretagne SudLorientFrance

Personalised recommendations