A Delphi-based expert judgment method applied to the validation of a mature Agile framework for Web development projects

  • C. J. Torrecilla-SalinasEmail author
  • O. De Troyer
  • M. J. Escalona
  • M. Mejías


The validation of any new methodological proposal demands several real-life implementations. However, organizations are reluctant to invest without the firm guarantee that they will be returned the entire expended amount of money. For this purpose, expert judgment techniques are very useful to provide a less-costly initial validation that, when positive, may encourage organizations to use these new proposals. Therefore, the primary goal of the paper will be to assess how expert judgment techniques based on the Delphi method can be applied to Web Engineering field and, more in particular, to assess the validity of the NDT-Agile framework. NDT-Agile is a framework that combines Agile and Web Engineering techniques to meet Capability Maturity Model Integration development goals. The paper presents a real example of an application of a Delphi-based expert judgment method to assess NDT-Agile framework validity, explaining the design as well as the selection and usage of the different techniques it involves. The application of the method will allow assessing benefits and limitations of use in Web Engineering. As a main conclusion, we will state the utility of the proposed methods to obtain a low-resource initial validation of a certain proposal. Finally, we will identify further lines of research related to the analyzed topics.


Agile Web Engineering CMMI Delphi Expert judgment Organizational issues 



This research has been supported by the Megus project (TIN2013-46928-C3-3-R) and by the SoftPLM Network (TIN2015-71938-REDT) of the Ministerio de Ciencia e Innovación, Spain. We would like to thank Dr. Pedro Antonio García, Dr. Diego Torrecilla de Amo and Dr. Diego Nieto Lugilde, all from the University of Granada, for their useful and helpful comments. Finally, we would like to thank all experts participating in the process for their time, help and useful contribution.


  1. 1.
    Alonso JAG, Santacruz MP (2015) Cálculo e interpretación del Alfa de Cronbach para el caso de validación de la consistencia interna de un cuestionario, con dos posibles escalas tipo Likert. Rev Publ 2(2):62–77Google Scholar
  2. 2.
    Ambler SW (2002) Lessons in agility from Internet-based development. IEEE Softw 19:66–73CrossRefGoogle Scholar
  3. 3.
    Beck K, Andres C (2004) Extreme programming explained: embrace change, 2nd edn. Addison-Wesley, BostonGoogle Scholar
  4. 4.
    Beck K et al (2001) Manifesto for Agile software development. Accessed 08 2016
  5. 5.
    Benzécri JP (1973) L’Analyse des Données. Volume II, L’Analyse des Correspondances. Dunod, Paris/Bruxelles/MontrealGoogle Scholar
  6. 6.
    Bougroun Z, Zeaaraoui A, Bouchentouf T (2014) The projection of the specific practices of the third level of CMMI model in agile methods: Scrum, XP and Kanban. In: Third IEEE international colloquium in proceedings of information science and technology (CIST). IEEE, pp 174–179Google Scholar
  7. 7.
    Brooks KW (1979) Delphi technique: expanding applications. North Cent Assoc Q 54(3):377–385Google Scholar
  8. 8.
    Carney O, McIntosh J, Worth A (1996) The use of the nominal group technique in research with community nurses. J Adv Nurs 23(5):1024–1029CrossRefGoogle Scholar
  9. 9.
    Chaffin WW, Talley WK (1980) Individual stability in Delphi studies. Technol Forecast Soc Change 16:67–73CrossRefGoogle Scholar
  10. 10.
    CMMI Product Team (2010) CMMI for development, version 1.3., Carnegie Mellon University, technical report. Accessed 08 2016
  11. 11.
    Cohen J (1968) Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit. Psychol Bull 70(4):213–220CrossRefGoogle Scholar
  12. 12.
    Creswell JW (2003) Research design: qualitative, quantitative, and mixed method approaches, 2nd edn. SAGE, Thousand OaksGoogle Scholar
  13. 13.
    Cronbach LJ (1951) Coefficient alpha and the internal structure of tests. Psychometrika 16(3):297–334CrossRefGoogle Scholar
  14. 14.
    Cyphert FR, Gant WL (1971) The Delphi technique: a case study. Phi Delta Kappan 52:272–273Google Scholar
  15. 15.
    Dajani JS, Sincoff MZ, Talley WK (1979) Stability and agreement criteria for the termination of Delphi studies. Technol Forecast Soc Change 13(1979):83–90CrossRefGoogle Scholar
  16. 16.
    Dalkey NC (1972) The Delphi method: an experimental study of group opinion. In: Dalkey NC, Rourke DL, Lewis R, Snyder D (eds) Studies in the quality of life: Delphi and decision-making. Lexington Books, Lexington, pp 13–54Google Scholar
  17. 17.
    Dalkey NC, Helmer O (1963) An experimental application of the Delphi method to the use of experts. Manag Sci 9:458–467CrossRefGoogle Scholar
  18. 18.
    Dell Software (2015) How to analyze simple two-way and multi-way table, correspondence analysis. Accessed 08 2016
  19. 19.
    Díaz J, Garbajosa J, Calvo-Manzano JA (2009) Mapping CMMI level 2 to Scrum practices: an experience report. SPI, Chennai, pp 93–104Google Scholar
  20. 20.
    Diehl M, Stroebe W (1987) Productivity loss in brainstorming groups: towards the solution of a riddle. J Pers Soc Psychol 53(3):497CrossRefGoogle Scholar
  21. 21.
    Escalona MJ, Aragón G (2008) NDT: a model-driven approach for web requirements. IEEE Trans Softw Eng 34(3):370–390Google Scholar
  22. 22.
    Escalona MJ, Mejías M, Torres J (2004) Developing systems with NDT and NDT-Tool. In: 13th International conference on information systems development: methods and tools, theory and practice, Vilna, Lithuania, pp 149–159Google Scholar
  23. 23.
    Falissard B (2012) psy: various procedures used in psychometry. R package version 1.1. Accessed 08 2016
  24. 24.
    Fleiss JL (1971) Measuring nominal scale agreement among many raters. Psychol Bull 76(5):378–382CrossRefGoogle Scholar
  25. 25.
    Gamer M, Lemon J, Puspendra Singh IF (2012) irr: various coefficients of interrater reliability and agreement. R package version 0.84. Accessed 11 2016
  26. 26.
    García-Crespo A, Colomo-Palacios R, Soto-Acosta P, Ruano-Mayoral M (2010) A qualitative study of hard decision making in managing global software development teams. Inf Syst Manag 27(3):247–252CrossRefGoogle Scholar
  27. 27.
    George D, Mallery P (2003) SPSS for windows step by step: a simple guide and reference. 11.0 update, 4th edn. Allyn & Bacon, BostonGoogle Scholar
  28. 28.
    Glazer H et al (2008) CMMI or Agile: why not embrace both!, Carnegie Mellon University. Accessed 08 2016
  29. 29.
    Gliem RR, Gliem JA (2003) Calculating, interpreting, and reporting Cronbach’s alpha reliability coefficient for Likert-type scales. In: Midwest research-to-practice conference in adult, continuing, and community educationGoogle Scholar
  30. 30.
    Goldenson DR, Gibson DL, Ferguson RL. Why make the switch? Evidence about the benefits of CMMI. Accessed 08 2016
  31. 31.
    Google. Google Forms. Accessed 08 2016
  32. 32.
    Heiko A (2012) Consensus measurement in Delphi studies: review and implications for future quality assurance. Technol Forecast Soc Change 79(8):1525–1536CrossRefGoogle Scholar
  33. 33.
    Hirschfeld HO (1935) A connection between correlation and contingency. Proc Camb Philos Soc 31:520–524CrossRefGoogle Scholar
  34. 34.
    Holey EA, Feeley JL, Dixon J, Whittaker VJ (2007) An exploration of the use of simple statistics to measure consensus and stability in Delphi studies. BMC Med Res Methodol 7:52. CrossRefGoogle Scholar
  35. 35.
    Hsu CC, Sandford BA (2007) The Delphi technique: making sense of consensus. Pract Assess Res Eval 12(10):1–8Google Scholar
  36. 36.
    Joshi JB, Aref WG, Ghafoor A, Spafford EH (2001) Security models for web-based applications. Commun ACM 44(2):38–44CrossRefGoogle Scholar
  37. 37.
    Kitchenham B et al (2009) Systematic literature reviews in software engineering—a systematic literature review. Inf Softw Technol 51(7–15):2009Google Scholar
  38. 38.
    Legendre P (2005) Species associations: the Kendall coefficient of concordance revisited. J Agric Biol Environ Stat 10(2):226–245CrossRefGoogle Scholar
  39. 39.
    Likert R (1932) A technique for the measurement of attitudes. Arch Psychol 22:1–55Google Scholar
  40. 40.
    Linstone HA, Turoff M (1975) The Delphi method: techniques and applications. Addison-Wesley Publishing Company, Reading, pp 3–12Google Scholar
  41. 41.
    Ludwig B (1997) Predicting the future: have you considered using the Delphi methodology? J Ext 35(5):1–4Google Scholar
  42. 42.
    Lukasiewicz K, Miler J (2012) Improving agility and discipline of software development with the Scrum and CMMI. Softw IET 6(5):416–422CrossRefGoogle Scholar
  43. 43.
    Marcal ASC, de Freitas BCC, Furtado Soares FS, Belchior AD (2008) Blending Scrum practices and CMMI project management process areas. ISSE 4:17–29Google Scholar
  44. 44.
    Mendes E, Mosley N (2005) Web cost estimation: an introduction. In: Web engineering: principles and techniques. IGI Global, Hershey, pp 182–202Google Scholar
  45. 45.
    Model Driven Web Engineering Workshop (2012) Satellite workshop of ICWE’2012 conference. Accessed 08 2016
  46. 46.
    Murugesan S, Deshpande Y, Hansen S, Ginige A (2001) Web engineering: a new discipline for development of web-based systems. In: Murugesan S, Deshpande Y (eds) Web engineering. Springer, Berlin, pp 3–13Google Scholar
  47. 47.
    Nakatsu RT, Iacovou CL (2009) A comparative study of important risk factors involved in offshore and domestic outsourcing of software development projects: a two-panel Delphi study. Inf Manag 46(1):57–68CrossRefGoogle Scholar
  48. 48.
    Nenadic O, Greenacre M (2007) Correspondence analysis in R, with two-and three-dimensional graphics: the ca package. J Stat Softw 20(3)Google Scholar
  49. 49.
    Nunnally JC (1967) Psychometric theory, 1st edn. McGraw-Hill, New YorkGoogle Scholar
  50. 50.
    Oh KH (1974) Forecasting through hierarchical Delphi. Unpublished doctoral dissertation, The Ohio State University, ColumbusGoogle Scholar
  51. 51.
    Osborn AF (1957) Applied imagination (rev. ed). Scribner, New YorkGoogle Scholar
  52. 52.
    Paulk MC (2001) Extreme programming from a CMM perspective. IEEE Softw 18(6):19–26CrossRefGoogle Scholar
  53. 53.
    Pikkarainen M et al (2008) The impact of Agile practices on communication in software development. Empir Softw Eng 13:303–337CrossRefGoogle Scholar
  54. 54.
    Pill J (1971) The Delphi method: substance, context, a critique and an annotated bibliography. Socio Econ Plan Sci 5:57–71CrossRefGoogle Scholar
  55. 55.
    Pressman RS (2000) What a tangled Web we weave. IEEE Softw 17:18–21CrossRefGoogle Scholar
  56. 56.
    R Core Team (2016) R: a language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. Accessed 08 2016
  57. 57.
    Reifer DJ (2000) Web development: estimating quick-to-market software. IEEE Softw 17:57–64CrossRefGoogle Scholar
  58. 58.
    Rieger WG (1986) Directions in Delphi developments: dissertations and their quality. Technol Forecast Soc Change 29(1986):195–204CrossRefGoogle Scholar
  59. 59.
    Rowe G, Wright G (2001) Expert opinions in forecasting: the role of the Delphi technique. In: Armstrong JS (ed) Principles of forecasting: a handbook for researchers and practitioners. Kluwer Academic Publishers, Boston, pp 125–144CrossRefGoogle Scholar
  60. 60.
    Scheibe M, Skutsch M, Schofer J (1975) Experiments in Delphi methodology. In: The Delphi method—techniques and applications. Addison-Wesley, Reading, pp 262–287Google Scholar
  61. 61.
    Schmidt R, Lyytinen K, Mark Keil PC (2001) Identifying software project risks: an international Delphi study. J Manag Inf Syst 17(4):5–36CrossRefGoogle Scholar
  62. 62.
    Schmitt N (1996) Uses and abuses of coefficient alpha. Psychol Assess 8(4):350–353CrossRefGoogle Scholar
  63. 63.
    Selleri Silva F, Santana Furtado Soares F, Lima Peres A, Monteiro de Azevedo I, Vasconcelos A, Kenji Kamei F, de Lemos Romero, Meira S (2015) Using CMMI together with agile software development: a systematic review. Inf Softw Technol 58:20–43CrossRefGoogle Scholar
  64. 64.
    Siegel S, Castellan NJ (1995) Estadística no paramétrica aplicada a las ciencias de la conducta. Trillas, Mexico CityGoogle Scholar
  65. 65.
    Staples M, Niazi M, Jeffery R, Abrahams A, Byatt P, Murphy R (2007) An exploratory study of why organizations do not adopt CMMI. J Syst Softw 80(6):883–895CrossRefGoogle Scholar
  66. 66.
    Sutherland J, Schwaber K (2011) The Scrum guide: the definitive guide to Scrum: the rules of the game. Accessed 08 2016
  67. 67.
    Torrecilla-Salinas CJ, Guardia T, De Troyer O, Mejías M, Sedeño J (2017) NDT-Agile: an Agile, CMMI-compatible framework for web engineering. In: International conference on software process improvement and capability determination. Springer, pp 3–16Google Scholar
  68. 68.
    Torrecilla Salinas CJ, Sedeño J, Escalona MJ, Mejías M (2014) An Agile approach to CMMI-DEV levels 4 and 5 in Web development projects. In: Information systems development (ISD2016 proceedings). Katowice, PolandGoogle Scholar
  69. 69.
    Torrecilla Salinas CJ, Sedeño J, Escalona MJ, Mejías M (2016) Agile, web engineering and capability maturity model integration: a systematic literature review. Inf Softw Technol 71(2016):92–107CrossRefGoogle Scholar
  70. 70.
    Torrecilla Salinas CJ, Sedeño J, Escalona MJ, Mejías M (2015) Estimating, planning and managing Agile Web development projects under a value-based perspective. Inf Softw Technol 61(2015):124–144CrossRefGoogle Scholar
  71. 71.
    Torrecilla Salinas CJ, Sedeño J, Escalona MJ, Mejías M (2014) Mapping Agile practices to CMMI-DEV level 3 in Web development environments. In: Information systems development: transforming organisations and society through information systems (ISD2014 proceedings). Varaždin, CroatiaGoogle Scholar
  72. 72.
    Torrecilla Salinas CJ, Escalona MJ, Mejías M (2012) A Scrum-based approach to CMMI maturity level 2 in Web development environments. In: Proceeding of international conference on information integration and web-based applications and services, Bali, Indonesia, December 3–5 2012. iiWAS, 12. ACMGoogle Scholar
  73. 73.
    VersionOne (2016) 9th Annual State of Agile survey. Accessed 08 2016
  74. 74.
    Warnes GR, Bolker B, Bonebakker L, Gentleman R, Huber W, Liaw A, Lumley T, Maechler M, Magnusson A, Moeller S, Schwartz M, Venables B (2016) gplots: various R programming tools for plotting data. R package version 3.0.1. Accessed 07 2018
  75. 75.
    Welch S, Comer J (1988) Quantitative methods for public administration: techniques and applications. Brooks/Cole Publishing Company, Pacific GroveGoogle Scholar
  76. 76.
    Wickham H (2009) ggplot2: Elegant Graphics for Data Analysis. Springer-Verlag, New YorkCrossRefGoogle Scholar
  77. 77.
    Wickham H (2007) Reshaping data with the reshape package. J Stat Softw 21(12):1–20.
  78. 78.
    Yelland PM (2010) An introduction to correspondence analysis. Accessed 08 2016

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  • C. J. Torrecilla-Salinas
    • 1
    Email author
  • O. De Troyer
    • 2
  • M. J. Escalona
    • 1
  • M. Mejías
    • 1
  1. 1.ETS Ingeniería InformaticaSevilleSpain
  2. 2.Department of Computer ScienceVrije Universiteit BrusselBrusselsBelgium

Personalised recommendations