Skip to main content
Log in

A Delphi-based expert judgment method applied to the validation of a mature Agile framework for Web development projects

  • Published:
Information Technology and Management Aims and scope Submit manuscript


The validation of any new methodological proposal demands several real-life implementations. However, organizations are reluctant to invest without the firm guarantee that they will be returned the entire expended amount of money. For this purpose, expert judgment techniques are very useful to provide a less-costly initial validation that, when positive, may encourage organizations to use these new proposals. Therefore, the primary goal of the paper will be to assess how expert judgment techniques based on the Delphi method can be applied to Web Engineering field and, more in particular, to assess the validity of the NDT-Agile framework. NDT-Agile is a framework that combines Agile and Web Engineering techniques to meet Capability Maturity Model Integration development goals. The paper presents a real example of an application of a Delphi-based expert judgment method to assess NDT-Agile framework validity, explaining the design as well as the selection and usage of the different techniques it involves. The application of the method will allow assessing benefits and limitations of use in Web Engineering. As a main conclusion, we will state the utility of the proposed methods to obtain a low-resource initial validation of a certain proposal. Finally, we will identify further lines of research related to the analyzed topics.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24
Fig. 25
Fig. 26
Fig. 27
Fig. 28
Fig. 29
Fig. 30
Fig. 31

Similar content being viewed by others


  1. Alonso JAG, Santacruz MP (2015) Cálculo e interpretación del Alfa de Cronbach para el caso de validación de la consistencia interna de un cuestionario, con dos posibles escalas tipo Likert. Rev Publ 2(2):62–77

    Google Scholar 

  2. Ambler SW (2002) Lessons in agility from Internet-based development. IEEE Softw 19:66–73

    Article  Google Scholar 

  3. Beck K, Andres C (2004) Extreme programming explained: embrace change, 2nd edn. Addison-Wesley, Boston

    Google Scholar 

  4. Beck K et al (2001) Manifesto for Agile software development. Accessed 08 2016

  5. Benzécri JP (1973) L’Analyse des Données. Volume II, L’Analyse des Correspondances. Dunod, Paris/Bruxelles/Montreal

  6. Bougroun Z, Zeaaraoui A, Bouchentouf T (2014) The projection of the specific practices of the third level of CMMI model in agile methods: Scrum, XP and Kanban. In: Third IEEE international colloquium in proceedings of information science and technology (CIST). IEEE, pp 174–179

  7. Brooks KW (1979) Delphi technique: expanding applications. North Cent Assoc Q 54(3):377–385

    Google Scholar 

  8. Carney O, McIntosh J, Worth A (1996) The use of the nominal group technique in research with community nurses. J Adv Nurs 23(5):1024–1029

    Article  Google Scholar 

  9. Chaffin WW, Talley WK (1980) Individual stability in Delphi studies. Technol Forecast Soc Change 16:67–73

    Article  Google Scholar 

  10. CMMI Product Team (2010) CMMI for development, version 1.3., Carnegie Mellon University, technical report. Accessed 08 2016

  11. Cohen J (1968) Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit. Psychol Bull 70(4):213–220

    Article  Google Scholar 

  12. Creswell JW (2003) Research design: qualitative, quantitative, and mixed method approaches, 2nd edn. SAGE, Thousand Oaks

    Google Scholar 

  13. Cronbach LJ (1951) Coefficient alpha and the internal structure of tests. Psychometrika 16(3):297–334

    Article  Google Scholar 

  14. Cyphert FR, Gant WL (1971) The Delphi technique: a case study. Phi Delta Kappan 52:272–273

    Google Scholar 

  15. Dajani JS, Sincoff MZ, Talley WK (1979) Stability and agreement criteria for the termination of Delphi studies. Technol Forecast Soc Change 13(1979):83–90

    Article  Google Scholar 

  16. Dalkey NC (1972) The Delphi method: an experimental study of group opinion. In: Dalkey NC, Rourke DL, Lewis R, Snyder D (eds) Studies in the quality of life: Delphi and decision-making. Lexington Books, Lexington, pp 13–54

    Google Scholar 

  17. Dalkey NC, Helmer O (1963) An experimental application of the Delphi method to the use of experts. Manag Sci 9:458–467

    Article  Google Scholar 

  18. Dell Software (2015) How to analyze simple two-way and multi-way table, correspondence analysis. Accessed 08 2016

  19. Díaz J, Garbajosa J, Calvo-Manzano JA (2009) Mapping CMMI level 2 to Scrum practices: an experience report. SPI, Chennai, pp 93–104

    Google Scholar 

  20. Diehl M, Stroebe W (1987) Productivity loss in brainstorming groups: towards the solution of a riddle. J Pers Soc Psychol 53(3):497

    Article  Google Scholar 

  21. Escalona MJ, Aragón G (2008) NDT: a model-driven approach for web requirements. IEEE Trans Softw Eng 34(3):370–390

    Google Scholar 

  22. Escalona MJ, Mejías M, Torres J (2004) Developing systems with NDT and NDT-Tool. In: 13th International conference on information systems development: methods and tools, theory and practice, Vilna, Lithuania, pp 149–159

  23. Falissard B (2012) psy: various procedures used in psychometry. R package version 1.1. Accessed 08 2016

  24. Fleiss JL (1971) Measuring nominal scale agreement among many raters. Psychol Bull 76(5):378–382

    Article  Google Scholar 

  25. Gamer M, Lemon J, Puspendra Singh IF (2012) irr: various coefficients of interrater reliability and agreement. R package version 0.84. Accessed 11 2016

  26. García-Crespo A, Colomo-Palacios R, Soto-Acosta P, Ruano-Mayoral M (2010) A qualitative study of hard decision making in managing global software development teams. Inf Syst Manag 27(3):247–252

    Article  Google Scholar 

  27. George D, Mallery P (2003) SPSS for windows step by step: a simple guide and reference. 11.0 update, 4th edn. Allyn & Bacon, Boston

    Google Scholar 

  28. Glazer H et al (2008) CMMI or Agile: why not embrace both!, Carnegie Mellon University. Accessed 08 2016

  29. Gliem RR, Gliem JA (2003) Calculating, interpreting, and reporting Cronbach’s alpha reliability coefficient for Likert-type scales. In: Midwest research-to-practice conference in adult, continuing, and community education

  30. Goldenson DR, Gibson DL, Ferguson RL. Why make the switch? Evidence about the benefits of CMMI. Accessed 08 2016

  31. Google. Google Forms. Accessed 08 2016

  32. Heiko A (2012) Consensus measurement in Delphi studies: review and implications for future quality assurance. Technol Forecast Soc Change 79(8):1525–1536

    Article  Google Scholar 

  33. Hirschfeld HO (1935) A connection between correlation and contingency. Proc Camb Philos Soc 31:520–524

    Article  Google Scholar 

  34. Holey EA, Feeley JL, Dixon J, Whittaker VJ (2007) An exploration of the use of simple statistics to measure consensus and stability in Delphi studies. BMC Med Res Methodol 7:52.

    Article  Google Scholar 

  35. Hsu CC, Sandford BA (2007) The Delphi technique: making sense of consensus. Pract Assess Res Eval 12(10):1–8

    Google Scholar 

  36. Joshi JB, Aref WG, Ghafoor A, Spafford EH (2001) Security models for web-based applications. Commun ACM 44(2):38–44

    Article  Google Scholar 

  37. Kitchenham B et al (2009) Systematic literature reviews in software engineering—a systematic literature review. Inf Softw Technol 51(7–15):2009

    Google Scholar 

  38. Legendre P (2005) Species associations: the Kendall coefficient of concordance revisited. J Agric Biol Environ Stat 10(2):226–245

    Article  Google Scholar 

  39. Likert R (1932) A technique for the measurement of attitudes. Arch Psychol 22:1–55

    Google Scholar 

  40. Linstone HA, Turoff M (1975) The Delphi method: techniques and applications. Addison-Wesley Publishing Company, Reading, pp 3–12

    Google Scholar 

  41. Ludwig B (1997) Predicting the future: have you considered using the Delphi methodology? J Ext 35(5):1–4

    Google Scholar 

  42. Lukasiewicz K, Miler J (2012) Improving agility and discipline of software development with the Scrum and CMMI. Softw IET 6(5):416–422

    Article  Google Scholar 

  43. Marcal ASC, de Freitas BCC, Furtado Soares FS, Belchior AD (2008) Blending Scrum practices and CMMI project management process areas. ISSE 4:17–29

    Google Scholar 

  44. Mendes E, Mosley N (2005) Web cost estimation: an introduction. In: Web engineering: principles and techniques. IGI Global, Hershey, pp 182–202

  45. Model Driven Web Engineering Workshop (2012) Satellite workshop of ICWE’2012 conference. Accessed 08 2016

  46. Murugesan S, Deshpande Y, Hansen S, Ginige A (2001) Web engineering: a new discipline for development of web-based systems. In: Murugesan S, Deshpande Y (eds) Web engineering. Springer, Berlin, pp 3–13

  47. Nakatsu RT, Iacovou CL (2009) A comparative study of important risk factors involved in offshore and domestic outsourcing of software development projects: a two-panel Delphi study. Inf Manag 46(1):57–68

    Article  Google Scholar 

  48. Nenadic O, Greenacre M (2007) Correspondence analysis in R, with two-and three-dimensional graphics: the ca package. J Stat Softw 20(3)

  49. Nunnally JC (1967) Psychometric theory, 1st edn. McGraw-Hill, New York

    Google Scholar 

  50. Oh KH (1974) Forecasting through hierarchical Delphi. Unpublished doctoral dissertation, The Ohio State University, Columbus

  51. Osborn AF (1957) Applied imagination (rev. ed). Scribner, New York

    Google Scholar 

  52. Paulk MC (2001) Extreme programming from a CMM perspective. IEEE Softw 18(6):19–26

    Article  Google Scholar 

  53. Pikkarainen M et al (2008) The impact of Agile practices on communication in software development. Empir Softw Eng 13:303–337

    Article  Google Scholar 

  54. Pill J (1971) The Delphi method: substance, context, a critique and an annotated bibliography. Socio Econ Plan Sci 5:57–71

    Article  Google Scholar 

  55. Pressman RS (2000) What a tangled Web we weave. IEEE Softw 17:18–21

    Article  Google Scholar 

  56. R Core Team (2016) R: a language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. Accessed 08 2016

  57. Reifer DJ (2000) Web development: estimating quick-to-market software. IEEE Softw 17:57–64

    Article  Google Scholar 

  58. Rieger WG (1986) Directions in Delphi developments: dissertations and their quality. Technol Forecast Soc Change 29(1986):195–204

    Article  Google Scholar 

  59. Rowe G, Wright G (2001) Expert opinions in forecasting: the role of the Delphi technique. In: Armstrong JS (ed) Principles of forecasting: a handbook for researchers and practitioners. Kluwer Academic Publishers, Boston, pp 125–144

    Chapter  Google Scholar 

  60. Scheibe M, Skutsch M, Schofer J (1975) Experiments in Delphi methodology. In: The Delphi method—techniques and applications. Addison-Wesley, Reading, pp 262–287

  61. Schmidt R, Lyytinen K, Mark Keil PC (2001) Identifying software project risks: an international Delphi study. J Manag Inf Syst 17(4):5–36

    Article  Google Scholar 

  62. Schmitt N (1996) Uses and abuses of coefficient alpha. Psychol Assess 8(4):350–353

    Article  Google Scholar 

  63. Selleri Silva F, Santana Furtado Soares F, Lima Peres A, Monteiro de Azevedo I, Vasconcelos A, Kenji Kamei F, de Lemos Romero, Meira S (2015) Using CMMI together with agile software development: a systematic review. Inf Softw Technol 58:20–43

    Article  Google Scholar 

  64. Siegel S, Castellan NJ (1995) Estadística no paramétrica aplicada a las ciencias de la conducta. Trillas, Mexico City

    Google Scholar 

  65. Staples M, Niazi M, Jeffery R, Abrahams A, Byatt P, Murphy R (2007) An exploratory study of why organizations do not adopt CMMI. J Syst Softw 80(6):883–895

    Article  Google Scholar 

  66. Sutherland J, Schwaber K (2011) The Scrum guide: the definitive guide to Scrum: the rules of the game. Accessed 08 2016

  67. Torrecilla-Salinas CJ, Guardia T, De Troyer O, Mejías M, Sedeño J (2017) NDT-Agile: an Agile, CMMI-compatible framework for web engineering. In: International conference on software process improvement and capability determination. Springer, pp 3–16

  68. Torrecilla Salinas CJ, Sedeño J, Escalona MJ, Mejías M (2014) An Agile approach to CMMI-DEV levels 4 and 5 in Web development projects. In: Information systems development (ISD2016 proceedings). Katowice, Poland

  69. Torrecilla Salinas CJ, Sedeño J, Escalona MJ, Mejías M (2016) Agile, web engineering and capability maturity model integration: a systematic literature review. Inf Softw Technol 71(2016):92–107

    Article  Google Scholar 

  70. Torrecilla Salinas CJ, Sedeño J, Escalona MJ, Mejías M (2015) Estimating, planning and managing Agile Web development projects under a value-based perspective. Inf Softw Technol 61(2015):124–144

    Article  Google Scholar 

  71. Torrecilla Salinas CJ, Sedeño J, Escalona MJ, Mejías M (2014) Mapping Agile practices to CMMI-DEV level 3 in Web development environments. In: Information systems development: transforming organisations and society through information systems (ISD2014 proceedings). Varaždin, Croatia

  72. Torrecilla Salinas CJ, Escalona MJ, Mejías M (2012) A Scrum-based approach to CMMI maturity level 2 in Web development environments. In: Proceeding of international conference on information integration and web-based applications and services, Bali, Indonesia, December 3–5 2012. iiWAS, 12. ACM

  73. VersionOne (2016) 9th Annual State of Agile survey. Accessed 08 2016

  74. Warnes GR, Bolker B, Bonebakker L, Gentleman R, Huber W, Liaw A, Lumley T, Maechler M, Magnusson A, Moeller S, Schwartz M, Venables B (2016) gplots: various R programming tools for plotting data. R package version 3.0.1. Accessed 07 2018

  75. Welch S, Comer J (1988) Quantitative methods for public administration: techniques and applications. Brooks/Cole Publishing Company, Pacific Grove

    Google Scholar 

  76. Wickham H (2009) ggplot2: Elegant Graphics for Data Analysis. Springer-Verlag, New York

    Book  Google Scholar 

  77. Wickham H (2007) Reshaping data with the reshape package. J Stat Softw 21(12):1–20.

  78. Yelland PM (2010) An introduction to correspondence analysis. Accessed 08 2016

Download references


This research has been supported by the Megus project (TIN2013-46928-C3-3-R) and by the SoftPLM Network (TIN2015-71938-REDT) of the Ministerio de Ciencia e Innovación, Spain. We would like to thank Dr. Pedro Antonio García, Dr. Diego Torrecilla de Amo and Dr. Diego Nieto Lugilde, all from the University of Granada, for their useful and helpful comments. Finally, we would like to thank all experts participating in the process for their time, help and useful contribution.

Author information

Authors and Affiliations


Corresponding author

Correspondence to C. J. Torrecilla-Salinas.

Additional information

This paper presents the authors’ views, which do not necessarily reflect those of their employers.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Torrecilla-Salinas, C.J., De Troyer, O., Escalona, M.J. et al. A Delphi-based expert judgment method applied to the validation of a mature Agile framework for Web development projects. Inf Technol Manag 20, 9–40 (2019).

Download citation

  • Published:

  • Issue Date:

  • DOI: