Empirical Software Engineering

, Volume 5, Issue 4, pp 357–390 | Cite as

An Instrument for Measuring the Key Factors of Success in Software Process Improvement

  • Tore Dyba


Understandinghow to implement SPI successfully is arguably the most challengingissue facing the SPI field today. The SPI literature containsmany case studies of successful companies and descriptions oftheir SPI programs. However, there has been no systematic attemptto synthesize and organize the prescriptions offered. The researchefforts to date are limited and inconclusive and without adequatetheoretical and psychometric justification.

This paper provides a synthesis of prescriptions for successfulquality management and process improvement found from an extensivereview of the quality management, organizational learning, andsoftware process improvement literature. The literature reviewwas confirmed by empirical studies among both researchers andpractitioners. The main result is an instrument for measuringthe key factors of success in SPI based on data collected from120 software organizations. The measures were found to have satisfactorypsychometric properties. Hence, managers can use the instrumentto guide SPI activities in their respective organizations andresearchers can use it to build models to relate the facilitatingfactors to both learning processes and SPI outcomes.

Software process improvement success factors measurement instrument 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Ahire, S. L., Golhar, D. Y. and Waller, M. A. 1996 Development and Validation of TQM Implementation Constructs, Decision Sciences, 27(1), 23-56.Google Scholar
  2. Anastasi, A. and Urbina, S. 1997 Psychological Testing, Seventh edition, Upper Saddle River, New Jersey: Prentice-Hall.Google Scholar
  3. Argyris, C. and Schön, D. A. 1978 Organizational Learning: A Theory of Action Perspective, Reading, Massachusetts: Addison-Wesley.Google Scholar
  4. Argyris, C. and Schön, D. A. 1996 Organizational Learning II: Theory, Method, and Practice, Reading, Massachusetts: Addison-Wesley.Google Scholar
  5. Bagozzi, R. P. 1996 Measurement in Marketing Research: Basic Principles of Questionnaire Design, in R. P. Bagozzi (Ed.), Principles of Marketing Research, Cambridge, Massachusetts: Blackwell, pp. 1-49.Google Scholar
  6. Baruch, Y. 1999 Response Rate in Academic Studies—A Comparative Analysis, Human Relations, 52(4), 421-438.Google Scholar
  7. Basili, V. R. 1996 The Role of Experimentation in Software Engineering: Past, Current, and Future, Proceedings of the 18th International Conference on Software Engineering (ICSE-18), Berlin, Germany, March 25–29, pp. 442-449.Google Scholar
  8. Basili, V. R. and Caldiera, G. 1995 Improve Software Quality by Reusing Knowledge and Experience, Sloan Management Review, 37, Autumn, pp. 55-64.Google Scholar
  9. Basili, V. R. and Rombach, H. D. 1988 The TAME Project: Towards Improvement-Oriented Software Environments, IEEE Transactions on Software Engineering, 14(6), 758-773.Google Scholar
  10. Basili, V. R. and Selby, R. W. 1991 Paradigms for Experimentation and Empirical Studies in Software Engineering, Reliability Engineering and System Safety, 32(1–2), pp. 171-191.Google Scholar
  11. Basili, V. R., Selby, R. W. and Hutchens, D. H. 1986 Experimentation in Software Engineering, IEEE Transactions on Software Engineering, 12(7), 733-743.Google Scholar
  12. Black, S. A. and Porter, L. J. 1996 Identification of the Critical Factors of TQM, Decision Sciences, 27(1), 1-21.Google Scholar
  13. Blalock, H. M. 1969 Theory Construction: From Verbal to Mathematical Formulations, Englewood Cliffs, N.J.: Prentice-Hall.Google Scholar
  14. Carmines, E. G. and Zeller, R. A. 1979 Reliability and Validity Assessment, Sage University Paper series on Quantitative Applications in the Social Sciences, series no. 07-017, Newbury Park, CA: Sage.Google Scholar
  15. Cattell, R. B. 1966 The Scree Test for the Number of Factors, Multivariate Behavioral Research, 1, 245-276.Google Scholar
  16. Choo, C. W. 1995 The Knowing Organization: How Organizations Use Information to Construct Meaning Create Knowledge and Make Decisions, International Journal of Information Management, 16(5), 329-340.Google Scholar
  17. Choo, C. W. 1998 The Knowing Organization: How Organizations Use Information to Construct Meaning, Create Knowledge, and Make Decisions, New York: Oxford University Press.Google Scholar
  18. Cliff, N. R. 1988 The Eigenvalues-greater-than-one Rule and the Reliability of Components, Psychological Bulletin, 103, 276-279.Google Scholar
  19. Cohen, J. 1988 Statistical Power Analysis for the Behavioral Sciences, Second Edition, Hillsdale, New Jersey: Laurence Erlbaum.Google Scholar
  20. Comrey, A. 1973 A First Course on Factor Analysis, London: Academic Press.Google Scholar
  21. Cronbach, L. J. 1951 Coefficient Alpha and the Internal Consistency of Tests, Psychometrica, 16, 297-334, September.Google Scholar
  22. Cronbach, L. J. 1971 Test Validation, in Educational Measurement, 2nd Edition, R. L. Thorndike (ed.), American Council on Education, Washington, D.C, pp. 443-507.Google Scholar
  23. Crosby, P. B. 1979 Quality is Free: The Art of Making Quality Certain, New York: McGraw-Hill.Google Scholar
  24. Crosby, P. B. 1984 Quality Without Tears, New York: McGraw-Hill.Google Scholar
  25. Crosby, P. B. 1996 Quality Is Still Free: Making Quality Certain in Uncertain Times, New York: McGraw-Hill.Google Scholar
  26. Davis, D. 1996 Business Research for Decision Making, Fourth Edition, Belmont, California: Duxbury Press.Google Scholar
  27. Deming, W. E. 1982 Quality, Productivity, and Competitive Position, Camebridge, Massachusetts: MIT Centre for Advanced Engineering Study.Google Scholar
  28. Deming, W. E. 1986 Out of the Crisis, Cambridge, Massachusetts: MIT Center for Advanced Engineering Study.Google Scholar
  29. Deephouse, C., Mukhopadhyay, T., Goldenson, D. R. and Kellner, M. I. 1996 Software Processes and Project Performance, Journal of Management Information Systems, 12(3), 187-205.Google Scholar
  30. DeVellis, R. F. 1991 Scale Development: Theory and Applications, Newbury Park, CA: Sage.Google Scholar
  31. Dybå, T. 2000a Improvisation in Small Software Organizations, IEEE Software, 17(5), Sept.–Oct.Google Scholar
  32. Dybå, T. (Ed.) 2000b SPIQ—Software Process Improvement for better Quality: Methodology Handbook, Version 3.0 (Final), IDI Report 2/2000, Norwegian University of Science and Technology, Trondheim, Norway (in Norwegian).Google Scholar
  33. EFQM 1999 The EFQM Excellence Model, Brussels: European Foundation for Quality Management.Google Scholar
  34. El Emam 1998 The Internal Consistency of the ISO/IEC 15504 Software Process Capability Scale, Proceedings of the Fifth International Symposium on Software Metrics, IEEE Computer Society Press, pp. 72-81.Google Scholar
  35. El Emam, K. and Birk, A. 2000 Validating the ISO/IEC 15504 Measure of Software Requirements Analysis Process Capability, IEEE Transactions on Software Engineering, 26(6), 541-566.Google Scholar
  36. El Emam, K. and Madhavji, N. H. 1995 The Reliability of Measuring Organizational Maturity, Software Process—Improvement and Practice, 1, 3-25.Google Scholar
  37. El Emam, K. and Madhavji, N. H. 1996 An Instrument for Measuring the Success of the Requirements Engineering Process in Information Systems Development, Empirical Software Engineering, 1(3), 201-240.Google Scholar
  38. El Emam, K., Fusaro, P. and Smith, B. 1999 Success Factors and Barriers for Software Process Improvement, in R. Messnarz, and C. Tully (Eds.) Better Software Practice for Business Benefit: Principles and Experiences, Los Alamitos, California: IEEE Computer Society Press, pp. 355-371.Google Scholar
  39. El Emam, K., Goldenson, D. R., McCurley J. and Herbsleb, J. 1998 Success or Failure? Modeling the Likelihood of Software Process Improvement, Technical Report, ISERN-98-15, International Software Engineering Research Network.Google Scholar
  40. Feigenbaum, A. V. 1991 Total Quality Control, Fortieth Anniversary Edition, New York: McGraw-Hill.Google Scholar
  41. Fink, A. and Kosecoff, J. 1998 How to Conduct Surveys: A Step-By-Step Guide, Second Edition, Thousand Oaks, California: Sage Publications.Google Scholar
  42. French, W. L. and Bell, C. H. Jr. 1999 Organization Development: Behavioral Science Interventions for Organization Improvement, Sixth Edition, Upper Saddle River, New Jersey: Prentice-Hall.Google Scholar
  43. Fusaro, P., El Emam, K. and Smith, B. 1998 The Internal Consistency of the 1987 SEI Maturity Questionnaire and the SPICE Capability Dimension, Empirical Software Engineering, 3(2), 179-201.Google Scholar
  44. Garvin, D. A. 1983 Quality on the Line, Harvard Business Review, 61(5), 65-75.Google Scholar
  45. Garvin, D. A. 1984 Japanese Quality Management, Columbia Journal of World Business, 19(3), 3-19.Google Scholar
  46. Garvin, D. A. 1993 Building a Learning Organization, Harvard Business Review, 71(4), 78-91.Google Scholar
  47. Goldenson, D. R. and Herbsleb, J. (1995) After the Appraisal: A Systematic Survey of Process Improvement, its Benefits, and Factors that Influence Success, Technical Report, CMU/SEI-95-TR-009, Carnegie Mellon University, Software Engineering Institute.Google Scholar
  48. Goldenson, D. R., El Emam, K., Herbsleb, J. and Deephouse, C. 1999 Empirical Studies of Software Process Assessment Methods, in K. El Emam and N. H. Madhavji (Eds.), Elements of Software Process Assessment and Improvement, Los Alamitos, California: IEEE Computer Society Press, pp. 177-218.Google Scholar
  49. Guilford, J. P. 1954 Psychometric Methods, 2nd edition, New York: McGraw-Hill.Google Scholar
  50. Harrison, R., Badoo, N., Barry, E., Biffl, S., Parra, A., Winter, B. and Wuest, J. 1999 Directions and Methodologies for Empirical Software Engineering Research, Empirical Software Engineering, 4(4), 405-410.Google Scholar
  51. Humphrey, W. S. 1989 Managing the Software Process, Reading, Massachusetts: Addison-Wesley.Google Scholar
  52. Humphrey, W. S. 1997 Managing Technical People: Innovation, Teamwork, and the Software Process, Reading, Massachusetts: Addison-Wesley.Google Scholar
  53. Hunt, R. and Buzan, T. 1999 Creating a Thinking Organization: Groundrules for Success, Hampshire, England: Gower.Google Scholar
  54. Hunter, J. E. and Schmidt, F. L. 1990 Methods of Meta-Analysis: Correcting Error and Bias in Research Findings, Newbury Park, California: Sage Publications.Google Scholar
  55. Ishikawa, K. 1986 Guide to Quality Control, Second Edition, New York: Quality Resources.Google Scholar
  56. Ishikawa, K. 1990 Introduction to Quality Control, London: Chapman & Hall.Google Scholar
  57. ISO/DIS 9000 2000 Quality Management Systems—Fundamentals and Vocabulary.Google Scholar
  58. ISO/DIS 9004 2000 Quality Management Systems—Guidelines for Performance Improvements.Google Scholar
  59. ISO/IEC TR 15504-7 1998 Information Technology—Software Process Assessment—Part 7: Guide for Use in Process Improvement.Google Scholar
  60. Jeffery, D. R. and Votta, L. G. 1999 Guest Editor's Special Section Introduction, IEEE Transactions on Software Engineering, 25(4), 435-437.Google Scholar
  61. Juran, J. M. 1992 Juran on Quality by Design: The New Steps for Planning Quality into Goods and Services, New York: Free Press.Google Scholar
  62. Juran, J. M. and Godfrey, A. B. (Eds.) 1999 Juran's Quality Handbook, Fifth Edition, New York: McGraw-Hill.Google Scholar
  63. Kaiser, H. F. 1960 The Application of Electronic Computers to Factor Analysis, Educational and Psychological Measurements, 20, 141-151.Google Scholar
  64. Kaiser, H. F. 1970 A Second Generation Little Jiffy, Psychometrika, 35, 401-417.Google Scholar
  65. Kanuk, L. and Berenson, C. 1975 Mail Surveys and Response Rates: A Literature Review, Journal of Marketing Research, 12, 440-453.Google Scholar
  66. Kerlinger, F. 1986 Foundations of Behavioral Research, Holt, Rinehart and Winston.Google Scholar
  67. Larsen, E. A, and Kautz, K. 1997 Quality Assurance and Software Process Improvement in Norway, Software Process—Improvement and Practice, 3(2), 71-86.Google Scholar
  68. Lawrence, P. R. and Lorsch, J. W. 1986 Organization and Environment: Managing Differentiation and Integration, 2nd Edition, Boston: Harvard Business School Press.Google Scholar
  69. Likert, R. 1932 A Technique for the Measurement of Attitudes, Archives of Psychology, 22(140).Google Scholar
  70. Likert, R. 1967 The Human Organization: Its Management and Value, New York: McGraw-Hill.Google Scholar
  71. Likert, R. and Roslow, S. 1934 The Effects Upon the Reliability of Attitude Scales of Using Three, Five or Seven Alternatives, Working Paper, New York University.Google Scholar
  72. Lissitz, R. W. and Green, S. B. 1975 Effects of the Number of Scale Points on Reliability: A Monte Carlo Approach, Journal of Applied Psychology, 60, 10-13, February.Google Scholar
  73. March, J. G. 1991 Exploration and Exploitation in Organizational Learning, Organization Science, 2(1), 71-87.Google Scholar
  74. March, J. G. 1999 The Pursuit of Organizational Intelligence, Malden, Massachusetts: Blackwell.Google Scholar
  75. Miller, G. A. 1956 The Magical Number Seven, Plus or Minus Two: Some Limits on Our Capacity for Processing Information, Psychological Review, 63, 81-97.Google Scholar
  76. Neuman, W. L. 2000 Social Research Methods: Qualitative and Quantitative Approaches, Fourth Edition, Boston: Allyn and Bacon.Google Scholar
  77. Nevis, E. C., DiBella, A. J. and Gould, J. M. 1995 Understanding Organizations as Learning Systems, Sloan Management Review, 37, 73-85.Google Scholar
  78. Nonaka, I. 1991 The Knowledge-Creating Company, Harvard Business Review, 69(6), 96-104.Google Scholar
  79. Nonaka, I. 1994 A Dynamic Theory of Organizational Knowledge Creation, Organization Science, 5(1), 14-37.Google Scholar
  80. Nonaka, I. and Takeuchi, H. 1995. The Knowledge-Creating Company: How Japanese Companies Create the Dynamics of Innovation, New York: Oxford University Press.Google Scholar
  81. Novick, M. and Lewis, G. 1967 Coefficient Alpha and the Reliability of Composite Measurements, Psychometrika, 32, 1-13.Google Scholar
  82. Nunnally, J. C. 1978 Psychometric Theory, Second edition, New York: McGraw-Hill.Google Scholar
  83. Nunnally, J. C. and Bernstein, I. A. 1994 Psychometric Theory, Third edition, New York: McGraw-Hill.Google Scholar
  84. Powell, T. C. 1995 Total Quality Management as Competitive Advantage: A Review and Empirical Study, Strategic Management Journal, 16, 15-37.Google Scholar
  85. Rogers, E. M. 1995 Diffusion of Innovations, Fourth Edition, New York: The Free Press.Google Scholar
  86. Sanders, M. (Ed.) 1998 The SPIRE Handbook: Better, Faster, Cheaper Software Development in Small Organisations, Dublin: Centre for Software Engineering Ltd.Google Scholar
  87. Saraph, J. V., Benson, P. G. and Schroeder, R. G. 1989. An Instrument for Measuring the Critical Factors of Quality Management, Decision Sciences, 20(4), 810-829.Google Scholar
  88. Schaffer, R. H. and Thomson, H. A. 1992. Successful Change Programs Begin with Results, Harvard Business Review, 70, 80-89.Google Scholar
  89. Senge, P. M. 1990 The Fifth Discipline: The Art and Practice of the Learning Organization, New York: Doubleday.Google Scholar
  90. Senge, P. M., Kleiner, A., Roberts, C., Ross, R. and Smith, B. 1994 The Fifth Discipline Fieldbook: Strategies and Tools for Building a Learning Organization, New York: Currency/Doubleday.Google Scholar
  91. Senge, P. M., Kleiner, A., Roberts, C., Ross, R., Roth, G. and Smith, B. 1999 The Dance of Change: The Challenges of Sustaining Momentum in Learning Organizations, New York: Currency/Doubleday.Google Scholar
  92. Spector, P. 1992 Summated Rating Scale Construction: An Introduction, Sage University Paper series on Quantitative Applications in the Social Sciences, series no. 07-082, Newbury Park, CA: Sage.Google Scholar
  93. SPSS 1999a SPSS Base 9.0: User's Guide, Chicago, IL: SPSS Inc.Google Scholar
  94. SPSS 1999b SPSS Base 9.0: Applications Guide, Chicago, IL: SPSS Inc.Google Scholar
  95. Stelzer, D. and Mellis, W. 1998 Success Factors of Organizational Change in Software Process Improvement, Software Process—Improvement and Practice, 4(4), 227-250.Google Scholar
  96. Stelzer, D., Mellis, W. and Herzwurm, G. 1996 Software Process Improvement via ISO 9000? Results of Two Surveys among European Software Houses, Proceedings of the 29th Hawaii International Conference on Systems Sciences, January 3–6, Wailea, Hawaii, USA.Google Scholar
  97. Stevens, J. 1992 Applied Multivariate Statistics for the Social Sciences, London: Lawrence Erlbaum.Google Scholar
  98. Straub, D. W. 1989 Validating Instruments in MIS Research, MIS Quarterly, 13(2), 147-169.Google Scholar
  99. Stålhane, T., Borgersen, P. C. and Arnesen, K. 1997 In Search of the Customer's Quality View, Journal of Systems Software, 38, 85-93.Google Scholar
  100. Saarinen, T. 1996 An Expanded Instrument for Evaluating Information System Success, Information & Management, 31, 103-118.Google Scholar
  101. Taguchi, G. 1986 Introduction to Quality Engineering: Designing Quality into Products and Processes, Tokyo: Asian Productivity Organization.Google Scholar
  102. Taguchi, G., Elsayed, E. A. and Hsiang, T. C. 1989 Quality Engineering in Production Systems, New York: McGraw-Hill.Google Scholar
  103. Teng, J. T. C., Jeong, S. R. and Grover, V. 1998 Profiling Successful Reengineering Projects, Communications of the ACM, 41(6), 96-102.Google Scholar
  104. Van de Ven, A. H. and Ferry, D. L. 1980 Measuring and Assessing Organizations, New York: John Wiley & Sons.Google Scholar
  105. Winter, S. G. 1994 Organizing for Continuous Improvement: Evolutionary Theory Meets the Quality Revolution, in J. A. C. Baum and J. Singh (Eds.) The Evolutionary Dynamics of Organizations, Oxford University Press.Google Scholar
  106. Yusof, S. M. and Aspinwall, E. 1999 Critical Success Factors for Total Quality Management Implementation in Small and Medium Enterprises, Total Quality Management, 10 (4&5), 803-809.Google Scholar
  107. Zahran, S. 1998 Software Process Improvement: Practical Guidelines for Business Success, Harlow, England: Addison-Wesley.Google Scholar
  108. Zwick, W. R. and Velicer, W. F. 1986 Comparison of Five Rules for Determining the Number of Components to Retain, Psychological Bulletin, 99, 432-442.Google Scholar

Copyright information

© Kluwer Academic Publishers 2000

Authors and Affiliations

  • Tore Dyba
    • 1
  1. 1.SINTEF Telecom and InformaticsNorway

Personalised recommendations