Evaluation of Software Mediated Process Assessments for IT Service Management

Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 526)


IT service organisations are cognisant that continual service improvement can be achieved by conducting regular process assessments. However, such assessments are expensive and so we have developed a Decision Support System (DSS) tool which uses the international standard for process assessment ISO/IEC 15504 to offer a transparent and efficient approach. This paper provides evidence of evaluation of this software-mediated process assessment (SMPA) approach which was based on ISO/IEC 15504, ISO/IEC 20000 and the IT Infrastructure Library (ITIL®). In a usability evaluation of the online tool, participants reported largely positive experiences finding the online survey easy to use trustworthy, comfortable, generally effective, and more transparent and less costly to implement than a manual assessment. However, to engage in process improvement, human judgment, and possibly expert assessment facilitators are necessary for assessment validation and improvement, that is, a fully automated online survey that is strictly standard-based is not very useful. Further clarification of the survey questions with relevant examples, clearer answer options and having more visible goal statements on every question page were suggested.


ITSM process assessment ISO/IEC 15504 Evaluation IT service management Process improvement 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Barafort, B., Betry, V., Cortina, S., Picard, M., St-Jean, M., Renault, A., Valdès, O.: ITSM Process Assessment Supporting ITIL. Public Research Centre Henri Tudor (ed.) Zaltbommel, Netherlands. Van Haren Publishing (2009)Google Scholar
  2. 2.
    Bernard, P.: Foundations of ITIL, Chittenden, J. (ed.) Zaltbommel, Netherlands. Van Haren Publishing (2012)Google Scholar
  3. 3.
    Lloyd, V.: ITIL Continual Service Improvement. The Stationery Office, London, UK (2011)Google Scholar
  4. 4.
    ISO/IEC, ISO/IEC 15504-2:2004 – Information Technology – Process Assessment – Part 2: Performing an Assessment. International Organization for Standardization: Geneva, Switzerland (2004)Google Scholar
  5. 5.
    Mesquida, A.L., Mas, A., Amengual, E., Calvo-Manzano, J.A.: IT Service Management Process Improvement based on ISO/IEC 15504: A Systematic Review. Information and Software Technology 54(3), 239–247 (2012)CrossRefGoogle Scholar
  6. 6.
    ISO/IEC, ISO/IEC TS 15504-8:2012 - Information Technology - Process Assessment - Part 8: An Exemplar Process Assessment Model for IT Service Management. International Organization for Standardization: Geneva, Switzerland (2012)Google Scholar
  7. 7.
    Shrestha, A., Cater-Steel, A., Toleman, M., Rout, T.: towards transparent and efficient process assessments for IT service Management. In: Mitasiunas, A., Rout, T., O’Connor, R.V., Dorling, A. (eds.) SPICE 2014. CCIS, vol. 477, pp. 165–176. Springer, Heidelberg (2014)CrossRefGoogle Scholar
  8. 8.
    CMMI, Standard CMMI® Appraisal Method for Process Improvement (SCAMPISM) A, Version 1.3: Method Definition Document. Software Engineering Institute: Carnegie Mellon University, MA, USA (2011)Google Scholar
  9. 9.
    MacDonald, I.: ITIL Process Assessment Framework. The Co-operative Financial Services: Manchester, UK (2010)Google Scholar
  10. 10.
    Deutskens, E., de Ruyter, K., Wetzels, M.: An Assessment of Equivalence Between Online and Mail Surveys in Service Research. Journal of Service Research 8(4), 346–355 (2006)CrossRefGoogle Scholar
  11. 11.
    Parasuraman, A., Zeithaml, V.A., Berry, L.L.: A Conceptual Model of Service Quality and its Implications for Future Research. Journal of Marketing 49(4), 41–50 (1985)CrossRefGoogle Scholar
  12. 12.
    Lepmets, M., Cater-Steel, A., Gacenga, F., Ras, E.: Extending the IT Service Quality Measurement Framework through a Systematic Literature Review. Journal of Service Science Research 4(1), 7–47 (2012)CrossRefGoogle Scholar
  13. 13.
    Jia, R., Reich, B.H.: IT Service Climate—An Essential Managerial Tool to Improve Client Satisfaction with IT Service Quality. Information Systems Management 28(2), 174–179 (2011)CrossRefGoogle Scholar
  14. 14.
    Spath, D., Bauer, W., Praeg, C.-P.: IT Service Quality Management: Assumptions, Frameworks and Effects on Business Performance. In: Quality Management for IT Services-Perspectives on Business and Process Performance. IGI Global, PA, pp. 1–21 (2011)Google Scholar
  15. 15.
    Edgeman, R.L., Bigio, D., Ferleman, T.: Six Sigma and Business Excellence: Strategic and Tactical Examination of IT Service Level Management at the Office of the Chief Technology Officer of Washington, DC. Quality and Reliability Engineering International 21(3), 257–273 (2005)CrossRefGoogle Scholar
  16. 16.
    Barafort, B., Di Renzo, B., Merlan, O.: Benefits resulting from the combined use of ISO/IEC 15504 with the information technology infrastructure library (ITIL). In: Oivo, M., Komi-Sirviö, M. (eds.) 4th International Conference on Product Focussed Software Development and Process Improvement (PROFES). Lecture Notes in Computer Science, vol. 2559, pp. 314–325. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  17. 17.
    Checkland, P., Scholes, J.: Soft Systems Methodology in Practice, 1st edn. J Wiley, Chichester (1990)Google Scholar
  18. 18.
    Hevner, A.R., March, S.T., Park, J., Ram, S.: Design Science in Information Systems Research. MIS Quarterly 28(1), 75–106 (2004)Google Scholar
  19. 19.
    Gregor, S., Jones, D.: The Anatomy of a Design Theory. Journal of the Association for Information Systems 8(5), 312–335 (2007)Google Scholar
  20. 20.
    Pries-Heje, J., Baskerville, R.: The Design Theory Nexus. MIS Quarterly 32(4), 731–755 (2008)Google Scholar
  21. 21.
    Sein, M.K., Henfridsson, O., Purao, S., Rossi, M., Lindgren, R.: Action Design Research. MIS Quarterly 35(1), 37–56 (2011)Google Scholar
  22. 22.
    Iivari, J.: A Paradigmatic Analysis of Information Systems As a Design Science. Scandinavian Journal of Information Systems 19(2) Article 5 (2007)Google Scholar
  23. 23.
    Walls, J.G., Widmeyer, G.R., El Sawy, O.A.: Building an Information System Design Theory for Vigilant EIS. Information Systems Research 3(1), 36–59 (1992)CrossRefGoogle Scholar
  24. 24.
    Van Aken, J.E.: The Nature of Organizing Design: Both Like and Unlike Material Object Design. Eindhoven Center for Innovation Studies (ECIS) (2006)Google Scholar
  25. 25.
    Carlsson, S.A., Henningsson, S., Hrastinski, S., Keller, C.: Socio-technical IS Design Science Research: Developing Design Theory for IS Integration Management. Information Systems and e-Business Management 9(1), 109–131 (2011)CrossRefGoogle Scholar
  26. 26.
    Venable, J., Pries-Heje, J., Baskerville, R.: A comprehensive framework for evaluation in design science research. In: Peffers, K., Rothenberger, M., Kuechler, B. (eds.) DESRIST 2012. LNCS, vol. 7286, pp. 423–438. Springer, Heidelberg (2012)Google Scholar
  27. 27.
    Peffers, K., Tuunanen, T., Rothenberger, M.A., Chatterjee, S.: A Design Science Research Methodology for Information Systems Research. Journal of Management Information Systems 24(3), 45–77 (2007)CrossRefGoogle Scholar
  28. 28.
    Pries-Heje, J., Baskerville, R., Venable, J.R.: Strategies for Design Science Research Evaluation. In: 16th European Conference on Information Systems. Galway, Ireland (2008)Google Scholar
  29. 29.
    Peffers, K., Tuunanen, T., Rothenberger, M.A., Chatterjee, S.: A Design Science Research Methodology for Information Systems Research. Journal of Management Information Systems 24(3), 45–77 (2008)CrossRefGoogle Scholar
  30. 30.
    ISO/IEC, ISO/IEC 25010:2011 – Systems and software engineering – Systems and software Quality Requirements and Evaluation (SQuaRE) - System and software quality models. International Organisation for Standardisation: Geneva, Switzerland (2011)Google Scholar
  31. 31.
    TRC, TRC Annual Report 2012/13. Toowoomba Regional Council: Toowoomba (2013)Google Scholar
  32. 32.
    ISO/IEC, ISO/IEC 33001:2015 – Information technology Process assessment – Concepts and terminology. International Organisation for Standardisation: Geneva, Switzerland (2015)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  1. 1.School of Management and EnterpriseUniversity of Southern QueenslandToowoombaAustralia

Personalised recommendations