Skip to main content
SpringerLink
Log in

CLOUD-QM: a quality model for benchmarking cloud-based enterprise information systems

  • Research
  • Published:
Software Quality Journal Aims and scope Submit manuscript

Abstract

Organizations are increasingly migrating from on-premise enterprise information systems (EIS) to cloud products due to cloud computing benefits, such as flexibility, elasticity, and on-demand service. However, identifying the most suitable option becomes challenging with the proliferation of Cloud-EIS solutions in the market. To address this challenge, this study introduces a novel quality model named Cloud-QM, based on ISO/IEC 250nn standards. It diagnoses the quality of Cloud-EIS products, benchmarks available options, and identifies the most suitable choice for the organization. Cloud-QM comprises 10 main dimensions, 33 sub-dimensions, and corresponding metrics for a systematic quality assessment. Furthermore, the practical use of Cloud-QM is illustrated through a case study that evaluates two substitute Cloud-EIS products. The results from the case study highlight the effectiveness of Cloud-QM in enabling decision-makers to delve into the quality dimensions and facilitate the selection of the most suitable product for their organizations. The main contributions are as follows: (1) proposing a comprehensive and hierarchically structured quality model for Cloud-EIS products; (2) offering a quantifiable and standardized assessment approach through a set of metrics for quality evaluation; and (3) demonstrating applicability and usability of Cloud-QM by benchmarking Cloud-EIS products.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Data availability

The datasets generated during and/or analyzed during the current study are available from the corresponding author on reasonable request. All data generated or analyzed during this study is included in this published article (and its supplementary information files). Data sharing not applicable to this article as no datasets were generated or analyzed during the current study.

References

  • Aletabi, H., & Abdallah, M. (2023). A Proposed Cloud Quality Model (IaaSQual) for “Infrastructure as a Service (IaaS)” from user’s perspective. International Conference on Information Technology (ICIT), 2023, 695–699. https://doi.org/10.1109/icit58056.2023.10225894

    Article  Google Scholar 

  • Alkhatib, A., Sabbagh, A. Al & Maraqa, R. (2021). Pubic Cloud computing: Big three vendors. 2021 International Conference on Information Technology, ICIT 2021 - Proceedings, 230–237. https://doi.org/10.1109/ICIT52682.2021.9491680

  • Alvaro, A., Santana de Almeida, E., & Romero de Lemos Meira, S. (2010). A software component quality framework. ACM SIGSOFT Software Engineering Notes, 35(1), 1–18.

    Article  Google Scholar 

  • Bayar, A., Şener, U., Kayabay, K., & Eren, P. E. (2023). Edge computing applications in industrial IoT: A literature review. In J. Á. Bañares, J. Altmann, O. Agmon Ben-Yehuda, K. Djemame, V. Stankovski, & B. Tuffin (Eds.), Economics of grids, clouds, systems, and services. GECON 2022. Lecture Notes in Computer Science. (Vol. 13430). Cham: Springer. https://doi.org/10.1007/978-3-031-29315-3_11

    Chapter  Google Scholar 

  • Behkamal, B., Kahani, M., & Akbari, M. K. (2009). Customizing ISO 9126 quality model for evaluation of B2B applications. Information and Software Technology, 51(3), 599–609.

    Article  Google Scholar 

  • Benlian, A., Koufaris, M., & Hess, T. (2012). Service Quality in Software-as-a-Service: Developing the SaaS-Qual measure and examining its role in usage continuance. Journal of Management Information Systems, 28(3), 85–126. https://doi.org/10.2753/MIS0742-1222280303

    Article  Google Scholar 

  • Boehm, B. W., Brown, J. R., Hans, K., Lipow, M., & MacLeod, G. (1978). Merritt.: Characteristics of software quality. Elsevier.

  • Bohn, R. B., Messina, J., Liu, F., Tong, J., & Mao, J. (2011). NIST cloud computing reference architecture. Proceedings - 2011 IEEE World Congress on Services. SERVICES, 2011, 594–596. https://doi.org/10.1109/SERVICES.2011.105

    Article  Google Scholar 

  • Bukhari, Z., Yahaya, J., & Deraman, A. (2019). Metric-based measurement and selection for software product quality assessment: Qualitative expert interviews. International Journal of Advanced Computer Science and Applications, 10(7), 223–231.

    Article  Google Scholar 

  • Bumpus, W. (2013). NIST Cloud Computing Standards Roadmap. In NIST Cloud Computing Standards. https://doi.org/10.6028/NIST.SP.500-291r2

    Article  Google Scholar 

  • Buonanno, G., Faverio, P., Pigni, F., Ravarini, A., Sciuto, D., & Tagliavini, M. (2005). Factors affecting ERP system adoption: A comparative analysis between SMEs and large companies. Journal of Enterprise Information Management, 18(4), 384–426. https://doi.org/10.1108/17410390510609572

    Article  Google Scholar 

  • Cavano, J. P., & McCall, J. A. (1978). A framework for the measurement of software quality. Proceedings of the Software Quality Assurance Workshop on Functional and Performance Issues, 133–139. https://doi.org/10.1145/800283.811113

  • Côté, M. A., Suryn, W., & Georgiadou, E. (2007). In search for a widely applicable and accepted software quality model for software quality engineering. Software Quality Journal, 15(4), 401–416. https://doi.org/10.1007/s11219-007-9029-0

    Article  Google Scholar 

  • Dadhich, M., Rathore, V. S., & Vyas, S. (2019). Essential parameters of ASMAN framework to compare SSPs in Cloud computing. In 2019 Amity International Conference on Artificial Intelligence (AICAI), 778–781.

  • Dhamdhere, S. N., (Ed.). (2013). Cloud computing and virtualization technologies in libraries. IGI Global.

  • Dromey, R. G. (2011). A model for software product quality. Software Process Improvement, 21(2), 269–285. https://doi.org/10.1109/9781118156667.ch6

    Article  Google Scholar 

  • Enríquez, J. G., Sánchez-Begines, J. M., Domínguez-Mayo, F. J., García-García, J. A., & Escalona, M. J. (2019). An approach to characterize and evaluate the quality of Product Lifecycle Management Software Systems. Computer Standards and Interfaces, 61(May 2018), 77–88. https://doi.org/10.1016/j.csi.2018.05.003

  • Garg, S. K., Versteeg, S., & Buyya, R. (2011). SMICloud: A framework for comparing and ranking cloud services. Proceedings - 2011 4th IEEE International Conference on Utility and Cloud Computing, UCC 2011, Vm, 210–218. https://doi.org/10.1109/UCC.2011.36

  • Gartner. (2023). Gartner forecasts worldwide public Cloud end-user spending to reach nearly $600 Billion in 2023. https://www.gartner.com/en/newsroom/press-releases/2023-04-19-gartner-forecasts-worldwide-public-cloud-end-user-spending-to-reach-nearly-600-billion-in-2023

  • Henrique, C., & Raj, B. (2022). Top four trends are shaping the future of public Cloud. https://www.gartner.com/en/doc/740943-top-four-trends-are-shaping-the-future-of-public-cloud

  • Hourani, H., & Abdallah, M. (2018). Cloud computing: Legal and security issues. 2018 8th International Conference on Computer Science and Information Technology, CSIT 2018, 13–16. https://doi.org/10.1109/CSIT.2018.8486161

  • IEEE. (1990). IEEE standard glossary of software engineering terminology. The Office, 121990(1), 1. https://doi.org/10.1109/IEEESTD.1990.101064

    Article  Google Scholar 

  • ISO/IEC 9126–1. (2001). Software engineering-software product quality-part 1: Quality model.

  • ISO/IEC 9126–2. (2003). Software engineering-product quality-part 2: External metrics.

  • ISO/IEC 25000. (2014). Systems and software engineering – Systems and software Quality Requirements and Evaluation (SQuaRE) – System and software quality models.

  • ISO/IEC 25010. (2011a). BSI Standards Publication Systems and software engineering — Systems and software quality requirements and evaluation (SQuaRE ) — System and software quality models. In BSI Standards Publication.

  • ISO/IEC 25010. (2011b). Systems and software engineering — Systems and software Quality Requirements and Evaluation ( SQuaRE ) — System and software quality models. In BSI Standards Publication.

  • ISO/IEC 25023. (2016). Systems and software engineering — Systems and software Quality Requirements and Evaluation ( SQuaRE ) — Measurement of system and software product quality. In BSI Standards Publication.

  • ISO/IEC 25040. (2011). BSI Standards Publication Systems and software engineering — Systems and software Quality Requirements and Evaluation (SQuaRE ) — Evaluation process. In BSI Standards Publication.

  • ISO/IEC 270017 Security Issues for Cloud Services. (2015). Information Technology –Security techniques- Code of practice for information security controls based on ISO/IEC 27002 for cloud services.

  • ISO/IEC 270018 Privacy Issues for Cloud Services. (2014). Information Technology –Security techniques- Code of practice for protection of personally identifiable information (PII) in public clouds acting as PII processors.

  • Karim, R., Ding, C., & Miri, A. (2013). An end-to-end QoS mapping approach for Cloud service selection. IEEE Ninth World Congress on Services, 2013, 341–348. https://doi.org/10.1109/SERVICES.2013.71

    Article  Google Scholar 

  • Kaynama, S. A., & Black, C. I. (2000). A proposal to assess the service quality of online travel agencies: An exploratory study. Journal of Professional Services Marketing, 21(1), 63–88. https://doi.org/10.1300/J090v21n01_05

    Article  Google Scholar 

  • Khosravi, K., & Guéhéneuc, Y. G. (2004). A quality model for design patterns. In In German Industry Standard (Vol. 12, Issue August). http://www-etud.iro.umontreal.ca/~ptidej/yann-gael/Work/Publications/Documents/041021+Kashayar+Khosravi+Technical+Report.doc.pdf

  • Koç, B., Şener, U., & Eren, P. E. (2022). Determinative factors of cloud computing adoption in government organizations. 2022 3rd International Informatics and Software Engineering Conference (IISEC) (pp. 1–6). IEEE. https://doi.org/10.1109/IISEC56263.2022.9998286

    Chapter  Google Scholar 

  • Koteska, B., Mishev, A., & Pejov, L. (2018). Quantitative measurement of scientific software quality: Definition of a novel quality model. International Journal of Software Engineering and Knowledge Engineering, 28(3), 407–425. https://doi.org/10.1142/S0218194018500146

    Article  Google Scholar 

  • Kumar, P. (2012). Aspect-oriented software quality model: The AOSQ model. Advanced Computing: An International Journal (ACIJ), 3(2), 105–118.

    Google Scholar 

  • Linstone, Harold A. & Turoff, M. (1975). The Delphi method: Techniques and applications, AddisonWesley. Reading, MA.

  • Loiacono, E. T., Watson, R. T., & Goodhue, D. L. (2007). WebQual: An instrument for consumer evaluation of web sites. International Journal of Electronic Commerce, 11(3), 51–87. https://doi.org/10.2753/JEC1086-4415110302

    Article  Google Scholar 

  • Loo, I. De., Bots, J., Louwrink, E., Meeuwsen, D., & Moorsel, P. Van. (2010). The effects of ERP-implementations on organizational benefits in small and medium-sized enterprises in the Netherlands. 8th International Conference on Enterprise Systems Accounting and Logistics, July, 11–12.

  • Ma, Q., Pearson, J. M., & Tadisina, S. (2005). An exploratory study into factors of service quality for application service providers. Information & Management, 42(8), 1067–1080. https://doi.org/10.1016/j.im.2004.11.007

    Article  Google Scholar 

  • Martin, P., & Brohman, K. (2014). CLOUDQUAL: A quality model for Cloud services. IEEE Transactions on Industrial Informatics, 10(2), 1527–1536. https://doi.org/10.1109/TII.2014.2306329

    Article  Google Scholar 

  • Nguyen, K., Wang, K., Bu, Y., Fang, L., & Xu, G. (2018). Understanding and combating memory bloat in managed data-intensive systems. ACM Transactions on Software Engineering and Methodology, 26(4), 1–41. https://doi.org/10.1145/3162626

    Article  Google Scholar 

  • Parasuraman, A. (1995). Measuring and monitoring service quality. Understanding services management.

  • Ramchand, K., Baruwal Chhetri, M., & Kowalczyk, R. (2021). Enterprise adoption of cloud computing with application portfolio profiling and application portfolio assessment. Journal of Cloud Computing, 10(1), 1. https://doi.org/10.1186/s13677-020-00210-w

    Article  Google Scholar 

  • Rawashdeh, A., & Matalkah, B. (2006). A new software quality model for evaluating COTS components. Journal of Computer Science, 2(4), 373–381.

    Article  Google Scholar 

  • Samoladas, I., Gousios, G., Spinellis, D., & Stamelos, I. (2008). The SQO-OSS quality model: Measurement based open source software evaluation. Open source development, communities and quality: IFIP 20th World Computer Congress, Working Group 2.3 on Open Source Software, September 7-10, 2008, Milano, Italy 4 (pp. 237–248). US: Springer.

    Chapter  Google Scholar 

  • Sandhu, R., Reich, J., Wolff, T., Krishnan, R., & Zachry, J. (2010). Towards a discipline of mission-aware Cloud computing categories and subject descriptors. Proceedings of the 2010 ACM Workshop on Cloud Computing Security Workshop, 13–17. https://doi.org/10.1145/1866835.1866839

  • Seffah, A., Kececi, N., & Donyaee, M. (2001). QUIM: A framework for quantifying usability metrics in software quality models. Proceedings - 2nd Asia-Pacific Conference on Quality Software, APAQS 2001, 2, 311–318. https://doi.org/10.1109/APAQS.2001.990036

  • Şener, U., Gökalp, E., & Eren, P. E. (2016). Cloud-based enterprise information systems: Determinants of adoption in the context of organizations. In International Conference on Information and Software Technologies (pp. 53–66). Springer, Cham. https://doi.org/10.1007/978-3-319-46254-7_5

  • Şener, U., Gökalp, E., & Eren, P. E. (2017). ClouDSS: A decision support system for cloud service selection. Economics of grids, clouds, systems, and services: 14th International Conference, GECON 2017, Biarritz, France, September 19-21, 2017, Proceedings 14 (pp. 249–261). Springer International Publishing. https://doi.org/10.1007/978-3-319-68066-8_19

    Chapter  Google Scholar 

  • Şener, U., Gökalp, E., & Erhan Eren, P. (2023). Intelligent digital transformation strategy management: Development of a measurement framework. In C. Kahraman & E. Haktanır (Eds.), Intelligent systems in digital transformation. Lecture notes in networks and systems. (Vol. 549). Cham: Springer. https://doi.org/10.1007/978-3-031-16598-6_4

    Chapter  Google Scholar 

  • Shepperd, M., Abreu, F. B. E., & Perez-Castillo, R. (2022). Special issue on information systems quality management in practice. Software Quality Journal, 281–282. https://doi.org/10.1007/s11219-022-09584-3

  • Stieninger, M., Nedbal, D., Wetzlinger, W., Wagner, G., & Erskine, M. A. (2014). Impacts on the organizational adoption of Cloud computing: A reconceptualization of influencing factors. Procedia Technology, 16, 85–93. https://doi.org/10.1016/j.protcy.2014.10.071

    Article  Google Scholar 

  • Sultan, N. A. (2011). Reaching for the “cloud”: How SMEs can manage. International Journal of Information Management, 31(3), 272–278. https://doi.org/10.1016/j.ijinfomgt.2010.08.001

    Article  Google Scholar 

  • Wilson, D. N., & Hall, T. (1998). Perceptions of software quality : A pilot study. Software Quality Journal, 7(1), 67–75. https://doi.org/10.1023/b:sqjo.0000042060.88173.fe

    Article  Google Scholar 

  • Yılmaz, N., & Kolukısa Tarhan, A. (2022). Quality evaluation models or frameworks for open source software: A systematic literature review. Journal of Software: Evolution and Process, 34(6), 1–34. https://doi.org/10.1002/smr.2458

    Article  Google Scholar 

  • Yin, R. K. (2013). Case study research: Design and methods. Sage publications.

    Google Scholar 

  • Zeithaml, V. A., Parasuraman, A., & Malhotra, A. (2002). Service quality delivery through web sites: A critical review of extant knowledge. Journal of the Academy of Marketing Science, 30(4), 362375.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Graduate School of Informatics, Middle East Technical University, Ankara, Turkey

    Umut Şener & P. Erhan Eren

  2. The Department of Computer Engineering, Hacettepe University, Ankara, Turkey

    Ebru Gökalp

Authors
  1. Umut Şener
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ebru Gökalp
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. P. Erhan Eren
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Umut Şener wrote the main manuscript text and prepared Fig. 2. Ebru Gökalp prepared Figs. 1 and 3. All authors conceptualized, reviewed and edited the manuscript.

Corresponding author

Correspondence to Umut Şener.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendices

Appendix 1. Definitions of quality and sub-quality dimensions of the ISO 25010 Software Quality Model

Fig. 4
figure 4

Quality dimensions of the ISO 25010 Software Quality Model (2011a)

Full size image
Fig. 5
figure 5

Sub-quality dimensions of the ISO 25010 Software Quality Model (2011b)

Full size image

Appendix 2. Mapping of quality dimensions in the existing models

Table 6 Tracing quality dimensions in the existing models
Full size table

Appendix 3. Metrics of the Cloud-QM

Table 7 Metrics for functionality (part 1)
Full size table
Table 8 Metrics for functionality (Cont.)
Full size table
Table 9 Metrics for reliability
Full size table
Table 10 Metrics for efficiency
Full size table
Table 11 Metrics for maintainability
Full size table
Table 12 Metrics for portability
Full size table
Table 13 Metrics for elasticity
Full size table
Table 14 Metrics for security & privacy
Full size table
Table 15 Metrics for policy & regulations
Full size table
Table 16 Metrics for customer service quality
Full size table

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Şener, U., Gökalp, E. & Eren, P.E. CLOUD-QM: a quality model for benchmarking cloud-based enterprise information systems. Software Qual J (2024). https://doi.org/10.1007/s11219-024-09669-1

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11219-024-09669-1

Keywords

Search

Navigation