Software quality assessment model: a systematic mapping study


Quality model is regarded as a well-accepted approach for assessing, managing and improving software product quality. There are three categories of quality models for software products, i.e., definition model, assessment model, and prediction model. Quality assessment model (QAM) is a metric-based approach to assess the software quality. It is typically regarded as of high importance for its clear method on how to assess a system. However, the current state-of-the-art in QAM research is under limited investigation. To address this gap, the paper provides an organized and synthesized summary of the current QAMs. In detail, we conduct a systematic mapping study (SMS) for structuring the relevant articles. We obtain a total of 716 papers from the five databases, and 31 papers are selected as relevant studies at last. In summary, our work focuses on QAMs from the following aspects: software metrics, quality factors, aggregation methods, evaluation methods and tool support. According to the analysis results, our work discovers five needs that researchers in this area should continue to address: (1) new method and criteria to tailor a quality framework (i.e., structure of software metrics and quality factors) according to different specifics, (2) systematic investigations on the effectiveness, strength and weakness of different aggregation methods to guide the method selection in different context, (3) more investigations on evaluating QAMs in the context of industrial cases, (4) further investigations or real-world case studies on the QAMs related tools, and (5) building a public and diverse software benchmark which can be adopted in different application context.

This is a preview of subscription content, access via your institution.


  1. 1

    ISO. Information technology, software product evaluation. ISO/IEC 14598–1. 1999.

  2. 2

    Bakota T, Hegedus P, Kortvelyesi P, et al. A probabilistic software quality model. In: Proceedings of the 27th IEEE International Conference on Software Maintenance, 2011. 243-252

  3. 3

    Boehm B W, Brown J R, Kaspar H. Characteristics of Software Quality. Amsterdam: North-Holland Publishing, 1978

  4. 4

    ISO. Software engineering-product quality. ISO/IEC 9126. 2001.

  5. 5

    ISO. Systems and software engineering-systems and software quality requirements and evaluation (SQuaRE)-system and software quality models. ISO/IEC 25010. 2011.

  6. 6

    Catal C, Diri B. A systematic review of software fault prediction studies. Expert Syst Appl, 2009, 36: 7346–7354

    Article  Google Scholar 

  7. 7

    Öztürk M M, Cavusoglu U, Zengin A. A novel defect prediction method for web pages using k-means++ Expert Syst Appl, 2015, 42: 6496–6506

    Article  Google Scholar 

  8. 8

    Deissenboeck F, Juergens E, Lochmann K, et al. Software quality models: purposes, usage scenarios and requirements. In: Proceedings of the ICSE Workshop on Software Quality, 2009. 9-14

  9. 9

    Dromey R G. A model for software product quality. IEEE Trans Softw Eng, 1995, 21: 146–162

    Article  Google Scholar 

  10. 10

    ISO. Systems and software engineering—vocabulary. ISO/IEC/IEEE 24765. 2010.

  11. 11

    Wagner S. Software Product Quality Control. Berlin: Springer, 2013

    Google Scholar 

  12. 12

    Chotjaratwanich U, Arpnikanondt C. A visualization technique for metrics-based hierarchical quality models. In: Proceedings of the 19th Asia-Pacific Software Engineering Conference (APSEC), 2012. 733-736

  13. 13

    Mordal-Manet K, Balmas F, Denier S, et al. The Squale model—a practice-based industrial quality model. In: Proceedings of the IEEE International Conference on Software Maintenance, 2009. 531-534

  14. 14

    Klas M, Heidrich J, Munch J, et al. CQML scheme: a classification scheme for comprehensive quality model landscapes. In: Proceedings of the EUROMICRO Conference on Software Engineering and Advanced Applications (SEAA), 2009. 243-250

  15. 15

    Montagud S, Abrahão S, Insfran E. A systematic review of quality attributes and measures for software product lines. Softw Qual J, 2012, 20: 425–486

    Article  Google Scholar 

  16. 16

    Riaz M, Mendes E, Tempero E. A systematic review of software maintainability prediction and metrics. In: Proceedings of the 3rd International Symposium on Empirical Software Engineering and Measurement, 2009. 367-377

  17. 17

    Febrero F, Calero C, Moraga M. A systematic mapping study of software reliability modeling. Inf Softw Tech, 2014, 56: 839–849

    Article  Google Scholar 

  18. 18

    Kitchenham B. What’s up with software metrics?—a preliminary mapping study. J Syst Softw, 2010, 83: 37–51

    Article  Google Scholar 

  19. 19

    Tomas P, Escalona M J, Mejias M. Open source tools for measuring the internal quality of Java software products. A survey. Comput Stand Inter, 2013, 36: 244–255

    Article  Google Scholar 

  20. 20

    Petersen K, Feldt R, Mujtaba S. Systematic mapping studies in software engineering. In: Proceedings of the International Conference on Evaluation and Assessment in Software Engineering (EASE), 2008. 68-77

  21. 21

    Yan M, Xia X, Zhang X H, et al. A systematic mapping study of quality assessment models for software products. In: Proceedings of the International Conference on Software Analysis, Testing and Evolution (SATE), 2017. 63-71

  22. 22

    Kitchenham B. Guidelines for Performing Systematic Literature Reviews in Software Engineering. EBSE Technical Report EBSE-2007-01. 2007

  23. 23

    Brereton P, Kitchenham B A, Budgen D, et al. Lessons from applying the systematic literature review process within the software engineering domain. J Syst Softw, 2007, 80: 571–583

    Article  Google Scholar 

  24. 24

    Chernyi A I. The ISI web of knowledge, a modern system for the information support of scientific research: a review. Sci Tech Inf Proc, 2009, 36: 351–358

    Article  Google Scholar 

  25. 25

    Webster J, Watson R T. Analyzing the past to prepare for the future: writing a literature review. Manag Inf Syst Quart, 2002, 26: 3

    Google Scholar 

  26. 26

    Kvam K, Lie R, Bakkelund D. Legacy system exorcism by Pareto’s principle. In: Proceedings of the Companion to the 20th Annual ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications, 2005. 250-256

  27. 27

    Morisio M, Stamelos I, Tsoukias A. Software product and process assessment through profile-based evaluation. Int J Soft Eng Knowl Eng, 2003, 13: 495–512

    Article  Google Scholar 

  28. 28

    Satrijandi N, Widyani Y. Efficiency measurement of java Android code. In: Proceedings of the International Conference on Data and Software Engineering, 2014

  29. 29

    Wagner S, Goeb A, Heinemann L, et al. Operationalised product quality models and assessment: the Quamoco approach. Inf Softw Tech, 2015, 62: 101–123

    Article  Google Scholar 

  30. 30

    Zheng X R, Martin P, Brohman K, et al. CLOUDQUAL: a quality model for cloud services. IEEE Trans Ind Inf, 2014, 10: 1527–1536

    Article  Google Scholar 

  31. 31

    Mayr A, Plosch R, Saft M. Objective safety compliance checks for source code. In: Proceedings of the 36th International Conference on Software Engineering, 2014. 115-124

  32. 32

    Gupta S, Singh H K, Venkatasubramanyam R D, et al. SCQAM: a scalable structured code quality assessment method for industrial software. In: Proceedings of the 22nd International Conference on Program Comprehension, 2014. 244-252

  33. 33

    Athanasiou D, Nugroho A, Visser J, et al. Test code quality and its relation to issue handling performance. IEEE Trans Softw Eng, 2014, 40: 1100–1125

    Article  Google Scholar 

  34. 34

    Srivastava S, Kumar R. Indirect method to measure software quality using CK-OO suite. In: Proceedings of the International Conference on Intelligent Systems and Signal Processing, 2013. 47-51

  35. 35

    Samarthyam G, Suryanarayana G, Sharma T. Midas: a design quality assessment method for industrial software. In: Proceedings of the International Conference on Software Engineering, 2013. 911-920

  36. 36

    Mayr A, Plosch R, Saft M. Objective measurement of safety in the context of IEC 61508–3. In: Proceedings of the EUROMICRO Conference on Software Engineering and Advanced Applications (SEAA), 2013. 45-52

  37. 37

    Mayr A, Plosch R, Klas M, et al. A comprehensive code-based quality model for embedded systems: systematic development and validation by industrial projects. In: Proceedings of the 23rd International Symposium on Software Reliability Engineering (ISSRE), 2012. 281-290

  38. 38

    Baggen R, Correia J P, Schill K, et al. Standardized code quality benchmarking for improving software maintainability. Softw Qual J, 2012, 20: 287–307

    Article  Google Scholar 

  39. 39

    Wagner S, Lochmann K, Heinemann L, et al. The Quamoco product quality modelling and assessment approach. In: Proceedings of the International Conference on Software Engineering, 2012. 1133-1142

  40. 40

    Challa J S, Paul A, Dada Y, et al. Integrated software quality evaluation: a fuzzy multi-criteria approach. J Inf Proc Syst, 2011, 7: 473–518

    Article  Google Scholar 

  41. 41

    Lan Y Q, Liu Y F, Kuang M X. Evaluate the quality of foundational software platform by Bayesian network. In: Proceedings of International Conference on Artificial Intelligence and Computational Intelligence, 2010

  42. 42

    Glott R, Groven A K, Haaland K, et al. Quality models for free/libre open source software towards the “silver bullet”? In: Proceedings of the 36th EUROMICRO Conference on Software Engineering and Advanced Applications (SEAA), 2010. 439-446

  43. 43

    Soley R M, Curtis B. The consortium for IT software quality. In: Proceedings of International Conference on Software Quality, 2013

  44. 44

    Letouzey J L, Coq T. The sqale analysis model: an analysis model compliant with the representation condition for assessing the quality of software source code. In: Proceedings of the International Conference on Advances in System Testing and Validation Lifecycle, 2010. 43-48

  45. 45

    Marchetto A. OQMw: an OO quality model for web applications. Tamkang J Sci Eng, 2009, 12: 459–470

    Google Scholar 

  46. 46

    Khomh F, Gueheneuc Y G. DEQUALITE: building design-based software quality models. In: Proceedings of the 15th Conference on Pattern Languages of Programs, 2008

  47. 47

    Li Z, Lin L, Hui G. 2-d software quality model and case study in software flexibility research. In: Proceedings of the International Conference on Intelligence for Modelling Control and Automation, 2008. 1147-1152

  48. 48

    Plosch R, Gruber H, Hentschel A, et al. The EMISQ method and its tool support-expert-based evaluation of internal software quality. Innov Syst Softw Eng, 2008, 4: 3–15

    Article  Google Scholar 

  49. 49

    Samoladas I, Gousios G, Spinellis D. The SQO-OSS quality model: measurement based open source software evaluation. In: Proceedings of IFIP International Conference on Open Source Systems, 2008. 237-248

  50. 50

    Ortega M, Pérez M, Rojas T. Construction of a systemic quality model for evaluating a software product. Softw Qual J, 2003, 11: 219–242

    Article  Google Scholar 

  51. 51

    Franch X, Carvallo J P. Using quality models in software package selection. IEEE Softw, 2003, 20: 34–41

    Article  Google Scholar 

  52. 52

    Bansiya J, Davis C G. A hierarchical model for object-oriented design quality assessment. IEEE Trans Softw Eng, 2002, 28: 4–17

    Article  Google Scholar 

  53. 53

    Blin M J, Tsoukias A. Multi-criteria methodology contribution to the software quality evaluation. Softw Qual J, 2001, 9: 113–132

    Article  Google Scholar 

  54. 54

    Pedrycz W, Peters J F, Ramanna S. Software quality measurement: concepts and fuzzy neural relational model. In: Proceedings of the IEEE International Conference on Fuzzy Systems Proceedings, 1998. 1026-1031

  55. 55

    Gomez O, Oktaba H, Piattini M, et al. A systematic review measurement in software engineering: state-of-the-art in measures. In: Poroceedings of International Conference on Software and Data Technologies, 2008. 165-176

  56. 56

    McCabe T J. A complexity measure. IEEE Trans Softw Eng, 1976, 2: 308–320

    MathSciNet  Article  MATH  Google Scholar 

  57. 57

    Halstead M H. Elements of Software Science. New York: Elsevier Science Inc., 1977

    Google Scholar 

  58. 58

    Chidamber S R, Kemerer C F. A metrics suite for object oriented design. IEEE Trans Softw Eng, 1994, 20: 476–493

    Article  Google Scholar 

  59. 59

    Martin R C. Agile Software Development: Principles, Patterns, and Practices. Upper Saddle River: Prentice Hall PTR, 2003

    Google Scholar 

  60. 60

    Fowler M. Refactoring: Improving the Design of Existing Code. Boston: Addison-Wesley Longman Publishing, 1999

    Google Scholar 

  61. 61

    Deissenboeck F, Wagner S, Pizka M, et al. An activity-based quality model for maintainability. In: Proceedings of the IEEE International Conference on Software Maintenance, 2007. 184-193

  62. 62

    Malhotra R. A systematic review of machine learning techniques for software fault prediction. Appl Soft Comput, 2015, 27: 504–518

    Article  Google Scholar 

  63. 63

    Ploesch R, Schuerz S, Koerner C. On the validity of the it-cisq quality model for automatic measurement of maintainability. In: Proceedings of the 39th Annual Computer Software and Applications Conference (COMPSAC), 2015. 326-334

  64. 64

    IEEE. IEEE recommended practice for software requirements specifications. In: Institute of Electrical and Electronics Engineers, 1998

  65. 65

    Barron F H, Barrett B E. Decision quality using ranked attribute weights. Manage Sci, 1996, 42: 1515–1523

    Article  MATH  Google Scholar 

  66. 66

    Correia J P, Kanellopoulos Y, Visser J. A survey-based study of the mapping of system properties to ISO/IEC 9126 maintainability characteristics. In: Proceedings of the IEEE International Conference on Software Maintenance, 2009. 61-70

  67. 67

    Ross T J. Fuzzy Logic with Engineering Applications. Hoboken: John Wiley and Sons, 2009

    Google Scholar 

  68. 68

    Heckman S, Williams L. A systematic literature review of actionable alert identification techniques for automated static code analysis. Inf Softw Tech, 2011, 53: 363–387

    Article  Google Scholar 

  69. 69

    Köksalan M, Ulu C. An interactive approach for placing alternatives in preference classes. Eur J Oper Res, 2003, 144: 429–439

    MathSciNet  Article  MATH  Google Scholar 

  70. 70

    Larichev O. An approach to ordinal classification problems. Int Trans Oper Res, 1994, 1: 375–385

    Article  MATH  Google Scholar 

  71. 71

    Mordal K, Anquetil N, Laval J, et al. Software quality metrics aggregation in industry. J Softw Evol Proc, 2013, 25: 1117–1135

    Article  Google Scholar 

  72. 72

    Jorgensen M, Shepperd M. A systematic review of software development cost estimation studies. IEEE Trans Softw Eng, 2007, 33: 33–53

    Article  Google Scholar 

Download references


This work was partially supported by National Key Research and Development Program of China (Grant No. 2018YFB1003904), National Natural Science Foundation of China (Grant No. 61602403), and China Postdoctoral Science Foundation (Grant No. 2017M621931).

Author information



Corresponding author

Correspondence to Xin Xia.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Yan, M., Xia, X., Zhang, X. et al. Software quality assessment model: a systematic mapping study. Sci. China Inf. Sci. 62, 191101 (2019).

Download citation


  • software quality
  • systematic mapping study
  • quality assessment model
  • aggregation method