Skip to main content
SpringerLink
Log in

A systematic review of quality attributes and measures for software product lines

  • Published:
Software Quality Journal Aims and scope Submit manuscript

Abstract

It is widely accepted that software measures provide an appropriate mechanism for understanding, monitoring, controlling, and predicting the quality of software development projects. In software product lines (SPL), quality is even more important than in a single software product since, owing to systematic reuse, a fault or an inadequate design decision could be propagated to several products in the family. Over the last few years, a great number of quality attributes and measures for assessing the quality of SPL have been reported in literature. However, no studies summarizing the current knowledge about them exist. This paper presents a systematic literature review with the objective of identifying and interpreting all the available studies from 1996 to 2010 that present quality attributes and/or measures for SPL. These attributes and measures have been classified using a set of criteria that includes the life cycle phase in which the measures are applied; the corresponding quality characteristics; their support for specific SPL characteristics (e.g., variability, compositionality); the procedure used to validate the measures, etc. We found 165 measures related to 97 different quality attributes. The results of the review indicated that 92% of the measures evaluate attributes that are related to maintainability. In addition, 67% of the measures are used during the design phase of Domain Engineering, and 56% are applied to evaluate the product line architecture. However, only 25% of them have been empirically validated. In conclusion, the results provide a global vision of the state of the research within this area in order to help researchers in detecting weaknesses, directing research efforts, and identifying new research lines. In particular, there is a need for new measures with which to evaluate both the quality of the artifacts produced during the entire SPL life cycle and other quality characteristics. There is also a need for more validation (both theoretical and empirical) of existing measures. In addition, our results may be useful as a reference guide for practitioners to assist them in the selection or the adaptation of existing measures for evaluating their software product lines.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

Notes

  1. In this study, we use the terms measure and metric as synonyms since the ISO/IEC 9126 (2000) standard uses the term metric to refer to the defined measurement method and the measurement scale whereas the new standard for Software Product Quality Requirements and Evaluation (SQuaRE) uses the term measure to refer to a variable to which a value is assigned as the result of measurement.

References

  • Abdelmoez, W., Nassar, D. M., Shereschevsky, M., Gradetsky, N., Gunnalan, R., Ammar, H. H., et al. (2004). Error propagation in software architectures. In 10th international symposium on software metrics (METRICS), Chicago, Illinois, USA.

  • Ajila, S. A., & Dumitrescu, R. T. (2007). Experimental use of code delta, code churn, and rate of change to understand software product line evolution. Journal of Systems and Software, 80, 74–91.

    Article  Google Scholar 

  • Aldekoa, G., Trujillo, S., Sagardui, G., & Díaz, O. (2006). Experience measuring maintainability in software product lines. In XV Jornadas de Ingeniería del Software y Bases de Datos (JISBD). Barcelona.

  • Aldekoa, G., Trujillo, S., Sagardui, G., & Díaz, O. (2008). Quantifying maintanibility in feature oriented product lines, Athens, Greece, pp. 243–247.

  • Alves de Oliveira Junior, E., Gimenes, I. M. S., & Maldonado, J. C. (2008). A metric suite to support software product line architecture evaluation. In XXXIV Conferencia Latinamericana de Informática (CLEI), Santa Fé, Argentina, pp. 489–498.

  • Alves, V., Niu, N., Alves, C., & Valença, G. (2010). Requirements engineering for software product lines: A systematic literature review. Information & Software Technology, 52(8), 806–820.

    Article  Google Scholar 

  • Bosch, J. (2000). Design and use of software architectures: Adopting and evolving a product line approach. USA: ACM Press/Addison-Wesley Publishing Co.

    Google Scholar 

  • Briand, L. C., Differing, C. M., & Rombach, D. (1996a). Practical guidelines for measurement-based process improvement. Software Process-Improvement and Practice, 2, 253–280.

    Article  Google Scholar 

  • Briand, L. C., Morasca, S., & Basili, V. R. (1996b). Property based software engineering measurement. IEEE Transactions on Software Eng., 22(1), 68–86.

    Article  Google Scholar 

  • Calero, C., Ruiz, J., & Piattini, M. (2005). Classifying web metrics using the web quality model. Online Information Review, 29(3): 227–248.

    Google Scholar 

  • Chen, L., Ali Babar, M., & Ali, N. (2009). Variability management in software product lines: A systematic review. In 13th international software product lines conferences (SPLC), San Francisco, USA.

  • Clements, P., & Northrop, L. (2002). Software product lines. 2003. Software product lines practices and patterns. Boston, MA: Addison-Wesley.

    Google Scholar 

  • Crnkovic, I., & Larsson, M. (2004). Classification of quality attributes for predictability in component-based systems. Journal of Econometrics, pp. 231–250.

  • Conference Rankings of Computing Research and Education Association of Australasia (CORE). (2010). Available in http://core.edu.au/index.php/categories/conference%20rankings/1.

  • Davis, A., Dieste, Ó., Hickey, A., Juristo, N., & Moreno, A. M. (2006). Effectiveness of requirements elicitation techniques: Empirical results derived from a systematic review. In 14th IEEE international conference requirements engineering, pp. 179–188.

  • de Souza Filho, E. D., de Oliveira Cavalcanti, R., Neiva, D. F. S., Oliveira, T. H. B., Barachisio Lisboa, L., de Almeida E. S., & de Lemos Meira, S. R. (2008). Evaluating domain design approaches using systematic review. In 2nd European conference on software architecture, Cyprus, pp. 50–65.

  • Ejiogu, L. (1991). Software engineering with formal metrics. QED Publishing.

  • Engström, E., & Runeson, P. (2011). Software product line testing—A systematic mapping study. Information & Software Technology, 53(1), 2–13.

    Article  Google Scholar 

  • Etxeberria, L., Sagarui, G., & Belategi, L. (2008). Quality aware software product line engineering. Journal of the Brazilian Computer Society, 14(1), Campinas Mar.

  • Ganesan, D., Knodel, J., Kolb, R., Haury, U., & Meier, G. (2007). Comparing costs and benefits of different test strategies for a software product line: A study from Testo AG. In 11th international software product line conference, Kyoto, Japan, pp. 74–83, September 2007.

  • Gómez, O., Oktaba, H., Piattini, M., & García, F. (2006). A systematic review measurement in software engineering: State-of-the-art in measures. In First international conference on software and data technologies (ICSOFT), Setúbal, Portugal, pp. 11–14.

  • IEEE standard for a software quality metrics methodology, IEEE Std 1061-1998, 1998.

  • Inoki, M., & Fukazawa, Y. (2007). Software product line evolution method based on Kaizen approach. In 22nd annual ACM symposium on applied computing, Korea.

  • Insfran, E., & Fernandez, A. (2008). A systematic review of usability evaluation in Web development. 2nd international workshop on web usability and accessibility (IWWUA’08), New Zealand, LNCS 5176, Springer, pp. 81–91.

  • ISO/IEC 25010. (2008). Systems and software engineering. Systems and software Quality Requirements and Evaluation (SQuaRE). System and software quality models.

  • ISO/IEC 9126. (2000). Software engineering. Product Quality.

  • Johansson, E., & Höst, R. (2002). Tracking degradation in software product lines through measurement of design rule violations. In 14th International conference on software engineering and knowledge engineering, Ischia, Italy, pp. 249–254.

  • Journal Citation Reports of Thomson Reuters. (2010). Available in http://thomsonreuters.com/products_services/science/science_products/a-z/journal_citation_reports/.

  • Khurum, M., & Gorschek, T. (2009). A systematic review of domain analysis solutions for product lines. The Journal of Systems and Software.

  • Kim, T., Ko, I. Y., Kang, S. W., & Lee, D. H. (2008). Extending ATAM to assess product line architecture. In 8th IEEE international conference on computer and information technology, pp. 790–797.

  • Kitchenham, B. (2007). Guidelines for performing systematic literature reviews in software engineering. Version 2.3, EBSE Technical Report, Keele University, UK.

  • Kitchenham, B., Pfleeger, S., & Fenton, N. (1995). Towards a framework for software measurement validation. IEEE Transactions on Software Engineering, 21(12).

  • Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159–174.

    Article  MathSciNet  MATH  Google Scholar 

  • Mendes, E. (2005). A systematic review of Web engineering research. International symposium on empirical software engineering. Noosa Heads, Australia.

  • Meyer, M. H., & Dalal, D. (2002). Managing platform architectures and manufacturing processes for non assembled products. Journal of Product Innovation Management, 19(4), 277–293.

    Article  Google Scholar 

  • Montagud, S., & Abrahão, S. (2009). Gathering Current knowledge about quality evaluation in software product lines. In 13th international software product lines conferences (SPLC), San Francisco, USA.

  • Montagud, S., & Abrahão, S. (2009). A SQuaRE-bassed quality evaluation method for software product lines. Master’s thesis, December 2009 (in Spanish).

  • Needham, D., & Jones, S. (2006). A software fault tree metric. In 22nd international conference on software maintenance (ICSM), Philadelphia, Pennsylvania, USA.

  • Niemelä, E., & Immonen, A. (2007). Capturing quality requirements of product family architecture. Information and Software Technology, 49(11–12), 1107–1120.

    Article  Google Scholar 

  • Odia, O. E. (2007). Testing in software product lines. Master Thesis Software Engineering of School of Engineering, Bleking Institute of Technology. Thesis no. MSE-2007:16, Sweden.

  • Olumofin, F. G., & Mišić, V. B. (2007). A holistic architecture assessment method for software product lines. Information and Software Technology, 49, 309–323.

    Article  Google Scholar 

  • Pérez Lamancha, B., Polo Usaola, M., & Piattini Velthius, M. (2009). Software product line testing—a systematic review. ICSOFT, (1), 23–30.

  • Poels, G., & Dedene, G. (2000). Distance-based software measurement: necessary and sufficient properties for software measures. Information and Software Technology, 42(I), 35–46.

    Article  Google Scholar 

  • Prehofer, C., van Gurp, J., & Bosch, J. (2008). Compositionality in software platforms. In Emerging methods, technologies and process management in software engineering. Wiley.

  • Rahman, A. (2004). Metrics for the structural assessment of product line architecture. Master Thesis on Software Engineering, Thesis no. MSE-2004:24. School of Engineering, Blekinge Institute of Technology, Sweden.

  • Sethi, K., Cai, Y., Wong, S., Garcia, A., & Sant’Anna, C. (2009). From retrospect to prospect: Assessing modularity and stability from software architecture. Joint working IEEE/IFIP conference on software architecture, 2009 & European conference on software architecture. WICSA/ECSA.

  • Shaik, I., Abdelmoez, W,. Gunnalan, R., Shereshevsky, M., Zeid, A., Ammar, H. H., et al. (2005). Change propagation for assessing design quality of software architectures. 5th working IEEE/IFIP conference on software architecture (WICSA’05).

  • Siegmund, N., Rosenmüller, M., Kuhlemann, M., Kästner, C., & Saake, G. (2008). Measuring non-functional properties in software product lines for product derivation. In 15th Asia-Pacific software engineering conference, Beijing, China.

  • Sun Her, J., Hyeok Kim, J., Hun Oh, S., Yul Rhew, S., & Dong Kim, S. (2007). A framework for evaluating reusability of core asset in product line engineering. Information and Software Technology, 49, 740–760.

    Article  Google Scholar 

  • Svahnberg, M., & Bosch, J. (2000). Evolution in software product lines. In 3rd international workshop on software architectures for products families (IWSAPF-3). Las Palmas de Gran Canaria.

  • Van der Hoek, A., Dincel, E., & Medidović, N. (2003). Using services utilization metrics to assess the structure of product line architectures. In 9th international software metrics symposium (METRICS), Sydney, Australia.

  • Van der Linden, F., Schmid, K., & Rommes, E. (2007). Software product lines in action. Springer.

  • Whitmire, S. (1997). Object oriented design measurement. John Wiley & Sons.

  • Wnuk, K., Regnell, B., & Karlsson, L. (2009). What happened to our features? Visualization and understanding of scope change dynamics in a large-scale industrial setting. In 17th IEEE international requirements engineering conference.

  • Yoshimura, K., Ganesan, D., & Muthig, D. (2006). Assessing merge potential of existing engine control systems into a product line. In International workshop on software engineering for automative systems, Shangai, China, pp. 61–67.

  • Zhang, T., Deng, L., Wu, J., Zhou, Q., & Ma, C. (2008). Some metrics for accessing quality of product line architecture. In International conference on computer science and software engineering (CSSE), Wuhan, China, pp. 500–503.

Download references

Acknowledgments

This research has been funded by the Spanish Ministry of Science and Innovation under the MULTIPLE (Multimodeling Approach For Quality-Aware Software Product Lines) project with ref. TIN2009-13838.

Author information

Authors and Affiliations

  1. Department of Computer Science and Computation, Universitat Politècnica de València, Camino de Vera, s/n, 46022, Valencia, Spain

    Sonia Montagud, Silvia Abrahão & Emilio Insfran

Authors
  1. Sonia Montagud
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Silvia Abrahão
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Emilio Insfran
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sonia Montagud.

Appendices

Appendix 1

List of papers selected in the systematic review.

S01:

Abdelmoez, W., Nassar, D.M., Shereschevsky, M., Gradetsky, N., Gunnalan, R., Ammar, H.H., Yu, B., Mili, A.: Error Propagation In Software Architectures. In 10th International Symposium on Software Metrics (METRICS), Chicago, Illinois, USA, 2004

S02:

Ajila, S. A., Dumitrescu, R. T.: Experimental use of code delta, code churn, and rate of change to understand software product line evolution. Journal of Systems and Software 80, pp. 74–91, 2007

S03:

Aldekoa, G., Trujillo, S., Sagardui, G., Díaz, O.: Quantifying Maintainability in Feature Oriented Product Lines. In 12th European Conference on Software Maintenance and Reengineering, Athens, Greece, pp. 243–247, 2008

S04:

Alves de Oliveira Junior, E., Gimenes, I. M. S., Maldonado, J. C.: A Metric Suite to Support Software Product Line Architecture Evaluation. In XXXIV Conferencia Latinamericana de Informática (CLEI), Santa Fé, Argentina, pp. 489–498, 2008

S05:

Axelsson, J. Evolutionary Architecting of Embedded Automotive Product Lines: An Industrial Case Study. In Joint Working IEEE/IFIP Conference on Software Architecture 2009 and European Conference on Software Architecture 2009, WICSA/ECSA 2009, Cambridge, UK, 14–17 September 2009

S06:

Dincel, E., Medvidovic, N., Van der Hoek, A.: Measuring Product Line Architectures. In 4th International Workshop on Product Family Engineering (PFE), Bilbao, Spain, 2001

S07:

Etxeberria, L., and Sagardui, G. Product-Line Architecture: New Issues for Evaluation. Software Product Lines Conference, 2005

S08:

Gallina, B. and Guelfi, N. A Product Line Perspective for Quality Reuse of Development Framework for Distributed Transactional Applications. In International Computer Software and Applications Conference, 2008

S09:

Gallina, B., and Guelfi, N. A Template for Requirement Elicitation of Dependable Product Lines. In International Workshop on Requirements Engineering: Foundation for Software Quality (RefSQ), 2007

S10:

Ganesan, D., Knodel, J., Kolb, R., Haury, U., Meier, G.: Comparing Costs and Benefits of Different Test Strategies for a Software Product Line: A Study from Testo AG. In 11th International Software Product Line Conference, Kyoto, Japan, pp. 74–83, September 2007

S11:

Gannod, G.C., and Lutz, R.R. An Approach to Architectural Analysis of Product Lines. In 22nd International Conference on Software Engineering, 2000, pp. 548–557

S12:

Geppert, B., Weiss, D.M. Goal-Oriented Assessment of Product-Line Domains. International Software Metrics Symposium (METRICS’03), 2003

S13:

Hwan, S., Kim, J., Kim, J. An Elicitation Approach of Measurement Indicator Based on Product Line Context. Proceedings of the Fourth International Conference on Software Engineering Research, 2006,

S14:

Inoki, M., Fukazawa, Y.: Software Product Line Evolution Method Based on Kaizen Approach. In 22nd Annual ACM Symposium on Applied Computing, Korea, 2007

S15:

Johansson, E., Höst, R.: Tracking Degradation in Software Product Lines through Measurement of Design Rule Violations. In 14th International Conference on Software Engineering and Knowledge Engineering, Ischia, Italy, pp. 249–254, 2002

S16:

Kim, T., Ko, I.Y., Kang, S.W., and Lee, D.H. “Extending ATAM to assess product line architecture”. In 8th IEEE International Conference on Computer and Information Technology, pp. 790–797, 2008

S17:

Land, R., Alvaro, A., Crnkovic, I. Towards Efficient Software Component Evaluation: An Examination of Component Selection and Certification. 34th Euromicro Conference Software Engineering and Advanced Applications, 2008

S18:

Lin, Y., Ye, H., and Tang, J. Measurement of the Complexity of Variation Points in Software Product Lines. World Congress on Software Engineering, 2009

S19:

Lutz, R.R., Gannod, G.C. Analysis of a software product line architecture: an experience report. Journal of Systems and Software 66 (2003) 253–267

S20:

Mellado, D., Rodríguez, J., Fernández-Medina, E., and Piattini, M. Automated Support for Security Requirements Engineering in Software Product Line Domain Engineering. International Conference on Availability, Reliability and Security, 2009

S21:

Meyer, M. H., and Dalal, D. Managing platform architectures and manufacturing processes for non assembled products. Journal of Product Innovation Management. Volume 19, Issue 4, pages 277–293, July 2002

S22:

Misic, V. B. Measuring the Coherence of SoftwareProduct Line Architectures. technical report TR 06/03, Department of Computer Science, University of Manitoba, June 2006

S23:

Mustapic, G., Andersson, J., Norström, C., and Wall, A. A Dependable Open Platform for Industrial Robotics – A Case Study. In Architecting Dependable Systems II. Lecture Notes in Computer Science, 2004, Volume 3069/2004, 307–329

S24:

Needham, D., Jones, S.: A Software Fault Tree Metric. In 22nd International Conference on Software Maintenance (ICSM), Philadelphia, Pennsylvania, USA, 2006

S25:

Rahman, A.: Metrics for the Structural Assessment of Product Line Architecture. Master Tesis on Software Engineering, Thesis no. MSE-2004:24. School of Engineering, Blekinge Institute of Technology, Sweden, 2004

S26:

Sethi, K. Cai, Y., Wong, S., Garcia, A., Sant’Anna, C. From Retrospect to Prospect: Assessing Modularity and Stability from Software Architecture. Joint Working IEEE/IFIP Conference on Software Architecture, 2009 & European Conference on Software Architecture. WICSA/ECSA 2009

S27:

Shaik I., Abdelmoez, W,. Gunnalan, R., Shereshevsky, M., Zeid, A., Ammar, H.H., Mili, A., and Fuhrman, C. Change Propagation for Assessing Design Quality of Software Architectures. 5th Working IEEE/IFIP Conference on Software Architecture (WICSA’05), 2005

S28:

Siegmund, N., Rosenmüller, M., Kuhlemann, M., Kästner, C., Saake, G.: Measuring Non-functional Properties in Software Product Lines for Product Derivation. In 15th Asia–Pacific Software Engineering Conference, Beijing, China, 2008

S29:

Stoll, P., Bass, L., Golden, E., John, B.E. Supporting Usability in Product Line Architectures. In Software Product Lines Conference, 2009

S30:

Stuikys, V., Damasevicius, R. Measuring Complexity of Domain Models represented by Feature Diagrams. Information Technology and Control, 2009, Vol.38, No.3

S31:

Sun Her, J., Hyeok Kim, J., Hun Oh, S., Yul Rhew, S., Dong Kim, S.: A framework for evaluating reusability of core asset in product line engineering. Information and Software Technology 49, pp. 740–760, 2007

S32:

Van der Hoek, A., Dincel, E., Medidović, N.: Using Services Utilization Metrics to Assess the Structure of Product Line Architectures. In 9th International Software Metrics Symposium (METRICS), Sydney, Australia, 2003

S33:

Wnuk, K., Regnell, B., Karlsson, L. What Happened to Our Features? Visualization and Understanding of Scope Change Dynamics in a Large-Scale Industrial Setting. In 17th IEEE International Requirements Engineering Conference, 2009

S34:

Yoshimura, K., Ganesan, D., Muthig, D.: Assessing Merge Potential of Existing Engine Control Systems into a Product Line. In International Workshop on Software Engineering for Automative Systems, Shangai, China, pp. 61–67, 2006

S35:

Zhang, T., Deng, L., Wu, J., Zhou, Q., Ma, C.: Some Metrics for Accessing Quality of Product Line Architecture. In International Conference on Computer Science and Software Engineering (CSSE), Wuhan, China, pp. 500–503, 2008

Appendix 2

See Table 7.

Table 7 Classification of software measures found in the systematic review
Full size table

Appendix 3

See Table 8.

Table 8 List of quality attributes for software product lines found in the systematic review
Full size table

Rights and permissions

Reprints and permissions

About this article

Cite this article

Montagud, S., Abrahão, S. & Insfran, E. A systematic review of quality attributes and measures for software product lines. Software Qual J 20, 425–486 (2012). https://doi.org/10.1007/s11219-011-9146-7

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11219-011-9146-7

Keywords

Search

Navigation