Skip to main content
SpringerLink
Log in

Benchmarking library and application software with Data Envelopment Analysis

  • Published:
Software Quality Journal Aims and scope Submit manuscript

Abstract

Library software is generally believed to be well-structured and follows certain design guidelines due to the need of continuous evolution and stability of the respective APIs. We perform an empirical study to investigate whether the design of open-source library software is actually superior to that of application software. By analyzing certain design principles and heuristics that are considered important for API design, we extract a set of software metrics that are expected to reflect the improved nature of libraries. An initial comparison by conventional statistical analysis confirms the overall belief that products of different software size scale should not be compared by simply examining metric values in isolation. In this paper, we propose the use of Data Envelopment Analysis (DEA), borrowed from production economics, as a means of measuring and benchmarking the quality of different object-oriented software designs captured by software metrics and apply this approach to the comparison of library and application software. The advantages offered by DEA and the differences between the application of DEA in an economic and a technological context are discussed. Results of the approach are presented for 44 open-source projects, equally divided between libraries and applications.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  • Anda, B. (2007). Assessing software system maintainability using structural measures and expert assessments. In Proceedings of the 23rd IEEE international conference on software maintenance (Paris, France, October 2–5, 2007, pp. 204–213). ICSM’07.

  • Anderson, T. R. (2004). Benchmarking in sports: Bonds or Ruth, determining the most dominant baseball batter using DEA. In W. W. Cooper, L. M. Seiford, & J. Zhu (Eds.), Handbook on Data Envelopment Analysis (pp. 443–454). Boston, MA: Kluwer.

    Google Scholar 

  • Arisholm, E., & Sjøberg, D. I. K. (2004). Evaluating the effect of a delegated versus centralized control style on the maintainability of object-oriented software. IEEE Transactions on Software Engineering, 30(8), 521–534.

    Article  Google Scholar 

  • Asmild, M., Paradi, J. C., & Kulkarni, A. (2006). Using Data Envelopment Analysis in software development productivity measurement. Software Process Improvement and Practice, 11(6), 561–572.

    Article  Google Scholar 

  • Banker, R. D., Charnes, A., & Cooper, W. W. (1984). Some models for estimating technical and scale inefficiencies in Data Envelopment Analysis. Management Science, 30(9), 1078–1092.

    Article  MATH  Google Scholar 

  • Basili, V. R., Caldiera, G., & Rombach, H. D. (1994). Goal question metrics paradigm. In J. J. Marciniak (Ed.), Encyclopedia of software engineering (Vol. I, pp. 528–532). New York: Wiley.

    Google Scholar 

  • Benestad, H. C., Anda, B., & Arisholm, E. (2006). Assessing software product maintainability based on class-level structural measures. In Proceedings of the 7th international conference on product-focused software process improvement (Amsterdam, Netherlands, June 12–14, 2006). PROFES’06.

  • Bloch, J. (2006). How to design a good API and why it matters. Companion to the 21st ACM SIGPLAN symposium on object-oriented programming systems, languages, and applications (Portland, Oregon, USA: ACM Press, October 22–26, 2006). OOPSLA’2006.

  • Bloch, J. (2008). Effective java (2nd ed.). Boston: Addison-Wesley.

    Google Scholar 

  • Bowman, M., Briand, L. C., & Labiche, Y. (2007). Multi-objective genetic algorithms to support class responsibility assignment. In Proceedings of the 23rd IEEE international conference on software maintenance (Paris, France, October 2–5, 2007, pp. 124–133). ICSM’07.

  • Briand, L. C., Daly, J. W., & Wüst, J. K. (1999). A unified framework for coupling measurement in object-oriented systems. IEEE Transactions on Software Engineering, 25(1), 91–121.

    Article  Google Scholar 

  • Charnes, A., Cooper, W. W., Lewin, A. Y., & Seiford, L. M. (Eds.). (1995). Data Envelopment Analysis: Theory, methodology and applications. Boston: Kluwer.

    Google Scholar 

  • Charnes, A., Cooper, W. W., & Rhodes, E. (1978). Measuring the efficiency of decision making units. European Journal of Operational Research, 2(6), 429–444.

    Article  MathSciNet  MATH  Google Scholar 

  • Coelli, T. J., Prasada Rao, D. S., O’Donnell, C. J., & Battese, G. E. (2005). An introduction to efficiency and productivity analysis. Berlin: Springer.

    Google Scholar 

  • Cook, W. D., Kazakov, A., & Roll, Y. (1995). On the measuring and monitoring of relative efficiency of highway maintenance patrols. In A. Charnes, W. W. Cooper, A. Lewin, & L. M. Seiford (Eds.), Data Envelopment Analysis: Theory, methodology and applications (pp. 195–210). Norwell, MA: Kluwer.

    Google Scholar 

  • Cooper, W. W., Seiford, L. M., & Tone, K. (2005). Introduction to Data Envelopment Analysis and its uses: With DEA-solver software and references. New York, USA: Springer.

    Google Scholar 

  • Cooper, W. W., Seiford, L. M., & Tone, K. (2007). Data Envelopment Analysis: A comprehensive text with models, applications, references and DEA-solver software. New York, USA: Springer.

    MATH  Google Scholar 

  • Criswell, D. R., & Thompson, R. G. (1996). Data Envelopment Analysis of space and terrestrially based large commercial power systems for earth: A prototype analysis of their relative economic advantages. Solar Energy, 56(1), 119–131.

    Article  Google Scholar 

  • Dyson, R. G., Allen, R., Camanho, A. S., Podinovski, V. V., Sarrico, C. S., & Shale, E. A. (2001). Pitfalls and protocols in DEA. European Journal of Operational Research, 132(2), 245–259.

    Article  MATH  Google Scholar 

  • Gamma, E., Helm, R., Johnson, R., & Vlissides, J. (1995). Design patterns: Elements of reusable object-oriented software. Boston: Addison-Wesley.

    Google Scholar 

  • ISO/IEC. (1991). International standard ISO, IEC 9126, International Organization for Standardization, Geneva.

  • Li, W., & Henry, S. (1993). Object-oriented metrics that predict maintainability. Journal of Systems and Software, 23(2), 111–122.

    Article  Google Scholar 

  • Liskov, B. (1988). Data abstraction and hierarchy. SIGPLAN Notices, 23(5), 17–34.

    Article  Google Scholar 

  • Lorenz, M., & Kidd, J. (1994). Object-oriented software metrics. Upper Saddle River, NJ: Prentice Hall.

    Google Scholar 

  • Martin, R. C. (2003). Agile software development: Principles, patterns and practices. Upper Saddle River, NJ: Prentice Hall.

    Google Scholar 

  • Meyer, B. (2000). Object-oriented software construction. Upper Saddle River, NJ: Prentice Hall PTR.

    Google Scholar 

  • O’Keeffe, M., & O’Cinneide, M. (2006). Search-based software maintenance. In Proceedings of the 10th European conference on software maintenance and reengineering (Bari, Italy, March 22–24, 2006). CSMR’06.

  • Parnas, D. L. (1972). On the criteria to be used in decomposing systems into modules. Communications of the ACM, 15(12), 1053–1058.

    Article  Google Scholar 

  • Riel, A. J. (1996). Object-oriented design heuristics. Boston: Addison-Wesley Professional.

    Google Scholar 

  • Ruggiero, J. (2004). Performance evaluation in education: Modeling educational production. In W. W. Cooper, L. M. Seiford, & J. Zhu (Eds.), Handbook on Data Envelopment Analysis (pp. 323–348). Boston, MA: Kluwer.

    Google Scholar 

  • Seng, O., Stammel, J., & Burkhart, D. (2006). Search-based determination of refactorings for improving the class structure of object-oriented systems. In Proceedings of the 8th annual conference on genetic and evolutionary computation (Seattle, WA, July 8–12, 2006). GECCO’06.

  • Stensrud, E., & Myrtveit, I. (2003). Identifying high performance ERP projects. IEEE Transactions on Software Engineering, 29(5), 398–416.

    Article  Google Scholar 

  • Tulach, J. (2008). Practical API design: Confessions of a java framework architect. APress.

  • von Mayrhauser, A., Wohlin, C., & Ohlsson, M. C. (2000). Assessing and understanding efficiency and success of software production. Empirical Software Engineering, 5(2), 125–154.

    Article  MATH  Google Scholar 

  • Wohlin, C., Runeson, P., Höst, M., Ohlsson, M. C., Regnell, B., & Wesslén, A. (2000). Experimentation in software engineering: An introduction. Boston, MA: Kluwer.

    MATH  Google Scholar 

  • Yamashita, A. F., Anda, B., Sjøberg, D. I. K., Benestad, H. C., Arnstad, P. E., & Moonen, L. (2009). Using concept mapping for maintainability assessments. In Proceedings of the 3rd international symposium on empirical software engineering and measurement (Florida, USA, October 15–16, 2009). ESEM’09.

  • Yang, Z. (2009). Assessing the performance of Canadian bank branches using Data Envelopment Analysis. Journal of the Operational Research Society, 60(6), 771–780.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Applied Informatics, University of Macedonia, 156 Egnatia Str., 54006, Thessaloniki, Greece

    Alexander Chatzigeorgiou & Emmanouil Stiakakis

Authors
  1. Alexander Chatzigeorgiou
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Emmanouil Stiakakis
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Alexander Chatzigeorgiou.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Chatzigeorgiou, A., Stiakakis, E. Benchmarking library and application software with Data Envelopment Analysis. Software Qual J 19, 553–578 (2011). https://doi.org/10.1007/s11219-010-9113-8

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11219-010-9113-8

Keywords

Search

Navigation