Improvement Opportunities and Suggestions for Benchmarking

  • Cigdem Gencel
  • Luigi Buglione
  • Alain Abran
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5891)


During the past 10 years, the amount of effort put on setting up benchmarking repositories has considerably increased at the organizational, national and even at international levels to help software managers to determine the performance of software activities and to make better software estimates. This has enabled a number of studies with an emphasis on the relationship between software product size, effort and cost drivers in order to either measure the average performance for similar software projects or to develop estimation models and then refine them using the collected data. However, despite these efforts, none of those methods are yet deemed to be universally applicable and there is still no agreement on which cost drivers are significant in the estimation process. This study discusses some of the possible reasons why in software engineering, practitioners and researchers have not yet been able to come up with reasonable and well quantified relationships between effort and cost drivers although considerable amounts of data on software projects have been collected. An improved classification of application types in benchmarking repositories is also proposed.


Benchmarking Repositories Performance Measurement Effort Estimation Cost Drivers 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Angelis, L., Stamelos, I., Morisio, M.: Building a Cost Estimation Model Based on Categorical Data. In: 7th IEEE Int. Software Metrics Symposium (METRICS 2001), London (April 2001)Google Scholar
  2. 2.
    Basili, V.R., Bomarius, F., Feldmann, R.L.: Get Your Experience Factory Ready for the Next Decade – Ten Years after "How to Build and Run One". In: Companion to the Proceedings of the 29th international Conference on Software Engineering, May 20-26, pp. 167–168. IEEE Computer Society, Washington (2007)Google Scholar
  3. 3.
    Basili, V., Caldiera, G., McGarry, F., Pajerski, R., Page, G., Waligora, S.: The software engineering laboratory: an operational software experience factory. In: Proceedings of the 14th intern. Conf. on Software Engineering, ICSE 1992, Melbourne, Australia, May 11-15, pp. 370–381. ACM, New York (1992)CrossRefGoogle Scholar
  4. 4.
    Boehm, B.W., Horowitz, E., Madachy, R., Reifer, D., Bradford, K.C., Steece, B., Brown, A.W., Chulani, S., Abts, C.: Software Cost Estimation with COCOMO II. Prentice Hall, New Jersey (2000)Google Scholar
  5. 5.
    Buglione, L.: Strengthening CMMI Maturity Levels with a Quantitative Approach to Root-Cause Analysis. In: Proceedings of the 5th Software Measurement European Forum (SMEF 2008), Milan, Italy, May 28-30, pp. 67–82 (2008) ISBN 9-788870-909999Google Scholar
  6. 6.
    Card, D., Zubrow, D.: Guest Editor’s introduction, Benchmarking Software Organizations. IEEE Software, 16–17 (September/October 2001)Google Scholar
  7. 7.
    CESMM. 1991: Civil Engineering Standard Method of Measurement, Thomas Telford Ltd., 3rd edn. (1991)Google Scholar
  8. 8.
    ISBSG: SC 7 Proposed New Work Item on Software and systems engineering – IT Performance Benchmarking Framework (2008),
  9. 9.
    Cukic, B.: The Promise of Public Software Engineering Data Repositories. IEEE Software, Guest Editor’s introduction 22(6), 20–22 (2005)Google Scholar
  10. 10.
    Experience Pro, (Last access: 2009/01/29)
  11. 11.
    Forselius, P.: Benchmarking Software-Development Productivity. IEEE Software 17(1), 80–88 (2000)CrossRefGoogle Scholar
  12. 12.
    ISBSG Dataset 10 (2007),
  13. 13.
    ISO/IEC 14143-1:2007: Information Technology – Software Measurement – Functional Size Measurement – Part 1: Definition of Concepts (2007)Google Scholar
  14. 14.
    ISO/IEC TR 12182:1998: Information technology – Categorization of softwareGoogle Scholar
  15. 15.
    ISO/IEC TR 14143-5:2004 Information Technology – Software Measurement – Functional Size Measurement – Part 5: Determination of Functional Domains for Use with Functional Size Measurement Google Scholar
  16. 16.
    Shirabad, J.S., Menzies, T.J.: PROMISE Software Engineering Repository, School of Information Technology and Eng., Univ. of Ottawa, Canada (2005),
  17. 17.
    Jørgensen, M., Shepperd, M.: A Systematic Review of Software Development Cost Estimation Studies. IEEE Transactions on Software Engineering 33(1), 33–53 (2007)CrossRefGoogle Scholar
  18. 18.
    Putnam, L.H.: A general empirical solution to the macro software sizing and estimating problem. IEEE Trans. Soft. Eng. 4(4), 345–361 (1978)CrossRefGoogle Scholar
  19. 19.
    Lokan, C., Wright, T., Hill, P.R., Stringer, M.: Organizational Benchmarking Using the ISBSG Data Repository. IEEE Software 18(5), 26–32 (2001)CrossRefGoogle Scholar
  20. 20.
    Maxwell, K.D.: Collecting Data for Comparability: Benchmarking Software Development Productivity. IEEE Software 18(5), 22–25 (2001)CrossRefGoogle Scholar
  21. 21.
    Morasca, S., Russo, G.: An Empirical Study of Software Productivity. In: Proc. of the 25th Annual International Computer Software and Applications Conference (COMPSAC 2001), Chicago, IL, USA, October 8-12, pp. 317–322 (2001)Google Scholar
  22. 22.
    Premraj, R., Shepperd, M.J., Kitchenham, B., Forselius, P.: An Empirical Analysis of Software Productivity over Time. In: 11th IEEE International Symposium on Software Metrics (Metrics 2005), p. 37. IEEE Computer Society, Los Alamitos (2005)CrossRefGoogle Scholar
  23. 23.
    PROMISE Data Repositories, (last access: 2009/01/29)
  24. 24.
    Park, R.E.: PRICE S: The calculation within and why. In: Proceedings of ISPA 10th Annual Conference, Brighton, England (July 1988)Google Scholar
  25. 25.
    Park, R.: Software Size Measurement: A Framework for Counting Source Statements. Technical Report CMU/SEI-92-TR-020Google Scholar
  26. 26.
    Tausworthe, R.: Deep Space Network Software Cost Estimation Model. Jet Propulsion Laboratory Publication 81-7 (1981)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  • Cigdem Gencel
    • 1
  • Luigi Buglione
    • 2
    • 3
  • Alain Abran
    • 2
  1. 1.Blekinge Institute of TechnologySweden
  2. 2.Ecole de Téchnologie Superieure (ETS) – Université du Québec à Montreal (UQAM)/ 
  3. 3.Nexen (Engineering Group)Italy

Personalised recommendations