Accurate Estimates without Calibration?

  • Tim Menzies
  • Oussama Elrawas
  • Barry Boehm
  • Raymond Madachy
  • Jairus Hihn
  • Daniel Baker
  • Karen Lum
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5007)


Most process models calibrate their internal settings using historical data. Collecting this data is expensive, tedious, and often an incomplete process.

Is it possible to make accurate software process estimates without historical data? Suppose much of uncertainty in a model comes from a small subset of the model variables. If so, then after (a) ranking variables by their ability to constrain the output; and (b) applying a small number of the top-ranked variables; then it should be possible to (c) make stable predictions in the constrained space.

To test that hypothesis, we combined a simulated annealer (to generate random solutions) with a variable ranker. The results where quite dramatic: in one of the studies in this paper, we found process options that reduced the median and variance of the effort estimates by a factor of 20. In ten case studies, we show that the estimates generated in this manner are usually similar to those produced by standard local calibration.

Our conclusion is that while it is always preferable to tune models to local data, it is possible to learn process control options without that data.


Effort Estimate Threat Model Software Development Project Local Calibration Master Variable 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Abdel-Hamid, T., Madnick, S.: Software Project Dynamics: An Integrated Approach. Prentice-Hall Software Series, Englewood Cliffs (1991)Google Scholar
  2. 2.
    Aguilar-Ruiz, J.S., Ramos, I., Riquelme, J., Toro, M.: An evolutionary approach to estimating software development projects. Information and Software Technology 43(14), 875–882 (2001)Google Scholar
  3. 3.
    Akhavi, M., Wilson, W.: Dynamic simulation of software process models. In: Proceedings of the 5th Software Engineering Process Group National Meeting, Costa Mesa, California, April 26-29, 1993, Software engineering Institute, Carnegie Mellon University (1993)Google Scholar
  4. 4.
    Alvarez, J.L., Mata, J., Riquelme, J.C., Ramos, I.: A data mining method to support decision making in software development projects. In: ICEIS 2003: Fifth International Conference on Enterprise Information Systems (2003)Google Scholar
  5. 5.
    Bailey, J.: Using monte carlo and cocomo-2 to model a large it system development (2002)Google Scholar
  6. 6.
    Baker, D.: A hybrid approach to expert and model-based effort estimation. Master’s thesis, Lane Department of Computer Science and Electrical Engineering, West Virginia University (2007),
  7. 7.
    Boehm, B.: Software Engineering Economics. Prentice-Hall, Englewood Cliffs (1981)Google Scholar
  8. 8.
    Boehm, B.: Safe and simple software cost analysis. IEEE Software, 14–17 (September/October 2000),
  9. 9.
    Boehm, B., Horowitz, E., Madachy, R., Reifer, D., Clark, B.K., Steece, B., Brown, A.W., Chulani, S., Abts, C.: Software Cost Estimation with Cocomo II. Prentice-Hall, Englewood Cliffs (2000)Google Scholar
  10. 10.
    Briand, L.C., Emam, K.E., Bomarius, F.,, C.: A hybrid method for software cost estimation, benchmarking, and risk assessment. In: ICSE, pp. 390–399 (1998)Google Scholar
  11. 11.
    Cass, A.G., Staudt Lerner, B., Sutton, S.M., Jr., McCall, E.K., Wise, A., Osterweil, L.J.,: Little-jil/juliette: A process definition language and interpreter. In: Proceedings of the 22nd International Conference on Software Engineering (ICSE 2000), June 2000, pp. 754–757 (2000)Google Scholar
  12. 12.
    Crawford, J., Baker, A.: Experimental results on the application of satisfiability algorithms to scheduling problems. In: AAAI (1994)Google Scholar
  13. 13.
    Harel, D.: Statemate: A working environment for the development of complex reactive systems. IEEE Transactions on Software Engineering 16(4), 403–414 (1990)Google Scholar
  14. 14.
    Jalali, O.: Evaluation bias in effort estimation. Master’s thesis, Lane Department of Computer Science and Electrical Engineering, West Virginia University (2007)Google Scholar
  15. 15.
    Jensen, R.: An improved macrolevel software development resource estimation model. In: 5th ISPA Conference, April 1983, pp. 88–92 (1983)Google Scholar
  16. 16.
    Kelton, D., Sadowski, R., Sadowski, D.: Simulation with Arena, second edition. McGraw-Hill, New York (2002)Google Scholar
  17. 17.
    Kemerer, C.: An empirical validation of software cost estimation models. Communications of the ACM 30(5), 416–429 (1987)Google Scholar
  18. 18.
    Kirkpatrick, S., Gelatt, C.D., Vecchi, M.P.: Optimization by simulated annealing. Science 220(4598), 671–680 (1983)Google Scholar
  19. 19.
    Law, A., Kelton, B.: Simulation Modeling and Analysis. McGraw-Hill, New York (2000)Google Scholar
  20. 20.
    Lum, K.: Software cost analysis tool user document (2005)Google Scholar
  21. 21.
    Lum, K., Bramble, M., Hihn, J., Hackney, J., Khorrami, M., Monson, E.: Handbook for software cost estimation (2003)Google Scholar
  22. 22.
    Lum, K., Powell, J., Hihn, J.: Validation of spacecraft software cost estimation models for flight and ground systems. In: ISPA Conference Proceedings, Software Modeling Track (May 2002)Google Scholar
  23. 23.
    Martin, R., Raffo, D.M.: A model of the software development process using both continuous and discrete models. International Journal of Software Process Improvement and Practice (June/July 2000)Google Scholar
  24. 24.
    Menzies, T., Chen, Z., Hihn, J., Lum, K.: Selecting best practices for effort estimation. IEEE Transactions on Software Engineering (November 2006),
  25. 25.
    Menzies, T., Elrawas, O., Baker, D., Hihn, J., Lum, K.: On the value of stochastic abduction (if you fix everything, you lose fixes for everything else). In: International Workshop on Living with Uncertainty (an ASE 2007 co-located event) (2007),
  26. 26.
    Menzies, T., Elwaras, O., Hihn, J., Feathear, M., Boehm, B., Madachy, R.: The business case for automated software engineerng. In: IEEE ASE 2007 (2007),
  27. 27.
    Metropolis, N., Rosenbluth, A., Rosenbluth, M., Teller, A., Teller, E.: J. Chem. Phys 21, 1087–1092 (1953)Google Scholar
  28. 28.
    Mi, P., Scacchi, W.: A knowledge-based environment for modeling and simulation software engineering processes. IEEE Transactions on Knowledge and Data Engineering, 283–294 (September 1990)Google Scholar
  29. 29.
    Park, R.: The central equations of the price software cost model. In: 4th COCOMO Users Group Meeting (November 1988)Google Scholar
  30. 30.
    Raffo, D.M., Vandeville, J.V., Martin, R.: Software process simulation to achieve higher cmm levels. Journal of Systems and Software 46(2/3) (April 1999)Google Scholar
  31. 31.
    Rela, L.: Evolutionary computing in search-based software engineering. Master’s thesis, Lappeenranta University of Technology (2004)Google Scholar
  32. 32.
    Williams, R., Gomes, C., Selman, B.: Backdoors to typical case complexity. In: Proceedings of IJCAI 2003 (2003),
  33. 33.
    Wise, A., Cass, A.G., Staudt Lerner, B., McCall, E.K., Osterweil, L.J., Sutton Jr., S.M.: Using little-jil to coordinate agents in software engineering. In: Proceedings of the Automated Software Engineering Conference (ASE 2000), Grenoble, France (September 2000),

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • Tim Menzies
    • 1
  • Oussama Elrawas
    • 1
  • Barry Boehm
    • 2
  • Raymond Madachy
    • 2
  • Jairus Hihn
    • 3
  • Daniel Baker
    • 1
  • Karen Lum
    • 3
  1. 1.LCSEEWest Virginia UniversityMorgantownUSA
  2. 2.CSUniversity of Southern CaliforniaLos AngelesUSA
  3. 3.JPLUSA

Personalised recommendations