Abstract
Most process models calibrate their internal settings using historical data. Collecting this data is expensive, tedious, and often an incomplete process.
Is it possible to make accurate software process estimates without historical data? Suppose much of uncertainty in a model comes from a small subset of the model variables. If so, then after (a) ranking variables by their ability to constrain the output; and (b) applying a small number of the top-ranked variables; then it should be possible to (c) make stable predictions in the constrained space.
To test that hypothesis, we combined a simulated annealer (to generate random solutions) with a variable ranker. The results where quite dramatic: in one of the studies in this paper, we found process options that reduced the median and variance of the effort estimates by a factor of 20. In ten case studies, we show that the estimates generated in this manner are usually similar to those produced by standard local calibration.
Our conclusion is that while it is always preferable to tune models to local data, it is possible to learn process control options without that data.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Abdel-Hamid, T., Madnick, S.: Software Project Dynamics: An Integrated Approach. Prentice-Hall Software Series, Englewood Cliffs (1991)
Aguilar-Ruiz, J.S., Ramos, I., Riquelme, J., Toro, M.: An evolutionary approach to estimating software development projects. Information and Software Technology 43(14), 875–882 (2001)
Akhavi, M., Wilson, W.: Dynamic simulation of software process models. In: Proceedings of the 5th Software Engineering Process Group National Meeting, Costa Mesa, California, April 26-29, 1993, Software engineering Institute, Carnegie Mellon University (1993)
Alvarez, J.L., Mata, J., Riquelme, J.C., Ramos, I.: A data mining method to support decision making in software development projects. In: ICEIS 2003: Fifth International Conference on Enterprise Information Systems (2003)
Bailey, J.: Using monte carlo and cocomo-2 to model a large it system development (2002)
Baker, D.: A hybrid approach to expert and model-based effort estimation. Master’s thesis, Lane Department of Computer Science and Electrical Engineering, West Virginia University (2007), https://eidr.wvu.edu/etd/documentdata.eTD?documentid=5443
Boehm, B.: Software Engineering Economics. Prentice-Hall, Englewood Cliffs (1981)
Boehm, B.: Safe and simple software cost analysis. IEEE Software, 14–17 (September/October 2000), http://www.computer.org/certification/beta/Boehm_Safe.pdf
Boehm, B., Horowitz, E., Madachy, R., Reifer, D., Clark, B.K., Steece, B., Brown, A.W., Chulani, S., Abts, C.: Software Cost Estimation with Cocomo II. Prentice-Hall, Englewood Cliffs (2000)
Briand, L.C., Emam, K.E., Bomarius, F.,, C.: A hybrid method for software cost estimation, benchmarking, and risk assessment. In: ICSE, pp. 390–399 (1998)
Cass, A.G., Staudt Lerner, B., Sutton, S.M., Jr., McCall, E.K., Wise, A., Osterweil, L.J.,: Little-jil/juliette: A process definition language and interpreter. In: Proceedings of the 22nd International Conference on Software Engineering (ICSE 2000), June 2000, pp. 754–757 (2000)
Crawford, J., Baker, A.: Experimental results on the application of satisfiability algorithms to scheduling problems. In: AAAI (1994)
Harel, D.: Statemate: A working environment for the development of complex reactive systems. IEEE Transactions on Software Engineering 16(4), 403–414 (1990)
Jalali, O.: Evaluation bias in effort estimation. Master’s thesis, Lane Department of Computer Science and Electrical Engineering, West Virginia University (2007)
Jensen, R.: An improved macrolevel software development resource estimation model. In: 5th ISPA Conference, April 1983, pp. 88–92 (1983)
Kelton, D., Sadowski, R., Sadowski, D.: Simulation with Arena, second edition. McGraw-Hill, New York (2002)
Kemerer, C.: An empirical validation of software cost estimation models. Communications of the ACM 30(5), 416–429 (1987)
Kirkpatrick, S., Gelatt, C.D., Vecchi, M.P.: Optimization by simulated annealing. Science 220(4598), 671–680 (1983)
Law, A., Kelton, B.: Simulation Modeling and Analysis. McGraw-Hill, New York (2000)
Lum, K.: Software cost analysis tool user document (2005)
Lum, K., Bramble, M., Hihn, J., Hackney, J., Khorrami, M., Monson, E.: Handbook for software cost estimation (2003)
Lum, K., Powell, J., Hihn, J.: Validation of spacecraft software cost estimation models for flight and ground systems. In: ISPA Conference Proceedings, Software Modeling Track (May 2002)
Martin, R., Raffo, D.M.: A model of the software development process using both continuous and discrete models. International Journal of Software Process Improvement and Practice (June/July 2000)
Menzies, T., Chen, Z., Hihn, J., Lum, K.: Selecting best practices for effort estimation. IEEE Transactions on Software Engineering (November 2006), http://menzies.us/pdf/06coseekmo.pdf
Menzies, T., Elrawas, O., Baker, D., Hihn, J., Lum, K.: On the value of stochastic abduction (if you fix everything, you lose fixes for everything else). In: International Workshop on Living with Uncertainty (an ASE 2007 co-located event) (2007), http://menzies.us/pdf/07fix.pdf
Menzies, T., Elwaras, O., Hihn, J., Feathear, M., Boehm, B., Madachy, R.: The business case for automated software engineerng. In: IEEE ASE 2007 (2007), http://menzies.us/pdf/07casease-v0.pdf
Metropolis, N., Rosenbluth, A., Rosenbluth, M., Teller, A., Teller, E.: J. Chem. Phys 21, 1087–1092 (1953)
Mi, P., Scacchi, W.: A knowledge-based environment for modeling and simulation software engineering processes. IEEE Transactions on Knowledge and Data Engineering, 283–294 (September 1990)
Park, R.: The central equations of the price software cost model. In: 4th COCOMO Users Group Meeting (November 1988)
Raffo, D.M., Vandeville, J.V., Martin, R.: Software process simulation to achieve higher cmm levels. Journal of Systems and Software 46(2/3) (April 1999)
Rela, L.: Evolutionary computing in search-based software engineering. Master’s thesis, Lappeenranta University of Technology (2004)
Williams, R., Gomes, C., Selman, B.: Backdoors to typical case complexity. In: Proceedings of IJCAI 2003 (2003), http://www.cs.cornell.edu/gomes/FILES/backdoors.pdf
Wise, A., Cass, A.G., Staudt Lerner, B., McCall, E.K., Osterweil, L.J., Sutton Jr., S.M.: Using little-jil to coordinate agents in software engineering. In: Proceedings of the Automated Software Engineering Conference (ASE 2000), Grenoble, France (September 2000), ftp://ftp.cs.umass.edu/pub/techrept/techreport/2000/UM-CS-2000-045.ps
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 2008 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Menzies, T. et al. (2008). Accurate Estimates without Calibration?. In: Wang, Q., Pfahl, D., Raffo, D.M. (eds) Making Globally Distributed Software Development a Success Story. ICSP 2008. Lecture Notes in Computer Science, vol 5007. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-79588-9_19
Download citation
DOI: https://doi.org/10.1007/978-3-540-79588-9_19
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-79587-2
Online ISBN: 978-3-540-79588-9
eBook Packages: Computer ScienceComputer Science (R0)