Software Engineering Data Collection for Field Studies
Software engineering is an intensely people-oriented activity, yet little is known about how software engineers perform their work. In order to improve software engineering tools and practice, it is therefore essential to conduct field studies, i.e., to study real practitioners as they solve real problems. To aid this goal, we describe a series of data collection techniques for such studies, organized around a taxonomy based on the degree to which interaction with software engineers is necessary. For each technique, we provide examples from the literature, an analysis of some of its advantages and disadvantages, and a discussion of special reporting requirements. We also talk briefly about recording options and data analysis.
Unable to display preview. Download preview PDF.
- An, K. H., Gustafson, D. A. & Melton, A. C. (1987), A model for software maintenance, in ‘Proceedings of the Conference in Software Maintenance’, Austin, Texas, pp. 57–62.Google Scholar
- Atkins, D., Ball, T., Graves, T. & Mockus, A. (1999), Using version control data to evaluate the effectiveness of software tools, in ‘1999 International Conference on Software Engineering’, ACM Press, pp. 324–333.Google Scholar
- Barnard, J. & Rubin, D. B. (1999), ‘Small sample degrees of freedom with multiple imputation’, Biometrika 86(4).Google Scholar
- Goldenson, D. R., Gopal, A. & Mukhopadhyay, T. (1999), Determinants of success in software measurement programs, in ‘Sixth International Symoposium on Software Metrics’, IEEE Computer Society, pp. 10–21.Google Scholar
- Graves, T. L. & Mockus, A. (1998), Inferring change effort from configuration management databases, in ‘Metrics 98: Fifth International Symposium on Software Metrics’, Bethesda, Maryland, pp. 267–273.Google Scholar
- Herbsleb, J. D. & Grinter, R. (1998), Conceptual simplicity meets organizational complexity: Case study of a corporate metrics program, in ‘20th International Conference on Software Engineering’, IEEE Computer Society, pp. 271–280.Google Scholar
- Herbsleb, J. D., Krishnan, M., Mockus, A., Siy, H. P. & Tucker, G. T. (2000), Lessons from ten years of software factory experience, Technical report, Bell Laboratories.Google Scholar
- Jönsson, P. & Wohlin, C. (2004), An evaluation of k-nearest neighbour imputation using likert data, in ‘Proc. of the 10th Int. Symp. on Software Metrics’, pp. 108–118.Google Scholar
- Kaplan, E. & Meyer, P. (1958), ‘Non-paramentric estimation from incomplete observations’, J Am Stat Assoc pp. 457–481.Google Scholar
- Little, R. & Hyonggin, A. (2003), Robust likelihood-based analysis of multivariate data with missing values, Technical Report Working Paper 5, The University of Michigan Department of Biostatistics Working Paper Series. http://www.bepress.com/umichbiostat/paper5.
- Little, R. J. A. & Rubin, D. B. (1987), Statistical Analysis with Missing Data, Willey Series in Probability and Mathematical Statistics, John Willey & Sons.Google Scholar
- Mockus, A. (2006), Empirical estimates of software availability of deployed systems, in ‘2006 International Symposium on Empirical Software Engineering’, ACM Press, Rio de Janeiro, Brazil, pp. 222–231.Google Scholar
- Mockus, A. (2007), Software support tools and experimental work, in V. Basili & et al, eds, ‘Empirical Software Engineering Issues: LNCS 4336:’, Springer, p. to appear.Google Scholar
- Mockus, A. & Votta, L. G. (1997), Identifying reasons for software changes using historic databases, Technical Report BL0113590–980410-04, Bell Laboratories.Google Scholar
- Novo, A. (2002), ‘Analysis of multivariate normal datasets with missing values’. Ported to R by Alvaro A. Novo. Original by J.L. Schafer.Google Scholar
- R Development Core Team (2005), R: A language and environment for statistical computing, R Foundation for Statistical Computing, Vienna, Austria. ISBN 3–900051-07–0. http://www.R-project.org.
- Rubin, D. B. (1987), Multiple Imputation for Nonresponse in Surveys, John Willey & Sons.Google Scholar
- Schafer, J. L. (1997), Analysis of Incomplete Data, Monograph on Statistics ans Applied Probability, Chapman & Hall.Google Scholar
- Schafer, J. S. (1999), ‘Software for multiple imputation’. http://www.stat.psu.edu/˜jls/misoftwa.html.
- Swanson, E. B. (1976), The dimensions of maintenance, in ‘Proc. 2nd Conf. on Software Engineering’, San Francisco, pp. 492–497.Google Scholar
- Twala, B., Cartwright, M. & Shepperd, M. (2006), Ensemble of missing data techniques to improve software prediction accuracy, in ‘ICSE’06’, ACM, Shanghai, China, pp. 909–912.Google Scholar