Abstract
The Personal Software Process (PSP) is used by software engineers to gather and analyze data about their work. Published studies typically use data collected using the PSP to draw quantitative conclusions about its impact upon programmer behavior and product quality. However, our experience using PSP led us to question the quality of data both during collection and its later analysis. We hypothesized that data quality problems can make a significant impact upon the value of PSP measures—significant enough to lead to incorrect conclusions regarding process improvement. To test this hypothesis, we built a tool to automate the PSP and then examined 89 projects completed by ten subjects using the PSP manually in an educational setting. We discovered 1539 primary errors and categorized them by type, subtype, severity, and age. To examine the collection problem we looked at the 90 errors that represented impossible combinations of data and at other less concrete anomalies in Time Recording Logs and Defect Recording Logs. To examine the analysis problem we developed a rule set, corrected the errors as far as possible, and compared the original and corrected data. We found significant differences for measures such as yield and the cost-performance ratio, confirming our hypothesis. Our results raise questions about the accuracy of manually collected and analyzed PSP data, indicate that integrated tool support may be required for high quality PSP data analysis, and suggest that external measures should be used when attempting to evaluate the impact of the PSP upon programmer behavior and product quality.
Similar content being viewed by others
References
Austin, R. D. 1996. Measuring and Managing Performance in Organizations. Dorset House Publishing.
Ceberio-Verghese, A. 1996. Personal Software Process: A user's perspective. In Nancy R. Mead, editor, Ninth Conference on Software Engineering Education, 10662 Los Vaqueros Circle, P. O. Box 3014, Los Alamitos, CA 90720–1264. IEEE Computer Society Press.
Dellien, O. 1997. The Personal Software Process in industry. Master's thesis, Lund Institute of Technology (Sweden), Department of Communication Systems.
El Emam, K., Shostak, B., and Madhavji, N. 1996. Implementing concepts from the Personal Software Process in an industrial setting. In Proceedings of the Fourth International Conference on the Software Process, Brighton, England.
Ferguson, G. A., and Takane, Y. 1989. Statistical Analysis In Psychology And Education. McGraw-Hill Book Company, 6th edition.
Ferguson, P., Humphrey, W. S., Khajenoori, S., Macke, S., and Matvya, A. 1997. Introducing the Personal Software Process: Three industry cases. IEEE Computer 30(5): 24–31.
Gilb, T., and Graham, D. 1993. Software Inspection. Addison-Wesley.
Hayes, W., and Over, J.W. 1997. The Personal Software Process (PSP): An empirical study of the impact of PSP on individual engineers. Technical Report CMU/SEI-97-TR-001, Software Engineering Institute, Pittsburgh, PA.
Henry, J. 1997. Personal Software Process studio. http://www-cs.etsu.edu/softeng/psp/.
Humphrey, W. 1994a. The personal process in software engineering. In Proceedings of the Third International Conference on the Software Process, 69–77.
Humphrey, W. 1994b. The Personal Software Process. Software Process Newsletter #1, pages 1–3.
Humphrey, W. 1995. Introducing the Personal Software Process. Annals of Software Engineering 1: 311–325.
Humphrey, W. 1995. The power of personal data. Software Process Improvement and Practice Journal 1: 69–81.
Humphrey, W. S. 1995. A Discipline for Software Engineering. New York: Addison-Wesley.
Humphrey, W. S. 1996. Using a defined and measured Personal Software Process. IEEE Software 13(3): 77–88.
Humphrey, W. S. 1997. Introduction to the Personal Software Process. New York: Addison-Wesley.
Moore, C. 1998. Project LEAP toolset. http://csdl.ics.hawaii.edu/Tools/LEAP/LEAP.html.
Paulk, M., Weber, C., Curtis, B., and Chrissis, M. B. 1995. The Capability Maturity Model: Guidelines for Improving the Software Process. New York: Addison-Wesley.
Sherdil, K., and Madhavji, N. H. 1996. Human-oriented improvement in the software process. In Proceedings of the 5th European Workshop on Software Process Technology.
Wohlin, C. 1998. The Personal Software Process as a context for empirical studies. Software Process Newsletter #12, pages 7–12.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Johnson, P.M., Disney, A.M. A Critical Analysis of PSP Data Quality: Results from a Case Study. Empirical Software Engineering 4, 317–349 (1999). https://doi.org/10.1023/A:1009801218527
Issue Date:
DOI: https://doi.org/10.1023/A:1009801218527