Skip to main content

A Critical Analysis of PSP Data Quality: Results from a Case Study

Abstract

The Personal Software Process (PSP) is used by software engineers to gather and analyze data about their work. Published studies typically use data collected using the PSP to draw quantitative conclusions about its impact upon programmer behavior and product quality. However, our experience using PSP led us to question the quality of data both during collection and its later analysis. We hypothesized that data quality problems can make a significant impact upon the value of PSP measures—significant enough to lead to incorrect conclusions regarding process improvement. To test this hypothesis, we built a tool to automate the PSP and then examined 89 projects completed by ten subjects using the PSP manually in an educational setting. We discovered 1539 primary errors and categorized them by type, subtype, severity, and age. To examine the collection problem we looked at the 90 errors that represented impossible combinations of data and at other less concrete anomalies in Time Recording Logs and Defect Recording Logs. To examine the analysis problem we developed a rule set, corrected the errors as far as possible, and compared the original and corrected data. We found significant differences for measures such as yield and the cost-performance ratio, confirming our hypothesis. Our results raise questions about the accuracy of manually collected and analyzed PSP data, indicate that integrated tool support may be required for high quality PSP data analysis, and suggest that external measures should be used when attempting to evaluate the impact of the PSP upon programmer behavior and product quality.

This is a preview of subscription content, access via your institution.

References

  • Austin, R. D. 1996. Measuring and Managing Performance in Organizations. Dorset House Publishing.

  • Ceberio-Verghese, A. 1996. Personal Software Process: A user's perspective. In Nancy R. Mead, editor, Ninth Conference on Software Engineering Education, 10662 Los Vaqueros Circle, P. O. Box 3014, Los Alamitos, CA 90720–1264. IEEE Computer Society Press.

  • Dellien, O. 1997. The Personal Software Process in industry. Master's thesis, Lund Institute of Technology (Sweden), Department of Communication Systems.

  • El Emam, K., Shostak, B., and Madhavji, N. 1996. Implementing concepts from the Personal Software Process in an industrial setting. In Proceedings of the Fourth International Conference on the Software Process, Brighton, England.

  • Ferguson, G. A., and Takane, Y. 1989. Statistical Analysis In Psychology And Education. McGraw-Hill Book Company, 6th edition.

  • Ferguson, P., Humphrey, W. S., Khajenoori, S., Macke, S., and Matvya, A. 1997. Introducing the Personal Software Process: Three industry cases. IEEE Computer 30(5): 24–31.

    Google Scholar 

  • Gilb, T., and Graham, D. 1993. Software Inspection. Addison-Wesley.

  • Hayes, W., and Over, J.W. 1997. The Personal Software Process (PSP): An empirical study of the impact of PSP on individual engineers. Technical Report CMU/SEI-97-TR-001, Software Engineering Institute, Pittsburgh, PA.

    Google Scholar 

  • Henry, J. 1997. Personal Software Process studio. http://www-cs.etsu.edu/softeng/psp/.

  • Humphrey, W. 1994a. The personal process in software engineering. In Proceedings of the Third International Conference on the Software Process, 69–77.

  • Humphrey, W. 1994b. The Personal Software Process. Software Process Newsletter #1, pages 1–3.

  • Humphrey, W. 1995. Introducing the Personal Software Process. Annals of Software Engineering 1: 311–325.

    Google Scholar 

  • Humphrey, W. 1995. The power of personal data. Software Process Improvement and Practice Journal 1: 69–81.

    Google Scholar 

  • Humphrey, W. S. 1995. A Discipline for Software Engineering. New York: Addison-Wesley.

    Google Scholar 

  • Humphrey, W. S. 1996. Using a defined and measured Personal Software Process. IEEE Software 13(3): 77–88.

    Google Scholar 

  • Humphrey, W. S. 1997. Introduction to the Personal Software Process. New York: Addison-Wesley.

    Google Scholar 

  • Moore, C. 1998. Project LEAP toolset. http://csdl.ics.hawaii.edu/Tools/LEAP/LEAP.html.

  • Paulk, M., Weber, C., Curtis, B., and Chrissis, M. B. 1995. The Capability Maturity Model: Guidelines for Improving the Software Process. New York: Addison-Wesley.

    Google Scholar 

  • Sherdil, K., and Madhavji, N. H. 1996. Human-oriented improvement in the software process. In Proceedings of the 5th European Workshop on Software Process Technology.

  • Wohlin, C. 1998. The Personal Software Process as a context for empirical studies. Software Process Newsletter #12, pages 7–12.

Download references

Author information

Authors and Affiliations

Authors

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Johnson, P.M., Disney, A.M. A Critical Analysis of PSP Data Quality: Results from a Case Study. Empirical Software Engineering 4, 317–349 (1999). https://doi.org/10.1023/A:1009801218527

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1023/A:1009801218527

  • Personal software process
  • defects
  • empirical software engineering
  • measurement dysfunction
  • automated process support