In 1997, Henry Lieberman stated that debugging is the dirty little secret of computer science. Since then, several promising debugging technologies have been developed such as back-in-time debuggers and automatic fault localization methods. However, the last study about the state-of-the-art in debugging is still more than 15 years old and so it is not clear whether these new approaches have been applied in practice or not. For that reason, we investigate the current state of debugging in a comprehensive study. First, we review the available literature and learn about current approaches and study results. Second, we observe several professional developers while debugging and interview them about their experiences. Third, we create a questionnaire that serves as the basis for a larger online debugging survey. Based on these results, we present new insights into debugging practice that help to suggest new directions for future research.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Tax calculation will be finalised during checkout.
Subscribe to journal
Immediate online access to all issues from 2019. Subscription will auto renew annually.
Tax calculation will be finalised during checkout.
The Pearson coefficient can be used to find linear relations between numeric variables. Values between \(-1\) and 1 indicate how well the actual relation can be approximated by a linear function (Pearson 1895).
As early results of the survey have shown a large number of parallel behavioral bugs, we added it as a new choice that was not represented in Eisenstadt’s original version.
A χ 2 test can be used for any kind of discretely categorized data with a large number of samples in each category. A large value indicates dependence and a low independence (Pearson 1900).
Agans, D. J. (2002). Debugging: The 9 indispensable rules for finding even the most elusive software and hardware problems. AMACOM Div American Mgmt Assn.
Agrawal, H., Horgan, J., London, S., & Wong, W. (1995). Fault localization using execution slices and dataflow tests. In conference on software reliability engineering (pp. 143–151).
Ahmadzadeh, M., Elliman, D., & Higgins, C. (2005). An analysis of patterns of debugging among novice computer science students. ACM SIGCSE Bulletin, 37(3), 84–88.
Artzi, S., Dolby, J., Tip, F., & Pistoia, M. (2010). Practical fault localization for dynamic web applications. In international conference on software engineering (pp. 265–274). ACM.
Arumuga Nainar, P., & Liblit, B. (2010). Adaptive bug isolation. In international conference on software engineering (pp. 255–264). ACM.
Baah, G. K., Podgurski, A., & Harrold, M. J. (2010). Causal inference for statistical fault localization. In international symposium on software testing and analysis (pp. 73–84). ACM.
Ballou, M. C. (2008). Improving software quality to drive business agility. IDC Survey and White Paper.
Chmiel, R., & Loui, M. C. (2004). Debugging: From novice to expert. ACM SIGCSE Bulletin, 36(1), 17–21.
Cleve, H., & Zeller, A. (2005). Locating causes of program failures. In international conference on software engineering (pp. 342–351). ACM.
Dallmeier, V., Lindig, C., & Zeller, A. (2005). Lightweight defect localization for java. In European conference on object-oriented programming (pp. 528–550). Springer.
Eisenstadt, M. (1997). My hairiest bug war stories. Communications of the ACM, 40(4), 30–37.
Gould, J. D. (1975). Some psychological evidence on how people debug computer programs. International Journal of Man-Machine Studies, 7(2), 151–182.
Gould, J. D., & Drongowski, P. (1974). An exploratory study of computer program debugging. The Journal of the Human Factors and Ergonomics Society, 16(3), 258–277.
Grötker, T., Holtmann, U., Keding, H., & Wloka, M. (2012). The developer’s guide to debugging (2nd ed.). NewYork: Self-publishing company.
Gupta, N., He, H., Zhang, X., & Gupta, R. (2005). Locating faulty code using failure-inducing chops. In international conference on automated software engineering (pp. 263–272). ACM.
Hailpern, B., & Santhanam, P. (2002). Software debugging, testing, and verification. IBM Systems Journal, 41(1), 4–12.
Hanks, B., & Brandt, M. (2009). Successful and unsuccessful problem solving approaches of novice programmers. ACM SIGCSE Bulletin, 41(1), 24–28.
James, S., Bidgoli, M., & Hansen, J. (2008). Why sally and joey can’t debug: Next generation tools and the perils they pose. Journal of Computing Sciences in Colleges, 24(1), 27–35.
Janssen, T., Abreu, R., & van Gemund, A. J. (2009). Zoltar: A toolset for automatic fault localization. In international conference on automated software engineering (pp. 662–664). IEEE Computer Society.
Jeffrey, D., Gupta, N., & Gupta, R. (2008). Fault localization using value replacement. In international symposium on software testing and analysis (pp. 167–178). ACM.
Jiang, L., & Su, Z. (2007). Context-aware statistical debugging: From bug predictors to faulty control flow paths. In international conference on automated software engineering (pp. 184–193). ACM.
Jones, J. A., Bowring, J. F., & Harrold, M. J. (2007). Debugging in parallel. In international symposium on software testing and analysis (pp. 16–26). ACM.
Kernighan, B. W., & Plauger, P. J. (1978). The elements of programming style (Vol. 1). NewYork: McGraw-Hill.
LaToza, T. D., & Myers, B. A. (2010). Developers ask reachability questions. In international conference on software engineering, vol. 1, (pp. 185–194). IEEE.
Lencevicius, R. (2000). On-the-fly query-based debugging with examples. arXiv.
Lewis, B. (2003). Debugging backwards in time. In proceedings of the international workshop on automated debugging, AADEBUG (pp. 225–235). Arxiv.
Liblit, B., Naik, M., Zheng, A. X., Aiken, A., & Jordan, M. I. (2005). Scalable statistical bug isolation. ACM SIGPLAN Notices, 40(6), 15–26.
Lieberman, H. (1997). The debugging scandal and what to do about it (introduction to the special section). Community ACM, 40(4), 26–29.
Liu, C., Yan, X., Fei, L., Han, J., & Midkiff, S. P. (2005). Sober: Statistical model-based bug localization. ACM SIGSOFT Software Engineering Notes, 30(5), 286–295.
Metzger, R. C. (2004). Debugging by thinking: A multidisciplinary approach. NewYork: Elsevier Digital Press.
Murphy, L., Lewandowski, G., McCauley, R., Simon, B., Thomas, L., & Zander, C. (2008). Debugging: The good, the bad, and the quirky-a qualitative analysis of novices’ strategies. ACM SIGCSE Bulletin, 40(1), 163–167.
Park, S., Vuduc, R. W., & Harrold, M. J. (2010). Falcon: Fault localization in concurrent programs. In international conference on software engineering (pp. 245–254). ACM.
Pearson, K. (1895). Notes on regression and inheritance in the case of two parents. In proceedings of the royal society of London (pp. 240–242). The royal society.
Pearson, K. (1900). On the criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling. In philosophical magazine (pp. 157–175).
Perscheid, M. (2013). Test-driven fault navigation for debugging reproducible failures. Ph.D. thesis, Hasso Plattner Institute, University of Potsdam.
Renieres, M., & Reiss, S. P. (2003). Fault localization with nearest neighbor queries. In international conference on automated software engineering (pp. 30–39). IEEE.
Vessey, I. (1985). Expertise in debugging computer programs: A process analysis. International Journal of Man-Machine Studies, 23(5), 459–494.
Weiser, M. (1982). Programmers use slices when debugging. Communications of the ACM, 25(7), 446–452.
Wong, W. E., & Debroy, V. (2009). A survey of software fault localization. Department of Computer Science, University of Texas at Dallas, Technical Report UTDCS-45-09.
Yilmaz, C., Paradkar, A., & Williams, C. (2008). Time will tell: Fault localization using time spectra. In international conference on software engineering (pp. 81–90). ACM.
Zeller, A. (2002). Isolating cause-effect chains from computer programs. In symposium on foundations of software engineering (pp. 1–10). ACM.
Zeller, A. (2009). Why programs fail: A guide to systematic debugging. Burlington: Morgan Kaufmann.
Zhang, X., Gupta, N., & Gupta, R. (2006). Locating faults through automated predicate switching. In international conference on software engineering (pp. 272–281). ACM.
About this article
Cite this article
Perscheid, M., Siegmund, B., Taeumel, M. et al. Studying the advancement in debugging practice of professional software developers. Software Qual J 25, 83–110 (2017). https://doi.org/10.1007/s11219-015-9294-2
- Literature review
- Field study
- Online survey