Z-Ranking: Using Statistical Analysis to Counter the Impact of Static Analysis Approximations
This paper explores z-ranking, a technique to rank error reports emitted by static program checking analysis tools. Such tools often use approximate analysis schemes, leading to false error reports. These reports can easily render the error checker useless by hiding real errors amidst the false, and by potentially causing the tool to be discarded as irrelevant. Empirically, all tools that effectively find errors have false positive rates that can easily reach 30–100%. Z-ranking employs a simple statistical model to rank those error messages most likely to be true errors over those that are least likely. This paper demonstrates that z-ranking applies to a range of program checking problems and that it performs up to an order of magnitude better than randomized ranking. Further, it has transformed previously unusable analysis tools into effective program error finders.
KeywordsError Report True Error Ranking Scheme Weak Hypothesis Real Error
Unable to display preview. Download preview PDF.
- 1.A. Aiken, M. Faehndrich, and Z. Su. Detecting races in relay ladder logic programs. In Proceedings of the 1st International Conference on Tools and Algorithms for the Construction and Analysis of Systems, April 1998.Google Scholar
- 2.T. Ball and S.K. Rajamani. Automatically validating temporal safety properties of interfaces. In SPIN 2001 Workshop on Model Checking of Software, May 2001.Google Scholar
- 3.A. Chou, J. Yang, B. Chelf, S. Hallem, and D. Engler. An empirical study of operating systems errors. In Proceedings of the Eighteenth ACM Symposium on Operating Systems Principles, 2001.Google Scholar
- 4.Manuvir Das, Sorin Lerner, and Mark Seigle. Path-sensitive program verification in polynomial time. In Proceedings of the ACM SIGPLAN 2002 Conference on Programming Language Design and Implementation, Berlin, Germany, June 2002.Google Scholar
- 5.D. Engler, B. Chelf, A. Chou, and S. Hallem. Checking system rules using system-specific, programmer-written compiler extensions. In Proceedings of Operating Systems Design and Implementation (OSDI), September 2000.Google Scholar
- 6.D. Evans, J. Guttag, J. Horning, and Y.M. Tan. Lclint: A tool for using specifications to check code. In Proceedings of the ACM SIGSOFT Symposium on the Foundations of Software Engineering, December 1994.Google Scholar
- 7.Cormac Flanagan and Stephen N. Freund. Type-based race detection for Java. In SIGPLAN Conference on Programming Language Design and Implementation, pages 219–232, 2000.Google Scholar
- 8.J.S. Foster, T. Terauchi, and Alex Aiken. Flow-sensitive type qualifiers. In Proceedings of the ACM SIGPLAN 2002 Conference on Programming Language Design and Implementation, June 2002.Google Scholar
- 9.D. Freedman, R. Pisani, and R. Purves. Statistics. W.W. Norton, third edition, 1998.Google Scholar
- 10.S. Hallem, B. Chelf, Y. Xie, and D. Engler. A system and language for building system-specific, static analyses. In SIGPLAN Conference on Programming Language Design and Implementation, 2002.Google Scholar
- 12.Thomas J. Santer and Dianne E. Duffy. The Statistical Analysis of Discrete Data. Springer-Verlag, December 1989.Google Scholar
- 13.D. Wagner, J. Foster, E. Brewer, and A. Aiken. A first step towards automated detection of buffer overrun vulnerabilities. In 2000 NDSSC, February 2000.Google Scholar