An Experimental Evaluation of Deliberate Unsoundness in a Static Program Analyzer
Many practical static analyzers are not completely sound by design. Their designers trade soundness to increase automation, improve performance, and reduce the number of false positives or the annotation overhead. However, the impact of such design decisions on the effectiveness of an analyzer is not well understood. This paper reports on the first systematic effort to document and evaluate the sources of unsoundness in a static analyzer. We developed a code instrumentation that reflects the sources of deliberate unsoundness in the .NET static analyzer Clousot and applied it to code from six open-source projects. We found that 33% of the instrumented methods were analyzed soundly. In the remaining methods, Clousot made unsound assumptions, which were violated in 2–26% of the methods during concrete executions. Manual inspection of these methods showed that no errors were missed due to an unsound assumption, which suggests that Clousot’s unsoundness does not compromise its effectiveness. Our findings can guide users of static analyzers in using them fruitfully, and designers in finding good trade-offs.
KeywordsTest Suite Access Path Explicit Assumption Object Invariant Assumed Statement
Unable to display preview. Download preview PDF.
- 4.Blazy, S., Laporte, V., Maroneze, A., Pichardie, D.: Formal verification of a C value analysis based on abstract interpretation. In: Logozzo, F., Fähndrich, M. (eds.) SAS 2013. LNCS, vol. 7935, pp. 324–344. Springer, Heidelberg (2013)Google Scholar
- 7.Cousot, P., Cousot, R., Feret, J., Miné, A., Mauborgne, L., Monniaux, D., Rival, X.: Varieties of static analyzers: A comparison with ASTRÉE. In: TASE, pp. 3–20. IEEE Computer Society (2007)Google Scholar
- 9.Fähndrich, M., Barnett, M., Logozzo, F.: Embedded contract languages. In: SAC, pp. 2103–2110. ACM (2010)Google Scholar
- 11.Liang, P., Tripp, O., Naik, M., Sagiv, M.: A dynamic evaluation of the precision of static heap abstractions. In: OOPSLA, pp. 411–427. ACM (2010)Google Scholar
- 12.Livshits, B., Lahiri, S.K. In: defense of probabilistic static analysis. In: APPROX (2014)Google Scholar
- 13.Logozzo, F., Lahiri, S.K., Fähndrich, M., Blackshear, S.: Verification modulo versions: Towards usable verification. In: PLDI, pp. 294–304. ACM (2014)Google Scholar
- 16.Summers, A.J., Müller, P.: Freedom before commitment: A lightweight type system for object initialisation. In: OOPSLA, pp. 1013–1032. ACM (2011)Google Scholar