Instrumentation Framework

Abstract

This chapter describes the process of going from unmodified application source code to native executables with sampled instrumentation. This process is managed by the instrumentor: a software tool whose external behavior mimics that of the native compiler, but that internally applies the instrumentation injection and sampling transformation steps depicted at the top center of Fig. 1.1. Our instrumentor, sampler-cc, is implemented as a source-to-source transformation for C using the CIL C front end [48]. Transformed code then proceeds to GCC for native compilation. From the developer’s perspective, the sampler-cc command behaves exactly like the gcc command with a few extra instrumentation-related command line flags.

Section 2.1 presents the basic strategy for managing fair, randomly sampled instrumentation. This sampling transformation is quite general, with potential applications beyond bug hunting. However, bug hunting is the focus of this book, and Sect. 2.2 describes several instrumentation schemes that may be used with the sampling transformation and that we have found to be helpful for bug isolation. Section 2.3 considers performance issues and examines several optimizations that may be applied atop the basic sampling transformation. Section 2.4 describes an adaptive, non-uniformly sampled generalization of the core random sampling model, while Sect. 2.5 closes the chapter with an informal discussion of realistic sampling rates in truly large scale deployments.

The major difference between a thing that might go wrong and a thing that cannot possibly go wrong is that when a thing that cannot possibly go wrong goes wrong, it usually turns out to be impossible to get at or repair.

Douglas Adams, Mostly Harmless