Examining the stability of logging statements

Abstract

Logging statements (embedded in the source code) produce logs that assist in understanding system behavior, monitoring choke-points and debugging. Prior work showcases the importance of logging statements in operating, understanding and improving software systems. The wide dependence on logs has lead to a new market of log processing and management tools. However, logs are often unstable, i.e., the logging statements that generate logs are often changed without the consideration of other stakeholders, causing sudden failures of log processing tools and increasing the maintenance costs of such tools. We examine the stability of logging statements in four open source applications namely: Liferay, ActiveMQ, Camel and CloudStack. We find that 20–45% of their logging statements change throughout their lifetime. The median number of days between the introduction of a logging statement and the first change to that statement is between 1 and 17 in our studied applications. These numbers show that in order to reduce maintenance effort, developers of log processing tools must be careful when selecting the logging statements on which their tools depend. In order to effectively mitigate the issues that are caused by unstable logging statements, we make an important first step towards determining whether a logging statement is likely to remain unchanged in the future. First, we use a random forest classifier to determine whether a just-introduced logging statement will change in the future, based solely on metrics that are calculated when it is introduced. Second, we examine whether a long-lived logging statement is likely to change based on its change history. We leverage Cox proportional hazards models (Cox models) to determine the change risk of long-lived logging statements in the source code. Through our case study on four open source applications, we show that our random forest classifier achieves a 83–91% precision, a 65–85% recall and a 0.95–0.96 AUC. We find that file ownership, developer experience, log density and SLOC are important metrics in our studied projects for determining the stability of logging statements in both our random forest classifiers and Cox models. Developers can use our approach to determine the risk of a logging statement changing in their own projects, to construct more robust log processing tools, by ensuring that these tools depend on logs that are generated by more stable logging statements.

Appendix A: Background on Survival Analysis

Survival analysis comprises a set of statistical modeling techniques that model the time taken for an event to occur (Miller 2011). These modeling techniques can be parametric, semi-parametric or non-parametric in form. However, they share the common goal of modeling the time that it takes between the start of an observation period (i.e., logging statement introduction) and an event (i.e., logging statement change) to occur i.e., they model the survival time of a logging statement. Survival analysis also helps in identifying the important metrics that affect the survival time of a logging statement. The following section discusses the crucial aspect of survival analysis as described in Syer et al. (2015): survival analysis data and measuring time to event.

1.1 Survival Analysis Data and Measuring Time to Event

Survival analysis uses the data that is collected at specific time intervals to observe the relation between how a subject changes over time and the occurrence of an event of interest (e.g., whether a log statement changes). We explain survival analysis using the stability of logging statements as an example.To model the time to change of a logging statement, we collect the data about content, context and developers (metric) for each release (observation period) after a logging statement (subject) is introduced into the application. Each observation in the survival data contains the following fields:

• 1. UID: Unique number of each logging statements.

• 2. Start: Time of introduction of a logging statement.

• 3. Stop: the time at which the logging statement changes.

• 4. Event: (1) if the logging statement was changed or (0) if the logging statement was not changed at the end of observation period.

• 5. Metrics: The content, context and developer metrics.

Table 10 shows the survival data for a logging statement (Log-1), where the observations are recorded at the beginning of a release. If a logging statement is changed (event occurs), the logging statement is not tracked and the study halted for that particular logging statement. However, some logging statements may never be changed and in such cases it is impractical to track them. Hence, the logging statements are tracked for a certain period of time (e.g., 3 years), during which they may or may not be changed.

Table 10 Data for survival analysis

To conduct the survival analysis we need to define how we measure the introducing event (i.e., the first release after introduction of a logging statement), the censored event (i.e., the subsequent months where the logging statement is not changed) and the terminating event (i.e., month the logging statement is changed). From Table 10, we find that the logging statement is changed in the second release, which makes it the terminating event. In the prior releases, the event of interest does not occur which makes the observations censored events. In addition, when a logging statement is not changed during the period of study (i.e., 3 years), their survival is considered equal to the period of study. We include both censored and terminating events for our survival analysis as the models can handle both censored and terminating events and can produce effective survival models without bias (Hosmer and Lemeshow 1999).