Measuring Cloud Service Health Using NetFlow/IPFIX: The WikiLeaks Case

Abstract

The increasing trend of outsourcing services to cloud providers is changing the way computing power is delivered to enterprises and end users. Although cloud services offer several advantages, they also make cloud consumers strongly dependent on providers. Hence, consumers have a vital interest to be immediately informed about any problems in their services. This paper aims at a first step toward a network-based approach to monitor cloud services. We focus on severe problems that affect most services, such as outages or extreme server overload, and propose a method to monitor these problems that relies solely on the traffic exchanged between users and cloud providers. Our proposal is entirely based on NetFlow/IPFIX data and, therefore, explicitly targets high-speed networks. By combining a methodology to reassemble and classify flow records with stochastic estimations, our proposal has the distinct characteristic of being applicable to both sampled and non-sampled data. We validate our proposal and show its applicability using data collected at both the University of Twente and an international backbone during the WikiLeaks Cablegate. Our results show that, in contrast to Anonymous’ claims, the users of the targeted services have been only marginally affected by the attacks.

Appendices

Appendix 1: Background Flow Monitoring

1.1 Definition

We assume flow records to be defined as in NetFlow/IPFIX [7]: a unidirectional set of packets crossing an observation point, sharing a common set of attributes (the flow key). NetFlow v9 [6] uses a fixed flow key composed of source and destination IP addresses and port numbers, IP protocol number, IP type of service and the input interface index. IPFIX offers more flexibility by supporting variable flow definitions [7]. More important for this work, NetFlow and IPFIX [6, 49] define the following reasons for a flow record to be exported:

• 1. Idle timeout: No packets belonging to a flow have been observed for a specified period of time;

• 2. Active timeout: A flow has been active for a specified period of time;

• 3. End of flow detected: The end of a flow has been identified. For instance, TCP flags can be used to expire flow records;

• 4. Forced end: An external event, such as a shutdown of the exporting device, forces all flow records to be exported;

• 5. Lack of resources: Special heuristics can expire flow records prematurely in case of resource constraints on the exporting device.

After expiration, flow records are sent to flow collectors. In IPFIX, only the protocol for transporting records from exporters to collectors is specified. All details related to flow measurement, including the configuration of expiration rules, are considered implementation decisions [50]. Therefore, flow data can differ per exporting device. Since our method aims to be robust against different flow definitions, it must handle data exported according to any of these rules. See Sect. 3 for a discussion about the consequences of different flow definitions to our methodology.

1.2 Why Raw Flow Records are Not Sufficient

All NetFlow/IPFIX expiration rules can cause TCP connections to be split into multiple flow records. However, since TCP is bidirectional, one should expect that those rules would produce the same number of records in both traffic directions for a healthy service. Figure 10 illustrates the invalidity of this reasoning using traffic to Google and Facebook in our network.

Fig. 10

Incoming/outgoing flow records for healthy Web services. a Google (non-sampled), b Facebook (non-sampled), c Google (sampled, p = 0.01), d Facebook (sampled, p = 0.01)

Figure 10a, b depict the number of flow records when packet sampling is not applied. The packet traces of our validations are converted into NetFlow records, with idle and active timeouts set to the typical values of our flow exporters (30 and 120 s, respectively). ⁶ The difference between the quantities is also shown, with negative values representing more incoming records. In general, more outgoing than incoming records are seen. This difference is mainly caused by the widespread use of TCP RST to terminate connections (see [23]). This can result in an extra flow record in either traffic direction, depending on timeout parameters.

Interestingly, an opposite pattern is seen when sampling is applied. For illustration, Fig. 10c, d depict the results of applying independent random sampling with probability p = 0.01 to the same packet traces. The figures show that the number of incoming records tends to be higher than the number of outgoing records. This happens because servers normally send more packets to clients, increasing the probability of sampling incoming flows.

Appendix 2: The Performance of Classification Models

As defined by [24], “a classification model (or classifier) is a mapping from instances to predicted classes”. In the classification problem studied in Sect. 3, the instances are TCP connections, which are mapped to discrete labels (healthy, unhealthy, unmatched). This appendix reviews the background on performance metrics commonly used to compare classifiers.

1.1 Confusion Matrix

The confusion matrix is a way of presenting results when evaluating classifiers [24]. It presents the original number of instances per class (i.e., the ground truth in a test set) versus the total number of instances that are predicted to belong to each class. Table 3 shows an example of such a matrix M, in a problem composed of n classes (n = 3 in our case). The diagonal contains correctly classified instances, whereas the remaining cells contain the errors per class.

Table 3 Confusion matrix M when classifying instances in n classes

Note that, in our problem, no instance can be correctly classified as unmatched, since no connections of this class exist in the ground truth set. This artificial state, however, is needed because the output of our approach may have different cardinality than the ground truth set. Instances in the unmatched state penalize our method when it wrongly merges flow records.

1.2 Performance Metrics

Several performance metrics to compare classification schemes can be derived from the confusion matrix. The most simplistic one is the accuracy A, which is defined as the overall fraction of correctly classified instances:

$$ A=\frac{\sum\nolimits _{i=1}^{n}M_{ii}} {\sum\nolimits _{i=1}^{n}\sum\nolimits _{j=1}^{n}M_{ij}} $$

(5)

The accuracy, however, fails to provide insights about the source of errors of a classifier. This is particularly the case when the different classes in the classification problem are unbalanced—that is, when the probability of observing instances of a specific class is much lower than the probability of observing the remaining classes. It is very common that the rarest class is exactly the most important one to be correctly classified. In our case, instances of the healthy class are much more frequent, accounting for almost 90 % of the samples in our test set. Any classification model could reach high accuracy by predicting that all instances belong to the healthy class, although such models would be of no use. Metrics that capture the performance of a classifier in a finer granularity are therefore needed.

Three metrics are of primary interest in our analysis: Precision (P _i ), recall (R _i ), and F-measure (F _i ) [24, 25]. Assuming $\varvec{\omega}$ to be the classes in a classification problem (i.e., ${\varvec{\omega}}=(healthy,unhealthy,unmatched)$), these metrics are defined as a function of the class ω _i . The precision P _i represents the fraction of instances correctly classified as being of class ω _i . It can be calculated from the confusion matrix, as follows.

$$ P_{i}=\frac{M_{ii}}{\sum\nolimits _{j=1}^{n}M_{ji}} $$

(6)

The recall R _i is the fraction of all original instances of class ω _i that are correctly classified:

$$ R_{i}=\frac{M_{ii}}{\sum\nolimits _{j=1}^{n}M_{ij}} $$

(7)

As for the overall accuracy, tuning a classifier to optimize only one of these metrics for a specific class may produce degenerated models. For instance, a classifier that labels all instances as ω _i has R _i  = 1, but P _i equals to the prior probability of ω _i . To overcome that, the F-measure (F _i ) can be applied to assess the classifier:

$$ F_{i}=\frac{P_{i}R_{i}}{(1-\alpha)P_{i}+\alpha R_{i}}\quad 0\leq\alpha\leq1 $$

(8)

The F-measure is a combination of precision and recall that increases faster when both metrics are simultaneously increased. The parameter α weights the importance of each metric in the results. This can be used, for instance, when failing to identify instances of a class has a different cost than making classification mistakes. In all our experiments, we calculate F _i using α = 0.5, which means that both precision and recall have equal importance. We refer the readers to [25] for a deeper discussion and to [51] for a practical example of the F-measure use in another domain.