Abstract
Summary statistics represent a key primitive for profiling and protecting operational networks. Many network operators routinely measure properties such as throughput, traffic mix, and heavy hitters. Likewise, security monitoring often deploys statistical anomaly detectors that trigger, e.g., when a source scans the local IP address range, or exceeds a threshold of failed login attempts. Traditionally, a diverse set of tools is used for such computations, each typically hard-coding either the features it operates on or the specific calculations it performs, or both. In this work we present a novel framework for calculating a wide array of summary statistics in real-time, independent of the underlying data, and potentially aggregated from independent monitoring points. We focus on providing a transparent, extensible, easy-to-use interface and implement our design on top of an open-source network monitoring system. We demonstrate a set of example applications for profiling and statistical anomaly detection that would traditionally require significant effort and different tools to compute. We have released our implementation under BSD license and report experiences from real-world deployments in large-scale network environments.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Barman, D., Satapathy, P., Ciardo, G.: Detecting Attacks in Routers using Sketches. In: Workshop on High Performance Switching and Routing, HPSR (2007)
Bro SumStat Scripts & Repos, http://www.icir.org/johanna/sumstats
Bro Network Security Monitor Web Site, http://www.bro.org
Cohen, E., Duffield, N., Kaplan, H., Lund, C., Thorup, M.: Composable, Scalable, and Accurate Weight Summarization of Unaggregated Data Sets. Proc. VLDB Endow. 2(1) (August 2009)
Das, S., Antony, S., Agrawal, D., El Abbadi, A.: Thread Cooperation in Multicore Architectures for Frequency Counting over Multiple Data Streams. Proc. VLDB Endow. 2(1) (August 2009)
Dean, J., Ghemawat, S.: MapReduce: Simplified Data Processing on Large Clusters. Commun. ACM 51(1) (January 2008)
Denning, D.E.: An Intrusion-Detection Model. IEEE TSE 13(2) (February 1987)
Estan, C., Varghese, G.: New Directions in Traffic Measurement and Accounting: Focusing on the Elephants, ignoring the Mice. ACM Trans. Comput. Syst. 21(3) (August 2003)
Estan, C., Varghese, G., Fisk, M.: Bitmap Algorithms for Counting Active Flows on High-Speed Links. IEEE/ACM Trans. Netw. 14(5) (October 2006)
Flajolet, P., Fusy, É., Gandouet, O., et al.: Hyperloglog: The Analysis of a Near-Optimal Cardinality Estimation Algorithm. In: Proc. of the International Conference of Analysis of Algorithms, AFOA (2007)
Flow-tools information, http://www.splintered.net/sw/flow-tools
Garcia-Teodoro, P., DÃaz-Verdejo, J.E., Maciá-Fernández, G., Vzquez, E.: Anomaly-Based Network Intrusion Detection: Techniques, Systems and Challenges. Computers & Security 28(1-2) (2009)
Heule, S., Nunkesser, M., Hall, A.: HyperLogLog in Practice: Algorithmic Engineering of a State of The Art Cardinality Estimation Algorithm. In: Proc. EDBT (2013)
Kane, D.M., Nelson, J., Woodruff, D.P.: An Optimal Algorithm for the Distinct Elements Problem. In: Proceedings ACM PODS (2010)
Keys, K., Moore, D., Estan, C.: A Robust System for Accurate Real-Time Summaries of Internet Traffic. In: Proc. SIGMETRICS (2005)
Kim, H.A., O’Hallaron, D.R.: Counting Network Flows in Real Time. In: Proc. IEEE Global Telecommunications Conference, vol. 7 (2003)
Metwally, A., Agrawal, D., El Abbadi, A.: Efficient Computation of Frequent and Top-k Elements in Data Streams. In: Proc. ICDT (2005)
Patcha, A., Park, J.M.: An Overview of Anomaly Detection Techniques: Existing Solutions and Latest Technological Trends. Computer Networks 51(12) (2007)
Paxson, V.: Bro: A System for Detecting Network Intruders in Real-Time. Computer Networks 31(23-24) (1999)
Peng, T., Leckie, C., Ramamohanarao, K.: Information Sharing for Distributed Intrusion Detection Systems. Journal of Network and Computer Applications 30(3) (August 2007)
Roesch, M.: Snort: Lightweight Intrusion Detection for Networks. In: LISA (1999)
SILK – System for Internet-Level Knowledge, http://tools.netsa.cert.org/silk/
Sommer, R., Paxson, V.: Exploiting Independent State For Network Intrusion Detection. In: ACSAC (2005)
Sridharan, A., Ye, T.: Tracking Port Scanners on the IP Backbone. In: Proc. Workshop on Large Scale Attack Defense, LSAD (2007)
Vallentin, M., Sommer, R., Lee, J., Leres, C., Paxson, V., Tierney, B.: The NIDS Cluster: Scalable, Stateful Network Intrusion Detection on Commodity Hardware. In: Kruegel, C., Lippmann, R., Clark, A. (eds.) RAID 2007. LNCS, vol. 4637, pp. 107–126. Springer, Heidelberg (2007)
Vitter, J.S.: Random Sampling with a Reservoir. ACM TOMS 11(1) (March 1985)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer International Publishing Switzerland
About this paper
Cite this paper
Amann, J., Hall, S., Sommer, R. (2014). Count Me In: Viable Distributed Summary Statistics for Securing High-Speed Networks. In: Stavrou, A., Bos, H., Portokalidis, G. (eds) Research in Attacks, Intrusions and Defenses. RAID 2014. Lecture Notes in Computer Science, vol 8688. Springer, Cham. https://doi.org/10.1007/978-3-319-11379-1_16
Download citation
DOI: https://doi.org/10.1007/978-3-319-11379-1_16
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-11378-4
Online ISBN: 978-3-319-11379-1
eBook Packages: Computer ScienceComputer Science (R0)