Count Me In: Viable Distributed Summary Statistics for Securing High-Speed Networks

  • Johanna Amann
  • Seth Hall
  • Robin Sommer
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8688)


Summary statistics represent a key primitive for profiling and protecting operational networks. Many network operators routinely measure properties such as throughput, traffic mix, and heavy hitters. Likewise, security monitoring often deploys statistical anomaly detectors that trigger, e.g., when a source scans the local IP address range, or exceeds a threshold of failed login attempts. Traditionally, a diverse set of tools is used for such computations, each typically hard-coding either the features it operates on or the specific calculations it performs, or both. In this work we present a novel framework for calculating a wide array of summary statistics in real-time, independent of the underlying data, and potentially aggregated from independent monitoring points. We focus on providing a transparent, extensible, easy-to-use interface and implement our design on top of an open-source network monitoring system. We demonstrate a set of example applications for profiling and statistical anomaly detection that would traditionally require significant effort and different tools to compute. We have released our implementation under BSD license and report experiences from real-world deployments in large-scale network environments.


Intrusion Detection Destination Address Work Node Memory Overhead Heavy Hitter 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Barman, D., Satapathy, P., Ciardo, G.: Detecting Attacks in Routers using Sketches. In: Workshop on High Performance Switching and Routing, HPSR (2007)Google Scholar
  2. 2.
    Bro SumStat Scripts & Repos,
  3. 3.
    Bro Network Security Monitor Web Site,
  4. 4.
    Cohen, E., Duffield, N., Kaplan, H., Lund, C., Thorup, M.: Composable, Scalable, and Accurate Weight Summarization of Unaggregated Data Sets. Proc. VLDB Endow. 2(1) (August 2009)Google Scholar
  5. 5.
    Das, S., Antony, S., Agrawal, D., El Abbadi, A.: Thread Cooperation in Multicore Architectures for Frequency Counting over Multiple Data Streams. Proc. VLDB Endow. 2(1) (August 2009)Google Scholar
  6. 6.
    Dean, J., Ghemawat, S.: MapReduce: Simplified Data Processing on Large Clusters. Commun. ACM 51(1) (January 2008)Google Scholar
  7. 7.
    Denning, D.E.: An Intrusion-Detection Model. IEEE TSE 13(2) (February 1987)Google Scholar
  8. 8.
    Estan, C., Varghese, G.: New Directions in Traffic Measurement and Accounting: Focusing on the Elephants, ignoring the Mice. ACM Trans. Comput. Syst. 21(3) (August 2003)Google Scholar
  9. 9.
    Estan, C., Varghese, G., Fisk, M.: Bitmap Algorithms for Counting Active Flows on High-Speed Links. IEEE/ACM Trans. Netw. 14(5) (October 2006)Google Scholar
  10. 10.
    Flajolet, P., Fusy, É., Gandouet, O., et al.: Hyperloglog: The Analysis of a Near-Optimal Cardinality Estimation Algorithm. In: Proc. of the International Conference of Analysis of Algorithms, AFOA (2007)Google Scholar
  11. 11.
  12. 12.
    Garcia-Teodoro, P., Díaz-Verdejo, J.E., Maciá-Fernández, G., Vzquez, E.: Anomaly-Based Network Intrusion Detection: Techniques, Systems and Challenges. Computers & Security 28(1-2) (2009)Google Scholar
  13. 13.
    Heule, S., Nunkesser, M., Hall, A.: HyperLogLog in Practice: Algorithmic Engineering of a State of The Art Cardinality Estimation Algorithm. In: Proc. EDBT (2013)Google Scholar
  14. 14.
    Kane, D.M., Nelson, J., Woodruff, D.P.: An Optimal Algorithm for the Distinct Elements Problem. In: Proceedings ACM PODS (2010)Google Scholar
  15. 15.
    Keys, K., Moore, D., Estan, C.: A Robust System for Accurate Real-Time Summaries of Internet Traffic. In: Proc. SIGMETRICS (2005)Google Scholar
  16. 16.
    Kim, H.A., O’Hallaron, D.R.: Counting Network Flows in Real Time. In: Proc. IEEE Global Telecommunications Conference, vol. 7 (2003)Google Scholar
  17. 17.
    Metwally, A., Agrawal, D., El Abbadi, A.: Efficient Computation of Frequent and Top-k Elements in Data Streams. In: Proc. ICDT (2005)Google Scholar
  18. 18.
    Patcha, A., Park, J.M.: An Overview of Anomaly Detection Techniques: Existing Solutions and Latest Technological Trends. Computer Networks 51(12) (2007)Google Scholar
  19. 19.
    Paxson, V.: Bro: A System for Detecting Network Intruders in Real-Time. Computer Networks 31(23-24) (1999)Google Scholar
  20. 20.
    Peng, T., Leckie, C., Ramamohanarao, K.: Information Sharing for Distributed Intrusion Detection Systems. Journal of Network and Computer Applications 30(3) (August 2007)Google Scholar
  21. 21.
    Roesch, M.: Snort: Lightweight Intrusion Detection for Networks. In: LISA (1999)Google Scholar
  22. 22.
    SILK – System for Internet-Level Knowledge,
  23. 23.
    Sommer, R., Paxson, V.: Exploiting Independent State For Network Intrusion Detection. In: ACSAC (2005)Google Scholar
  24. 24.
    Sridharan, A., Ye, T.: Tracking Port Scanners on the IP Backbone. In: Proc. Workshop on Large Scale Attack Defense, LSAD (2007)Google Scholar
  25. 25.
    Vallentin, M., Sommer, R., Lee, J., Leres, C., Paxson, V., Tierney, B.: The NIDS Cluster: Scalable, Stateful Network Intrusion Detection on Commodity Hardware. In: Kruegel, C., Lippmann, R., Clark, A. (eds.) RAID 2007. LNCS, vol. 4637, pp. 107–126. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  26. 26.
    Vitter, J.S.: Random Sampling with a Reservoir. ACM TOMS 11(1) (March 1985)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Johanna Amann
    • 1
  • Seth Hall
    • 1
    • 2
  • Robin Sommer
    • 1
    • 2
  1. 1.International Computer Science InstituteUSA
  2. 2.Lawrence Berkeley National LaboratoryUSA

Personalised recommendations