On-Line Predictive Load Shedding for Network Monitoring

  • Pere Barlet-Ros
  • Diego Amores-López
  • Gianluca Iannaccone
  • Josep Sanjuàs-Cuxart
  • Josep Solé-Pareta
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4479)


Building robust network monitoring applications is hard given the unpredictable nature of network traffic. Complex analysis on streaming network data usually leads to overload situations when presented with anomalous traffic, extreme traffic mixes or highly variable rates. We present an on-line predictive load shedding scheme for monitoring systems that quickly reacts to overload situations by gracefully degrading the accuracy of analysis methods. The main novelty of our approach is that it does not require any knowledge of the monitoring applications. This way we preserve a high degree of flexibility, increasing the potential uses of these systems. We implemented our scheme in an existing network monitoring system and deployed it in a research ISP network. Our experiments show a 10-fold improvement in the accuracy of the results during long-lived executions with several concurrent monitoring applications. The system efficiently handles extreme load situations, while being always responsive and without undesired packet losses.


Network monitoring load shedding resource management traffic sampling resource usage monitoring resource usage prediction 


  1. 1.
    The OneLab project:
  2. 2.
    claffy, k., et al.: Community-oriented network measurement infrastructure (CONMI) workshop report. SIGCOMM Comput. Commun. Rev. 36(2), 41–48 (2006)CrossRefGoogle Scholar
  3. 3.
    Cranor, C., et al.: Gigascope: A stream database for network applications. In: Proceedings of ACM Sigmod, June 2003, pp. 647–651. ACM Press, New York (2003)Google Scholar
  4. 4.
    Iannaccone, G.: Fast prototyping of network data mining applications. In: Proceedings of Passive and Active Measurement Conference (March 2006)Google Scholar
  5. 5.
    Stankovic, J.A., et al.: The case for feedback control real-time scheduling. In: Proceedings of the 11th Euromicro Conference on Real-Time Systems, Jun 1999, pp. 11–20 (1999)Google Scholar
  6. 6.
    Keys, K., Moore, D., Estan, C.: A robust system for accurate real-time summaries of internet traffic. In: Proceedings of ACM Sigmetrics, Banff, Alberta, Canada, pp. 85–96. ACM Press, New York (2005)Google Scholar
  7. 7.
    Barlet-Ros, P., et al.: Predicting resource usage of arbitrary network traffic queries. Technical report, Technical University of Catalonia (December 2006),
  8. 8.
    Cisco Systems: NetFlow services and applications. White Paper (2000)Google Scholar
  9. 9.
    Estan, C., et al.: Building a better NetFlow. In: Proceedings of ACM Sigcomm, August 2004, pp. 245–256. ACM Press, New York (2004)Google Scholar
  10. 10.
    Dreger, H., et al.: Operational experiences with high-volume network intrusion detection. In: Proceedings of ACM Conference on Computer and Communications Security, Washington DC, USA, pp. 2–11. ACM Press, New York (2004)Google Scholar
  11. 11.
    Paxson, V.: Bro: A system for detecting network intruders in real-time. Computer Networks 31, 2435–2463 (1999)CrossRefGoogle Scholar
  12. 12.
    Gonzalez, J., Paxson, V.: Enhancing network intrusion detection with integrated sampling and filtering. In: Proceedings of International Symposium on Recent Advances in Intrusion Detection, pp. 272–289 (2006)Google Scholar
  13. 13.
    Tatbul, N.: Load shedding in a data stream manager. In: Proceedings of International Conference on Very Large Data Bases, pp. 309–320 (2003)Google Scholar
  14. 14.
    Reiss, F., Hellerstein, J.M.: Declarative network monitoring with an underprovisioned query processor. In: Proceedings of International Conference on Data Engineering, pp. 56–67. IEEE Computer Society Press, Los Alamitos (2006)Google Scholar
  15. 15.
    Welsh, M., Culler, D.E., Brewer, E.A.: SEDA: An architecture for well-conditioned, scalable internet services. In: Proceedings of ACM Symposium on Operating System Principles, Banff, Alberta, Canada, pp. 230–243. ACM Press, New York (2001)Google Scholar
  16. 16.
    Stevens, W.R.: TCP Slow Start, Congestion Avoidance, Fast Retransmit, and Fast Recovery algorithms. RFC 2001 (January 1997)Google Scholar
  17. 17.
    Duffield, N.: Sampling for passive internet measurement: A review. Statistical Science 19(3), 472–498 (2004)MathSciNetCrossRefzbMATHGoogle Scholar
  18. 18.
    Carter, J.L., Wegman, M.N.: Universal classes of hash functions. Journal of Computer and System Sciences 18(2), 143–154 (1979)MathSciNetCrossRefzbMATHGoogle Scholar
  19. 19.
  20. 20.
    Barakat, C., Iannaccone, G., Diot, C.: Ranking flows from sampled traffic. In: Proceedings of ACM conference on Emerging network experiment and technology, Toulouse, France, pp. 188–199. ACM Press, New York (2005)Google Scholar
  21. 21.
    Estan, C., Varghese, G., Fisk, M.: Bitmap algorithms for counting active flows on high speed links. In: Proceedings of ACM SIGCOMM Conference on Internet Measurement, pp. 153–166. ACM Press, New York (2003)Google Scholar
  22. 22.
    Duffield, N., Lund, C., Thorup, M.: Flow sampling under hard resource constraints. In: Proceedings of ACM Sigmetrics, pp. 85–96. ACM Press, New York (2004)Google Scholar

Copyright information

© IFIP International Federation for Information Processing 2007

Authors and Affiliations

  • Pere Barlet-Ros
    • 1
  • Diego Amores-López
    • 1
  • Gianluca Iannaccone
    • 2
  • Josep Sanjuàs-Cuxart
    • 1
  • Josep Solé-Pareta
    • 1
  1. 1.Technical University of Catalonia (UPC), Computer Architecture Dept., Jordi Girona, 1-3 (Campus Nord D6), Barcelona 08034Spain
  2. 2.Intel Research, 15 JJ Thomson Avenue, Cambridge CB3 0FDUK

Personalised recommendations