Counting Flows over Sliding Windows in High Speed Networks

  • Josep Sanjuàs-Cuxart
  • Pere Barlet-Ros
  • Josep Solé-Pareta
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5550)

Abstract

Counting the number of flows present in network traffic is not trivial, given that the naive approach of using a hash table to track the active flows is too slow for the current backbone network speeds. Several algorithms have been proposed in the recent literature that can calculate an approximate count using small amount of memory and few memory accesses per packet. Fewer works have addressed the more complex problem of counting flows over sliding windows, where the main challenge is to continuously expire old information. One of the existing proposals is a straightforward adaptation of the direct bitmaps technique to the sliding window model. We present an algorithm called Countdown Vector that also builds upon the direct bitmaps technique. Our algorithm, however, obtains significant cost reductions both in terms of memory and CPU, by introducing an extra approximation in the mechanism in charge of the expiration of old information.

Keywords

traffic measurement counting active flows sliding windows 

References

  1. 1.
    Estan, C., Varghese, G., Fisk, M.: Bitmap algorithms for counting active flows on high speed links. In: Proc. of ACM SIGCOMM Internet Measurement Conf. (October 2003)Google Scholar
  2. 2.
    Fusy, E., Giroire, F.: Estimating the number of Active Flows in a Data Stream over a Sliding Window. In: Proc. of SIAM Workshop on Analytic Algorithmics and Combinatorics (January 2007)Google Scholar
  3. 3.
    Kim, H., O’Hallaron, D.: Counting network flows in real time. In: Proc. of IEEE GLOBECOM (December 2003)Google Scholar
  4. 4.
    Fang, W., Peterson, L.: Inter-AS traffic patterns and their implications. In: Proc. of IEEE GLOBECOM (December 1999)Google Scholar
  5. 5.
    Cisco Systems: NetFlow services and applications. White Paper (2000)Google Scholar
  6. 6.
    Barlet-Ros, P., Iannaccone, G., Sanjuàs-Cuxart, J., Amores-López, D., Solé-Pareta, J.: Load shedding in network monitoring applications. In: Proc. of USENIX Annual Technical Conf. (June 2007)Google Scholar
  7. 7.
    Duffield, N., Lund, C., Thorup, M.: Properties and prediction of flow statistics from sampled packet streams. In: Proc. of ACM SIGCOMM Internet Measurement Workshop (November 2002)Google Scholar
  8. 8.
    Whang, K.Y., Vander-Zanden, B.T., Taylor, H.M.: A linear-time probabilistic counting algorithm for database applications. ACM Trans. Database Syst. 15(2) (June 1990)Google Scholar
  9. 9.
    Durand, M., Flajolet, P.: Loglog Counting of Large Cardinalities. In: Proc. of Annual European Symposium on Algorithms (September 2003)Google Scholar
  10. 10.
    Giroire, F.: Order statistics and estimating cardinalities of massive data sets. In: Proc. of Intl. Conf. on Analysis of Algorithms (June 2005)Google Scholar
  11. 11.
    Metwally, A., Agrawal, D., El Abbadi, A.: Why go logarithmic if we can go linear?: Towards effective distinct counting of search traffic. In: Proc. of Intl. Conf. on Extending Database Technology: Advances in Database Technology (March 2008)Google Scholar
  12. 12.
    Babcock, B., Babu, S., Datar, M., Motwani, R., Widom, J.: Models and issues in data stream systems. In: Proc. of ACM SIGMOD-SIGACT-SIGART Symp. on Principles of Database Systems (June 2002)Google Scholar
  13. 13.
    Golab, L., Özsu, T.M.: Issues in data stream management. SIGMOD Record 32 (June 2003)Google Scholar
  14. 14.
    Cranor, C., Johnson, T., Spataschek, O., Shkapenyuk, V.: Gigascope: A stream database for network applications. In: Proc. of ACM SIGMOD (June 2003)Google Scholar
  15. 15.
    Iannaccone, G.: Fast prototyping of network data mining applications. In: Proc. of Passive and Active Measurement Conf. (March 2006)Google Scholar
  16. 16.
    Reiss, F., Hellerstein, J.M.: Declarative network monitoring with an underprovisioned query processor. In: Proc. of IEEE Intl. Conf. on Data Engineering (April 2006)Google Scholar
  17. 17.
    Muthukrishnan, S.: Data Streams: Algorithms And Applications. Now Publishers Inc. (2005)Google Scholar
  18. 18.
  19. 19.
    Datar, M., Gionis, A., Indyk, P., Motwani, R.: Maintaining stream statistics over sliding windows. In: Proc. of ACM-SIAM Symp. on Discrete Algorithms (January 2002)Google Scholar
  20. 20.
    Carter, J.L., Wegman, M.N.: Universal classes of hash functions. J. Comput. Syst. Sci. 18 (April 1979)Google Scholar
  21. 21.
    Jacobson, V., Leres, C., McCanne, S. (libpcap) Lawrence Berkeley Laboratory, Berkeley, CA. Initial public release (June 1994), http://www.tcpdump.org
  22. 22.
    Endace: DAG network monitoring cards, http://www.endace.com
  23. 23.
    Golab, L., DeHaan, D., Demaine, E., Lopez-Ortiz, A., Munro, J.: Identifying frequent items in sliding windows over on-line packet streams. In: Proc. of ACM SIGCOMM Internet Measurement Conf. (October 2003)Google Scholar
  24. 24.
    Sanjuàs-Cuxart, J., Barlet-Ros, P., Solé-Pareta, J.: Counting network flows over sliding windows in high-speed networks. UPC Technical Report, http://loadshedding.ccaba.upc.edu/papers/counting-network-flows.techrep2008.pdf

Copyright information

© IFIP International Federation for Information Processing 2009

Authors and Affiliations

  • Josep Sanjuàs-Cuxart
    • 1
  • Pere Barlet-Ros
    • 1
  • Josep Solé-Pareta
    • 1
  1. 1.Computer Architecture Dept.Universitat Politècnica de Catalunya (UPC)BarcelonaSpain

Personalised recommendations