Skip to main content
SpringerLink
Log in

Per-Flow Size Estimators

  • Chapter
  • First Online:
Traffic Measurement on the Internet

Part of the book series: SpringerBriefs in Computer Science ((BRIEFSCOMPUTER))

  • 665 Accesses

Abstract

This chapter discusses the measurement of per-flow sizes for high-speed links. It is a particularly difficult problem because of the need to process and store a huge amount of information, which makes it difficult for the measurement module to fit in the small but fast SRAM space (in order to operate at the line speed). We provide a novel measurement function that estimates the sizes of all flows. It delivers good performance in tight memory space where other approaches no longer work. The effectiveness of the online per-flow measurement approach is analyzed and confirmed through extensive experiments based on real network traffic traces.The rest of this chapter is organized as follows: Sect. 2.1 discusses the performance metrics. Section 2.2 gives an overview of the system design. Section 2.3 discusses the state of the art. Section 2.4 presents the online data encoding module. Sections 2.5–2.6 present two offline data decoding modules. Section 2.7 discusses the problem of setting counter length. Section 2.8 addresses the problem of collecting flow labels. Section 2.9 presents the experimental results. Section 2.10 extends the estimators for large flow sizes. Section 2.11 gives the summary.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    At the end of each measurement period, about half of the bits in the filters of MRSCBF are set to ones.

  2. 2.

    At the end of each measurement period, less than half of the bits in the filters of MRSCBF are set to ones.

References

  1. Abramowitz, M., Stegun, I.: Handbook of Mathematical Functions: with Formulas, Graphs, and Mathematical Tables. Dover Publications, New York (1964)

    Google Scholar 

  2. Bloom, B.H.: Space/Time trade-offs in hash coding with allowable errors. Commun. ACM 13(7), 422–426 (1970)

    Article  MATH  Google Scholar 

  3. Broder, A., Mitzenmacher, M.: Network applications of bloom filters: a survey. Internet Math. 1(4), 485–509 (2002)

    Article  MathSciNet  Google Scholar 

  4. CAIDA: Analyzing UDP Usage in Internet Traffic. http://www.caida.org/research/traffic-analysis/tcpudpratio/ (2009)

  5. Cao, J., Jin, Y., Chen, A., Bu, T., Zhang, Z.: Identifying high cardinality internet hosts. In: Proceedings of IEEE INFOCOM (2009)

    Google Scholar 

  6. Casella, G., Berger, R.: Statistical Inference. Duxbury Press, Pacific Grove (2001)

    Google Scholar 

  7. Cohen, S., Matias, Y.: Spectral bloom filters. In: Proceedings of ACM SIGMOD (2003)

    Google Scholar 

  8. Cvetkovski, A.: An algorithm for approximate counting using limited memory resources. Proceedings of ACM SIGMETRICS (2007)

    Google Scholar 

  9. Demaine, E., Lopez-Ortiz, A., Munro, J.: Frequency estimation of internet packet streams with limited space. In: Proceedings of 10th ESA Annual European Symposium on Algorithms (2002)

    Google Scholar 

  10. Dimitropoulos, X., Hurley, P., Kind, A.: Probabilistic lossy counting: an efficient algorithm for finding heavy hitters. ACM SIGCOMM Comput. Comm. Rev. 38(1), 7–16 (2008)

    Google Scholar 

  11. Duffield, N., Lund, C., Thorup, M.: Estimating flow distributions from sampled flow statistics. In: Proceedings of ACM SIGCOMM (2003)

    Google Scholar 

  12. Estan, C., Varghese, G.: New directions in traffic measurement and accounting. In: Proceedings of ACM SIGCOMM (2002)

    Google Scholar 

  13. Estan, C., Varghese, G., Fish, M.: Bitmap algorithms for counting active flows on high-speed links. IEEE/ACM Trans. Netw. (TON) 14(5), 925–937 (2006)

    Article  Google Scholar 

  14. Hao, F., Kodialam, M., Lakshman, T.V.: ACCEL-RATE: A faster mechanism for memory efficient per-flow traffic estimation. In: Proceedings of ACM SIGMETRICS/Performance (2004)

    Google Scholar 

  15. Kamiyama, N., Mori, T.: Simple and accurate identification of high-rate flows by packet sampling. In: Proceedings of IEEE INFOCOM (2006)

    Google Scholar 

  16. Karp, R., Shenker, S., Papadimitriou, C.: A simple algorithm for finding frequent elements in streams and bags. ACM Trans. Database Syst. 28(1), 51–55 (2003)

    Article  Google Scholar 

  17. Kodialam, M., Lakshman, T.V., Mohanty, S.: Runs based traffic estimator (RATE): a simple, memory efficient scheme for per-flow rate estimation. In: Proceedings of IEEE INFOCOM (2004)

    Google Scholar 

  18. Kumar, A., Sung, M., Xu, J., Wang, J.: Data streaming algorithms for efficient and accurate estimation of flow size distribution. In: Proceedings of ACM SIGMETRICS (2004)

    Google Scholar 

  19. Kumar, A., Xu, J., Wang, J., Spatschek, O., Li, L.: Space-code bloom filter for efficient per-flow traffic measurement. In: Proceedings of IEEE INFOCOM (2004)

    Google Scholar 

  20. Lehmann, E., Casella, G.: Theory of Point Estimation. Springer, New York (1998)

    Google Scholar 

  21. Lu, Y., Montanari, A., Prabhakar, B., Dharmapurikar, S., Kabbani, A.: Counter braids: a novel counter architecture for per-flow measurement. In: Proceedings of ACM SIGMETRICS (2008)

    Google Scholar 

  22. Lu, Y., Prabhakar, B.: Robust Counting via counter braids: an error-resilient network measurement architecture. In: Proceedings of IEEE INFOCOM (2009)

    Google Scholar 

  23. Qiao, Y., Chen, S., Li, T., Chen, S.: Energy-efficient polling protocols in RFID systems. In: Proceedings of ACM MobiHoc (2011)

    Google Scholar 

  24. Ramakrishna, M., Fu, E., Bahcekapili, E.: Efficient hardware hashing functions for high performance computers. IEEE Trans. Comput. 46(12), 1378–1381 (1997)

    Article  Google Scholar 

  25. Stanojevic, R.: Small active counters. In: Proceedings of IEEE INFOCOM (2007)

    Google Scholar 

  26. Venkatataman, S., Song, D., Gibbons, P., Blum, A.: New streaming algorithms for fast detection of superspreaders. In: Proceedings of NDSS (2005)

    Google Scholar 

  27. Yoon, M., Li, T., Chen, S., Peir, J.: Fit a spread estimator in small memory. In: Proceedings of IEEE INFOCOM (2009)

    Google Scholar 

  28. Yoon, M., Li, T., Chen, S., Peir, J.: Fit a compact spread estimator in small high-speed memory. IEEE/ACM Trans. Netw. 19(5), 1253–1264 (2011)

    Article  Google Scholar 

  29. Zhao, H., Wang, H., Lin, B., Xu, J.: Design and performance analysis of a DRAM-based statistics counter array architecture. In: Proceedings of ACM/IEEE ANCS (2009)

    Google Scholar 

  30. Zhao, Q., Kumar, A., Xu, J.: Joint data streaming and sampling techniques for detection of super sources and destinations. In: Proceedings of USENIX/ACM Internet Measurement Conference (2005)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer and Information, University of Florida, CSE Building E460, Gainesville, 32611, Florida, USA

    Tao Li & Shigang Chen

Authors
  1. Tao Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shigang Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tao Li .

Rights and permissions

Reprints and permissions

Copyright information

© 2012 The Author(s)

About this chapter

Cite this chapter

Li, T., Chen, S. (2012). Per-Flow Size Estimators. In: Traffic Measurement on the Internet. SpringerBriefs in Computer Science. Springer, New York, NY. https://doi.org/10.1007/978-1-4614-4851-8_2

Download citation

  • DOI: https://doi.org/10.1007/978-1-4614-4851-8_2

  • Published:

  • Publisher Name: Springer, New York, NY

  • Print ISBN: 978-1-4614-4850-1

  • Online ISBN: 978-1-4614-4851-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation