Skip to main content
SpringerLink
Log in

Introduction

  • Chapter
  • First Online:
Traffic Measurement for Big Network Data

Part of the book series: Wireless Networks ((WN))

  • 908 Accesses

Abstract

Traffic measurement has many important applications in capacity planning, accounting and billing, anomaly detection, service provision, etc. In the era of big network data, traffic measurement becomes a daunting task that requires tremendous resources. To keep up with the line speeds of modern routers, the measurement modules should be implemented in the limited on-chip cache memory, thereby minimizing the per-packet processing time. This book aims to develop new compact and fast online measurement methods that reduce big network data to measurement summaries orders-of-magnitude smaller than what the traditional methods can do. The new methods hold the promise of allowing routers to perform measurement on large network traffic in real time using small cache memory on network processors.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
eBook
USD 16.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Bar-yossef, Z., Jayram, T.S., Kumar, R., Sivakumar, D., Trevisan, L.: Luca: counting distinct elements in a data stream. In: Proceedings of RANDOM: Workshop on Randomization and Approximation (2002)

    Google Scholar 

  2. Cao, J., Jin, Y., Chen, A., Bu, T., Zhang, Z.: Identifying high cardinality internet hosts. In: Proceedings of IEEE INFOCOM (2009)

    Book  Google Scholar 

  3. Cisco IOS NetFlow: Available at http://www.cisco.com/c/en/us/products/ios-nx-os-software/ios-netflow/index.html (2005)

  4. Chen, S., Tang, Y.: Slowing down internet worms. In: Proceedings of IEEE ICDCS (2004)

    Book  Google Scholar 

  5. Chen, A., Cao, J., Bu, T.: A simple and efficient estimation method for stream expression cardinalities. In: Proceedings of the 33rd International Conference on Very Large Data Bases, VLDB ’07, pp. 171–182 (2007)

    Google Scholar 

  6. Cisco: The Zettabyte EraTrends and Analysis (2015). Available at http://www.cisco.com/c/en/us/solutions/collateral/service-provider/visual-networking-index-vni/VNI_Hyperconnectivity_WP.html

  7. Duffield, N., Lund, C., Thorup, M.: Estimating flow distributions from sampled flow statistics. In: Proceedings of ACM SIGCOMM (2003)

    Book  Google Scholar 

  8. Durand, M., Flajolet, P.: Loglog counting of large cardinalities. In: ESA: European Symposia on Algorithms, pp. 605–617 (2003)

    Google Scholar 

  9. Estan, C., Varghese, G., Fish, M.: Bitmap algorithms for counting active flows on high-speed links. IEEE/ACM Trans. Netw. 14 (5), 925–937 (2006)

    Article  Google Scholar 

  10. Flajolet, P., Martin, G.N.: Probabilistic counting algorithms for database applications. J. Comput. Syst. Sci. 31 (2), 182–209 (1985)

    Article  MathSciNet  MATH  Google Scholar 

  11. Flajolet, P., Fusy, E., Gandouet, O., Meunier., F.: HyperLogLog: the analysis of a near-optimal cardinality estimation algorithm. In: Proceedings of AOFA: International Conference on Analysis of Algorithms (2007)

    Google Scholar 

  12. Google Trends: Available at https://www.google.com/trends/

  13. Hao, F., Kodialam, M., Lakshman, T.V.: ACCEL-RATE: a faster mechanism for memory efficient per-flow traffic estimation. In: Proceedings of ACM SIGMETRICS/Performance (2004)

    Book  Google Scholar 

  14. Heule, S., Nunkesser, M., Hall, A.: HyperLogLog in practice: algorithmic engineering of a state-of-the-art cardinality estimation algorithm. In: Proceedings of EDBT (2013)

    Book  Google Scholar 

  15. Kumar, A., Sung, M., Xu, J., Wang, J.: Data streaming algorithms for efficient and accurate estimation of flow size distribution. In: Proceedings of ACM SIGMETRICS (2004)

    Book  Google Scholar 

  16. Kumar, A., Xu, J., Wang, J., Spatschek, O., Li, L.: Space-code bloom filter for efficient per-flow traffic measurement. In: Proceedings of IEEE INFOCOM (2004). IEEE JSAC 24 (12), 2327–2339 (2006)

    Google Scholar 

  17. Li, T., Chen, S., Ling, Y.: Fast and compact per-flow traffic measurement through randomized counter sharing. In: Proceedings of IEEE INFOCOM, pp. 1799–1807 (2011)

    Google Scholar 

  18. Li, T., Chen, S., Luo, W., Zhang, M., Qiao, Y.: Spreader classification based on optimal dynamic bit sharing. IEEE/ACM Trans. Netw. 21 (3), 817–830 (2013)

    Article  Google Scholar 

  19. Lieven, P., Scheuermann, B.: High-speed per-flow traffic measurement with probabilistic multiplicity counting. In: Proceedings of IEEE INFOCOM, pp. 1–9 (2010). doi:10.1109/INFCOM.2010.5461921

  20. Lu, Y., Montanari, A., Prabhakar, B., Dharmapurikar, S., Kabbani, A.: Counter Braids: a novel counter architecture for per-flow measurement. In: Proceedings of ACM SIGMETRICS (2008)

    Book  Google Scholar 

  21. Mahajan, P., Bellovin, S.M., Floyd, S., Ioannidis, J., Paxson, V., Shenker, S.: Controlling high bandwidth aggregates in the network. Comput. Commun. Rev. 32 (3), 62–73 (2002)

    Article  Google Scholar 

  22. Moore, D., Voelker, G., Savage, S.: Inferring internet denial of service activity. In: Proceedings of USENIX Security Symposium’ 2001 (2001)

    Google Scholar 

  23. Moore, D., Shannon, C., Voelker, G.M., Savage, S.: Internet quarantine: requirements for containing self-propagating code. In: Proceedings of IEEE INFOCOM (2003)

    Google Scholar 

  24. Moshref, M., Yu, M., Govindan, R., Vahdat, A.: Dream: dynamic resource allocation for software-defined measurement. In: Proceedings of ACM SIGCOMM, pp. 419–430 (2014)

    Google Scholar 

  25. Park, K., Lee, H.: On the effectiveness of route-based packet filtering for distributed DoS attack prevention in power-law internets. In: Proceedings of ACM SIGCOMM’2001 (2001)

    Google Scholar 

  26. Plonka, D.: FlowScan: a network traffic flow reporting and visualization tool. In: Proceedings of USENIX LISA (2000)

    Google Scholar 

  27. Smith, G.: By the numbers: 80+ amazing google search statistics and facts (2015). Available at http://expandedramblings.com/index.php/by-the-numbers-a-gigantic-list-of-google-stats-and-facts/

    Book  Google Scholar 

  28. Song, H., Hao, F., Kodialam, M., Lakshman, T.: IPv6 lookups using distributed and load balanced bloom filters for 100Gbps core router line cards. In: Proceedings of IEEE INFOCOM (2009)

    Book  Google Scholar 

  29. Staniford, S., Hoagland, J., McAlerney, J.: Practical automated detection of stealthy portscans. J. Comput. Secur. 10, 105–136 (2002)

    Article  Google Scholar 

  30. Staniford, S., Paxson, V., Weaver, N.: How to 0wn the internet in your spare time. In: Proceedings of USENIX Security Symposium (2002)

    Google Scholar 

  31. The CAIDA UCSD Anonymized 2013 Internet Traces - January 17: http://www.caida.org/data/passive/passive_2013_dataset.xml (2013)

  32. Twitter Usage Statistics: Available at http://www.internetlivestats.com/twitter-statistics/ (2013)

  33. Venkatataman, S., Song, D., Gibbons, P., Blum, A.: New streaming algorithms for fast detection of superspreaders. In: Proceedings of NDSS (2005)

    Google Scholar 

  34. Wang, H., Zhang, D., Shin, K.G.: SYN-dog: sniffing SYN Flooding Sources. In: Proceedings of 22nd International Conference on Distributed Computing Systems (ICDCS’02) (2002)

    Google Scholar 

  35. Whang, K.Y., Vander-Zanden, B.T., Taylor, H.M.: A linear-time probabilistic counting algorithm for database applications. ACM Trans. Database Syst. 15 (2), 208–229 (1990)

    Article  Google Scholar 

  36. Yoon, M., Li, T., Chen, S., Peir, J.K.: Fit a spread estimator in small memory. In: Proceedings of IEEE INFOCOM (2009)

    Book  Google Scholar 

  37. Zhao, Q., Xu, J., Kumar, A.: Detection of super sources and destinations in high-speed networks: algorithms, analysis and evaluation. IEEE JASC 24 (10), 1840–1852 (2006)

    Google Scholar 

  38. Zhao, Q., Xu, J., Kumar, A.: Detection of super sources and destinations in high-speed networks: algorithms, analysis and evaluation. IEEE JSAC 24 (10) (2006)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer & Information Science, University of Florida, Gainesville, FL, USA

    Shigang Chen & Min Chen

  2. School of Computer Science and Engineering, Southeast University of China, Nanjing, Jiangsu, China

    Qingjun Xiao

Authors
  1. Shigang Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Min Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Qingjun Xiao
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this chapter

Cite this chapter

Chen, S., Chen, M., Xiao, Q. (2017). Introduction. In: Traffic Measurement for Big Network Data. Wireless Networks. Springer, Cham. https://doi.org/10.1007/978-3-319-47340-6_1

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-47340-6_1

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-47339-0

  • Online ISBN: 978-3-319-47340-6

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation