Time-Out Bloom Filter: A New Sampling Method for Recording More Flows

  • Shijin Kong
  • Tao He
  • Xiaoxin Shao
  • Changqing An
  • Xing Li
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3961)


Packet sampling is widely deployed to generate flow records on high speed links. However, random sampling in which 1 in N packets is chosen suffers from omitting majority of flows, most of which are short flows (within N packets). Although usage-based applications work well by sampling long flows and neglecting short ones, there are many other applications which depend on nearly per-flow information. In this paper, a novel sampling method is proposed to remedy the flow loss flaw. We use a Time-out Bloom Filter to alleviate the sampling bias towards long flows. Compared with random sampling, short flows have a much greater probability to be sampled while long flows are always sampled, but with much fewer sampled packets. Experimental results show that, with the same sampling rate, our solution records several times more short flows than random sampling. Particularly, up to 99% original flows can be retrieved. Besides, we also propose an adaptive TBF system in fast SRAM to perform online sampling.


Hash Function Bloom Filter False Positive Error Flow Length Incoming Packet 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Feldmann, A., Rexford, J., Cáceres, R.: Efficient Policies for Carrying Web Traffic over Flow-switched Networks. IEEE/ACM Transactions on Networking 6(6), 673–685 (1999)CrossRefGoogle Scholar
  2. 2.
    Karagiannis, T., Broido, A., Faloutsos, M., et al.: Transport Layer Identification of P2P Traffic. In: ACM SIGCOMM Internet Measurement Conference (IMC) (2004)Google Scholar
  3. 3.
    Duffield, N.G.: Sampling for Passive Internet Measurement: A Review. Statistical Science 19(3), 472–498 (2004)MATHCrossRefMathSciNetGoogle Scholar
  4. 4.
  5. 5.
    Estan, C., Keys, K., Moore, D., et al.: Building a Better NetFlow. In: ACM SIGCOMM (2004)Google Scholar
  6. 6.
    Duffield, N.G., Lund, C., Thorup, M.: Charging from Sampled Network Usage. In: ACM SIGCOMM Internet Measurement Workshop (IMW) (2001)Google Scholar
  7. 7.
    Duffield, N.G., Lund, C.: Predicting Resource Usage and Estimation Accuracy in an IP Flow Measurement Collection Infrastructure. In: ACM SIGCOMM IMC (2003)Google Scholar
  8. 8.
    Duffield, N.G., Lund, C., Thorup, M.: Estimating Flow Distributions from Sampled Flow Statistics. In: ACM SIGCOMM (2003)Google Scholar
  9. 9.
    Hohn, N., Veitch, D.: Inverting Sampled Traffic. In: ACM SIGCOMM IMC (2003)Google Scholar
  10. 10.
    Duffield, N.G., Grossglauser, M.: Trajectory Sampling for Direct Traffic Observation. IEEE/ACM Transactions on Networking 9(3), 280–292 (2001)CrossRefGoogle Scholar
  11. 11.
    Duffield, N.G., Grossglauser, M.: Trajectory Engine: A Backend for Trajectory Sampling. In: IEEE Network Operations and Management Symposium (2002)Google Scholar
  12. 12.
    Kumar, A., Xu, J., Wang, J., et al.: Space-Code Bloom Filter for Efficient Per-Flow Traffic Measurement. In: IEEE INFOCOM (2004)Google Scholar
  13. 13.
    Estan, C., Varghese, G.: New Directions in Traffic Measurement and Accounting. In: ACM SIGCOMM (2002)Google Scholar
  14. 14.
    Kompella, R.R., Singh, S., Varghese, G.: On Scalable Attack Detection in the Network. In: ACM SIGCOMM Internet Measurement Conference (2004)Google Scholar
  15. 15.
    Bloom, B.H.: Space/time Tradeoffs in Hash Coding with Allowable Errors. ACM Communicationsm 13(7) (1970)Google Scholar
  16. 16.
    Levchenko, K., Paturi, R., Varghese, G.: On the Difficulty of Scalably Detecting Network Attacks. In: ACM CCS (2004)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Shijin Kong
    • 1
  • Tao He
    • 2
  • Xiaoxin Shao
    • 1
  • Changqing An
    • 2
  • Xing Li
    • 2
  1. 1.Department of Electronic EngineeringTsinghua UniversityBeijingP.R. China
  2. 2.China Education and Research Network (CERNET)BeijingP.R. China

Personalised recommendations