CLEF: Limiting the Damage Caused by Large Flows in the Internet Core

  • Hao WuEmail author
  • Hsu-Chun Hsiao
  • Daniele E. Asoni
  • Simon Scherrer
  • Adrian Perrig
  • Yih-Chun Hu
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11124)


The detection of network flows that send excessive amounts of traffic is of increasing importance to enforce QoS and to counter DDoS attacks. Large-flow detection has been previously explored, but the proposed approaches can be used on high-capacity core routers only at the cost of significantly reduced accuracy, due to their otherwise too high memory and processing overhead. We propose CLEF, a new large-flow detection scheme with low memory requirements, which maintains high accuracy under the strict conditions of high-capacity core routers. We compare our scheme with previous proposals through extensive theoretical analysis, and with an evaluation based on worst-case-scenario attack traffic. We show that CLEF outperforms previously proposed systems in settings with limited memory.


Large-flow detection Damage metric Memory and computation efficiency 



We thank Pratyaksh Sharma and Prateesh Goyal for early work on this project as part of their summer internship at ETH in Summer 2015. We also thank the anonymous reviewers, whose feedback helped to improve the paper.

The research leading to these results has received funding from the European Research Council under the European Union’s Seventh Framework Programme (FP7/2007-2013), ERC grant agreement 617605, the Ministry of Science and Technology of Taiwan under grant number MOST 107-2636-E-002-005, and the US National Science Foundation under grant numbers CNS-1717313 and CNS-0953600. We also gratefully acknowledge support from ETH Zurich and from the Zurich Information Security and Privacy Center (ZISC).


  1. 1.
    Andersen, D.G., Balakrishnan, H., Feamster, N., Koponen, T., Moon, D., Shenker, S.: Accountable internet protocol (AIP). In: Proceedings of ACM SIGCOMM (2008).
  2. 2.
    Anderson, T., et al.: The NEBULA future internet architecture. In: Galis, A., Gavras, A. (eds.) FIA 2013. LNCS, vol. 7858, pp. 16–26. Springer, Heidelberg (2013). Scholar
  3. 3.
    Antonakakis, M., et al.: Understanding the Mirai botnet. In: USENIX Security Symposium (2017)Google Scholar
  4. 4.
    Basescu, C., et al.: SIBRA: scalable internet bandwidth reservation architecture. In: Proceedings of Network and Distributed System Security Symposium (NDSS), February 2016Google Scholar
  5. 5.
    Braden, R., Clark, D., Shenker, S.: Integrated services in the internet architecture: an overview. RFC 1633 (Informational), June 1994.
  6. 6.
    CAIDA: CAIDA Anonymized Internet Traces 2016 (2016).
  7. 7.
    Chen, M., Chen, S., Cai, Z.: Counter tree: a scalable counter architecture for per-flow traffic measurement. IEEE/ACM Trans. Netw. (TON) 25(2), 1249–1262 (2017)CrossRefGoogle Scholar
  8. 8.
    Claise, B.: Cisco Systems NetFlow Services Export Version 9, RFC 3954 (Informational), October 2004.
  9. 9.
    Cormode, G., Muthukrishnan, S.: An improved data stream summary: the count-min sketch and its applications. J. Algorithms 55(1), 58–75 (2005). Scholar
  10. 10.
    Demaine, E.D., López-Ortiz, A., Munro, J.I.: Frequency estimation of internet packet streams with limited space. In: Möhring, R., Raman, R. (eds.) ESA 2002. LNCS, vol. 2461, pp. 348–360. Springer, Heidelberg (2002). Scholar
  11. 11.
    Estan, C.: Internet traffic measurement: what’s going on in my network? Ph.D. thesis (2003)Google Scholar
  12. 12.
    Estan, C., Varghese, G.: New directions in traffic measurement and accounting: focusing on the elephants, ignoring the mice. ACM Trans. Comput. Syst. (TOCS) 21(3), 270–313 (2003). Scholar
  13. 13.
    Fang, M., Shivakumar, N.: Computing iceberg queries efficiently. In: Proceedings of VLDB (1999).
  14. 14.
    Han, D., et al.: XIA: efficient support for evolvable internetworking. In: Proceedings of the 9th USENIX NSDI, San Jose, CA, April 2012Google Scholar
  15. 15.
  16. 16.
    Karp, R.M., Shenker, S., Papadimitriou, C.H.: A simple algorithm for finding frequent elements in streams and bags. ACM Trans. Database Syst. 28(1), 51–55 (2003). Scholar
  17. 17.
    Kim, T.H.J., Basescu, C., Jia, L., Lee, S.B., Hu, Y.C., Perrig, A.: Lightweight source authentication and path validation. In: ACM SIGCOMM Computer Communication Review, vol. 44, pp. 271–282. ACM (2014)Google Scholar
  18. 18.
    Kumar, A., Xu, J., Wang, J.: Space-code bloom filter for efficient per-flow traffic measurement. IEEE J. Sel. Areas Commun. 24(12), 2327–2339 (2006)CrossRefGoogle Scholar
  19. 19.
    Lee, S.B., Kang, M.S., Gligor, V.D.: CoDef: collaborative defense against large-scale link-flooding attacks. In: Proceedings of CoNext (2013)Google Scholar
  20. 20.
    Li, A., Liu, X., Yang, X.: Bootstrapping accountability in the internet we have. In: Proceedings of USENIX/ACM NSDI, March 2011Google Scholar
  21. 21.
    Liu, X., Li, A., Yang, X., Wetherall, D.: Passport: secure and adoptable source authentication. In: Proceedings of USENIX/ACM NSDI (2008).
  22. 22.
    Liu, Z., Manousis, A., Vorsanger, G., Sekar, V., Braverman, V.: One sketch to rule them all: rethinking network flow monitoring with UnivMon. In: ACM SIGCOMM (2016).
  23. 23.
    Liu, Z., Jin, H., Hu, Y.C., Bailey, M.: MiddlePolice: toward enforcing destination-defined policies in the middle of the internet. In: Proceedings of ACM CCS, October 2016Google Scholar
  24. 24.
    Manku, G., Motwani, R.: Approximate frequency counts over data streams. In: Proceedings of VLDB (2002).
  25. 25.
    Metwally, A., Agrawal, D., El Abbadi, A.: Efficient computation of frequent and top-k elements in data streams. In: Eiter, T., Libkin, L. (eds.) ICDT 2005. LNCS, vol. 3363, pp. 398–412. Springer, Heidelberg (2004). Scholar
  26. 26.
    Misra, J., Gries, D.: Finding repeated elements. Sci. Comput. Program. 2(2), 143–152 (1982)MathSciNetCrossRefGoogle Scholar
  27. 27.
    Naous, J., Walfish, M., Nicolosi, A., Mazières, D., Miller, M., Seehra, A.: Verifying and enforcing network paths with ICING. In: Proceedings of ACM CoNEXT (2011).
  28. 28.
    Pagh, R., Rodler, F.F.: Cuckoo hashing. In: auf der Heide, F.M. (ed.) ESA 2001. LNCS, vol. 2161, pp. 121–133. Springer, Heidelberg (2001). Scholar
  29. 29.
    Shenker, S., Partridge, C., Guerin, R.: Specification of guaranteed quality of service, RFC 2212 (Proposed Standard), September 1997.
  30. 30.
    Sivaraman, V., Narayana, S., Rottenstreich, O., Muthukrishnan, S., Rexford, J.: Heavy-hitter detection entirely in the data plane. In: Proceedings of the Symposium on SDN Research, pp. 164–176. ACM (2017)Google Scholar
  31. 31.
    Tong, D., Prasanna, V.: High throughput sketch based online heavy hitter detection on FPGA. ACM SIGARCH Comput. Arch. News 43(4), 70–75 (2016)CrossRefGoogle Scholar
  32. 32.
    Trybulec, W.A.: Pigeon hole principle. J. Formaliz. Math. 2, 4 (1990)Google Scholar
  33. 33.
    Wu, H., Hsiao, H.C., Asoni, D.E., Scherrer, S., Perrig, A., Hu, Y.C.: CLEF: limiting the damage caused by large flows in the internet core. Technical report, arXiv:1807.05652 [cs.NI], arXiv (2018).
  34. 34.
    Wu, H., Hsiao, H.C., Hu, Y.C.: Efficient large flow detection over arbitrary windows: an algorithm exact outside an ambiguity region. In: Proceedings of the 2014 Conference on Internet Measurement Conference, pp. 209–222. ACM (2014)Google Scholar
  35. 35.
    Xiao, Q., Chen, S., Chen, M., Ling, Y.: Hyper-compact virtual estimators for big network data based on register sharing. In: ACM SIGMETRICS Performance Evaluation Review, vol. 43, pp. 417–428. ACM (2015)Google Scholar
  36. 36.
    Zhang, X., Hsiao, H.C., Hasker, G., Chan, H., Perrig, A., Andersen, D.G.: SCION: scalability, control, and isolation on next-generation networks. In: IEEE Symposium on Security and Privacy, pp. 212–227 (2011)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.University of Illinois at Urbana ChampaignChampaignUSA
  2. 2.Rubrik, Inc.Palo AltoUSA
  3. 3.National Taiwan UniversityTaipeiTaiwan
  4. 4.ETH ZurichZürichSwitzerland

Personalised recommendations