Efficient flow detection and scheduling for SDN-based big data centers

  • Heteng Zhang
  • Feilong TangEmail author
  • Leonard Barolli
Original Research


In Software defined networking (SDN) based datacenters, flow-level management seriously limits system scalability due to large amount of control messages between data and control planes; and mice flows often are blocked by elephant flows because of the indiscriminate flow scheduling. To improve management efficiency and system performance, it is prerequisite to schedule elephant and mice flows respectively. Unfortunately, existing flow scheduling approaches in SDN consider only elephant flows. In this paper, we firstly propose an efficient flow detection mechanism. Then, we propose a novel DIFFERENtiated sChEduling (DIFFERENCE) approach that dynamically sets up paths for elephant and mice flows separately, based on current link workload. Our DIFFERENCE schedules mice flows with proactively installed weighted multipath routing algorithm and adjusts path weight according to link utilization. Instead, we propose a blocking island based path setup algorithm for elephant flows, which find the least congested path with shorter searching space. To balance traffic in a SDN networks, we design an algorithm to dynamically reschedule data flows in terms of current link utilization ratio. Experiment results on real public datacenter traces demonstrate that our approach outperforms related proposals in terms of various system performance.


Flow detection Scheduling Load balancing SDN Data center 



This work was supported in part by the National Natural Science Foundation of China Projects under Grant 91438121, Grant 61373156 and Grant 61672351, in part by the National Basic Research Program under Grant 2015CB352403, and in part by Huawei Technologies Co. Ltd. Projects under Grant YBN2017090053 and Grant YBN2017050015.


  1. Afek Y, Bremler-Barr A, Landau Feibish S et al (2015) Sampling and large flow detection in SDN. ACM SIGCOMM Comput Commun Rev 45(4):345–346CrossRefGoogle Scholar
  2. Al-Fares M, Loukissas A, Vahdat A (2008) A scalable, commodity data center network architecture. ACM SIGCOMM Comput Commun Rev 38(4):63–74CrossRefGoogle Scholar
  3. Al-Fares M, Radhakrishnan S, Raghavan B, Huang N, Vadhat A (2010) Hedera: dynamic flow scheduling for data center networks. Proc. NSDI 10:19Google Scholar
  4. Benson T, Akella A, Maltz DA (2010) Network traffic characteristics of data centers in the wild. In: Proc. the 10th ACM SIGCOMM conference on internet measurement, Melbourne, pp 267–280Google Scholar
  5. Cao Y, Xu M, Fu X, Dong E (2013) Explicit multipath congestion control for data center networks. In: Proc. 9th ACM conf. emerging netw. exp. technol. (CoNEXT), Santa Barbara, CA, pp 73–84Google Scholar
  6. Cao ZZ, Kodialam M, Lakshman TV (2014) Joint static and dynamic traffic scheduling in data center networks. In: Proc. IEEE INFOCOM, Toronto, pp 2445–2553Google Scholar
  7. Curtis AR, Mogul JC, Tourrilhes al (2011a) DevoFlow: scaling flow management for high-performance networks. ACM SIGCOMM Comput Commun Rev 41(4):254–265CrossRefGoogle Scholar
  8. Curtis AR, Kim W, Yalagandula P (2011b) Mahout: low-overhead datacenter traffic management using end-host-based elephant detection. In: Proc. IEEE INFOCOM, Shanghai, pp 1629–1637Google Scholar
  9. Data set for imc 2010 data center measurement. Data.html
  10. Estan C, Varghese G (2003) New directions in traffic measurement and accounting: focusing on the elephants, ignoring the mice. ACM Trans Comput Syst TOCS 21(3):270–313CrossRefGoogle Scholar
  11. Gill P, Jain N, Nagappan N (2011) Understanding network failures in data centers: measurement, analysis, and implications. In: Proc. ACM SIGCOMM, TorontoGoogle Scholar
  12. Greenberg A, Hamilton JR, Jain al (2009) VL2: a scalable and flexible data center network. ACM SIGCOMM Comput Commun Rev 39(4):51–62CrossRefGoogle Scholar
  13. Guo B, Chen HH, Han Q, Yu ZW, Zhang DQ, Wang Y (2016) Worker-contributed data utility measurement for visual crowdsensing systems. IEEE Trans Mob Comput 16(8):2379–2391CrossRefGoogle Scholar
  14. Handigol N, Heller B, Jeyakumar V, Lantz B, McKeown N (2012) Reproducible network experiments using container-based emulation. In: Proc. the 8th international conference on emerging networking experiments and technologies. ACM, Nice, pp 253–264Google Scholar
  15. Higashino WA, Capretz MAM, Toledo MBFD., Bittencourt LF (2016) A hybrid particle swarm optimisation-genetic algorithm applied to grid scheduling. Int J Grid Util Comput 7(2):113–129CrossRefGoogle Scholar
  16. Hong C-Y, Caesar M, Godfrey PB (2012) Finishing flows quickly with preemptive scheduling. In: Proc. ACM SIGCOMM, HelsinkiGoogle Scholar
  17. Kandula S, Sengupta S, Greenberg A et al (2009) The nature of data center traffic: measurements & analysis. In: Proc. 9th ACM SIG-COMM conference on internet measurement conference, Chicago, IL, pp 202–208Google Scholar
  18. Kuzuno H, Magata K (2016) Detecting and characterising of mobile advertisement network traffic using graph modelling. Int J Space Based Situat Comput 6(2):90–101CrossRefGoogle Scholar
  19. Li ZT, Xiao F, Wang SG, Pei TR, Li J (2018) Achievable rate maximization for cognitive hybrid satellite-terrestrial networks with AF-relays. IEEE J Sel Areas Commun Spec Issue Adv Satell Commun PP(99):1Google Scholar
  20. Lin CY, Chen C, Chang JW et al (2014) Elephant flow detection in datacenters using OpenFlow-based hierarchical statistics pulling. In: Proc. 2014 IEEE global communications conference, Austin, pp 2264–2269Google Scholar
  21. Liu CY, He L, Li ZT, Li J (2017) Feature-driven active learning for hyperspectral image classification. IEEE Trans Geosci Rem Sens PP(99):1–14Google Scholar
  22. Luo CZ, Li ZT, Huang KZ, Feng JS, Wang M (2017) Zero-shot learning via attribute regression and class prototype rectification. IEEE Trans Image Process PP(99):1–1zbMATHGoogle Scholar
  23. McKeown N, Anderson T, Balakrishnan H et al (2008) OpenFlow: enabling innovation in campus networks. ACM CCRGoogle Scholar
  24. Mori T, Uchida M, Kawahara R et al (2004) Identifying elephant flows through periodically sampled packets. In: Proc. the 4th ACM SIGCOMM conference on internet measurement, Taormina, Sicily, pp 115–120Google Scholar
  25. Nakamura S, Duolikun D, Enokido T, Takizawa M (2016) A read–write abortion protocol to prevent illegal information flow in role-based access control systems. Int J Space Based Situat Comput 6(1):43–53CrossRefGoogle Scholar
  26. Popal L, Raiciu C, Stoica I, Rosenblum D (2006) Reducing congestion effects in wireless networks by multipath routing. In: Proc. IEEE int’l conf. network protocols (ICNP), Santa Barbara, CAGoogle Scholar
  27. Su Z, Wang T, Xia Y, Hamdi M (2014) CheetahFlow: towards low latency software-defined network. In: Proc. 2014 IEEE international conference on communications (ICC), Sydney, pp 3076–3081Google Scholar
  28. Tang FL, Li J (2017) Joint rate adaptation, channel assignment and routing to maximize social welfare in multi-hop cognitive radio networks. IEEE Trans Wirel Commun 16(4):2097–2110CrossRefGoogle Scholar
  29. Tang FL, Yang LT, Tang C, Li J, Guo MY (2016) A dynamical and load-balanced flow scheduling approach for big data centers in clouds. IEEE Trans Cloud Comput PP(99):1Google Scholar
  30. Tang FL, Yang YQ, Yang LT, Zhou T, Guo MY (2017) Delay-minimized routing in mobile cognitive networks for time-critical applications. IEEE Trans Ind Inf 13(3):1398–1409CrossRefGoogle Scholar
  31. Tang FL, Zhang HT, Li J (2018) Joint topology control and stable routing based on pu prediction for multi-hop mobile cognitive networks. IEEE Trans Wirel Commun 17(3):1713–1726CrossRefGoogle Scholar
  32. Thaman J, Singh M (2017) Cost-effective task scheduling using hybrid approach in cloud. Int J Grid Util Comput 8(3):241–253CrossRefGoogle Scholar
  33. Vasudevan V, Phanishayee A, Shah H et al (2009) Safe and effective fine-grained TCP retransmissions for datacenter communication. In: Proc. SIGCOMM, Barcelona, pp 303–314Google Scholar
  34. Wang SG, Ruby R, Leung VCM, Yao ZQ, Liu XR, Li ZT (2017) Sum-power minimization problem in multi-source single-AF-relay networks: a new revisit to study the optimality. IEEE Trans Veh Technol 66(11):9958–9971CrossRefGoogle Scholar
  35. Wette P, Karl H (2013) Which flows are hiding behind my wildcard rule? Adding packet sampling to OpenFlow. ACM SIGCOMM Comput Commun Rev 43(4):541–542CrossRefGoogle Scholar
  36. Yu Z, Xu H, Yang Z, Guo B (2016) Personalized travel package with multi-point-of-interest recommendation based on crowdsourced user footprints. IEEE Trans Hum Mach Syst 46(1):151–158CrossRefGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2018

Authors and Affiliations

  1. 1.School of SoftwareShanghai Jiao Tong UniversityShanghaiChina
  2. 2.Department of Computer Science and EngineeringShanghai Jiao Tong UniversityShanghaiChina
  3. 3.Faculty of Information EngineeringFukuoka Institute of TechnologyFukuokaJapan

Personalised recommendations