Journal of Network and Systems Management

, Volume 24, Issue 1, pp 1–33 | Cite as

A Network Management System for Handling Scientific Data Flows

  • Zhenzhen Yan
  • Chris Tracy
  • Malathi Veeraraghavan
  • Tian Jin
  • Zhengyang Liu


Large scientific data transfers often occur at high rates causing increased burstiness in Internet traffic. To limit the adverse effects of these high-rate large-sized flows, which are referred to as \(\alpha \) flows, on delay-sensitive audio/video flows, a network management system called Alpha Flow Traffic Engineering System (AFTES) is proposed for intra-domain traffic engineering. An offline approach is used in which AFTES analyzes NetFlow records collected by routers, extracts source–destination address prefixes of \(\alpha \) flows, and uses these prefixes to configure firewall filters at ingress routers of a provider’s network to redirect future \(\alpha \) flows to traffic-engineered paths and isolated queues. The effectiveness of this scheme was evaluated through an analysis of 7 months of NetFlow data obtained from an ESnet router. For this data set, 91 % of bytes generated by \(\alpha \) flows during high-rate intervals would have been directed had AFTES been deployed. The negative aspect of using address prefixes in firewall filters, i.e., the redirection of \(\beta \) flows to \(\alpha \)-flow paths/queues, was also quantified.


NetFlow traffic analysis Elephant flows Scientific computing Research and education networks (RENs) MPLS Virtual circuits 


  1. 1.
    USDOE Office of Science ASCR: Terabit Networks for Extreme-Scale Science Workshop Report (2011).
  2. 2.
    Liu, Z., Veeraraghavan, M., Yan, Z., Tracy, C., Tie, J., Foster, I., Dennis, J., Hick, J., Li, Y., Yang, W.: On using virtual circuits for GridFTP transfers. In: The International Conference for High Performance Computing, Networking, Storage and Analysis 2012 (SC 2012), pp. 81:1–81:11, Nov 10–16, 2012Google Scholar
  3. 3.
    Leith, D., Shorten, R.: H-TCP: TCP for high-speed and long-distance networks. In: Protocols for Fast Long Distance Networks Workshop (PFLDnet), Feb 16–17, 2004Google Scholar
  4. 4.
    Sarvotham, S., Riedi, R., Baraniuk, R.: Connection-level analysis and modeling of nework traffic. In: ACM SIGCOMM Internet Measurement Workshop 2001, pp. 99–104, Nov 2001Google Scholar
  5. 5.
  6. 6.
    Jin, T., Tracy, C., Veeraraghavan, M., Yan, Z.: Traffic engineering of high-rate large-sized flows. In: 2013 IEEE 14th International Conference on High Performance Switching and Routing (HPSR), pp. 128–135 (2013)Google Scholar
  7. 7.
    Yan, Z., Tracy, C., Veeraraghavan, M.: A hybrid network traffic engineering system, In: Proceedings of the IEEE 13th High Performance Switching and Routing (HPSR), Jun 24–27, 2012Google Scholar
  8. 8.
    Yan, Z., Veeraraghavan, M., Tracy, C., Guok, C.: On how to provision Quality of Service (QoS) for large dataset transfers, In: Proceedings of the Sixth International Conference on Communication Theory, Reliability, and Quality of Service (CTRQ), Apr 21–26, 2013Google Scholar
  9. 9.
  10. 10.
    The Lambda Station Project.
  11. 11.
    TeraPaths: Configuring End-to-End Virtual Network Paths with QoS Guarantees.
  12. 12.
    Circuit Switched High-speed End-to-End Transport Architecture (CHEETAH).
  13. 13.
  14. 14.
    Hybrid Network Traffic Engineering System (HNTES).
  15. 15.
    Spragins, J.: Asynchronous transfer mode: solution for broadband ISDN, third edition [New Books]. IEEE Netw. 10, 7 (1996)Google Scholar
  16. 16.
    Braden, R., Zhang, L., Berson, S., Herzog, S., Jamin, S.: Resource ReSerVation Protocol (RSVP)—Version 1 Functional Specification. RFC 2205 (Proposed Standard), Sept 1997. Updated by RFCs 2750, 3936, 4495, 5946, 6437Google Scholar
  17. 17.
  18. 18.
  19. 19.
  20. 20.
    Wallerich, J., Dreger, H., Feldmann, A., Krishnamurthy, B., Willinger, W.: A methodology for studying persistency aspects of Internet flows. ACM SIGCOMM Commun. Rev. 35(2), 23–36 (2005)Google Scholar
  21. 21.
  22. 22.
  23. 23.
  24. 24.
    Next-generation network testbed JGN-X.
  25. 25.
    On-Demand Secure Circuits and Advance Reservation System (OSCARS).
  26. 26.
    Liakopoulos, A., Maglaris, B., Bouras, C., Sevasti, A.: Providing and verifying advanced IP services in hierarchical DiffServ networks-the case of GEANT. Int. J. Commun. Syst. 17(4), 321–336 (2004)CrossRefGoogle Scholar
  27. 27.
    Claise, B.: Cisco Systems NetFlow Services Export Version 9. RFC 3954 (Informational), Oct 2004Google Scholar
  28. 28.
    Claise, B.: Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of IP Traffic Flow Information. RFC 5101 (Proposed Standard), Jan 2008Google Scholar
  29. 29.
    Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z., Weiss, W.: An Architecture for Differentiated Service. RFC 2475 (Informational), Dec 1998. Updated by RFC 3260Google Scholar
  30. 30.
    Lan, Kun-chan, Heidemann, John: A measurement study of correlations of Internet flow characteristics. Comput. Netw. 50(1), 46–62 (2006)CrossRefGoogle Scholar
  31. 31.
    Crovella, M.E., Taqqu, M.S.: Estimating the heavy tail index from scaling properties. Methodol. Comput. Appl. Probab. 1, 55–79 (1999)MathSciNetCrossRefzbMATHGoogle Scholar
  32. 32.
    Brownlee, N., Claffy, K.: Understanding Internet traffic streams: dragonflies and tortoises. IEEE Commun. Mag. 40, 110–117 (2002)CrossRefGoogle Scholar
  33. 33.
    Nguyen, T.T.T., Armitage, G.J.: A survey of techniques for Internet traffic classification using machine learning. IEEE Commun. Surv. Tutor. 10(4), 56–76 (2008)CrossRefGoogle Scholar
  34. 34.
    Awduche, D.O., Jabbari, B.: Internet traffic engineering using multi-protocol label switching (MPLS). Comput. Netw. 40(1), 111–129 (2002)CrossRefGoogle Scholar
  35. 35.
    Wang, N., Ho, K., Pavlou, G., Howarth, M.: An overview of routing optimization for Internet traffic engineering. IEEE Commun. Surv. Tutor. 10(1), 36–56 (2008)CrossRefGoogle Scholar
  36. 36.
    Papagiannaki, K., Taft, N., Bhattacharyya, S., Thiran, P., Salamatian, K., Diot, C.: A pragmatic definition of elephants in Internet backbone traffic, In: Proceedings of the 2nd ACM SIGCOMM Workshop on Internet measurment (IMW ’02), pp. 175–176 (2002)Google Scholar
  37. 37.
    Callado, A., Kamienski, C., Szabo, G., Gero, B., Kelner, J., Fernandes, S., Sadok, D.: A survey on Internet traffic identification. IEEE Commun. Surv. Tutor. 11, 37–52 (2009)CrossRefGoogle Scholar
  38. 38.
    Kamiyama, N., Mori, T.: Simple and accurate identification of high-rate flows by packet sampling, In: Proceedings of INFOCOM 2006. 25th IEEE International Conference on Computer Communications, pp. 1–13 (2006)Google Scholar
  39. 39.
    Mori, T., Uchida, M., Kawahara, R., Pan, J., Goto, S.: Identifying elephant flows through periodically sampled packets, In: Proceedings of the 4th ACM SIGCOMM conference on Internet measurement (IMC ’04) (New York, NY, USA), pp. 115–120, ACM (2004)Google Scholar
  40. 40.
    Zhang, Y., Fang, B., Zhang, Y.: Identifying high-rate flows based on bayesian single sampling, In: 2010 2nd International Conference on Computer Engineering and Technology (ICCET), vol. 1, pp. V1-370–V1-374 (2010)Google Scholar
  41. 41.
    Duffield, N., Lund, C., Thorup, M.: Estimating flow distributions from sampled flow statistics. IEEE/ACM Trans. Netw. 13(5), 933–946 (2005)MathSciNetCrossRefGoogle Scholar
  42. 42.
    Fioreze, T., Pras, A.: Self-management of hybrid optical and packet switching networks. In: 2011 IFIP/IEEE International Symposium on Integrated Network Management (IM), pp. 946–951 (2011)Google Scholar
  43. 43.
    Caria, M., Jukan, A.: A novel approach to accurately compute an IP traffic matrix using optical bypass. In: 2013 IFIP/IEEE International Symposium on Integrated Network Management (IM 2013), pp. 1135–1141 (2013)Google Scholar
  44. 44.
    Lu, Y., Wang, M., Prabhakar, B., Bonomi, F.: ElephantTrap: A low cost device for identifying large flows. In: 15th Annual IEEE Symposium on High-Performance Interconnects, 2007 (HOTI 2007), pp. 99–108 (2007)Google Scholar
  45. 45.
    Kodialam, M., Lakshman, T.V., Mohanty, S.: Runs based traffic estimator (rate): a simple, memory efficient scheme for per-flow rate estimation. In: INFOCOM 2004. Twenty-third Annual Joint Conference of the IEEE Computer and Communications Societies, vol. 3, pp. 1808–1818 (2004)Google Scholar
  46. 46.
    Hao, F., Kodialam, M., Lakshman, T.V., Zhang, H.: Fast, memory-efficient traffic estimation by coincidence counting, In: Proceedings of IEEE INFOCOM 2005. 24th Annual Joint Conference of the IEEE Computer and Communications Societies, vol. 3, pp. 2080–2090 (2005)Google Scholar
  47. 47.
    Zadnik, M., Canini, M., Moore, A., Miller, D., Li, W.: Tracking elephant flows in Internet backbone traffic with an FPGA-based cache. In: International Conference on Field Programmable Logic and Applications, 2009 (FPL 2009), pp. 640–644 (2009)Google Scholar
  48. 48.
    Paisley, J., Sventek, J.: Real-time detection of grid bulk transfer traffic, In: 10th IEEE/IFIP Network Operations and Management Symposium (NOMS), pp. 66–72, Apr 2006Google Scholar
  49. 49.
    Hohn, N., Veitch, D.: Inverting sampled traffic. IEEE/ACM Trans. Netw. 14(1), 68–80 (2006)CrossRefGoogle Scholar
  50. 50.
    Chen, K., Singla, A., Singh, A., Ramachandran, K., Xu, L., Zhang, Y., Wen, X., Chen, Y.: Osa: An optical switching architecture for data center networks with unprecedented flexibility, In: Proceedings of the 9th USENIX conference on Networked Systems Design and Implementation, pp. 18–18, USENIX Association (2012)Google Scholar
  51. 51.
    Farrington, N., Porter, G., Radhakrishnan, S., Bazzaz, H., Subramanya, V., Fainman, Y., Papen, G., Vahdat, A.: Helios: a hybrid electrical/optical switch architecture for modular data centers, In: ACM SIGCOMM Computer Communication Review, vol. 40, pp. 339–350, ACM (2010)Google Scholar
  52. 52.
    Wang, G., Andersen, D., Kaminsky, M., Papagiannaki, K., Ng, T., Kozuch, M., Ryan, M.: c-through: Part-time optics in data centers. In: ACM SIGCOMM Computer Communication Review, vol. 40, pp. 327–338, ACM (2010)Google Scholar
  53. 53.
    Open Networking Foundation.
  54. 54.
  55. 55.
    Qazi, Z.A., Lee, J., Jin, T., Bellala, G., Arndt, M., Noubir, G.: Application-awareness in sdn, In: Proceedings of the ACM SIGCOMM 2013 Conference on SIGCOMM (SIGCOMM ’13) (New York, NY, USA), pp. 487–488, ACM (2013)Google Scholar
  56. 56.
    Wang, G., Ng, T.E., Shaikh, A.: Programming your network at run-time for big data applications, In: Proceedings of the First Workshop on Hot Topics in Software Defined Networks (HotSDN ’12) (New York, NY, USA), pp. 103–108, ACM (2012)Google Scholar
  57. 57.
    Xiao, X., Hannan, A., Bailey, B., Ni, L.: Traffic engineering with MPLS in the internet. IEEE Netw. 14(2), 28–33 (2000)CrossRefGoogle Scholar
  58. 58.
    Paolucci, F., Cugini, F., Giorgetti, A., Sambo, N., Castoldi, P.: A survey on the path computation element (pce) architecture. IEEE Commun. Surv. Tutor. 15, 1819–1841 (2013)Google Scholar
  59. 59.
    Sharma, A., Mishra, A., Kumar, V., Venkataramani, A.: Beyond MLU: An application-centric comparison of traffic engineering schemes. In: 2011 Proceedings of the IEEE INFOCOM, pp. 721–729, IEEE (2011)Google Scholar
  60. 60.
    Jin, T., Tracy, C., Veeraraghavan, M.: Characterization of high-rate large-sized flows. In: 2014 IEEE International Black Sea Conference on Communications and Networking (BlackSeaCom), pp. 73–76 (2014)Google Scholar
  61. 61.
  62. 62.
    The R Project for Statistical Computing.
  63. 63.
    Balman, M., Pouyoul, E., Yao, Y., Bethel, E.W., Loring, B., Prabhat, M., Shalf, J., Sim, A., Tierney, B.L.: Experiences with 100 gbps network applications, In: Proceedings of the Fifth International Workshop on Data-Intensive Distributed Computing Date (DIDC ’12) (New York, NY, USA), pp. 33–42, ACM (2012)Google Scholar
  64. 64.
    Thompson, K., Miller, G., Wilder, R.: Wide-area Internet traffic patterns and characteristics. IEEE Netw. 11, 10–23 (1997)CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2014

Authors and Affiliations

  • Zhenzhen Yan
    • 1
  • Chris Tracy
    • 2
  • Malathi Veeraraghavan
    • 1
  • Tian Jin
    • 3
  • Zhengyang Liu
    • 3
  1. 1.Department of Electrical and Computer EngineeringUniversity of VirginiaCharlottesvilleUSA
  2. 2.Energy Sciences Network (ESnet)Lawrence Berkeley National LaboratoryBerkeleyUSA
  3. 3.Department of Computer ScienceUniversity of VirginiaCharlottesvilleUSA

Personalised recommendations