Skip to main content
Log in

Abstract

Large scientific data transfers often occur at high rates causing increased burstiness in Internet traffic. To limit the adverse effects of these high-rate large-sized flows, which are referred to as \(\alpha \) flows, on delay-sensitive audio/video flows, a network management system called Alpha Flow Traffic Engineering System (AFTES) is proposed for intra-domain traffic engineering. An offline approach is used in which AFTES analyzes NetFlow records collected by routers, extracts source–destination address prefixes of \(\alpha \) flows, and uses these prefixes to configure firewall filters at ingress routers of a provider’s network to redirect future \(\alpha \) flows to traffic-engineered paths and isolated queues. The effectiveness of this scheme was evaluated through an analysis of 7 months of NetFlow data obtained from an ESnet router. For this data set, 91 % of bytes generated by \(\alpha \) flows during high-rate intervals would have been directed had AFTES been deployed. The negative aspect of using address prefixes in firewall filters, i.e., the redirection of \(\beta \) flows to \(\alpha \)-flow paths/queues, was also quantified.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

Notes

  1. Unlike in residential networks where link capacity is the bottleneck, in scientific laboratories, the bottleneck is the end-system computing/disk resource, not links.

  2. This system was proposed in our conference papers [6, 7]. It was called Hybrid Network Traffic Engineering System (HNTES). The name was changed to AFTES to reflect its functionality more accurately.

  3. It is difficult to procure NetFlow data from providers for privacy reasons and therefore we could not test our hypothesis with other providers’ data. But we did extend our prior 2-month analysis [7] to a 7-month analysis presented in this paper.

  4. In 2012, ESnet had a major upgrade with all backbone link capacities increased to 100 Gbps.

References

  1. USDOE Office of Science ASCR: Terabit Networks for Extreme-Scale Science Workshop Report (2011). http://science.energy.gov/~/media/ascr/pdf/program-documents/docs/Terabit_networks_workshop_report.pdf

  2. Liu, Z., Veeraraghavan, M., Yan, Z., Tracy, C., Tie, J., Foster, I., Dennis, J., Hick, J., Li, Y., Yang, W.: On using virtual circuits for GridFTP transfers. In: The International Conference for High Performance Computing, Networking, Storage and Analysis 2012 (SC 2012), pp. 81:1–81:11, Nov 10–16, 2012

  3. Leith, D., Shorten, R.: H-TCP: TCP for high-speed and long-distance networks. In: Protocols for Fast Long Distance Networks Workshop (PFLDnet), Feb 16–17, 2004

  4. Sarvotham, S., Riedi, R., Baraniuk, R.: Connection-level analysis and modeling of nework traffic. In: ACM SIGCOMM Internet Measurement Workshop 2001, pp. 99–104, Nov 2001

  5. ESnet. http://www.es.net/

  6. Jin, T., Tracy, C., Veeraraghavan, M., Yan, Z.: Traffic engineering of high-rate large-sized flows. In: 2013 IEEE 14th International Conference on High Performance Switching and Routing (HPSR), pp. 128–135 (2013)

  7. Yan, Z., Tracy, C., Veeraraghavan, M.: A hybrid network traffic engineering system, In: Proceedings of the IEEE 13th High Performance Switching and Routing (HPSR), Jun 24–27, 2012

  8. Yan, Z., Veeraraghavan, M., Tracy, C., Guok, C.: On how to provision Quality of Service (QoS) for large dataset transfers, In: Proceedings of the Sixth International Conference on Communication Theory, Reliability, and Quality of Service (CTRQ), Apr 21–26, 2013

  9. GridFTP. http://globus.org/toolkit/docs/3.2/gridftp/

  10. The Lambda Station Project. http://www.lambdastation.org/

  11. TeraPaths: Configuring End-to-End Virtual Network Paths with QoS Guarantees. https://www.racf.bnl.gov/terapaths/

  12. Circuit Switched High-speed End-to-End Transport Architecture (CHEETAH). http://www.ece.virginia.edu/cheetah/

  13. NetFlow. http://www.cisco.com/c/en/us/products/ios-nx-os-software/ios-netflow/index.html

  14. Hybrid Network Traffic Engineering System (HNTES). http://www.ece.virginia.edu/mv/research/DOE09/index.html

  15. Spragins, J.: Asynchronous transfer mode: solution for broadband ISDN, third edition [New Books]. IEEE Netw. 10, 7 (1996)

    Google Scholar 

  16. Braden, R., Zhang, L., Berson, S., Herzog, S., Jamin, S.: Resource ReSerVation Protocol (RSVP)—Version 1 Functional Specification. RFC 2205 (Proposed Standard), Sept 1997. Updated by RFCs 2750, 3936, 4495, 5946, 6437

  17. Vietzke, R.P.: Internet2 Headroom Practice, 15 Aug 2008. https://wiki.internet2.edu/confluence/download/attachments/17383/Internet2+Headroom+Practice+8-14-08.pdf?version=1

  18. ESnet Graphite. https://stats.es.net/graphite/

  19. nuttcp. http://www.nuttcp.net/

  20. Wallerich, J., Dreger, H., Feldmann, A., Krishnamurthy, B., Willinger, W.: A methodology for studying persistency aspects of Internet flows. ACM SIGCOMM Commun. Rev. 35(2), 23–36 (2005)

  21. ESnet Backbone Topology Map Summer 2010. http://es.net/introducing-esnet5/network-maps/historical-network-maps/

  22. GEANT2. http://www.geant2.net/

  23. Interoperable Ondemand Network (ION). http://www.internet2.edu/products-services/advanced-networking/layer-2-services/

  24. Next-generation network testbed JGN-X. http://www.jgn.nict.go.jp/english/

  25. On-Demand Secure Circuits and Advance Reservation System (OSCARS). http://www.es.net/OSCARS/docs/index.html

  26. Liakopoulos, A., Maglaris, B., Bouras, C., Sevasti, A.: Providing and verifying advanced IP services in hierarchical DiffServ networks-the case of GEANT. Int. J. Commun. Syst. 17(4), 321–336 (2004)

    Article  Google Scholar 

  27. Claise, B.: Cisco Systems NetFlow Services Export Version 9. RFC 3954 (Informational), Oct 2004

  28. Claise, B.: Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of IP Traffic Flow Information. RFC 5101 (Proposed Standard), Jan 2008

  29. Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z., Weiss, W.: An Architecture for Differentiated Service. RFC 2475 (Informational), Dec 1998. Updated by RFC 3260

  30. Lan, Kun-chan, Heidemann, John: A measurement study of correlations of Internet flow characteristics. Comput. Netw. 50(1), 46–62 (2006)

    Article  Google Scholar 

  31. Crovella, M.E., Taqqu, M.S.: Estimating the heavy tail index from scaling properties. Methodol. Comput. Appl. Probab. 1, 55–79 (1999)

    Article  MathSciNet  MATH  Google Scholar 

  32. Brownlee, N., Claffy, K.: Understanding Internet traffic streams: dragonflies and tortoises. IEEE Commun. Mag. 40, 110–117 (2002)

    Article  Google Scholar 

  33. Nguyen, T.T.T., Armitage, G.J.: A survey of techniques for Internet traffic classification using machine learning. IEEE Commun. Surv. Tutor. 10(4), 56–76 (2008)

    Article  Google Scholar 

  34. Awduche, D.O., Jabbari, B.: Internet traffic engineering using multi-protocol label switching (MPLS). Comput. Netw. 40(1), 111–129 (2002)

    Article  Google Scholar 

  35. Wang, N., Ho, K., Pavlou, G., Howarth, M.: An overview of routing optimization for Internet traffic engineering. IEEE Commun. Surv. Tutor. 10(1), 36–56 (2008)

    Article  Google Scholar 

  36. Papagiannaki, K., Taft, N., Bhattacharyya, S., Thiran, P., Salamatian, K., Diot, C.: A pragmatic definition of elephants in Internet backbone traffic, In: Proceedings of the 2nd ACM SIGCOMM Workshop on Internet measurment (IMW ’02), pp. 175–176 (2002)

  37. Callado, A., Kamienski, C., Szabo, G., Gero, B., Kelner, J., Fernandes, S., Sadok, D.: A survey on Internet traffic identification. IEEE Commun. Surv. Tutor. 11, 37–52 (2009)

    Article  Google Scholar 

  38. Kamiyama, N., Mori, T.: Simple and accurate identification of high-rate flows by packet sampling, In: Proceedings of INFOCOM 2006. 25th IEEE International Conference on Computer Communications, pp. 1–13 (2006)

  39. Mori, T., Uchida, M., Kawahara, R., Pan, J., Goto, S.: Identifying elephant flows through periodically sampled packets, In: Proceedings of the 4th ACM SIGCOMM conference on Internet measurement (IMC ’04) (New York, NY, USA), pp. 115–120, ACM (2004)

  40. Zhang, Y., Fang, B., Zhang, Y.: Identifying high-rate flows based on bayesian single sampling, In: 2010 2nd International Conference on Computer Engineering and Technology (ICCET), vol. 1, pp. V1-370–V1-374 (2010)

  41. Duffield, N., Lund, C., Thorup, M.: Estimating flow distributions from sampled flow statistics. IEEE/ACM Trans. Netw. 13(5), 933–946 (2005)

    Article  MathSciNet  Google Scholar 

  42. Fioreze, T., Pras, A.: Self-management of hybrid optical and packet switching networks. In: 2011 IFIP/IEEE International Symposium on Integrated Network Management (IM), pp. 946–951 (2011)

  43. Caria, M., Jukan, A.: A novel approach to accurately compute an IP traffic matrix using optical bypass. In: 2013 IFIP/IEEE International Symposium on Integrated Network Management (IM 2013), pp. 1135–1141 (2013)

  44. Lu, Y., Wang, M., Prabhakar, B., Bonomi, F.: ElephantTrap: A low cost device for identifying large flows. In: 15th Annual IEEE Symposium on High-Performance Interconnects, 2007 (HOTI 2007), pp. 99–108 (2007)

  45. Kodialam, M., Lakshman, T.V., Mohanty, S.: Runs based traffic estimator (rate): a simple, memory efficient scheme for per-flow rate estimation. In: INFOCOM 2004. Twenty-third Annual Joint Conference of the IEEE Computer and Communications Societies, vol. 3, pp. 1808–1818 (2004)

  46. Hao, F., Kodialam, M., Lakshman, T.V., Zhang, H.: Fast, memory-efficient traffic estimation by coincidence counting, In: Proceedings of IEEE INFOCOM 2005. 24th Annual Joint Conference of the IEEE Computer and Communications Societies, vol. 3, pp. 2080–2090 (2005)

  47. Zadnik, M., Canini, M., Moore, A., Miller, D., Li, W.: Tracking elephant flows in Internet backbone traffic with an FPGA-based cache. In: International Conference on Field Programmable Logic and Applications, 2009 (FPL 2009), pp. 640–644 (2009)

  48. Paisley, J., Sventek, J.: Real-time detection of grid bulk transfer traffic, In: 10th IEEE/IFIP Network Operations and Management Symposium (NOMS), pp. 66–72, Apr 2006

  49. Hohn, N., Veitch, D.: Inverting sampled traffic. IEEE/ACM Trans. Netw. 14(1), 68–80 (2006)

    Article  Google Scholar 

  50. Chen, K., Singla, A., Singh, A., Ramachandran, K., Xu, L., Zhang, Y., Wen, X., Chen, Y.: Osa: An optical switching architecture for data center networks with unprecedented flexibility, In: Proceedings of the 9th USENIX conference on Networked Systems Design and Implementation, pp. 18–18, USENIX Association (2012)

  51. Farrington, N., Porter, G., Radhakrishnan, S., Bazzaz, H., Subramanya, V., Fainman, Y., Papen, G., Vahdat, A.: Helios: a hybrid electrical/optical switch architecture for modular data centers, In: ACM SIGCOMM Computer Communication Review, vol. 40, pp. 339–350, ACM (2010)

  52. Wang, G., Andersen, D., Kaminsky, M., Papagiannaki, K., Ng, T., Kozuch, M., Ryan, M.: c-through: Part-time optics in data centers. In: ACM SIGCOMM Computer Communication Review, vol. 40, pp. 327–338, ACM (2010)

  53. Open Networking Foundation. https://www.opennetworking.org/

  54. Software-Defined Networking (SDN). https://www.opennetworking.org/sdn-resources/sdn-definition

  55. Qazi, Z.A., Lee, J., Jin, T., Bellala, G., Arndt, M., Noubir, G.: Application-awareness in sdn, In: Proceedings of the ACM SIGCOMM 2013 Conference on SIGCOMM (SIGCOMM ’13) (New York, NY, USA), pp. 487–488, ACM (2013)

  56. Wang, G., Ng, T.E., Shaikh, A.: Programming your network at run-time for big data applications, In: Proceedings of the First Workshop on Hot Topics in Software Defined Networks (HotSDN ’12) (New York, NY, USA), pp. 103–108, ACM (2012)

  57. Xiao, X., Hannan, A., Bailey, B., Ni, L.: Traffic engineering with MPLS in the internet. IEEE Netw. 14(2), 28–33 (2000)

    Article  Google Scholar 

  58. Paolucci, F., Cugini, F., Giorgetti, A., Sambo, N., Castoldi, P.: A survey on the path computation element (pce) architecture. IEEE Commun. Surv. Tutor. 15, 1819–1841 (2013)

  59. Sharma, A., Mishra, A., Kumar, V., Venkataramani, A.: Beyond MLU: An application-centric comparison of traffic engineering schemes. In: 2011 Proceedings of the IEEE INFOCOM, pp. 721–729, IEEE (2011)

  60. Jin, T., Tracy, C., Veeraraghavan, M.: Characterization of high-rate large-sized flows. In: 2014 IEEE International Black Sea Conference on Communications and Networking (BlackSeaCom), pp. 73–76 (2014)

  61. Flow-tools. http://www.splintered.net/sw/flow-tools/docs/flow-tools.html

  62. The R Project for Statistical Computing. http://www.r-project.org/

  63. Balman, M., Pouyoul, E., Yao, Y., Bethel, E.W., Loring, B., Prabhat, M., Shalf, J., Sim, A., Tierney, B.L.: Experiences with 100 gbps network applications, In: Proceedings of the Fifth International Workshop on Data-Intensive Distributed Computing Date (DIDC ’12) (New York, NY, USA), pp. 33–42, ACM (2012)

  64. Thompson, K., Miller, G., Wilder, R.: Wide-area Internet traffic patterns and characteristics. IEEE Netw. 11, 10–23 (1997)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Malathi Veeraraghavan.

Additional information

The UVA portion was supported by NSF grants OCI-1127340, CNS-1116081, ACI-1340910, CNS-1405171 and U.S. DOE grants DE-SC0002350 and DE-SC0007341. The ESnet portion was supported by the Director, Office of Science, Office of Basic Energy Sciences, of the U.S. DOE under Contract No. DE-AC02-05CH11231. This research used resources of the ESnet ANI Testbed, which is supported by the Office of Science of the U.S. DOE under contract DE-AC02-05CH11231, funded through the American Recovery and Reinvestment Act of 2009.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yan, Z., Tracy, C., Veeraraghavan, M. et al. A Network Management System for Handling Scientific Data Flows. J Netw Syst Manage 24, 1–33 (2016). https://doi.org/10.1007/s10922-014-9336-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10922-014-9336-2

Keywords

Navigation