Abstract
Dedicated wide-area network connections are employed in big data and high-performance computing scenarios, since the absence of cross-traffic promises to make it easier to analyze and optimize data transfers over them. However, nonlinear transport dynamics and end-system complexity due to multi-core hosts and distributed filesystems make these tasks surprisingly challenging. We present an overview of methods to analyze memory and disk file transfers using extensive measurements over 10 Gbps physical and emulated connections with 0–366 ms round-trip times (RTTs). For memory transfers, we derive performance profiles of TCP and UDT throughput as a function of RTT, which show concave regions in contrast to entirely convex regions predicted by previous models. These highly desirable concave regions can be expanded by utilizing large buffers and more parallel flows. We also present Poincaré maps and Lyapunov exponents of TCP and UDT throughput traces that indicate complex throughput dynamics. For disk file transfers, we show that throughput can be optimized using a combination of parallel I/O and network threads under direct I/O mode. Our initial throughput measurements of Lustre filesystems mounted over long-haul connections using LNet routers show convex profiles indicative of I/O limits.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Alligood, K.T., Sauer, T.D., Yorke, J.A.: Chaos: An Introduction to Dynamical Systems. Springer, Reading, MA (1996)
Aspera high-speed file transfer. http://www-03.ibm.com/software/products/en/high-speed-file-transfer
Cardwell, N., Cheng, Y., Gunn, C.S., Yeganeh, S.H., Jacobson, V.: BBR: congestion based congestion control. ACM Queue 14(5), 1–34 (2016)
Energy Sciences Network. http://www.es.net
GT 4.0 GridFTP. http://www.globus.org
Gu, Y., Grossman, R.L.: UDT: UDP-based data transfer for high-speed wide area networks. Comput. Netw. 51(7) (2007)
Hassan, M., Jain, R.: High Performance TCP/IP Networking: Concepts, Issues, and Solutions. Prentice Hall, Upper Saddle River, NJ (2004)
Henschel, R., Simms, S., Hancock, D., Michael, S., Johnson, T., Heald, N., William, T., et al.: Demonstrating lustre over a 100 Gbps wide area network of 3,500 km. In: High Performance Computing, Networking, Storage and Analysis (SC), 2012 International Conference on, pp. 1–8 (2012)
Jain, S., Kumar, A., Mandal, S., et al.: B4: experience with a globally-deployed software defined Wan. SIGCOMM Comput. Commun. Rev. 43(4), 3–14 (2013)
Kelly, T.: Scalable TCP: Improving performance in high speed wide area networks. Comput. Commun. Rev. 33(2), 83–91 (2003)
Liu, Q., Rao, N.S.V., Wu, C.Q., Yun, D., Kettimuthu, R., Foster, I.: Measurement-based performance profiles and dynamics of udt over dedicated connections. In: International Conference on Network Protocols, Singapore (2016)
Liu, Z., Balaprakash, P., Kettimuthu, R., Foster, I.T.: Explaining wide area data transfer performance. In: ACM Symposium on High-Performance Parallel and Distributed Computing (HPDC), Washington, DC (2017)
Mathis, M., Semke, J., Mahdavi, J., Ott, T.: The mascroscopic behavior of the TCP congestion avoidance algorithm. Comput. Commun. Rev. 27(3), 62–82 (1997)
Michael, S., Zhen, L., Henschel, R., Simms, S., Barton, E., Link, M.: A study of Lustre networking over a 100 gigabit wide area network with 50 milliseconds of latency. In: Proceedings of the fifth international workshop on Data-Intensive Distributed Computing, pp. 43–52 (2012)
Padhye, J., Firoiu, V., Towsley, D.F., Kurose, J.F.: Modeling TCP Reno performance: a simple model and its empirical validation. IEEE/ACM Trans. Networking 8(2), 133–145 (2000)
Palencia, J., Budden, R., Sullivan, K.: Kerberized Lustre 2.0 over the WAN. In: Proceedings of the 2010 TeraGrid Conference (2010)
Rao, N.S.V., Gao, J., Chua, L.O.: On dynamics of transport protocols in wide-area internet connections. In: Complex Dynamics in Communication Networks. Springer, Berlin (2005)
Rao, N.S.V., Towsley, D., Vardoyan, G., Settlemyer, B.W., Foster, I.T., Kettimuthu, R.: Sustained wide-area TCP memory transfers over dedicated connections. In: IEEE International Conference on High Performance and Smart Computing, New York, NY (2015)
Rao, N.S.V., Liu, Q., Sen, S., Hinkel, G., Imam, N., Settlemyer, B.W., Foster, I.T., et al.: Experimental analysis of file transfer rates over wide-area dedicated connections. In: 18th IEEE International Conference on High Performance Computing and Communications (HPCC), Sydney, Australia, pp. 198–205 (2016)
Rao, N.S.V., Liu, Q., Sen, S., Henley, J., Foster, I.T., Kettimuthu, R., Towsley, D., Vardoyan, G.: TCP throughput profiles using measurements over dedicated connections. In: ACM Symposium on High-Performance Parallel and Distributed Computing (HPDC), Washington, DC (2017)
Rao, N.S.V., Liu, Q., Sen, S., Towsley, D., Vardoyan, G., Foster, I.T., Kettimuthu, R.: Experiments and analyses of data transfers over wide-area dedicated connections. In: The 26th International Conference on Computer Communications and Network(ICCCN 2017), Vancouver, Canada (2017)
Rhee, I., Xu, L.: CUBIC: a new TCP-friendly high-speed TCP variant. In: Proceedings of the Third International Workshop on Protocols for Fast Long-Distance Networks (2005)
Sen, S., Rao, N.S.V., Liu, Q., Imam, N., Foster, I.T., Kettimuthu, R.: On analytics of file transfer rates over dedicated wide-area connections. In: First International Workshop on Workflow Science (WOWS) in conjunction with 13th IEEE International Conference on e-Science, Auckland, New Zealand, (2017)
Settlemyer, B.W., Dobson, J.D., Hodson, S.W., Kuehn, J.A., Poole, S.W., Ruwart, T.M.: A technique for moving large data sets over high-performance long distance networks. In: IEEE 27th Symposium on Mass Storage Systems and Technologies (MSST), pp. 1–6 (2011)
Shorten, R.N., Leith, D.J.: H-TCP: TCP for high-speed and long-distance networks. In: Proceedings of the Third International Workshop on Protocols for Fast Long-Distance Networks (2004)
Srikant, Y., Ying, L.: Communication Networks: An Optimization, Control, and Stochastic Networks Perspective. Cambridge University Press, Cambridge (2014)
Walgenbach, J., Simms, S.C., Westneat, K., Miller, J.P.: Enabling Lustre WAN for production use on the TeraGrid: a lightweight UID mapping scheme. In: Proceedings of the 2010 TeraGrid Conference (2010)
XDD - The eXtreme dd toolset. https://github.com/bws/xdd
Yee, T., Leith, D., Shorten, R.: Experimental evaluation of high-speed congestion control protocols. Trans. Netw. 15(5), 1109–1122 (2007)
Acknowledgements
This work is supported in part by the RAMSES and Net2013 projects, Office of Advanced Computing Research, US Department of Energy, and by the Extreme Scale Systems Center, sponsored by US Department of Defense, and performed at Oak Ridge National Laboratory managed by UT-Battelle, LLC for US Department of Energy under Contract DE-AC05-00OR22725.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG, part of Springer Nature
About this paper
Cite this paper
Rao, N.S.V., Liu, Q. (2018). On Data Transfers Over Wide-Area Dedicated Connections. In: Rao, N., Brooks, R., Wu, C. (eds) Proceedings of International Symposium on Sensor Networks, Systems and Security. ISSNSS 2017. Springer, Cham. https://doi.org/10.1007/978-3-319-75683-7_2
Download citation
DOI: https://doi.org/10.1007/978-3-319-75683-7_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-75682-0
Online ISBN: 978-3-319-75683-7
eBook Packages: EngineeringEngineering (R0)