Cluster Computing

, Volume 21, Issue 3, pp 1493–1506 | Cite as

Network slicing to improve multicasting in HPC clusters

  • Izzat AlsmadiEmail author
  • Abdallah Khreishah
  • Dianxiang Xu


In high performance computing (HPC) resources’ extensive experiments are frequently executed. HPC resources (e.g. computing machines and switches) should be able to handle running several experiments in parallel. Typically HPC utilizes parallelization in programs, processing and data. The underlying network is seen as the only non-parallelized HPC component (i.e. no dynamic virtual slicing based on HPC jobs). In this scope we present an approach in this paper to utilize software defined networking (SDN) to parallelize HPC clusters among the different running experiments. We propose to accomplish this through two major components: A passive module (network mapper/remapper) to select for each experiment as soon as it starts the least busy resources in the network, and an SDN-HPC active load balancer to perform more complex and intelligent operations. Active load balancer can logically divide the network based on experiments’ host files. The goal is to reduce traffic to unnecessary hosts or ports. An HPC experiment should multicast, rather than broadcast to only cluster nodes that are used by the experiment. We use virtual tenant network modules in Opendaylight controller to create VLANs based on HPC experiments. In each HPC host, virtual interfaces are created to isolate traffic from the different experiments. The traffic between the different physical hosts that belong to the same experiment can be distinguished based on the VLAN ID assigned to each experiment. We evaluate the new approach using several HPC public benchmarks. Results show a significant enhancement in experiments’ performance especially when HPC cluster experiences running several heavy load experiments simultaneously. Results show also that this multi-casting approach can significantly reduce casting overhead that is caused by using a single cast for all resources in the HPC cluster. In comparison with InfiniBand networks that offer interconnect services with low latency and high bandwidth, HPC services based on SDN can provide two distinguished objectives that may not be possible with InfiniBand: The first objective is the integration of HPC with Ethernet enterprise networks and hence expanding HPC usage to much wider domains. The second objective is the ability to enable users and their applications to customize HPC services with different QoS requirements that fit the different needs of those applications and optimize the usage of HPC clusters.


HPC Network slicing SDN 


  1. 1.
    Graham, R.L., Woodall, T.S., Squyres J.M.: Open MPI: A flexible high performance MPI. In: Proceedings of the 6th Annual International Conference on Parallel Processing and Applied Mathematics, Poznan, Poland (2005)Google Scholar
  2. 2.
    Pilla, L.L., Ribeiro, C.P., Coucheney, P., Broquedis, F., Gaujal, B., Navaux, P.O., Méhaut, J.F.: A topology-aware load balancing algorithm for clustered hierarchical multi-core machines. Future Gener. Comput. Syst. 30, 191–201 (2014)CrossRefGoogle Scholar
  3. 3.
    Lowekamp, B., Miller, N., Sutherland, D., Gross, T., Steenkiste, P., Subhlok, J.: A resource query interface for network-aware applications. In: Proceedings of The Seventh International Symposium on High Performance Distributed Computing, 1998 (pp. 189–196). IEEE (1998)Google Scholar
  4. 4.
    Mathis, M., Mahdavi, J.: Diagnosing internet congestion with a transport layer performance tool. In: Proceedings of INET, vol. 96 (1996)Google Scholar
  5. 5.
    Jacobson, V.: Pathchar: a tool to infer characteristics of Internet paths (1997)Google Scholar
  6. 6.
    Carter, R.L., Crovella, M.E.: Dynamic Server Selection Using Bandwidth Probing in Wide-Area Networks. Boston University Computer Science Department, Boston (1996)Google Scholar
  7. 7.
    Vasile, M.A., Pop, F., Tutueanu, R.I., Cristea, V, Koodziej J.: Resource-aware hybrid scheduling algorithm in heterogeneous distributed computing. Future Gener. Comput. Syst. (2014). CrossRefGoogle Scholar
  8. 8.
    Sfrent, A., Pop, F.: Asymptotic scheduling for many task computing in big data platforms. Inf. Sci. 319, 7191 (2015)MathSciNetCrossRefGoogle Scholar
  9. 9.
    Sirbu, A., Pop, C., Serbanescu, C., Pop, F.: Predicting provisioning and booting times in a Metal-as-a-service system. Future Gener. Comput. Syst. 72, 180–192 (2017)CrossRefGoogle Scholar
  10. 10.
    Bessis, N., Sotiriadis, S., Pop, F., Cristea, V.: Using a novel message-exchanging optimization (MEO) model to reduce energy consumption in distributed systems. Simul. Model. Pract. Theory 39, 104–120 (2013)CrossRefGoogle Scholar
  11. 11.
    Watashiba, Y., Date, S., Abe, H., Kido, Y., Ichikawa, K., Yamanaka, H., Takemura, H.: Performance characteristics of an SDN-enhanced job management system for cluster systems with fat-tree interconnect. In: 2014 IEEE 6th International Conference on Cloud Computing Technology and Science (CloudCom) (pp. 781–786). IEEE (2014)Google Scholar
  12. 12.
    Dashdavaa, K., Date, S., Yamanaka, H., Kawai, E., Watashiba, Y., Ichikawa, K., Shimojo, S.: Architecture of a high-speed mpi_bcast leveraging software-defined network. In: European Conference on Parallel Processing (pp. 885–894). Springer, Berlin (2013)CrossRefGoogle Scholar
  13. 13.
    Alsmadi, I., Khamaiseh, S., Xu D.: Network parallelization in HPC clusters. In: The 2016 International Conference on Computational Science and Computational Intelligence (CSCI’16), Symposium of Parallel and Distributed Computing and Computational Science (CSCI-ISPD), December 15–17, Las Vegas (2016)Google Scholar
  14. 14.
    Huang, L.H., Hung, H.J., Lin, C.C., Yang, D.N.: Scalable and bandwidth-efficient multicast for software-defined networks. In: 2014 IEEE Global Communications Conference (GLOBECOM) (pp. 1890–1896). IEEE (2014)Google Scholar
  15. 15.
    Takahashi, K., Khureltulga, D., Watashiba, Y., Kido, Y., Date, S., Shimojo, S.: Performance evaluation of SDN-enhanced MPI allreduce on a cluster system with fat-tree interconnect. In: 2014 International Conference on High Performance Computing & Simulation (HPCS) (pp. 784–792). IEEE (2014)Google Scholar
  16. 16.
    Arap, O., Brown, G., Himebaugh, B., Swany, M.: Software defined multicasting for MPI collective operation offloading with the NetFPGA. In: European Conference on Parallel Processing (pp. 632–643). Springer, Heidelberg (2014)Google Scholar
  17. 17.
    Goglin, B., Hursey, J., Squyres, J.M.: Netloc: towards a comprehensive view of the HPC system topology. In: 2014 43rd International Conference on Parallel Processing Workshops (ICCPW) (pp. 216–225). IEEE (2014)Google Scholar
  18. 18.
    Zhang, X., Jiang, S.: InterferenceRemoval: removing interference of disk access for MPI programs through data replication. In: Proceedings of the 24th ACM International Conference on Supercomputing (pp. 223–232). ACM, New York (2010)Google Scholar
  19. 19.
    Newhall, T., Libeks, J., Greenwood, R., Knerr, J.: Peermon: a peer-to-peer network monitoring system. In: Proceedings of the 24th International Conference on Large Installation System Administration (pp. 1–12). USENIX Association, Berkeley (2010)Google Scholar
  20. 20.
    White, S., Verosky, N., Newhall, T.: A CUDA-MPI hybrid bitonic sorting algorithm for GPU clusters. In: 2012 41st International Conference on Parallel Processing Workshops (ICPPW) (pp. 588–589). IEEE (2012)Google Scholar
  21. 21.
    Bhatelé, A., Kalé, L. V., Kumar, S.: Dynamic topology aware load balancing algorithms for molecular dynamics applications. In: Proceedings of the 23rd International Conference on Supercomputing (pp. 110–116). ACM, New York (2009)Google Scholar
  22. 22.
    Hoefler, T., Snir, M.: Generic topology mapping strategies for large-scale parallel architectures. In: Proceedings of the International Conference on Supercomputing (pp. 75–84). ACM, New York (2011)Google Scholar
  23. 23.
    Liao, W.H., Kuai, S.C., Lu, C.H. Dynamic load-balancing mechanism for software-defined networking. In: 2016 International Conference on Networking and Network Applications (NaNA) (pp. 336–341). IEEE (2016)Google Scholar
  24. 24.
    Wu, Z., Lu, K., Wang, X., Chi, W.: Alleviating network congestion for HPC clusters with fat-tree interconnection leveraging software-defined networking. In: 2016 3rd International Conference on Systems and Informatics (ICSAI) (pp. 808–813). IEEE (2016)Google Scholar
  25. 25.
    Xavier, M., Neves, M., Rossi, F., Ferreto, T., Lange, T., De Rose, C.: Performance evaluation of container-based virtualization for high performance computing environments. In: 2013 21st Euromicro International Conference on Parallel, Distributed and Network-Based Processing (PDP), pp. 233–240 (2013)Google Scholar
  26. 26.
    Prout, A., Arcand, W., Bestor, D., Bergeron, D., Byun, C., Gadepally, V., Hubbell, M., Houle, M., Jones, M., Michaleas, P., Milechin, L., Mullen, J., Rosa, A., Samsi, S., Reuther, A., Kepner, J.: Enhancing HPC security with a user-based firewall. In: IEEE High Performance Extreme Computing (HPEC) Conference, Sep 13–15, Waltham, MA (2016)Google Scholar
  27. 27.
    Zahid, F., Gran, E.G, Bogdaski, B., Johnsen, BD., Skeie, T.: Efficient network isolation and load balancing in multi-tenant HPC clusters. Future Gener. Comput. Syst. (2016). CrossRefGoogle Scholar
  28. 28.
    Subramoni, H., Lu, X., Panda Dhabaleswar, K.: A scalable network-based performance analysis tool for MPI on large-scale HPC systems. In: CLUSTER, pp. 354–358 (2017)Google Scholar
  29. 29.
    Bright Computing, Breaking the Rules: Bright Cluster Manager with Cisco UCS, a Complete HPC Solution. (2014)
  30. 30.
    Seger, M.: Collect. Accessed 27 Jan 2018
  31. 31.
    Thakur, R., Rabenseifner, R., Gropp, W.: Optimization of collective communication operations in MPICH. Int. J. High Perform. Comput. Appl. 19(1), 49–66 (2005)CrossRefGoogle Scholar
  32. 32.
    Luszczek, P. R., Bailey, D. H., Dongarra, J. J., Kepner, J., Lucas, R. F., Rabenseifner, R., Takahashi, D.: The HPC challenge (HPCC) benchmark suite. In: Proceedings of the 2006 ACM/IEEE Conference on Supercomputing (p. 213) (2006)Google Scholar
  33. 33.
    Reussner, R., Sanders, P., Prechelt, L., Müller, M.: SKaMPI: a detailed, accurate MPI benchmark. In: European Parallel Virtual Machine/Message Passing Interface Users Group Meeting (pp. 52–59). Springer, Berlin (1998)CrossRefGoogle Scholar
  34. 34.
    Qiang, G., Weiqin, T., Kausar, S.: A high-performance DSM leveraging software defined network. In: 2015 44th International Conference on Parallel Processing Workshops (ICPPW) (pp. 105–110). IEEE (2015)Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Department of Computing and Cyber SecurityTexas A&MSan AntonioUSA
  2. 2.Electrical and Computer Engineering DepartmentNew Jersey Institute of TechnologyNewarkUSA
  3. 3.Department of Computer ScienceBoise State UniversityBoiseUSA

Personalised recommendations