Characterization and Understanding Machine-Specific Interconnects

  • Vitali Morozov
  • Jiayuan Meng
  • Venkatram Vishwanath
  • Kalyan Kumaran
  • Michael E. Papka
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7979)


The use of efficient communication patterns is becoming increasingly important to utilize high concurrency and achieve high throughput and low latency of modern high-performance supercomputers. Efficacy is not only dictated by an application communication pattern, but also driven by the interconnect properties, the node architecture, and the quality of runtime communication libraries. Different systems require different tradeoffs with respect to communication mechanisms, which can impact the choice of application implementations. We use the in-house MPI benchmark suite to study the communication behavior of the interconnects and guide the performance tuning of scientific applications. We report the results of our investigation of four supercomputer systems located at ALCF, and present lessons learned from our experience.


interconnect performance tuning topology-aware MPI 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Grinberg, L., Morozov, V., Fedosov, D., Insley, J.A., Papka, M.E., Kumaran, K., Karniadakis, G.E.: A New Computational Paradigm in Multiscale Simulations with Applications to Brain Blood Flow. Gordon Bell Honorable Mention. In: Int. Conf. for High Performance Computing, Networking, Storage, and Analysis, Seattle, WA, November 12-18 (2011)Google Scholar
  2. 2.
    Müller, M.S., van Waveren, M., Lieberman, R., Whitney, B., Saito, H., Kumaran, K., Baron, J., Brantley, W.C., Parrott, C., Elken, T., Feng, H., Ponder, C.: SPEC MPI2007 – an Application Benchmark Suite for Parallel Systems using MPI. J. Concurrency and Computation: Practice and Experience – Int. Supercomputing Conf. 22(2), 191–205 (2010)Google Scholar
  3. 3.
    Intel MPI Benchmarks 3.2.3. Intel Corporation,
  4. 4.
    Phloem MPI Benchmarks. Lawrence Livermore National Laboratory Technical report LLNL-MI-400479,
  5. 5.
    The Ohio State University MPI Benchmarks. The Ohio State University,
  6. 6.
    Augustin, W., Worsch, T.: The SKaMPI 5 Manual. Technical report (2008),
  7. 7.
    Morozov, V., Meng, J., Vishwanath, V., Hammond, J., Kumaran, K., Papka, M.: ALCF MPI Benchmarks: Understanding Machine-Specific Communication Behavior. In: Fifth Int. Workshop on Parallel Programming Models and Systems Software for High-End Computing, Pittsburgh, MA, September 10 (2012)Google Scholar
  8. 8.
    NAS Parallel Benchmarks. NASA Advanced Supercomputing Division,
  9. 9.
    Underwood, K.D., Brightwell, R.: The Impact of MPI Queue Usage on Message Latency. In: Proc. Int. Conf. on Parallel Processing, vol. 1, pp. 152–160 (2004)Google Scholar
  10. 10.
    Brodsky, A., Pedersen, J.B., Wagner, A.S.: On the Complexity of Buffer Allocation in Message Passing Systems. J. Parallel Distrib. Comput. 65(6), 692–713 (2005)CrossRefzbMATHGoogle Scholar
  11. 11.
    Chen, D., Eisley, N.A., Heidelberger, P., Senger, R.M., Sugawara, Y., Kumar, S., Salapura, V., Satterfield, D.L., Steinmacher-Burow, B., Parker, J.J.: The IBM Blue Gene/Q Interconnection Network and Message Unit. In: Int. Conf. on High Performance Computing, Networking, Storage and Analysis, Seattle, WA, November 12-18 (2011)Google Scholar
  12. 12.
    Overview of the IBM Blue Gene/P project. IBM J. Res. Dev. 52(1/2), 199–220 (2008)Google Scholar
  13. 13.
    Hoefler, T., Snir, M.: Generic Topology Mapping Strategies for Large-Scale Parallel Architectures. In: Proc. Int. Conf. on Supercomputing, Tucson, Arizona, pp. 78–84 (2011)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Vitali Morozov
    • 1
  • Jiayuan Meng
    • 1
  • Venkatram Vishwanath
    • 1
  • Kalyan Kumaran
    • 1
  • Michael E. Papka
    • 1
  1. 1.Argonne Leadership Computing FacilityArgonne National LaboratoryArgonneUSA

Personalised recommendations