Frontiers of Computer Science

, Volume 13, Issue 6, pp 1228–1242 | Cite as

Analyzing time-dimension communication characterizations for representative scientific applications on supercomputer systems

  • Juan ChenEmail author
  • Wenhao Zhou
  • Yong DongEmail author
  • Zhiyuan Wang
  • Chen Cui
  • Feihao Wu
  • Enqiang Zhou
  • Yuhua Tang
Research Article


Exascale computing is one of the major challenges of this decade, and several studies have shown that communications are becoming one of the bottlenecks for scaling parallel applications. The analysis on the characteristics of communications can effectively aid to improve the performance of scientific applications. In this paper, we focus on the statistical regularity in time-dimension communication characteristics for representative scientific applications on supercomputer systems, and then prove that the distribution of communication-event intervals has a power-law decay, which is common in scientific interests and human activities. We verify the distribution of communication-event intervals has really a power-law decay on the Tianhe-2 supercomputer, and also on the other six parallel systems with three different network topologies and two routing policies. In order to do a quantitative study on the power-law distribution, we exploit two groups of statistics: bursty vs. memory and periodicity vs. dispersion. Our results indicate that the communication events show a “strong-bursty and weak-memory” characteristic and the communication event intervals show the periodicity and the dispersion. Finally, our research provides an insight into the relationship between communication optimizations and time-dimension communication characteristics.


power-law distributions supercomputer systems time-dimension communication characteristics Tianhe-2 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.



The authors would like to thank to the funding from the National Key Research and Development Program of China (2017YFB0202200), the Advanced Research Project of China (31511010203), Open Fund (201503-02) from State Key Laboratory of High Performance Computing, and Research Program of NUDT (ZK18-03-10).

Supplementary material

11704_2018_7239_MOESM1_ESM.pdf (299 kb)
Analyzing time-dimension communication characterizations for representative scientific applications on supercomputer systems


  1. 1.
    Liao X K, Pang Z B, Wang K F, Lu Y T, Xie M, Xia J, Dong D Z, Suo G. High performance interconnect network for tianhe system. Journal of Computer Science and Technology, 2015, 30(2): 259–272Google Scholar
  2. 2.
    Geist A, Lucas R. Major computer science challenges at exascale. The International Journal of High Performance Computing Applications, 2009, 23(4): 427–436Google Scholar
  3. 3.
    Shao B B M, Rao H R. A parallel hypercube algorithm for discrete resource allocation problems. IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans, 2006, 36(1): 233–242Google Scholar
  4. 4.
    Dongarra J J, Luszczek P, Petitet A. The linpack benchmark: past, present and future. Concurrency and Computation: Practice and Experience, 2003, 15(9): 803–820Google Scholar
  5. 5.
    Li Y, Zhai J D, Li K Q. Communication analysis and performance prediction of parallel applications on large-scale machines. Innovative Research and Applications in Next-Generation High Performance Computing, 2016, 5: 80–105Google Scholar
  6. 6.
    Yang X J, Du J, Wang Z Y. An effective speedup metric for measuring productivity in large-scale parallel computer systems. The Journal of Supercomputing, 2011, 56(2): 164–181Google Scholar
  7. 7.
    Chen J, Tang Y H, Dong Y, Xue J L, Wang Z Y, and Zhou W H. Reducing static energy in supercomputer interconnection networks using topology-aware partitioning. IEEE Transactions on Computers, 2016, 65(8): 2588–2602MathSciNetzbMATHGoogle Scholar
  8. 8.
    Zhou WH, Chen J, Cui C, Wang Q, Dong D Z, Tang Y H. Detailed and clock-driven simulation for HPC interconnection network. Frontiers of Computer Science, 2016, 10(5): 797–811Google Scholar
  9. 9.
    Raponi P G, Petrini F, Walkup R, Checconi F. Characterization of the communication patterns of scientific applications on blue gene/P. In: Proceedings of 2011 IEEE International Symposium on Parallel and Distributed Processing Workshops and Phd Forum, 2011, 1017–1024Google Scholar
  10. 10.
    Almasi G, Asaad S, Bellofatto R E, Bickford H R, Blumrich M A, Brezzo B, Bright A A, Brunheroto J R, Castanos J G, Chen D. Overview of the IBM blue gene/p project. IBM Journal of Research and Development, 2008, 52(1-2): 199–220Google Scholar
  11. 11.
    Landge A G, Levine J A, Bhatele A, Isaacs K E, Gamblin T, Schulz M, Langer S H, Bremer P T, Pascucci V. Visualizing network traffic to understand the performance of massively parallel simulations. IEEE Transactions on Visualization and Computer Graphics, 2012, 18(12): 2467–2476Google Scholar
  12. 12.
    Yuan X, Mahapatra S, Lang M, Pakin S. LFTI: a new performance metric for assessing interconnect designs for extreme-scale HPC systems. In: Proceedings of the 28th International Parallel and Distributed Processing Symposium, 2014, 273–282Google Scholar
  13. 13.
    Zhou W H, Chen J, Wang Z Y, Xu X H, Xu L Y, Tang Y H. Timedimension communication characterization of representative scientific applications on Tianhe-2. In: Proceedings of the 17th IEEE International Conference on High Performance Computing and Communications. 2015, 423–429Google Scholar
  14. 14.
    Simon H A. On a class of skew distribution functions. Biometrika, 1955, 42(3/4): 425–440MathSciNetzbMATHGoogle Scholar
  15. 15.
    Mitzenmacher M. A brief history of generative models for power law and lognormal distributions. Internet Mathematics, 2004, 1(2): 226–251MathSciNetzbMATHGoogle Scholar
  16. 16.
    Newman M E J. Power laws, pareto distributions and zipf’s law. Contemporary Physics, 2005, 46(5): 323–351Google Scholar
  17. 17.
    Sornette D. Critical Phenomena in Natural Sciences: Chaos, Fractals, Selforganization and Disorder: Concepts and Tools. Spnhger Science & Business Media, 2004Google Scholar
  18. 18.
    Faloutsos M, Faloutsos P, Faloutsos C. On power-law relationships of the internet topology. ACM SIGCOMM Computer Communication Review, 1999, 29(4): 251–262zbMATHGoogle Scholar
  19. 19.
    Clauset A, Shalizi C R, Newman M E J. Power-law distributions in empirical data. SIAM Review, 2009, 51(4): 661–703MathSciNetzbMATHGoogle Scholar
  20. 20.
    Bailey D H, Barszcz E, Barton J T, Browning D S, Carter R L, Dagum L, Fatoohi R A, Frederickson P O, Lasinski T A, Schreiber R S, Simon H D, Venkatakrishnan V, Weeratunga S K. The nas parallel benchmarks. International Journal of High Performance Computing Applications, 1991, 5(3): 63–73Google Scholar
  21. 21.
    Jasak H, Jemcov A, Tukovic Z. Openfoam: A C++ library for complex physics simulations. International Workshop on Coupled Methods in Numerical Dynamics, 2007, 1000: 1–20Google Scholar
  22. 22.
    Accelerated Strategic Computing Initiative. The ASCI sweep3d benchmark code, 1995Google Scholar
  23. 23.
    Kim J, Esler K, McMinis J, Clark B, Gergely J, Chiesa S, Delaney K, Vincent J, Ceperley D. Qmcpack simulation suite, 2014Google Scholar
  24. 24.
    Plimpton S, Crozier P, Thompson A. Lammps-large-scale atomic/molecular massively parallel simulator. Sandia National Laboratories, 2007, 18: 43Google Scholar
  25. 25.
    Berendsen H J C, van der Spoel D, van Drunen R. GROMACS: a message-passing parallel molecular dynamics implementation. Computer Physics Communications, 1995, 91(1): 43–56Google Scholar
  26. 26.
    Zhai J D, Chen W G, Zheng W M. Phantom: predicting performance of parallel applications on large-scale parallel machines using a single node. ACM Sigplan Notices, 2010, 45(5): 305–314Google Scholar
  27. 27.
    Gutenberg B, Richter C F. Frequency of earthquakes in california. Bulletin of the Seismological Society of America, 1944, 34(4): 185–188Google Scholar
  28. 28.
    Neukum G, Ivanov B A. Crater size distributions and impact probabilities on earth from lunar, terrestrial-planet, and asteroid cratering data. Hazards due to Comets and Asteroids, 1994, 359–416Google Scholar
  29. 29.
    Lu E T, Hamilton R J. Avalanches and the distribution of solar flares. The Astrophysical Journal, 1991, 380: L89–L92Google Scholar
  30. 30.
    Roberts D C, Turcotte D L. Fractality and self-organized criticality of wars. Fractals, 1998, 6(4): 351–357Google Scholar
  31. 31.
    Zipf G K. Human Behavior and The Principle of Least Effort. Addisonwesley Press, 1949, 1–721Google Scholar
  32. 32.
    Estoup J B. Les gammes stenographiques. Institut Stenographique de France, 1916Google Scholar
  33. 33.
    Zanette D H, Manrubia S C. Vertical transmission of culture and the distribution of family names. Physica A: Statistical Mechanics and its Applications, 2001, 295(1): 1–8MathSciNetzbMATHGoogle Scholar
  34. 34.
    Coile R C. Lotka’s frequency distribution of scientific productivity. Journal of the american society for information science, 1977, 28(6): 366–370Google Scholar
  35. 35.
    de Solla Price D J. Networks of scientific papers. Science, 1965, 149(3683): 510–515Google Scholar
  36. 36.
    Cox R A K, Felton J M, Chung K H. The concentration of commercial success in popular music: an analysis of the distribution of gold records. Journal of cultural economics, 1995, 19(4): 333–340Google Scholar
  37. 37.
    Kohli R, Sah R K. Market shares: some power law results and observations. Management Science, 2006, 52(11): 1792–1798Google Scholar
  38. 38.
    Willis J C, Yule G U. Some statistics of evolution and geographical distribution in plants and animals, and their significance. Nature, 1922, 109(2728): 177–179Google Scholar
  39. 39.
    Pareto V. Cours D’économie Politique. Librairie Droz, 1964, 1–429Google Scholar
  40. 40.
    Adamic L A, Huberman B A. The nature of markets in the world wide Web. Quarterly Joural of Electronic Commerce, 2000, 1(1): 5–12Google Scholar
  41. 41.
    Crovella M E, Bestavros A. Self-similarity in World Wide Web traffic: evidence and possible causes. IEEE/ACM Transactions on Networking, 1997, 5(6): 835–846Google Scholar
  42. 42.
    Goh K I, Barabási A L. Burstiness and memory in complex systems. EPL (Europhysics Letters), 2008, 81(4): 48002.MathSciNetGoogle Scholar
  43. 43.
    Hidalgo R C A. Conditions for the emergence of scaling in the interevent time of uncorrelated and seasonal systems. Physica A: Statistical Mechanics and its Applications, 2006, 369(2): 877–883MathSciNetGoogle Scholar
  44. 44.
    Zhou T, Zhao Z D, Yang Z M, Zhou C S. Relative clock verifies endogenous bursts of human dynamics. EPL (Europhysics Letters), 2012, 97(1): 18006Google Scholar
  45. 45.
    Lee Rodgers J, Nicewander W A. Thirteen ways to look at the correlation coefficient. The American Statistician, 1988, 42(1): 59–66Google Scholar
  46. 46.
    Legates D R, McCabe G J. Evaluating the use of goodness-of-fitaś measures in hydrologic and hydroclimatic model validation. Water Resources Research, 1999, 35(1): 233–241Google Scholar
  47. 47.
    Engelen R V. Efficient symbolic analysis for optimizing compilers. In: Proceedings of the 10th International Conference on Compiler Construction. 2001, 118–132Google Scholar
  48. 48.
    Kelefouras V, Kritikakou A, Goutis C. A methodology for speeding up loop kernels by exploiting the software information and the memory architecture. Computer Languages, Systems & Structure, 2015, 41: 21–41zbMATHGoogle Scholar
  49. 49.
    Liao X K, Xiao L Q, Yang C Q, Lu Y T. Milkyway-2 supercomputer: system and application. Frontiers of Computer Science, 2014, 8(3): 345–356MathSciNetGoogle Scholar
  50. 50.
    Dally WJ, Towles B P. Principles and Practices of Interconnection Networks, Elsevier, Amsterdam, 2004, 1–550Google Scholar
  51. 51.
    Morgan J A, Tatar J F. Calculation of the residual sum of squares for all possible regressions. Technometrics, 1972, 14(2): 317–325zbMATHGoogle Scholar
  52. 52.
    Boccaletti S, Latora V, Moreno Y, Chavezf M, Hwang D U. Complex networks: structure and dynamics. Physics Reports, 2006, 424(4-5): 175–308MathSciNetzbMATHGoogle Scholar
  53. 53.
    Tabe T B, Stout Q F. The use of the MPI communication library in the NAS parallel benchmarks. Ann Arbor, 1999, 1001: 48109Google Scholar
  54. 54.
    Malmgren R D, Stouffer D B, Motter A E, Amaral L A. A poissonian explanation for heavy tails in e-mail communication. Proceedings of the National Academy of Sciences, 2008, 105(47): 18153–18158.Google Scholar
  55. 55.
    Kay S M, Marple Jr S L. Spectrum analysis — a modern perspective. Proceedings of the IEEE, 1981, 69(11): 1380–1419.Google Scholar
  56. 56.
    Woodbury G. An Introduction to Statistics, Cengage Learning, 2001, 1–720Google Scholar
  57. 57.
    Bland J M, Altman D G. Statistics notes: measurement error. Bmj, 1996, 313(7059): 744Google Scholar
  58. 58.
    Gropp W, Lusk E, Skjellum A. Using MPI: Portable Parallel Programming with The Message-Passing Interface, Massachusetts: MIT press, 1999, 1–275Google Scholar
  59. 59.
    Matsuda M, Kudoh T, Kodama Y, Takano R, Ishikawa Y. Efficient MPI collective operations for clusters in long-and-fast networks. In: Proceedings of 2006 IEEE International Conference on Cluster Computing. 2006, 1–9Google Scholar

Copyright information

© Higher Education Press and Springer-Verlag GmbH Germany, part of Springer Nature 2018

Authors and Affiliations

  • Juan Chen
    • 1
    Email author
  • Wenhao Zhou
    • 1
  • Yong Dong
    • 1
    Email author
  • Zhiyuan Wang
    • 1
  • Chen Cui
    • 1
  • Feihao Wu
    • 1
  • Enqiang Zhou
    • 1
  • Yuhua Tang
    • 1
  1. 1.State Key Laboratory of High Performance Computing, College of ComputerNational University of Defense TechnologyChangshaChina

Personalised recommendations