Preliminary Evaluation of a Parallel Trace Replay Tool for HPC Network Simulations

  • Bilge AcunEmail author
  • Nikhil Jain
  • Abhinav Bhatele
  • Misbah Mubarak
  • Christopher D. Carothers
  • Laxmikant V. Kale
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9523)


This paper presents a preliminary evaluation of TraceR, a trace replay tool built upon the ROSS-based CODES simulation framework. TraceR can be used for predicting network performance and understanding network behavior by simulating messaging on interconnection networks. It addresses two major shortcomings in current network simulators. First, it enables fast and scalable simulations of large-scale supercomputer networks. Second, it can simulate production HPC applications using BigSim’s emulation framework. In addition to introducing TraceR, this paper studies the impact of input parameters on simulation performance. We also compare TraceR with other network simulators such as SST and BigSim, and demonstrate TraceR ’s scalability using various case studies.


Execution Time Batch Size Optimistic Mode Kernel Process Global Synchronization 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344. This work was funded by the LDRD Program at LLNL under project tracking code 13-ERD-055 (LLNL-CONF-667225).


  1. 1.
    DUMPI: The mpi trace library.
  2. 2.
    Bhatele, A., et al.: Overcoming scaling challenges in biomolecular simulations across multiple platforms. In: Proceedings of IEEE International Parallel and Distributed Processing Symposium 2008 (2008)Google Scholar
  3. 3.
    Bohm, S., Engelmann, C.: xSim: the extreme-scale simulator. In: HPCS (2011)Google Scholar
  4. 4.
    Carothers, C.D., Bauer, D., Pearce, S.: ROSS: a high-performance, low-memory, modular time warp system. J. Parallel Distr. Com. 62(11), 1648–1669 (2002)zbMATHCrossRefGoogle Scholar
  5. 5.
    Casanova, H., et al.: Versatile, scalable, and accurate simulation of distributed applications and platforms. J. Parallel Distr. Comput. 74(10), 2899–2917 (2014)CrossRefGoogle Scholar
  6. 6.
    Cope, J., et al.: Codes: enabling co-design of multilayer exascale storage architectures. In: Proceedings of the Workshop on Emerging Supercomp. Technologies (2011)Google Scholar
  7. 7.
    Girona, S., Labarta, J.: Sensitivity of performance prediction of message passing programs. J. Supercomput. 17, 291–298 (2000)zbMATHCrossRefGoogle Scholar
  8. 8.
    Hoefler, T., Schneider, T., Lumsdaine, A.: LogGOPSim - simulating large-scale applications in the LogGOPS model. In: Proceedings of the 19th ACM International Symposium on HPDC, pp. 597–604. ACM (2010)Google Scholar
  9. 9.
    Huang, C., Lawlor, O., Kalé, L.V.: Adaptive MPI. In: Rauchwerger, L. (ed.) LCPC 2003. LNCS, vol. 2958, pp. 306–322. Springer, Heidelberg (2004) CrossRefGoogle Scholar
  10. 10.
    Janssen, C.L., et al.: A simulator for large-scale parallel computer architectures. IJDST 1(2), 57–73 (2010)Google Scholar
  11. 11.
    Jiang, N., et al.: A detailed and flexible cycle-accurate network-on-chip simulator. In: IEEE Internationsl Symposium on Performance Analysis of Systems and Software (2013)Google Scholar
  12. 12.
    Kale, L.V., Bhatele, A. (eds.): Parallel Science and Engineering Applications: The Charm++ Approach. CRC Press, Taylor & Francis Group, Boca Raton, Florida, USA (2013)Google Scholar
  13. 13.
    Minkenberg, C., Rodriguez, G.: Trace-driven co-simulation of high-performance computing systems using OMNeT++. In: Proceedings of the 2nd International Conference on Simulation Tools and Techniques, p. 65 (2009)Google Scholar
  14. 14.
    Mubarak, M., Carothers, C.D., Ross, R.B., Carns, P.: A case study in using massively parallel simulation for extreme-scale torus network codesign. In: Proceedings of the 2nd ACM SIGSIM PADS, pp. 27–38. ACM (2014)Google Scholar
  15. 15.
    Penoff, B., Wagner, A., Tuxen, M., Rungeler, I.: Mpi-netsim: a network simulation module for mpi. In: Parallel and Distributed Systems (ICPADS), IEEE (2009)Google Scholar
  16. 16.
    Wilmarth, T., Kalé, L.V.: POSE: Getting over grainsize in parallel discrete event simulation. In: International Conference on Parallel Processing, pp. 12–19 (2004)Google Scholar
  17. 17.
    Zheng, G., et al.: Simulation-based performance prediction for large parallel machines. Intl. J. Parallel Program. 33, 183–207 (2005)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Bilge Acun
    • 1
    Email author
  • Nikhil Jain
    • 1
  • Abhinav Bhatele
    • 2
  • Misbah Mubarak
    • 3
  • Christopher D. Carothers
    • 4
  • Laxmikant V. Kale
    • 1
  1. 1.Department of Computer ScienceUniversity of Illinois at Urbana-ChampaignUrbanaUSA
  2. 2.Center for Applied Scientific ComputingLawrence Livermore National LaboratoryLivermoreUSA
  3. 3.Mathematics and Computer Science DivisionArgonne National LaboratoryArgonneUSA
  4. 4.Department of Computer ScienceRensselaer Polytechnic InstituteTroyUSA

Personalised recommendations