The Journal of Supercomputing

, Volume 74, Issue 7, pp 3168–3192 | Cite as

System-wide trade-off modeling of performance, power, and resilience on petascale systems

  • Li YuEmail author
  • Zhou Zhou
  • Yuping Fan
  • Michael E. Papka
  • Zhiling Lan


While performance remains a major objective in the field of high-performance computing (HPC), future systems will have to deliver desired performance under both reliability and energy constraints. Although a number of resilience methods and power management techniques have been presented to address the reliability and energy concerns, the trade-offs among performance, power, and resilience are not well understood, especially in HPC systems with unprecedented scale and complexity. In this work, we present a co-modeling mechanism named TOPPER (system-wide Trade-Off modeling for Performance, PowEr, and Resilience). TOPPER is build with colored Petri nets which allow us to capture the dynamic, complicated interactions and dependencies among different factors such as workload characteristics, hardware reliability, runtime system operation, on a petascale machine. Using system traces collected from a production supercomputer, we conducted a series of experiments to analyze various resilience methods, power capping techniques, and job characteristics in terms of system-wide performance and energy consumption. Our results provide interesting insights regarding performance–power–resilience trade-offs on HPC systems.


Performance–power–resilience modeling Trade-off analysis Petaflop systems Colored Petri nets 



This work is supported in part by US National Science Foundation Grant CCF-1618776 and CCF-1422009. It used data of the Argonne Leadership Computing Facility, which is a DOE Office of Science User Facility supported under Contract DE-AC02-06CH11357.


  1. 1.
    Balbo G (2007) Introduction to generalized stochastic petri nets. In: Proceedings of SFMGoogle Scholar
  2. 2.
    Bautista-Gomez L, Komatitsch D, Maruyama N, Tsuboi S, Cappello F, Matsuoka S (2011) FTI: high performance fault tolerance interface for hybrid systems. In: Proceedings of SCGoogle Scholar
  3. 3.
    Bircher W, John L (2008) Analysis of dynamic power management on multi-core processors. In: Proceedings of ICSGoogle Scholar
  4. 4.
    Bodas D, Song J, Rajappa M, Hoffman A (2014) Simple power-aware scheduler to limit power consumption by HPC system within a budget. In: Proceedings of E2SCGoogle Scholar
  5. 5.
    Chen X, Xu C, Dick R, Mao Z (2010) Performance and power modeling in a multi-programmed multi-core environment. In: Proceedings of DACGoogle Scholar
  6. 6.
    Chiesi M, Vanzolini L, Mucci C, Scarselli E, Guerrieri R (2015) Power-aware job scheduling on heterogeneous multicore architectures. IEEE Trans Parallel Distrib Syst 26:868–877CrossRefGoogle Scholar
  7. 7.
  8. 8.
    Crovella M, Bianchini R, Leblanc T, Markatos E, Wisniewski R (1992) Using communication-to-computation ratio in parallel program design and performance prediction. In: Proceedings of IPDPSGoogle Scholar
  9. 9.
    CPN Tools (2015)
  10. 10.
    Curtis-Maury M, Dzierwa J, Antonopoulos C, Nikolopoulos D (2006) Online power-performance adaptation of multithreaded programs using hardware event-based prediction. In: Proceedings of ICSGoogle Scholar
  11. 11.
    Daly J (2006) A higher order estimate of the optimum checkpoint interval for restart dumps. Future Gener Comput Syst 22:303–312CrossRefGoogle Scholar
  12. 12.
    Di S, Bouguerra M-S, Bautista-Gomez LA, Cappello F (2014) Optimization of multi-level checkpoint model for large scale HPC applications. In: Proceedings of IPDPSGoogle Scholar
  13. 13.
    Elliott J, Kharbas K, Fiala D, Mueller F, Ferreira K, Engelmann C (2012) Combining partial redundancy and checkpointing for HPC. In: Proceedings of ICDCSGoogle Scholar
  14. 14.
    ExSpecT (2015)
  15. 15.
    Fan X, Weber W-D, Barroso L (2007) Power provisioning for a warehouse-sized computer. In: Proceedings of ISCAGoogle Scholar
  16. 16.
    Feitelson D, Rudolph L, Schwiegelshohn U, Sevcik K, Wong P (1997) Theory and practice in parallel job scheduling. In: Proceedings of JSSPPGoogle Scholar
  17. 17.
    Feng X, Ge R, Cameron K (2005) Power and energy profiling of scientific applications on distributed systems. In: Proceedings of IPDPSGoogle Scholar
  18. 18.
    Ferreira K, Stearley J, Laros III J, Oldfield R et al (2011) Evaluating the viability of process replication reliability for exascale systems. In: Proceedings of SCGoogle Scholar
  19. 19.
    Gandhi A, Harchol-Balter M, Adan I (2010) Server farms with setup costs. Perform Eval 67:1123–1138CrossRefGoogle Scholar
  20. 20.
    Ge R, Feng X, Cameron K (2005) Performance-constrained distributed DVS scheduling for scientific applications on power-aware clusters. In: Proceedings of SCGoogle Scholar
  21. 21.
    Ge R, Feng X, Song S, Chang H-C, Li D, Cameron K (2010) PowerPack: energy profiling and analysis of high-performance systems and applications. IEEE Trans Parallel Distrib Syst 21:658–671CrossRefGoogle Scholar
  22. 22.
    Gniady C, Butt A, Hu Y, Lu Y-H (2006) Program counter-based prediction techniques for dynamic power management. IEEE Trans Comput 55:641–658CrossRefGoogle Scholar
  23. 23.
    Goiri I, Kien L, Haque M, Beauchea R, Nguyen T, Guitart J, Torres J, Bianchini R (2011) GreenSlot: scheduling energy consumption in green datacenters. In: Proceedings of SCGoogle Scholar
  24. 24.
    Guenter B, Jain N, Williams C (2011) Managing cost, performance, and reliability tradeoffs for energy-aware server provisioning. In: Proceedings of INFOCOMGoogle Scholar
  25. 25.
    Jensen K (1981) Colored petri nets and the invariant-method. Theoret Comput Sci 14:317–336MathSciNetCrossRefzbMATHGoogle Scholar
  26. 26.
    Kanev S, Hazelwood KM, Wei G-Y, Brooks DM (2014) Tradeoffs between power management and tail latency in warehouse-scale applications. In: Proceedings of IISWCGoogle Scholar
  27. 27.
    LeBlanc T, Anand R, Gabriel E, Subhlok J (2009) Volpexmpi: an MPI Library for execution of parallel applications on volatile nodes. In: European PVM/MPI users’ group meetingGoogle Scholar
  28. 28.
    Lefurgy C, Wang X, Ware M (2007) Server-level power control. In: Proceedings of ICACGoogle Scholar
  29. 29.
    LLview (2013) Graphical monitoring of loadleveler controlled cluster.
  30. 30.
    Martin T, Siewiorek D (2001) Non-ideal battery and main memory effects on CPU speed-setting for low power. IEEE Trans VLSI System 9:29–34CrossRefGoogle Scholar
  31. 31.
    Marwan W, Rohr C, Heiner M (2012) Petri nets in snoopy: a unifying framework for the graphical display, computational modelling, and simulation of bacterial regulatory networks. Humana Press, New YorkGoogle Scholar
  32. 32.
    Mira (2012) Next-generation supercomputer.
  33. 33.
    Moody A, Bronevetsky G, Mohror K, Supinski B (2010) Design, modeling, and evaluation of a scalable multi-level checkpointing system. In: Proceedings of SCGoogle Scholar
  34. 34.
    NSF Cyberinfrastructure Framework for \(21^{st}\) Century Science and Engineering Vision.
  35. 35.
    Patki T, Lowenthal D, Rountree B, Schulz M, de Supinski B (2013) Exploring hardware overprovisioning in power-constrained, high performance computing. In: Proceedings of ICSGoogle Scholar
  36. 36.
    Qiu Q, Pedram M (1999) Dynamic power management based on continuous-time Markov decision processes. In: Proceedings of DACGoogle Scholar
  37. 37.
    Reed D, Lu C, Mendes C (2003) Big systems and big reliability challenges. In: Proceedings of ParCoGoogle Scholar
  38. 38.
    ReNeW (2015)
  39. 39.
    Riesen R, Ferreira K, Silva D, Lemarinier P, Arnold D, Bridges P (2012) Alleviating scalability issues of checkpointing protocols. In: Proceedings of SCGoogle Scholar
  40. 40.
    Rong P, Pedram M (2006) Battery-aware power management based on Markovian decision processes. In: Proceedings of ICCADGoogle Scholar
  41. 41.
    Sancho J, Petrini F, Davis K, Gioiosa R, Jiang S (2005) Current practice and a direction forward in checkpoint/restart implementations for fault tolerance. In: Proceedings of IPDPSGoogle Scholar
  42. 42.
    Srinivasan J, Adve S, Bose P, Rivers J (2004) The impact of technology scaling on lifetime reliability. In: Proceedings of DSNGoogle Scholar
  43. 43.
    Tang W, Desai N, Buettner D, Lan Z (2010) Analyzing and adjusting user runtime estimates to improve job scheduling on blue gene/P. In: Proceedings of IPDPSGoogle Scholar
  44. 44.
  45. 45.
    Tian Y, Lin C, Yao M (2012) Modeling and analyzing power management policies in server farms using stochastic petri nets. In: Proceedings of e-EnergyGoogle Scholar
  46. 46.
    Tiwari A, Laurenzano M, Carrington L, Snavely A (2012) Modeling power and energy usage of HPC Kernels. In: Proceedings of IPDPSWGoogle Scholar
  47. 47.
  48. 48.
    Wallace S, Vishwanath V, Coghlan S, Lan Z, Papka M (2013) Application profilling benchmarks on IBM blue gene/Q. In: Proceedings of clusterGoogle Scholar
  49. 49.
    Wingstrom J (2009) Overcoming the difficulties created by the volatile nature of desktop grids through understanding. Technical report, Ph.D. thesis, University of Hawai’i, ManoaGoogle Scholar
  50. 50.
    Yang X, Zhou Z, Wallace S, Lan Z, Tang W, Coghlan S, Papka M (2013) Integrating dynamic pricing of electricity into energy aware scheduling for HPC systems. In: Proceedings of SCGoogle Scholar
  51. 51.
    Yu L, Zhou Z, Wallace S, Papka M, Lan Z (2015) Quantitative modeling of power-performance tradeoffs on extreme scale systems. J Parallel Distrib Comput Comput 84:1–14CrossRefGoogle Scholar
  52. 52.
    Zhou Z, Lan Z, Tang W, Desai N (2013) Reducing energy costs for IBM blue gene/P via power-aware job scheduling. In: Proceedings of JSSPPGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Department of Computer ScienceIllinois Institute of TechnologyChicagoUSA

Personalised recommendations