, Volume 38, Issue 2, pp 111–127 | Cite as

Electrical Grid and Supercomputing Centers: An Investigative Analysis of Emerging Opportunities and Challenges

  • Natalie BatesEmail author
  • Girish Ghatikar
  • Ghaleb Abdulla
  • Gregory A. Koenig
  • Sridutt Bhalachandra
  • Mehdi Sheikhalishahi
  • Tapasya Patki
  • Barry Rountree
  • Stephen Poole


Some of the largest supercomputing centers (SCs) in the United States are developing new relationships with their electricity service providers (ESPs). These relationships, similar to other commercial and industrial partnerships, are driven by a mutual interest to reduce energy costs and improve electrical grid reliability. While SCs are concerned about the quality, cost, environmental impact, and availability of electricity, ESPs are concerned about electrical grid reliability, particularly in terms of energy consumption, peak power demands, and power fluctuations. The power demand for SCs can be 20 MW or more – the theoretical peak power requirements are greater than 45 MW – and recurring intra-hour variability can exceed 8 MW. As a result of this, ESPs may request large SCs to engage in demand response and grid integration.

This paper evaluates today’s relationships, potential partnerships, and possible integration between SCs and their ESPs. The paper uses feedback from a questionnaire submitted to supercomputing centers on the Top100 List in the United States to describe opportunities for overcoming the challenges of HPC-grid integration.


Power Management Demand Response Lawrence Berkeley National Laboratory Advance Reservation Grid Integration 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Top500 Supercomputer Sites, Scholar
  2. 2.
    Aikema D, Simmonds R (2011) Electrical Cost Savings and Clean Energy Usage Potential for HPC Workloads. In: 2011 IEEE International Symposium on Sustainable Systems and Technology (ISSST), pp 1–6Google Scholar
  3. 3.
    Aikema D, Simmonds R, Zareipour H (2012) Data Centres in the Ancillary Services Market. In: 2012 International Green Computing Conference (IGCC), IEEE, pp 1–10Google Scholar
  4. 4.
    ASHRAE (2012) Thermal Guidelines for Data Processing Environments, Special Publication, third edition. American Society of Heating, Refrigerating and Air-Conditioning Engineers, Inc., Atlanta, GAGoogle Scholar
  5. 5.
    Cameron KW, Feng X, Ge R (2005) Performance-Constrained Distributed DVS Scheduling for Scientific Applications on Power-Aware Clusters. In: Supercomputing, Seattle, WashingtonGoogle Scholar
  6. 6.
    Chen H, Caramanis MC, Coskun AK (2014) The Data Center as a Grid Load Stabilizer. In: 19th Asia and South Pacific Design Automation Conference (ASP-DAC), IEEE, pp 105–112Google Scholar
  7. 7.
    Chiu D, Stewart C, McManus B (2012) Electric Grid Balancing Through Lowcost Workload Migration. SIGMETRICS Perform Eval Rev 40(3):48–52CrossRefGoogle Scholar
  8. 8.
    Curtis-Maury M, Jevic FB, Antonopoulos C, Nikolopoulos DS (2008) Prediction-Based Power-Performance Adaptation of Multithreaded Scientific Codes. IEEE Trans Parallel Distrib Syst 19(10):1396–1410CrossRefGoogle Scholar
  9. 9.
    Curtis-Maury M, Dzierwa J, Antonopoulos CD, Nikolopoulos DS (2006) Online Power-Performance Adaptation of Multithreaded Programs Using Hardware Event-Based Prediction. In: International Conference on Supercomputing, New York, NY, USA, ACMGoogle Scholar
  10. 10.
    Curtis-Maury M, Shah A, Jevic FB, Nikolopoulos DS, de Supinski BR, Schulz M (2008) Prediction Models for Multi-Dimensional Power-Performance Optimization on Many Cores. In: International Conference on Parallel Architectures and Compilation techniques, New York, NY, USA, ACMGoogle Scholar
  11. 11.
    David H, Gorbatov E, Hanebutte UR, Khanna R, Le C (2010) RAPL: Memory Power Estimation and Capping. In: Proceedings of the 16th ACM/IEEE International Symposium on Low Power Electronics and Design, ISLPED ’10, New York, NY, USA, ACM, pp 189–194Google Scholar
  12. 12.
    Etinski M, Corbalan J, Labarta J, Valero M (2012) Parallel Job Scheduling for Power Constrained HPC Systems. Parallel Comput 38(12):615–630CrossRefMathSciNetGoogle Scholar
  13. 13.
    Etinski M, Corbalan J, Labarta J, Valero M (2010) Optimizing Job Performance Under a Given Power Constraint in HPC Centers. In: Green Computing Conference, pp 257–267Google Scholar
  14. 14.
    Etinski M, Corbalan J, Labarta J, Valero M (2010) Utilization Driven Power-Aware Parallel Job Scheduling. Comput Sci R&D 25(3-4):207–216Google Scholar
  15. 15.
    Etinski M, Corbalan J, Labarta J, Valero M (2011) Linear Programming Based Parallel Job Scheduling for Power Constrained Systems. In: International Conference on High Performance Computing and Simulation, pp 72–80Google Scholar
  16. 16.
    Fan X, Weber W-D, Barroso LA (2007) Power Provisioning for a Warehouse-Sized Computer. In: The 34th ACM International Symposium on Computer ArchitectureGoogle Scholar
  17. 17.
    Feitelson DG, Schwiegelshohn U, Rudolph L (2004) Parallel Job Scheduling – a Status Report. In: Lecture Notes in Computer Science, Springer-Verlag, pp 1–16Google Scholar
  18. 18.
    Femal ME, Freeh VW (2005) Safe Overprovisioning: Using Power Limits to Increase Aggregate Throughput. In: International Conference on Power-Aware Computer SystemsGoogle Scholar
  19. 19.
    Foster I (2001) The Anatomy of the Grid: Enabling Scalable Virtual Organizations. In: Proceedings of the First IEEE/ACM International Symposium on Cluster Computing and the Grid, pp 6–7Google Scholar
  20. 20.
    Foster I, Kesselman C, Lee C, Lindell B, Nahrstedt K, and Roy A (1999) A Distributed Resource Management Architecture That Supports Advance Reservations and Co-Allocation. In: Seventh International Workshop on Quality of Service, IWQoS ’99, pp 27–36Google Scholar
  21. 21.
    Ganti V, Ghatikar G (2012) Smart Grid as a Driver for Energy-Intensive Industries: A Data Center Case Study. In: Grid-Interop 2012Google Scholar
  22. 22.
    Ge R, Feng X, Feng W, Cameron KW (2007) CPU Miser: A Performance-Directed, Run-Time System for Power-Aware Clusters. In: International Conference on Parallel Processing, Xi’An, ChinaGoogle Scholar
  23. 23.
    Ghamkhari M, Mohsenian-Rad H (2012) Data Centers to Offer Ancillary Services. In: IEEE Third International Conference on Smart Grid Communications (SmartGridComm), pp 436–441, IEEEGoogle Scholar
  24. 24.
    Ghatikar G, Ganti V, Matson N, Piette MA (2012) Demand Response Opportunities and Enabling Technologies for Data Centers: Findings From Field Studies. In: PG&E/SDG&E/CEC/LBNLGoogle Scholar
  25. 25.
    Ghatikar G, Riess D, Piette MA (2014) Analyis of Open Automated Demand Response Deployments in California and Guidelines to Transition to Industry Standards. Technical Report LBNL-6560E, Lawrence Berkeley National Laboratory, 1 Cyclotron Rd, Berkeley, CA 94720Google Scholar
  26. 26.
    Hsu C-H, Feng W-C (2005) A Power-Aware Run-Time System for High-Performance Computing. In: SupercomputingGoogle Scholar
  27. 27.
    Intel (2011) Intel-64 and IA-32 Architectures Software Developer’s Manual, Volumes 3A and 3B: System Programming GuideGoogle Scholar
  28. 28.
    Kappiah N, Freeh VW, Lowenthal DK, Pan F (2005) Exploiting Slack Time in Power-Aware, High-Performance Programs. In: SupercomputingGoogle Scholar
  29. 29.
    Kaushik RT, Nahrstedt K (2012) T*: A Data-Centric Cooling Energy Costs Reduction Approach for Big Data Analytics Cloud, SC’12, Los Alamitos, CA, USA, IEEE Computer Society Press, Article no. 52Google Scholar
  30. 30.
    Li J, Martìnez JF (2006) Dynamic Power-Performance Adaptation of Parallel Computation on Chip Multiprocessors. In: 12th International Symposium on High-Performance Computer Architecture, Austin, TexasGoogle Scholar
  31. 31.
    Lifka DA (1995) The ANL/IBM SP Scheduling System. In: Job Scheduling Strategies for Parallel Processing, Springer-Verlag, pp 295–303Google Scholar
  32. 32.
    Liu Z, Chen Y, Bash C, Wierman A, Gmach D, Wang Z, Marwah M, Hyser C (2012) Renewable and Cooling Aware Workload Management for Sustainable Data Centers. ACM SIGMETRICS Perform Eval Rev 40:175–186CrossRefGoogle Scholar
  33. 33.
    Mu’alem AW, Feitelson DG (2001) Utilization, Predictability, Workloads, and User Runtime Estimates in Scheduling the IBM SP2 with Backfilling. IEEE Trans Parallel Distrib Syst 12(6):329–543Google Scholar
  34. 34.
    Niccolai J (2013) New Data Center Survey Shows Mediocre Results for Energy Efficiency. IT World IDG News Service, Scholar
  35. 35.
    Palensky P, Dietrich D (2011) Demand Side Management: Demand Response, Intelligent Energy Systems, and Smart Loads. IEEE T Ind Inform 7(3):381–388CrossRefGoogle Scholar
  36. 36.
    Patki T, Lowenthal DK, Rountree B, Schulz M, de Supinski BR (2013) Exploring Hardware Overprovisioning in Power-Constrained, High Performance Computing. In: International Conference on Supercomputing, pp 173–182Google Scholar
  37. 37.
    Porterfield AK, Olivier SL, Bhalachandra S, and Prins JF (2013) Power Measurement and Concurrency Throttling for Energy Reduction in OpenMP Programs. In: 27th International Parallel and Distributed Processing Symposium Workshops & PhD Forum (IPDPSW), IEEE, pp 884–891Google Scholar
  38. 38.
    Rountree B, Lowenthal D, de Supinski BR, Schulz M, Freeh V, Bletch T (2009) Adagio: Making DVS Practical for Complex HPC Applications. In: International Conference on SupercomputingGoogle Scholar
  39. 39.
    Rountree B, Ahn DH, de Supinski BR, Lowenthal DK, Schulz M (2012) Beyond DVFS: A First Look at Performance Under a Hardware-Enforced Power Bound. In: IPDPS Workshops, IEEE Computer Society, pp 947–953Google Scholar
  40. 40.
    Rountree B, Lowenthal DK, Funk S, Freeh VW, de Supinski B, Schulz M (2007) Bounding Energy Consumption in Large-Scale MPI Programs. In: SupercomputingGoogle Scholar
  41. 41.
    Sarood O, Kalé LV (2011) A “Cool” Load Balancer for Parallel Applications. In: Proceedings of the 2011 ACM/IEEE Conference on Supercomputing, Seattle, WAGoogle Scholar
  42. 42.
    Springer R, Lowenthal DK, Rountree B, Freeh VW (2006) Minimizing Execution Time in MPI Programs on an Energy-Constrained, Power-Scalable Cluster. In: ACM Symposium on Principles and Practice of Parallel ProgrammingGoogle Scholar
  43. 43.
    Torriti J, Hassan MG, Leach M (2010) Demand Response Experience in Europe: Policies, Programmes and Implementation. Energy 35(4):1575–1583CrossRefGoogle Scholar
  44. 44.
    Yang X, Zhou Z, Wallace S, Lan Z, Tang W, Coghlan S, Papka ME (2013) Integrating Dynamic Pricing of Electricity Into Energy Aware Scheduling for HPC Systems. In: Proceedings of SC13: International Conference for High Performance Computing, Networking, Storage and Analysis, SC ’13, New York, NY, USA, ACMGoogle Scholar
  45. 45.
    Zhou Z, Lan Z, Tang W, Desai N (2013) Reducing Energy Costs for IBM Blue Gene/P via Power-Aware Job Scheduling. IPDPS Workshop, IEEE Computer SocietyGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  • Natalie Bates
    • 1
    Email author
  • Girish Ghatikar
    • 2
  • Ghaleb Abdulla
    • 3
  • Gregory A. Koenig
    • 4
  • Sridutt Bhalachandra
    • 5
  • Mehdi Sheikhalishahi
    • 6
  • Tapasya Patki
    • 7
  • Barry Rountree
    • 3
  • Stephen Poole
    • 4
  1. 1.Energy Efficient HPC Working GroupWashingtonUSA
  2. 2.Lawrence Berkeley National LaboratoryCaliforniaUSA
  3. 3.Lawrence Livermore National LaboratoryCaliforniaUSA
  4. 4.Oak Ridge National LaboratoryTennesseeUSA
  5. 5.University of North CarolinaNorth CarolinaUSA
  6. 6.University of CalabriaCalabriaItaly
  7. 7.University of ArizonaArizonaUSA

Personalised recommendations