Computer Science - Research and Development

, Volume 31, Issue 4, pp 185–193 | Cite as

Optimizing performance-per-watt on GPUs in high performance computing

Temperature, frequency and voltage effects
  • D. C. Price
  • M. A. Clark
  • B. R. Barsdell
  • R. Babich
  • L. J. Greenhill
Special Issue Paper


The magnitude of the real-time digital signal processing challenge attached to large radio astronomical antenna arrays motivates use of high performance computing (HPC) systems. The need for high power efficiency at remote observatory sites parallels that in HPC broadly, where efficiency is a critical metric. We investigate how the performance-per-watt of graphics processing units (GPUs) is affected by temperature, core clock frequency and voltage. Our results highlight how the underlying physical processes that govern transistor operation affect power efficiency. In particular, we show experimentally that GPU power consumption increases non-linearly (quadratic) with both temperature and supply voltage, as predicted by physical transistor models. We show lowering GPU supply voltage and increasing clock frequency while maintaining a low die temperature increases the power efficiency of an NVIDIA K20 GPU by up to 37–48 % over default settings when running xGPU, a compute-bound code used in radio astronomy. We discuss how automatic temperature-aware and application-dependent voltage and frequency scaling (T-DVFS and A-DVFS) may provide a mechanism to achieve better power efficiency for a wider range of compute codes running on GPUs.


Performance per watt Power efficiency Radio astronomy HPC GPU DVFS  


  1. 1.
    Broekema PC, van Nieuwpoort RV, Bal HE (2012) ExaScale high performance computing in the Square Kilometer Array. In: Workshop on high-performance computing for astronomy data, p. 9. ACM Press, New YorkGoogle Scholar
  2. 2.
    Clark MA, Plante PL, Greenhill LJ (2013) Accelerating radio astronomy cross-correlation with graphics processing units. Int. J High Perform. Comput. Appl. 27(2):178–192CrossRefGoogle Scholar
  3. 3.
    Collange S, Defour D, Tisserand A (2009) Power consumption of GPUs from a software perspective. Computational science—ICCS 2009. Springer, Berlin, Heidelberg, pp 914–923Google Scholar
  4. 4.
    Ellingson SW, Taylor GB, Craig J et al (2013) The LWA1 radio telescope. IEEE Trans. Antennas Propag. 61(5):2540–2549CrossRefGoogle Scholar
  5. 5.
    Ge R, Vogt R, Majumder J, Alam A et al (2013) Effects of dynamic voltage and frequency scaling on a K20 GPU. In: Parallel processing (ICPP), 42nd International Conference on, pp 826–833Google Scholar
  6. 6.
    Hong S, Kim H (2010) An integrated GPU power and performance model. ACM SIGARCH Computer Architecture News - ISCA ’10, 38(3):280–289Google Scholar
  7. 7.
    Januszewskia R, Gillyb L, Yilmazc E, Auweterd A (2013) Cooling-making efficient choices. Tech rep, partnership for advanced computing in EuropeGoogle Scholar
  8. 8.
    Jiao Y, Lin H, Balaji P, Feng W (2010) Power and performance characterization of computational kernels on the GPU. In: Green computing and communications (GreenCom), IEEE/ACM Int’l Conf on, pp 221–228Google Scholar
  9. 9.
    Kasichayanula K, Terpstra D, Luszczek P et al (2012) Power aware computing on GPUs. In: Symposium on Application Accelerators in High Performance Computing (SAAHPC), 10–11 July 2012, pp 64–73Google Scholar
  10. 10.
    Kocz J, Greenhill LJ, Barsdell BR et al (2014) A scalable hybrid 543 FPGA/GPU FX correlator. J Astro. Instrum. 03(01):1450,002Google Scholar
  11. 11.
    Leng J, Zu Y, Reddi VJ (2014) Energy efficiency benefits of reducing the voltage guardband on the Kepler GPU architecture. In: 10th IEEE workshop on silicon errors in logic—system effectsGoogle Scholar
  12. 12.
    Liao W, He L (2005) Coupled power and thermal simulation with active cooling. In: Power-aware computer systems, pp 148–163. Springer, BerlinGoogle Scholar
  13. 13.
    Liao W, He L, Lepak KM (2005) Temperature and supply voltage aware performance and power modeling at microarchitecture level. IEEE Trans. Comput. Aided Des. Integr. Circ Syst 552. 24(7):1042–1053Google Scholar
  14. 14.
    Liu Y, Dick RP, Shang L, Yang H (2007) Accurate temperature-dependent integrated circuit leakage power estimation is easy. In: Design, automation and test in Europe conf and exhibition, DATE ’07, pp 1–6Google Scholar
  15. 15.
    Magro A, Adami KZ, Ord S (2014) Suitability of NVIDIA GPUs for SKA1-low. arXiv:1407.4698v3
  16. 16.
    Mei X, Yung LS, Zhao K, Chu X (2013) A measurement study of GPU DVFS on energy conservation. In: Workshop on power-aware computing and systems. ACM Press, New York, pp 1–5Google Scholar
  17. 17.
    Nagasaka H, Maruyama N, Nukada A et al (2010) Statistical power modeling of GPU kernels using performance counters. In: International conference on green computing (green comp), pp 115–122 IEEEGoogle Scholar
  18. 18.
    Nugteren C, van den Braak GJ, Corporaal H (2014) Roofline-aware DVFS for GPUs. International workshop on Adaptive Self-tuning Computing Systems. ACM Press, New York, pp 8–10CrossRefGoogle Scholar
  19. 19.
    Perley RA, Chandler CJ, Butler BJ, Wrobel JM (2011) The expanded Very Large Array: a new telescope for new science. Astrophys. J. Lett. 739(1):L1Google Scholar
  20. 20.
    Ren DQ, Suda R (2010) Investigation on the power efficiency of multi-core and GPU processing element in large scale SIMD computation with CUDA. In: International conference on green computing (green comp), pp 309–316. IEEEGoogle Scholar
  21. 21.
    Rofouei M, Stathopoulos T, Ryffel S (2008) Energy-aware high performance computing with graphic processing units. In: HotPower’08 proceedings of the 2008 conference on power aware computing and systemsGoogle Scholar
  22. 22.
    Wootten A, Thompson A (2009) The Atacama Large Millimeter/submillimeter Array. In: Proceedings of the IEEE 97(8):1463–1471Google Scholar
  23. 23.
    Zimmermann S, Meijer I, Tiwari MK, Paredes S et al (2012) Aquasar: a hot water cooled data center with direct energy reuse. Energy 43(1):237–245CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2015

Authors and Affiliations

  • D. C. Price
    • 1
  • M. A. Clark
    • 2
  • B. R. Barsdell
    • 1
  • R. Babich
    • 2
  • L. J. Greenhill
    • 1
  1. 1.Harvard-Smithsonian Center for Astrophysics, MS 42CambridgeUSA
  2. 2.NVIDIASanta ClaraUSA

Personalised recommendations