The Journal of Supercomputing

, Volume 59, Issue 1, pp 486–498 | Cite as

Using SPEC CPU2006 to evaluate the sequential and parallel code generated by commercial and open-source compilers

  • Sergio Aldea
  • Diego R. Llanos
  • Arturo González-Escribano
Article

Abstract

The role of the compiler is fundamental to exploit the hardware capabilities of a system running a particular application, minimizing the sequential execution time and, in some cases, offering the possibility of parallelizing part of the code automatically. This paper relies on the SPEC CPU2006 v1.1 benchmark suite to evaluate the performance of the code generated by three widely-used compilers (Intel C++/Fortran Compiler 11.0, Sun Studio 12 and GCC 4.3.2). Performance is measure in terms of base speed for reference problem sizes. Both sequential and automatic parallel performance obtained is analyzed, using different hardware architectures and configurations. The study includes a detailed description of the different problems that arise while compiling SPEC CPU2006 benchmarks with these tools, an information difficult to obtain elsewhere.

Having in mind that performance is a moving target in the field of compilers, our evaluation shows that the sequential code generated by both Sun and Intel compilers for the SPEC CPU2006 integer benchmarks present a similar performance, while the floating-point code generated by Intel compiler is faster than its competitors. With respect to the auto-parallelization options offered by Intel and Sun compilers, our study shows that their benefits only apply to some floating-point benchmarks, with an average speedup of 1.2× with four processors. Meanwhile, the GCC suite evaluated is not capable of compiling the SPEC CPU2006 benchmark with auto-parallelization options enabled.

Keywords

Compiler performance Automatic parallelization Benchmarking 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    IBM posts SPEC CPU2006 scores for next-generation System x scalable server. ftp://ftp.software.ibm.com/eserver/benchmarks/news/newsblurb_x3850M2_speccpu_090507.pdf. IBM Corporation. Accessed 8 April 2010
  2. 2.
    Intel Xeon Processor performance summary. http://www.intel.com/performance/work-station/xeon/summary.htm. Intel Corporation. Accessed 8 April 2010
  3. 3.
    Performance and benchmarks: Performance reports for CELSIUS R640. http://www.fujitsu-siemens.com/products/deskbound/workstations/performance/celsius_r640.html. Fujitsu Siemens. Accessed 8 April 2010
  4. 4.
    Quick reference guide to optimization with Intel compilers version 11 Intel Software development products. http://www.intel.com/software/products/compilers/docs/qr_guide.htm. Accessed 20 January 2010
  5. 5.
    SPEC releases CPU2006 benchmarks. http://www.spec.org/cpu2006/press/release.html. Standard Performance Evaluation Corporation. Accessed 8 April 2010
  6. 6.
    Sun Fire X4450 server performance. http://www.sun.com/servers/x64/x4450/benchmarks.jsp?display=1. Sun Microsystems. Accessed 8 April 2010
  7. 7.
    Sun Studio 12: C user’s guide. http://dlc.sun.com/pdf/819-5265/819-5265.pdf. Sun Microsystems. Accessed 8 April 2010
  8. 8.
    Sun Studio 12: Fortran user’s guide. http://dlc.sun.com/pdf/819-5263/819-5263.pdf. Sun Microsystems. Accessed 8 April 2010
  9. 9.
    Using the gnu compiler collection. http://gcc.gnu.org/onlinedocs/gcc-4.3.2/gcc.pdf. Accessed 8 April 2010
  10. 10.
    Chan Y, Sudarsanam A, Wolfe A (1994) The effect of compiler-flag tuning on SPEC benchmark performance. SIGARCH Comput Archit News 22(4):60–70 CrossRefGoogle Scholar
  11. 11.
    Kejariwal A, Veidenbaum AV, Nicolau A, Tian X, Girkar M, Saito H, Banerjee U (2008) Comparative architectural characterization of SPEC CPU2000 and CPU2006 benchmarks on the Intel Core 2 Duo processor. In: International conference on embedded computer systems: architectures, modeling, and simulation, 2008, SAMOS 2008, pp 132–141 Google Scholar
  12. 12.
    Li S, Qiao L, Tang Z, Cheng B, Gao X (2009) Performance characterization of SPEC CPU2006 benchmarks on Intel and AMD platform. In: First international workshop on education technology and computer science, 2009. ETCS ’09, vol 2, pp 116–121 Google Scholar
  13. 13.
    Munafo R Reference machine times for SPEC CPU2006. http://www.mrob.com/pub/comp/benchmarks/spec.html. Accessed 8 April 2010
  14. 14.
    Polsson K (2010) Chronology of workstation computers. http://www.islandnet.com/~kpolsson/workstat/work1987.htm. Accessed 3 June 2010
  15. 15.
    Schouten D, Tian X, Bik A, Girkar M (2003) Inside the Intel compiler. Linux J 2003(106):4 Google Scholar
  16. 16.
    Spradling CD (2007) SPEC CPU2006 benchmark tools. SIGARCH Comput Archit News 35(1):130–134 CrossRefGoogle Scholar
  17. 17.
    Tatkar V Build high performance applications on multicore systems using sun studio compilers and tools. In: Sun TechDays India 2008, 27th–29th Feb 2008, Hyderabad, India. http://developers.sun.com/events/techdays/presentations/locations-2008/india_techdays/solaris_track/td_hyd_sun_studio_tatkar.pdf. Accessed 8 April 2010
  18. 18.
    Ye D, Ray J, Harle C, Kaeli D (2006) Performance characterization of SPEC CPU2006 integer benchmarks on x86-64 architecture. In: IEEE international symposium on workload characterization, 2006, pp 120–127 Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2010

Authors and Affiliations

  • Sergio Aldea
    • 1
  • Diego R. Llanos
    • 1
  • Arturo González-Escribano
    • 1
  1. 1.Dpto. InformáticaUniv. ValladolidValladolidSpain

Personalised recommendations