Using SPEC CPU2006 to evaluate the sequential and parallel code generated by commercial and open-source compilers
The role of the compiler is fundamental to exploit the hardware capabilities of a system running a particular application, minimizing the sequential execution time and, in some cases, offering the possibility of parallelizing part of the code automatically. This paper relies on the SPEC CPU2006 v1.1 benchmark suite to evaluate the performance of the code generated by three widely-used compilers (Intel C++/Fortran Compiler 11.0, Sun Studio 12 and GCC 4.3.2). Performance is measure in terms of base speed for reference problem sizes. Both sequential and automatic parallel performance obtained is analyzed, using different hardware architectures and configurations. The study includes a detailed description of the different problems that arise while compiling SPEC CPU2006 benchmarks with these tools, an information difficult to obtain elsewhere.
Having in mind that performance is a moving target in the field of compilers, our evaluation shows that the sequential code generated by both Sun and Intel compilers for the SPEC CPU2006 integer benchmarks present a similar performance, while the floating-point code generated by Intel compiler is faster than its competitors. With respect to the auto-parallelization options offered by Intel and Sun compilers, our study shows that their benefits only apply to some floating-point benchmarks, with an average speedup of 1.2× with four processors. Meanwhile, the GCC suite evaluated is not capable of compiling the SPEC CPU2006 benchmark with auto-parallelization options enabled.
KeywordsCompiler performance Automatic parallelization Benchmarking
Unable to display preview. Download preview PDF.
- 1.IBM posts SPEC CPU2006 scores for next-generation System x scalable server. ftp://ftp.software.ibm.com/eserver/benchmarks/news/newsblurb_x3850M2_speccpu_090507.pdf. IBM Corporation. Accessed 8 April 2010
- 2.Intel Xeon Processor performance summary. http://www.intel.com/performance/work-station/xeon/summary.htm. Intel Corporation. Accessed 8 April 2010
- 3.Performance and benchmarks: Performance reports for CELSIUS R640. http://www.fujitsu-siemens.com/products/deskbound/workstations/performance/celsius_r640.html. Fujitsu Siemens. Accessed 8 April 2010
- 4.Quick reference guide to optimization with Intel compilers version 11 Intel Software development products. http://www.intel.com/software/products/compilers/docs/qr_guide.htm. Accessed 20 January 2010
- 5.SPEC releases CPU2006 benchmarks. http://www.spec.org/cpu2006/press/release.html. Standard Performance Evaluation Corporation. Accessed 8 April 2010
- 6.Sun Fire X4450 server performance. http://www.sun.com/servers/x64/x4450/benchmarks.jsp?display=1. Sun Microsystems. Accessed 8 April 2010
- 7.Sun Studio 12: C user’s guide. http://dlc.sun.com/pdf/819-5265/819-5265.pdf. Sun Microsystems. Accessed 8 April 2010
- 8.Sun Studio 12: Fortran user’s guide. http://dlc.sun.com/pdf/819-5263/819-5263.pdf. Sun Microsystems. Accessed 8 April 2010
- 9.Using the gnu compiler collection. http://gcc.gnu.org/onlinedocs/gcc-4.3.2/gcc.pdf. Accessed 8 April 2010
- 11.Kejariwal A, Veidenbaum AV, Nicolau A, Tian X, Girkar M, Saito H, Banerjee U (2008) Comparative architectural characterization of SPEC CPU2000 and CPU2006 benchmarks on the Intel Core 2 Duo processor. In: International conference on embedded computer systems: architectures, modeling, and simulation, 2008, SAMOS 2008, pp 132–141 Google Scholar
- 12.Li S, Qiao L, Tang Z, Cheng B, Gao X (2009) Performance characterization of SPEC CPU2006 benchmarks on Intel and AMD platform. In: First international workshop on education technology and computer science, 2009. ETCS ’09, vol 2, pp 116–121 Google Scholar
- 13.Munafo R Reference machine times for SPEC CPU2006. http://www.mrob.com/pub/comp/benchmarks/spec.html. Accessed 8 April 2010
- 14.Polsson K (2010) Chronology of workstation computers. http://www.islandnet.com/~kpolsson/workstat/work1987.htm. Accessed 3 June 2010
- 15.Schouten D, Tian X, Bik A, Girkar M (2003) Inside the Intel compiler. Linux J 2003(106):4 Google Scholar
- 17.Tatkar V Build high performance applications on multicore systems using sun studio compilers and tools. In: Sun TechDays India 2008, 27th–29th Feb 2008, Hyderabad, India. http://developers.sun.com/events/techdays/presentations/locations-2008/india_techdays/solaris_track/td_hyd_sun_studio_tatkar.pdf. Accessed 8 April 2010
- 18.Ye D, Ray J, Harle C, Kaeli D (2006) Performance characterization of SPEC CPU2006 integer benchmarks on x86-64 architecture. In: IEEE international symposium on workload characterization, 2006, pp 120–127 Google Scholar