MinneSPEC: A New SPEC Benchmark Workload for Simulation-Based Computer Architecture Research

The MinneSPEC inputs can be obtained by sending your SPEC license number to the SPEC webmaster at info@spec.org. Upon confirming your license, the webmaster will send you a userid and password, which you can use to access the SPEC ftp site to obtain the MinneSPEC files.

Computer architects must determine how to most effectively use finite computational resources when running simulations to evaluate new architectural ideas. To facilitate efficient simulations with a range of benchmark programs, we have developed the MinneSPEC input set for the SPEC CPU 2000 benchmark suite. This new workload allows computer architects to obtain simulation results in a reasonable time using existing simulators. While the MinneSPEC workload is derived from the standard SPEC CPU 2000 workload, it is a valid benchmark suite in and of itself for simulation-based research. MinneSPEC also may be used to run large numbers of simulations to find ``sweet spots'' in the evaluation parameter space. This small number of promising design points subsequently may be investigated in more detail with the full SPEC reference workload. In the process of developing the MinneSPEC datasets, we quantify its differences in terms of function-level execution patterns, instruction mixes, and memory behaviors compared to the SPEC programs when executed with the reference inputs. We find that for some programs, the MinneSPEC profiles match the SPEC reference dataset program behavior very closely. For other programs, however, the MinneSPEC inputs produce significantly different program behavior. The MinneSPEC workload has been recognized by SPEC and is distributed with Version 1.2 and higher of the SPEC CPU 2000 benchmark suite.

These benchmarks are maintained by AJ KleinOsowski in the Department of Electrical and Computer Engineering at the University of Minnesota. Comments, questions, or problems can be directed to ajko@ece.umn.edu

For more information, see: "MinneSPEC: A New SPEC Benchmark Workload for Simulation-Based Computer Architecture Research '', AJ KleinOsowski and David J. Lilja, Computer Architecture Letters, Volume 1, June, 2002. (A small correction, stated below, for this paper is available as: Laboratory for Advanced Research in Computing Technology and Compilers Technical Report No. ARCTiC 02-08, October, 2002). If you use the MinneSPEC inputs in your work, please cite the Computer Architecture Letters paper in any resulting publications. Thank you.

The April, 2003, release includes reduced input sets and profiles for all of the benchmarks in the SPEC CPU 2000 suite. Profiles are available on this site by clicking on the links for each benchmark. Input sets can be obtained by sending a request to info@spec.org. In your request, be sure to include your SPEC CPU 2000 license number.

Note #1: The reference run of several SPEC CPU 2000 benchmarks use multiple runs of the same executable with different command line arguments or different input files. For these programs, we treat each of these different command lines as separate sub-benchmarks.

Note #2: The profiles below were generated by compiling the benchmarks for either the SimpleScalar PISA instruction set or the Alpha instruction set, then collecting profile information with the SimpleScalar suite of simulators. The Simulation ISA column in the table below documents which instruction set architecture was used for our profiles.

Note #3: The Input Reduction column documents what methods were used to develop the MinneSPEC workloads. The FuncProf and InstMix columns document where the MinneSPEC function-level execution profile and the instruction mix are nearly identical to the SPEC reference workload, as determined by the chi-squared test at O3 optimization for pisa ISA profiles, or O4 optimization for alpha ISA profiles. For programs with [YN], the MinneSPEC profile is compared to the multiple SPEC reference workloads for that program. The Run Length column documents the dynamic instruction count of each program, at O0 optimization.

 

Erratum to "MinneSPEC: A New SPEC Benchmark Workload for Simulation-Based Computer Architecture Research," AJ KleinOsowski and David J. Lilja, Computer Architecture Letters, Volume 1, June 2002.

It was recently discovered that, in the sim-profile simulator from the SimpleScalar version 3.0 suite, the counters used to track instruction mixes are explicitly declared to be unsigned ints, which makes them only 32 bits. As a result, when simulating programs with more than 4 billion (2**32) instructions, these counters overflow and produce incorrect outputs.

Since the benchmark programs in the SPEC CPU 2000 benchmark suite execute more than 4 billion (2**32) instructions with SPEC's standard reference input sets, simulating these programs with sim-profile causes the counts of the different types of instructions executed to overflow. Consequently, the instruction mix profile results discussed in this paper and posted on this web site prior to July, 2002, may contain errors.

The authors have modified sim-profile to allow it to correctly profile programs with more than 4 billion instructions and have collected corrected instruction mix profiles. These updated profiles are available below.

This error with the instruction mix profiles does not affect the use or distribution of the MinneSPEC workload. Only the instruction mix profiles, which are provided primarily as a convenience to MinneSPEC users, are affected. The other reported results, including the cache behavior gathered with sim-cache and the execution time profiles gathered with gprof, are unaffected by this counter overflow problem.

Furthermore, please note that the limited size of the counters in the standard distribution of sim-profile will affect all sim-profile users who run simulations longer than 4 billion instructions. If you fall into this category, we suggest that you download the stats.h and stats.c files. These two files can directly replace the existing stats.h and stats.c files in the SimpleScalar version 3.0 distribution. These files change the 32 bit sim-profile counters to 64 bit counters.

Many thanks to Hans Vandierendonck and Lieven Eeckhout of Ghent University for their detailed evaluation of the MinneSPEC workload which led to the discovery of these errors.

 

 

Benchmark Type Description Input Reduction Method FuncProf InstMix Run Length Profiles Simulation ISA
164.gzip,graphic Int Compression modified command line, new input file Y Y 2.5 billion ref.graphic pisa
164.gzip,log Int Compression modified command line, truncated ref file N Y 1.0 billion ref.log pisa
164.gzip,program Int Compression modified command line, new input file Y Y 4.4 billion ref.program pisa
164.gzip,random Int Compression modified command line, modified ref file Y Y 1.9 billion ref.random pisa
164.gzip,source Int Compression modified command line, truncated ref file N Y 2.4 billion ref.source pisa
168.wupwise Float Quantum Chromodynamics modified test file N Y 14.6 billion ref alpha
171.swim Float Shallow Water Modeling same as test N Y 2.8 billion ref alpha
172.mgrid Float Muli-grid Solver modified test file N Y 993 million ref alpha
173.applu Float Partial Differential Equations modified test file Y Y 952 million ref alpha
175.vpr,place Int FPGA Place Route same as test Y Y 3.3 billion ref.place pisa
175.vpr,route Int FPGA Place Route same as test Y Y 2.0 billion ref.route pisa
176.gcc Int C Compiler same as train 5Y 5Y 6.4 billion ref.166 , ref.200 , ref.expr , ref.integrate , ref.scilab pisa
177.mesa Float 3-D Graphics modified command line, modified test file N N 1.3 billion ref pisa
178.galgel Float Computational Fluid Dynamics modified test file Y N 1.5 billion ref alpha
179.art Float Image Recognition modified command line 2N 2Y 7.7 billion ref.110 , ref.470 pisa
181.mcf Float Combinatorial Optimization modified ref file N Y 1.7 billion ref pisa
183.equake Float Seismic Wave Propagation modified train file N N 1.7 billion ref pisa
186.crafty Int Chess Playing modified test file Y Y 1.2 billion ref alpha
187.facerec Float Image Processing modified test file N N 859 million ref alpha
188.ammp Float Chemistry truncated ref file N N 2.2 billion ref pisa
189.lucas Float Number Theory modified test file N Y 1.4 billion ref alpha
191.fma3d Float Crash Simulation modified test file N N 774 million ref alpha
197.parser Int Word Processing sampled ref file N Y 5.6 billion ref pisa
200.sixtrack Float Nuclear Physics Accelerator Design modified test file N N 4.1 billion ref alpha
252.eon,cook Int Computer Visualization modified test file Y Y 2.9 billion ref alpha
252.eon,kajiya Int Computer Visualization modified test file Y Y 3.6 billion ref alpha
252.eon,rushmeier Int Computer Visualization modified test file Y Y 3.1 billion ref alpha
253.perlbmk Int PERL Language same as ref.makerand 1Y5N 2Y4N 1.8 billion ref.535 , ref.704 , ref.850 , ref.957 , ref.makerand , ref.perfect pisa
254.gap Int Group Theory modified test file Y Y 1.0 billion ref alpha
255.vortex Int Database modified train file 3Y 3Y 1.5 billion ref.1 , ref.2 , ref.3 pisa
256.bzip2,graphic Int Compression modified command line, modified ref file Y Y 6.4 billion ref.graphic pisa
256.bzip2,program Int Compression modified command line, new input file Y Y 5.0 billion ref.program pisa
256.bzip2,source Int Compression modified command line, sampled ref file Y Y 3.8 billion ref.source pisa
300.twolf Int Place and Route sampled train file Y Y 1.5 billion ref pisa
301.apsi Float Pollutant Distribution modified test file Y N 1.8 billion ref alpha

For information about this project from SPEC, visit http://www.spec.org/cpu2000/research/umn.

This work was supported in part by the National Science Foundation under grants EIA-9971666 and CCR-9900605, by the Minnesota Supercomputing Institute, and by Compaq's Alpha Development Group.