MinneSPEC: A New SPEC Benchmark
Workload for Simulation-Based
Computer Architecture Research
The MinneSPEC inputs can be
obtained by sending your SPEC license number to the SPEC webmaster
at info@spec.org. Upon confirming your license, the webmaster will send
you a userid and password, which you can use to access the SPEC ftp site
to obtain the MinneSPEC files.
Computer architects must determine how to most effectively use
finite computational resources when running simulations
to evaluate new architectural ideas.
To facilitate efficient simulations
with a range of benchmark programs,
we have developed the MinneSPEC input set
for the SPEC CPU 2000 benchmark suite.
This new workload allows
computer architects
to obtain simulation results in a reasonable time
using existing simulators.
While the MinneSPEC workload is derived
from the standard SPEC CPU 2000 workload,
it is a valid benchmark
suite in and of itself for
simulation-based research.
MinneSPEC also may be
used to run large numbers of simulations
to find ``sweet spots'' in the evaluation parameter space.
This small number of
promising design points
subsequently may be
investigated in more detail
with the full SPEC reference workload.
In the process of developing the MinneSPEC
datasets, we quantify
its differences in terms of
function-level execution patterns,
instruction mixes, and memory behaviors
compared to the SPEC programs
when executed with the reference inputs.
We find that
for some programs, the MinneSPEC profiles match the SPEC
reference dataset program behavior very closely. For other
programs, however, the
MinneSPEC inputs produce significantly different program
behavior.
The MinneSPEC workload
has been recognized
by SPEC and is distributed
with Version 1.2 and higher of the
SPEC CPU 2000
benchmark suite.
These benchmarks are maintained by AJ KleinOsowski in the Department of
Electrical and Computer Engineering at the University of Minnesota.
Comments, questions, or problems can be directed to
ajko@ece.umn.edu
For more information, see:
``
MinneSPEC: A New SPEC Benchmark Workload for
Simulation-Based Computer Architecture Research
'',
AJ KleinOsowski and David J. Lilja,
Computer Architecture Letters,
Volume 1,
June, 2002.
(A small correction, stated below, for this paper is available as:
Laboratory for Advanced Research in Computing
Technology and Compilers Technical Report No. ARCTiC 02-08,
October, 2002.)
If you use the MinneSPEC inputs in your work,
please cite the Computer Architecture Letters paper in
any resulting publications.
Thank you.
The April, 2003, release includes reduced input sets and
profiles for all of the benchmarks in the SPEC CPU 2000
suite.
Profiles are available
on this site
by clicking on the links for each benchmark. Input sets can be
obtained by sending a request to
info@spec.org.
In your request, be sure to include your SPEC CPU 2000
license number.
Note #1: The reference run of several SPEC CPU 2000 benchmarks
use multiple runs of the same executable
with different command line arguments or different input files.
For these programs, we treat each of these
different command lines as separate sub-benchmarks.
Note #2: The profiles below were generated by compiling
the benchmarks for either the SimpleScalar PISA instruction set
or the Alpha instruction set, then
collecting profile information with the SimpleScalar suite of
simulators. The Simulation ISA column in the table below
documents which instruction set architecture was used for our
profiles.
Note #3: The Input Reduction column documents what methods were used to
develop the MinneSPEC workloads.
The FuncProf and InstMix columns document where the
MinneSPEC function-level execution profile and the instruction mix are
nearly identical to the SPEC reference workload,
as determined by the chi-squared test at O3 optimization for pisa ISA
profiles, or O4 optimization for alpha ISA profiles.
For programs with
[YN], the MinneSPEC profile is compared to the multiple SPEC reference
workloads for that program. The Run Length column documents the
dynamic instruction count of each program, at O0 optimization.
Erratum to "MinneSPEC: A New SPEC
Benchmark Workload for Simulation-Based Computer Architecture
Research," AJ KleinOsowski and David J. Lilja,
Computer Architecture Letters, Volume 1, June 2002.
It was recently discovered that, in the sim-profile simulator from
the SimpleScalar version 3.0 suite, the counters used to track
instruction mixes are explicitly declared to be unsigned ints, which
makes them only 32 bits. As a result, when simulating programs with
more than 4 billion (2**32) instructions, these counters overflow
and produce incorrect outputs.
Since the benchmark programs in the SPEC CPU 2000 benchmark suite
execute more than 4 billion (2**32) instructions with SPEC's
standard reference input sets, simulating these programs with
sim-profile causes the counts of the different types of instructions
executed to overflow. Consequently, the instruction mix profile
results discussed in this paper and posted on this web site
prior to July, 2002, may contain errors.
The authors have modified sim-profile to allow it to correctly
profile programs with more than 4 billion instructions and have
collected corrected instruction mix profiles. These
updated profiles are available below.
This error with the instruction mix profiles does not affect the use
or distribution of the MinneSPEC workload. Only the instruction mix
profiles, which are provided primarily as a convenience to MinneSPEC
users, are affected. The other reported results, including the
cache behavior gathered with sim-cache and the execution time
profiles gathered with gprof, are unaffected by this counter
overflow problem.
Furthermore, please note that the limited size of the counters in
the standard distribution of sim-profile will affect all sim-profile
users who run simulations longer than 4 billion instructions. If
you fall into this category, we suggest that you
download the stats.h and stats.c
files. These two files can directly replace the existing stats.h and
stats.c files in the SimpleScalar version 3.0 distribution.
These files change the 32 bit sim-profile counters to 64 bit counters.
Many thanks to Hans Vandierendonck and Lieven Eeckhout of Ghent
University for their detailed evaluation of the MinneSPEC workload
which led to the discovery of these errors.
| Benchmark |
Type |
Description |
Input Reduction Method |
FuncProf |
InstMix |
Run Length |
Profiles |
Simulation ISA |
| 164.gzip,graphic |
Int |
Compression |
modified command line, new input file |
Y |
Y |
2.5 billion |
ref.graphic |
pisa |
| 164.gzip,log |
Int |
Compression |
modified command line, truncated ref file |
N |
Y |
1.0 billion |
ref.log |
pisa |
| 164.gzip,program |
Int |
Compression |
modified command line, new input file |
Y |
Y |
4.4 billion |
ref.program |
pisa |
| 164.gzip,random |
Int |
Compression |
modified command line, modified ref file |
Y |
Y |
1.9 billion |
ref.random |
pisa |
| 164.gzip,source |
Int |
Compression |
modified command line, truncated ref file |
N |
Y |
2.4 billion |
ref.source |
pisa |
| 168.wupwise |
Float |
Quantum Chromodynamics |
modified test file |
N |
Y |
14.6 billion |
ref |
alpha |
| 171.swim |
Float |
Shallow Water Modeling |
same as test |
N |
Y |
2.8 billion |
ref |
alpha |
| 172.mgrid |
Float |
Muli-grid Solver |
modified test file |
N |
Y |
993 million |
ref |
alpha |
| 173.applu |
Float |
Partial Differential Equations |
modified test file |
Y |
Y |
952 million |
ref |
alpha |
| 175.vpr,place |
Int |
FPGA Place Route |
same as test |
Y |
Y |
3.3 billion |
ref.place |
pisa |
| 175.vpr,route |
Int |
FPGA Place Route |
same as test |
Y |
Y |
2.0 billion |
ref.route |
pisa |
| 176.gcc |
Int |
C Compiler |
same as train |
5Y |
5Y |
6.4 billion |
ref.166 ,
ref.200 ,
ref.expr ,
ref.integrate ,
ref.scilab |
pisa |
| 177.mesa |
Float |
3-D Graphics |
modified command line, modified test file |
N |
N |
1.3 billion |
ref |
pisa |
| 178.galgel |
Float |
Computational Fluid Dynamics |
modified test file |
Y |
N |
1.5 billion |
ref |
alpha |
| 179.art |
Float |
Image Recognition |
modified command line |
2N |
2Y |
7.7 billion |
ref.110 ,
ref.470 |
pisa |
| 181.mcf |
Float |
Combinatorial Optimization |
modified ref file |
N |
Y |
1.7 billion |
ref |
pisa |
| 183.equake |
Float |
Seismic Wave Propagation |
modified train file |
N |
N |
1.7 billion |
ref |
pisa |
| 186.crafty |
Int |
Chess Playing |
modified test file |
Y |
Y |
1.2 billion |
ref |
alpha |
| 187.facerec |
Float |
Image Processing |
modified test file |
N |
N |
859 million |
ref |
alpha |
| 188.ammp |
Float |
Chemistry |
truncated ref file |
N |
N |
2.2 billion |
ref |
pisa |
| 189.lucas |
Float |
Number Theory |
modified test file |
N |
Y |
1.4 billion |
ref |
alpha |
| 191.fma3d |
Float |
Crash Simulation |
modified test file |
N |
N |
774 million |
ref |
alpha |
| 197.parser |
Int |
Word Processing |
sampled ref file |
N |
Y |
5.6 billion |
ref |
pisa |
| 200.sixtrack |
Float |
Nuclear Physics Accelerator Design |
modified test file |
N |
N |
4.1 billion |
ref |
alpha |
| 252.eon,cook |
Int |
Computer Visualization |
modified test file |
Y |
Y |
2.9 billion |
ref |
alpha |
| 252.eon,kajiya |
Int |
Computer Visualization |
modified test file |
Y |
Y |
3.6 billion |
ref |
alpha |
| 252.eon,rushmeier |
Int |
Computer Visualization |
modified test file |
Y |
Y |
3.1 billion |
ref |
alpha |
| 253.perlbmk |
Int |
PERL Language |
same as ref.makerand |
1Y5N |
2Y4N |
1.8 billion |
ref.535 ,
ref.704 ,
ref.850 ,
ref.957 ,
ref.makerand ,
ref.perfect |
pisa |
| 254.gap |
Int |
Group Theory |
modified test file |
Y |
Y |
1.0 billion |
ref |
alpha |
| 255.vortex |
Int |
Database |
modified train file |
3Y |
3Y |
1.5 billion |
ref.1 ,
ref.2 ,
ref.3 |
pisa |
| 256.bzip2,graphic |
Int |
Compression |
modified command line, modified ref file |
Y |
Y |
6.4 billion |
ref.graphic |
pisa |
| 256.bzip2,program |
Int |
Compression |
modified command line, new input file |
Y |
Y |
5.0 billion |
ref.program |
pisa |
| 256.bzip2,source |
Int |
Compression |
modified command line, sampled ref file |
Y |
Y |
3.8 billion |
ref.source |
pisa |
| 300.twolf |
Int |
Place and Route |
sampled train file |
Y |
Y |
1.5 billion |
ref |
pisa |
| 301.apsi |
Float |
Pollutant Distribution |
modified test file |
Y |
N |
1.8 billion |
ref |
alpha |
For information about this project from SPEC, visit
http://www.spec.org/cpu2000/research/umn.
This work was supported in part by the National Science Foundation
under grants EIA-9971666 and CCR-9900605,
by the
Minnesota Supercomputing Institute,
and by Compaq's Alpha Development Group.
Return
to: U of M Home
