link to homepage

Institute for Advanced Simulation (IAS)

Navigation and service


Detailed Configuration of Benchmarking Steps

Defining a platform

The JuBE system helps bringing a set of applications to a new platform with the introduction of several levels of configuration options. The highest of these levels is the platform itself. Here, all library paths and configuration switches for all applications in the benchmark a given for a specific platform. This also means, that all configuration parameters have to be held as general as possible to allow different applications to use application specific library and compilation parameters. The next listing shows an example platform definition.

Listing of the platform.xml containing one platform:


<platform name="example-system">


make = "gmake"

rm = "rm -f"

ar = "ar"

arflags = "-rs"

ranlib = "/usr/bin/ranlib"

cpp = "/usr/lib/cpp"

cppflags = "-P"

f77 = "xlf_r"

f77flags = "-qtune=pwr4 -qarch=pwr4"

f90 = "xlf90_r"

f90flags = "-qtune=pwr4 -qarch=pwr4"

cc = "xlc_r"

cflags = "-qtune=pwr4 -qarch=pwr4"

cxx = "xlC_r"

cxxflags = "-qtune=pwr4 -qarch=pwr4"

mpi_f90 = "mpxlf90_r"

mpi_f77 = "mpxlf_r"

mpi_cc = "mpcc_r"

mpi_cxx = "mpCC_r"

ldflags = "-qtune=pwr4 -qarch=pwr4"

mpi_dir = ""

mpi_lib = ""

mpi_inc = ""

mpi_bin = ""

blas_dir = ""

blas_lib = "-lessl"

lapack_dir = "-L/usr/local/lapack/LOCALlib"

lapack_lib = "-llapack -lessl"

fftw3_dir = "-L/usr/local/fftw/LOCALlib"

fftw3_lib = "-lfftw3 -lfftw3_threads -lm"

fftw3_inc = "-I/usr/local/fftw/LOCALinclude"

fftw2_dir = "-L/usr/local/fftw/LOCALlib"

fftw2_lib = "-ldfftw -ldrfftw -ldfftw_threads -ldrfftw_threads -ldfftw_mpi -ldrfftw_mpi -lm"

fftw2_inc = "-I/usr/local/fftw/LOCALinclude"

netcdf3_dir = "-L/usr/local/netcdf/LOCALlib"

netcdf3_lib = "-lnetcdf"

netcdf3_inc = "-I/usr/local/netcdf/LOCALinclude"

module_cmd = "module load"




The platform tag defines a new platform. Its name is used in different configuration steps later on. For each platform defined, a subdirectory to platform/ should hold any platform specific files, such as templates to batch queue submission scripts, etc..

The name of the platform will be used throughout the benchmark run for configuration purposes, thus a meaningful name can ease later maintenance of the benchmark suite.

Top-Level configuration of a benchmark

The top level configuration file defines the benchmark runs for a specific benchmark program. The outermost element <bench> has only two parameters, the benchmark name (here IMB) and the platform on which the benchmark will be performed. Both values are used for building the unique identifier, and the platform name has to be one of the platforms defined in platform.xml. The bench-element can contain one or more <benchmark> elements describing a set of benchmark runs. This element has also an attribute for specifying a unique name. The second attribute 'active' can be used to manage different benchmark run sets. Only benchmark elements which are active will be processed by JuBE.

Listing of an example top-level configuration:

<!-- JuBE benchmark config schema for: IMB -->

<bench name = "IMB" platform= "example-system" >

<benchmark name="scaling_1" active="1">

<!-- version="reuse|new" -->

<compile cname="$platform" version="new" />




nodes="1" />






mapy="$taskspernode" />

<prepare cname="bsp" />

<execution iteration="1" cname="$platform" />

<verify cname="imb" />

<analyse cname="$platform" />



The compile tag describes the compile steps. The cname attribute is a reference to an entry in compile.xml, one of the additional xml-files. The attribute version defines if the executable should be generated (new) or if an executable from a former benchmark run with the same configuration can be used (reuse).

The tasks tag is the definition of the number of cpus used for the benchmark run. Threadspertask is the number of OpenMP-Threads, taskspernode the number of MPI task per node and nodes the number of nodes. The total number of used processors is the product of these three values. In the first example, the attribute taskpernode contain a comma-seperated list of values. The benchmark run will be performed for each of this values.

The params tag defines the input parameters which should be used in the benchmark runs. The attributes of this element depends on the benchmark program and will be used in the preparation step for building the program input file. Each of this attribute van contain a comma separated list of value. JuBE will automatically generate benchmark runs for all possible combinations of these parameters and the tasks ranges given in the tasks element.

The prepare tag describes the preparation step. The corresponding element can be found in the file prepare.xml.

The execution tag contains a reference (cname) to a element in execute.xml. This file should contain for each batch system one entry which can be referenced here. The attribute iteration defines the number of repetitions of a benchmark run. This can be used for measuring random runtime variations between different runs.

The postprocess tag is not used in this example. This step could be used for collecting data, e.g. gathering data from output files of each task.

The analyse tag contains a reference (cname) to a element in analyse.xml. In the analyse step the output file of the benchmark run will be scanned in respect to the patterns defined in analyse.xml.

The compile step

This file defines all relevant parameters needed for the configuration and compilation of the source code. This file can contain, like the other second-level XML files, one or more different elements. The attribute cname must be unique in the file and defines the name under which it can be selected in the top-level XML file.

Example compile step definition:


<!-- predefined: $outdir->output for temp. compile files, $id -> id of this benchmark run -->

<compile cname="example-system">

<params fflags="" cflags = "-O3 -q64 -qlist -qsource -g -qfullpath -DnoCHECK" lflags="-O3 -q64 -qlist -qsource -qlist -qlistopt -g -qfullpath" fc="mpxlf90_r" cc="mpcc" />

<src directory="./src" files="*.c *.h Makefile makeb.temp" />

<substitute infile="makeb.temp" outfile="makeb">

<sub from="#FC#" to="$fc" />

<sub from="#CC#" to="$cc" />

<sub from="#LD#" to="$cc" />

<sub from="#CFLAGS#" to="$cflags" />

<sub from="#LFLAGS#" to="$lflags" />

<sub from="#OPTFLAGS#" to="" />

<sub from="#OUTDIR#" to="$outdir" />

<sub from="#EXECNAME#" to="$execname" />

<sub from="#MPIINCLUDE#" to="/usr/lpp/include" />


<!-- issue build command -->

<command>(make -f Makefile IMB-MPI1; mv IMB-MPI1 ../$id.exe)</command>



The params tag defines parameter for the compilation. Typical parameters are the compiler flags, the compiler name and loader name. The values specified in this element will also be part of the XML file containing the results of a benchmark program.

The src tag describes the source needed for the compilation. All files specified by the two attributes directory and files are copied to the temporary working directory of the benchmark run. The src element can specified more than once for copying files from different directories.

The substitute tag describes the replacement of the parameters in a template file. Each substitute can handle the replacement in one file. This element can also be iterated. The template has also to be copied by the src element. The placeholders are defined in the from attribute of the sub element. It will be replaced by the value of the to attribute. Possible variable names which can be used here are the attribute names defines in the params element or the attributes of the elements in the top-level XML file. There are also some pre-defined variables like $execname for the path and name of the executable.

The command tag defines the command which should be used to start the compilation. Typically this will be a call of make or gmake. It is also possible to start a shell script or to run configure in this place.

The prepare step

The prepare control file defines how the input file of benchmark should be generated. In principal this is defined by a substitution of placeholders in a template file.

Example prepare step definition (not from IMB):


<prepare cname="bsp">

<mkdir directory="data" />

<mkdir directory="dumps" />

<mkdir directory="log" />

<input files="inp/ inp/" />

<substitute infile="" outfile="run_bench.h">

<sub from="#NPART#" to="$npart" />

<sub from="#WSCHEME#" to="$wscheme" />

<sub from="#NT#" to="$nt" />



( cd $rundir;

perl $benchhome/run/ $ncpus;







The input tag describes the location of template files.

The substitute tag defines the replacements, further description see compile step.

The command tag specifies a optional command which should executed after the replacements. In the example above this is a script which creates temporary subdirectories for each processor.

The execution step

This file contains the description how a the parallel program will be executed. Typically this is done by a batch job which will be submitted to the local batch system. It can also be a interactive run of the benchmark program. In this case, the program will be executed directly from the script. Otherwise the job will be submitted and the script returns directly.

Example execution step definition:


<execute cname="example-system">

<input files="../../platform/example-system/"/>

<substitute infile="" outfile="submit.job">

<sub from="#OUTDIR#" to="$outdir"/>

<sub from="#LOGDIR#" to="$logdir"/>

<sub from="#STDOUTLOGFILE#" to="$stdoutlogfile"/>

<sub from="#STDERRLOGFILE#" to="$stderrlogfile"/>

<sub from="#ID#" to="$id" />

<sub from="#SUBID#" to="$subid" />

<sub from="#CLASS#" to="bench" />

<sub from="#BENCHNAME#" to="$benchname" />

<sub from="#NODEUSAGE#" to="not_shared" />

<sub from="#TIME_LIMIT#" to="00:30:00" />

<sub from="#NODES#" to="$nodes" />

<sub from="#TASKSPERNODE#" to="$taskspernode"/>

<sub from="#NOTIFICATION#" to="never" />

<sub from="#NOTIFY_EMAIL#" to="email" />

<sub from="#THREADSPERTASK#" to="$threadspertask"/>

<sub from="#DATA_LIMIT#" to="3.0Gb" />

<sub from="#STACK_LIMIT#" to="0.5GB" />

<sub from="#MEMORYPERTASK#" to="`3.5*$threadspertask`Gb"/>

<sub from="#EXECUTABLE#" to="$executable"/>

<sub from="#ENV#" to="$env" />

<sub from="#MEASUREMENT#" to="time hpmcount"/>

<sub from="#ARGS_EXECUTABLE#" to="" />

<sub from="#TYPE#" to="$type" />

<sub from="#MULTI#" to="`($multi?'-multi 0':'')`" />

<sub from="#NPMIN#" to="-npmin $npmin" />

<sub from="#MAPX#" to="$mapx" />

<sub from="#MAPY#" to="$mapy" />



<env var="MP_LABELIO" value="yes" />

<env var="MP_INFOLEVEL" value="2" />

<env var="MP_SHARED_MEMORY" value="yes" />

<env var="MP_TASK_AFFINITY" value="MCM" />

<env var="MEMORY_AFFINITY" value="MCM" />

<env var="OMP_NUM_THREADS" value="$threadspertask"



<command>llsubmit submit.job</command>



The input tag describes the input file for the batch job or an interactive started script.

The substitute tag defines the replacements, further description see compile step.

The environment tag defines additional environment variables, the variable $env will contain this.

The command tag specifies the which should be used to submit the job or to start the program interactively.

The verification step

Example verification step definition:


<!-- predefined vars:

$subdir -> execution dir of benchmark run

$stdoutfile -> $stdout file of bnechmark run

$stderrfile -> $stderr file of bnechmark run

+ params of benchmark specification in toplevel dir


<verify cname="IMB">

<command>run/ $subdir/verify.xml $stdoutfile $stderrfile $subdir</command>



The analysis step

The configuration file for the analysis step contains search pattern for scanning the output files (stdout, stderr) of the benchmark run. JuBE accept regular expression as defined in Perl. It is also possible to define derived results varables as expressions of other variables of the analysis step.

Example analysis step definition:


<!-- Input is stdout and stderr of benchmark run -->

<!-- Standard result parameter: walltime -->

<analyse cname="example-system">

<includepattern file="./analyse-pattern-imb.xml" />

<includepattern file="../../skel/hpm3patterns.xml"/>


<!-- for old runs of jube with cname IMB -->

<analyse cname="imb">

<includepattern file="./analyse-pattern-imb.xml" />

<includepattern file="../../skel/hpm3patterns.xml"/>



The name tag identifier for the varible, which will be used as a unique identifier in the result XML file

The unit tag String describing the unit of the measured value, will be used in output tables and also stored in the result XML file.

The mode tag can contain one of the following keywords: line (scan line-by-line), line,add (scan line-by-line and add values if they occurs more than once), derived (no scan, compute value as expression of other variables)

Generating result tables

There is one addition configuration file for describing how results tables printed by JuBE should be build.

Example result step definition:









<!-- do not show


<!-- all HPMFMPperc,HPMflopsWCTtot,HPMflopstot,HPMressize,HPMstime,HPMtasks,HPMtflips,twalkser, HPMutime,HPMwtime,analyse_cname,commtime,compile_cname,execute_cname,identifier, iteration,itertime,n,name,ncpus,nel,nfactor,nion,nt,postp_cname,prepare_cname,twalkcomm, subdir,subid,taskspernode,tbuild,tdiag,tdomain,template,tforces,tprefetch,tpusher,ttotal


The name tag identifier for the varible, which will be used as a unique identifier in the result XML file

The show tag defines a list of variables for the columns of the output table. Possible variable names are the identifier specified in the analysis step and the params variables of the top-level XML file

The sort tag defines sort order by a list of variables, each variable can have a flag (-,+) indicating whether to sort in descending or ascending order.