link to homepage

Institute for Advanced Simulation (IAS)

Navigation and service

Usage of PETSc on JUWELS

PETSc version 3.11.1 has been installed on JUWELS in a several versions. All versions use mkl and include several other libraries.
Those libraries are static libraries and can be found in $PETSC_DIR/lib.
There are versions with Intel compiler and ParaStationMPI and IntelMPI and with GCC and ParaStationMPI.

PETSc/3.11.1 is configured with standard double precision and integer values
PETSc/3.11.1_debug is configured with --with-debugging=1
PETSc/3.11.1_int8 is configured with --with-64-bit-indices=1
PETSc/3.11.1_int8_debug is configured with --with-64-bit-indices=1 and --with-debugging=1
PETSc/3.11.1_complex is configured with --with-scalar-type=complex
PETSc/3.11.1_complex_debug is configured with --with-scalar-type=complex and --with-debugging=1

The _debug versions are only available in Stages/Devel-2019a

Preparations necessary to use PETSc on JUWELS

First you have to load the PETSc version you want, for example:

module load intel-para [Intel InelMPI] [GCC ParaStationMPI]
module load PETSc/3.11.1
With that command the variables PETSC_DIR and PETSC_ARCH are set.

If you don't want to use the PETSc makefiles make sure that your makefile contains the statements

include ${PETSC_DIR}/${PETSC_ARCH}/lib/petsc/conf/petscvariables
include ${PETSC_DIR}/${PETSC_ARCH}/lib/petsc/conf/petscrules


Examples are available in $PETSC_DIR/src.

To run for instance ex1 from $PETSC_DIR/src/vec/vec/examples/tutorials you have to do the following:

# load the PETSc you want

module load intel-para [Intel InelMPI] [GCC ParaStationMPI]
module load PETSc/3.11.1

# copy ex1.c and the makefile to the current directory

cp $PETSC_DIR/share/petsc/examples/src/vec/vec/examples/tutorials/ex1.c .
cp $PETSC_DIR/share/petsc/examples/src/vec/vec/examples/tutorials/makefile .

# compile and link the example code
make ex1

To execute the example on 2 processors you have to write an batchfile runex1 with the following commands:

#!/bin/bash -x
#SBATCH --nodes=1
#SBATCH --ntasks=2
#SBATCH --partition=batch
srun -np 2 ex1

To submit the batchfile to the slurm batchsystem say
sbatch batchfile

In makefile you can also find how to run some testexamples. There is a line starting with
Hereafter you see which examles were tested by the PETSc team and in runex?? you see how it is called if you substitute $MPIEXEC by srun.