link to homepage

Institute for Advanced Simulation (IAS)

Navigation and service


Massively parallel finite element simulation of multiphysics problems

Scientific area

Numerical modeling and simulation, multiphysics PDEs, finite element methods, domain decomposition solvers, block preconditioners.

Short description

FEMPAR is a parallel hybrid OpenMP/MPI, object-oriented framework for the massively parallel FE simulation of multiphysics problems governed by PDEs. It provides tools for the numerical simulation of a wide range of different physical phenomena, including compressible/incompressible flows, magnetics, solid mechanics, fluid-structure interaction, or thermal coupling. FEMPAR has been designed to tackle multi-physics, non-linear, and multi-scale problems. FEMPAR makes use of scalable implicit massively parallel solvers that are based on Balancing Domain Decomposition by Constraints (BDDC) preconditioning and fully-coupled block LU preconditioning for multiphysics problems. In particular, within the domain decomposition kernel, FEMPAR provides a novel, fully-distributed, communicator-aware, recursive, and overlapped implementation of the multilevel BDDC preconditioner. This code weakly scales up to 458,752 JUQUEEN cores for coercive three-dimensional problems (the Poisson problem and linear elasticity). The largest problem solved with FEMPAR up to now involved 30 billion unknowns. As far as we know, it is the first time that such degree of scalability is reported for a domain decomposition preconditioner. FEMPAR is released under the GNU GPL v3 license, and is more than 200K lines of Fortran77/95/2003 code long.

FEMPARVorticity isosurfaces for the incompressible Taylor-Green vortex problem at Re=1600 at four different time steps, starting with the initial condition at the top-left corner and evolving in time from top to bottom, and left to right. A segregated velocity/pressure algorithm, that involves a pressure Poisson solver per time step, was used.


  • 458,752 cores on BlueGene/Q (JUQUEEN)
  • 27,648 cores on x86-64 (MN-3, HELIOS, CURIE, and HERMIT)

Performance of FEMPAR on JUQUEENWeak scalability on JUQUEEN for the number of PCG iterations (left) and total time (right) of the FEMPAR's highly scalable implementation of the 3-level BDDC solver when applied to a 3D Laplacian problem with a local problem size of 30x30x30 (27K) hexahedra/core. Total time includes both preconditiner set-up and PCG stages. The coarse-grid space was supplied with corner, edges and faces continuity constraints (cef).

Programming language and model

  • Fortran 95/2003/2008
  • MPI, OpenMP

Tested on platforms

  • BlueGene/Q
  • Cray XE-6
  • x86-64

Application developers and contributors

Large Scale Computational Physics (LSCP) team
International Center for Numerical Methods in Engineering (CIMNE)
Universitat Politècnica de Catalunya
Parc Mediterrani de la Tecnologia
Esteve Terradas 5, Building C3
08860 Castelldefels (Barcelona)


Prof. Dr. Santiago Badia

Dr. Alberto F. Martín

Prof. Dr. Javier Principe

(Text and images provided by the developers)