HORSES3D-GPU – A high-order discontinuous Galerkin solver for multi-GPU systems

  • Name: HORSES3D-GPU – A high-order discontinuous Galerkin solver for multi-GPU systems
  • EuroHPC machine used: MareNostrum 5
  • Topic: Computer and information sciences; Mechanical engineering

Overview of the project

HORSES3D-GPU is an open-source, high-order discontinuous Galerkin spectral-element solver for computational fluid dynamics on multi-GPU supercomputers. It is based on the HORSES3D framework and solves compressible Navier-Stokes flows, including laminar, transitional and turbulent regimes, shocked flows, wall-bounded flows and complex geometries. The GPU implementation uses Fortran 2008, OpenACC and MPI, while maintaining a common CPU/GPU code base and keeping the main computational kernels resident on the GPUs. The solver has been assessed on NVIDIA H100 GPUs at MareNostrum 5, including high-lift CRM simulations with up to 20.8 million elements and approximately 10.7 billion degrees of freedom.

 

How did EPICURE support the project and what were the benefits of the support?

“The overall challenge was to make a mature, Fortran-based, high-order CFD solver suitable for execution on modern multi-GPU supercomputers while preserving its numerical formulation and maintaining a common CPU/GPU code base.

The EPICURE contribution was targeted and limited to selected parts of this broader development effort. The EPICURE developer involved in these modifications contributed to some code refactoring, bug correction and the implementation and validation of additional GPU-supported features, particularly time-integration schemes and shock-capturing techniques. These contributions improved the robustness and functional coverage of the GPU branch.

The support reduced the effort required to debug and extend the GPU implementation, improved code maintainability and made additional time-integration and shock-capturing capabilities available on GPUs. This helped increase the readiness of HORSES3D-GPU for large-scale multi-GPU CFD simulations. No separate measurement of time or energy savings was performed for the EPICURE-specific contributions, so we would not attribute the full performance results of the project directly to EPICURE. The broader HORSES3D-GPU project has nevertheless demonstrated execution and scaling studies on MareNostrum 5 using up to 2048 NVIDIA H100 GPUs.” – Gonzalo Rubio

Project website:

Project GitHub repository