1. Energy efficient SCalable
Algorithms for weather Prediction at Exascale
Peter Bauer

2. ESCAPE key objectives
Define fundamental algorithm building blocks (“Weather & Climate Dwarfs”) to co-design, advance, benchmark and efficiently run the next generation of NWP and climate models on energy-efficient, heterogeneous HPC architectures.
Combine frontier research on algorithm development and extreme-scale, high-performance computing applications with novel hardware technology, to create a flexible and sustainable weather and climate prediction system.
Foster the future design of Earth-system models and commercialisation of weather-dependent innovative products and services in Europe through enabling open-source technology.
Pairing world-leading NWP with innovative HPC solutions.

3. ESCAPE European impact map
34 countries
7 countries
11 countries
16 countries

4. ESCAPE partners & expertise
Global operational NWP
Petaflop data centre
Numerical methods, parallel computing
Regional operational NWP
Numerical methods
Key technology development
Parallel computing
Regional operational NWP
Teraflop data centre
Physical processes
Numerical methods
Teraflop data centre
Parallel computing
Teraflop data centre
Parallel computing
Performance modelling
Teraflop data centre
Parallel computing
Global/regional operational NWP
Teraflop data centre
Numerical methods
Global/regional operational NWP
Petaflop data centre
Numerical methods
Key technology development
Parallel computing
Key technology development
Parallel computing
Regional operational NWP
Petaflop data centre
Parallel computing

5. Traditional science workflow
[Schulthess 2015]

6. Future science workflow
Energy efficient SCalable Algorithms for weather Prediction at Exascale 
 science specific code
 generic code
[Schulthess 2015]

7. Energy efficiency
.. aiming at minimizing
Watts per forecast

8. Model development: ESCAPE
Energy efficient SCalable Algorithms for weather Prediction at Exascale (hpc-escape.eu)
Disassemble global … extract, redesign … optimize for energy … reassemble global
NWP model… key components = dwarfs… efficiency on new hardware… NWP model

9. Computational cost
Global, spectral Regional, spectral Regional, Eulerian

10. What’s wrong with weather and climate codes?
Advection: halo-communication, scalability
Transforms: communication bandwidth, memory
Physics: expensive calculations, scalable
Ocean waves: expensive calculations, load balancing
3D-solver: iterations, memory, communication bandwidth
No 1: sequential with time steps (10d x 24h x 3600s / 450s = 9km
Dwarfs:
Spectral transforms (FT/LT and bi-FT) ………….. very memory and communication bandwidth intensive, possibly limited scalability
2 & 3-dimensional elliptic solver ……………….... compute and communication latency intensive, possibly limited scalability
Semi-Lagrangian advection …………………….... communication intensive, possibly limited scalability
Flux-form finite-volume advection ……………….. local communication, latency intensive, limited scalability
Cloud physics parameterization …………………. expensive computations, independent vertical columns, scalable
Radiation parameterization ………………………. expensive computations, spatial, temporal and wavenumber space, scalable

11. Dwarfs & adaptation

12. Separation of concerns: Atlas

13. Atlas support of grids
Grid object is collection of grid points (structured or unstructured), Global or Regional model set-up:
with hierarchical interpretations:

14. Atals & GridTools GridTools library and interfaces:
Atlas data structures & functionalities:

15. Algorithmic flexibility: path and control volume

16. Algorithmic flexibility: spectral and grid-point

17. Dwarf performance
more fields

18. Porting of LAITRI dwarf
LAITRI performs interpolations for semi-Lagrangian advection scheme
The settings to achieve this: correct use of OpenMP (e.g. ensuring correct ‘first touch’ of data), suitable compiler data alignment directives to good vectorisation performance, and prudent setting of runtime variables such as KMP AFFINITY and KMP HW SUBSET.
Aim: integrate these settings in Atlas, away from science code!

19. Weather & Climate in H2020 PantaRhei Reference application
Dissemination across community
Showcase of new technology
Reference application
Co-design hardware
& programming models
Co-design hardware
& programming models
Ingestion of dwarfs in full-scale ESM
Dissemination platform for dwarfs
Reference application
Reference application
Evaluation against alternative programming models
High-impact application demonstrator
Performance assessment and optimization tools
Feedback on tool applicability and value
PantaRhei
Finite-volume fully compressible core
Global NWP implementation

20. ESCAPE Staff

21. The 10-year production challenge
Data acquisition
Single model run
Product generation
Dissemination
RMDCN
Internet
Web services
Archive
Data Handling System
Scenarios
Multi-model run
Climate Data Store
Observations shared worldwide
O(GB)
Dissemination
To 300 destinations
National met services and commercial customers
O(TB)
No.1 goal: Global 1 km simulations at >1 SYPD
Computing and data challenges grow by factor

22. The 10-year technical&programmatic challenge
Top-3 computing bottlenecks: Memory bandwidth, Memory Size/Layout, Network latency… but not FLOPS!
[G. Kalbe, 2017, European HPC, Time line overview]

23. The 10-year impact challenge
&
Climate
Observations
Satellites
Radar
Ground stations
Nowcasting
Weather forecasting
Climate prediction
Reanalyses
+ Computing
Direct data links
Internet
TV, radio
Other
Threat
Impact
Probability
Reliability
Warning
Protection
Buying/selling
Investing
[J. Lazo 2016, the NWP value chain]