1. Experiences and Decisions in Met Office coupled ESM Development
Richard Hill, February 2013
© Crown copyright Met Office

2. Contents Where did we come from? Where are we now?
Where do we want to go?
Questions
© Crown copyright Met Office

3. 1: Where did we come from? 2004: A single exec with plenty of bottlenecks
Run the
UM on
n PEs
Atmos
Ocean
Sea ice
The Unified Model
© Crown copyright Met Office

4. What Changed? Adoption of NEMO ocean and CICE sea-ice models
Not owned by Met Office! Code management trickier
Not suitable for the single executable approach
NEMO and CICE use of tri-polar grids!
In-house coupling much harder
Desire for more “plug compatible” components and higher resolution
OASIS4 (initially) to the rescue!
© Crown copyright Met Office

5. MPMD Coupling Code configuration influenced by:
NEMO and CICE MUCH FASTER than atmosphere
NEC-SX6 vector machine
NEMO and CICE essentially use the same tri-polar grid
Not necessarily the same decomposition
Coupling frequencies.
NEMO  CICE: Every time step.
NEMO-CICE  Atmosphere: Once a day
Led to: NEMO-CICE combined in single executable
© Crown copyright Met Office

6. OASIS3 Coupling “Upgrading” to OASIS3 (~2007)
Switch to stable (but serial) OASIS3 for production
Pseudo-parallel OASIS3
Relatively straightforward to adapt components
Tricky bits:
Lack of effective debuggers
When something’s wrong it’s not always obvious what!
Creating, debugging and managing control files. “Framework” issues
Control files (OASIS “namcouple” control, regridding weights etc) add to configuration and system management costs
© Crown copyright Met Office

7. 2: Where are we now? Current system.
Adapted UM control system in lieu of true coupling “framework”. (GUI, pre-processing, post-processing, data archiving.)
Designed from the “inside-out”
Pros: No huge (and potentially speculative) up-front development cost; Develop/extend as required; Intuitive for existing users and developers; All model definition info in a single place (traceability); building on existing well worn procedures for exchange and development of models. Used successfully by external partners.
Cons: Modifying existing systems beyond original design constraints; Crowbar NEMO and CICE into system designed purely for the UM; Not inherently extensible; More components stretch infrastructure to/beyond limits.
© Crown copyright Met Office

8. Current OASIS3 models
Routinely run configurations (Typically 3-hour coupling)
For Higher resolutions and coupling frequencies,
e.g. N512-OASIS3-ORCA025 (The largest coupled model routinely run.)
OASIS3 costs become more evident
But still not crippling!
Component memory increasingly a limitation on load balancing
i.e. Size of model may inhibit ability to balance for optimal use of resources!
© Crown copyright Met Office

9. Issues with direct bespoke coupling as used in NEMO-CICE
Bespoke direct coupling:
May be quicker than going via separate coupler
But ongoing code maintenance costs are significant!
Extra responsibility of developing, maintaining, upgrading code which is owned by someone else.
Little chance of getting local interfacing code into someone else’s code repository!
Coordinating coupled model releases featuring local modifications, while keeping up to date with official component releases takes a lot of work.
© Crown copyright Met Office

10. To attach or detach components?
E.g.: The way CICE/Internal coupling throttles NEMO
© Crown copyright Met Office

11. Recent developments Installation of OASIS3-MCT
Upgrade costs lie in alterations to “framework” and management processes.
No separate coupler exec
Simplifies resource allocation, load balancing AND to some extent, management of control files (since no pseudo- parallel cases need to be considered.)
Long parallel assessment climate runs already performed
So far so good – coupling costs appear lower than OASIS3
Components still the limiting factor – not the coupler!
2nd Order Conservative remapping important?
© Crown copyright Met Office

12. 3: Where do we want to go? Increased resolution
N512-ORCA 1/12th global
Will be a challenge (~4000+ cores probably.)
Remapping weights files (SCRIP format ) at high resolutions are a worry – size, generation time and manipulation?
OASIS3-MCT results so far suggest that the components continue to be the bottlenecks, not the coupler!
Increased component numbers
ESM plans to include wave, biogeochemical, ice sheet models, etc.
Make the “right” decisions about generic/bespoke coupler use
Load balance an N component system!
© Crown copyright Met Office

13. Future models From this.....
Chemistry
Atmos
Coupler
NEMO
Land
CICE
© Crown copyright Met Office

14. Future shock To this.....? Wave IO Server Chemistry Atmos Coupler NEMO
Land
CICE
Biological
© Crown copyright Met Office

15. Future N-component ESMS
Detached components may be more attractive:
HPC architectures have changed since original decisions
Optimal core count for each component more achievable???
More flexibility to load balance?
Combined exec performance restricted by least scalable component
e.g. component X may have scope for increased scalability while component Y doesn’t (or even reverses gains made by X.)
Larger (higher res) models make NEMO-CICE a more significant cost (too many cores for CICE?)
Could a more generic framework help development, usability, traceability, system and configuration management?
© Crown copyright Met Office

16. Questions and answers
© Crown copyright Met Office