1. The Joint Effort for Data assimilation Integration (JEDI)
Model interfacing
Joint Center for Satellite Data Assimilation (JCSDA)
JEDI Academy June 2019

2. Building an application C++/Fortran binding
Outline
Introduction
Model space classes
GetValues
Building an application
C++/Fortran binding

3. Introduction
OOPS provides the algorithms that combine generic building blocks into applications such as variational assimilation, forecast, EnKF, FSOI etc.
OOPS (by design) knows nothing about the actual implementation of the building blocks and carries no information about the underlying data. The classes that need to be implemented for a specific model are called interface classes.
Often models are written in Fortran so a mixed language approach is required and a binding between the languages is implemented.
Once the interface to a specific model is ready it can be used to create applications by passing traits and information about factories.

4. Models being interfaced to JEDI
TYPE
INTERFACE
CENTER
FV3GFS
Atmosphere
fv3-jedi
NOAA-EMC
GEOS
NASA-GMAO
FV3GFS GSDChem
Atmospheric chemistry
NOAA-ESRL
GEOS-AERO
Atmospheric aerosols
MPAS
mpas
NCAR
WRF
wrf-jedi
LFRic
lfric
Met Office (UK)
MOM6
Ocean
soca
SIS2
Sea ice
CICE6
soca-cice6
NEPTUNE
neptune
NRL QG
Toy model
oops
JCSDA
Lorenz 95
ShallowWater
shallow-water

5. From interface to implementation
All the model (and UFO) classes follow basically the same file structure for the mixed C++/Fortran languages:
StateFV3JEDI.h
fv3jedi_state_interface_mod.F90
State.h
(OOPS)
StateFV3JEDI.cc
fv3jedi_state_mod.f90
StateFV3JEDIFortran.h
For a Fortran based model almost all the work and memory is here.

6. Model space classes

7. Model space interface classes
Geometry
State
Increment
InterpolatorTraj
Model
LinearModel
ErrorCovariance
Localization
ModelAuxControl
ModelAuxCovariance
ModelAuxIncrement
VariableChange
LinearVariableChange
Geometry, State, Increment, interpolation
Models
Background error covariance
Model bias correction or parameter estimation
(empty for now)
Nonlinear and linear change of variables

8. Building the FV3-JEDI building blocks
Model
GEOS
GEOS
FV3-JEDI-LM
NEMSfv3gfs LM
NEMS
GetValues
ufo
ioda
State
Geometry
LinearModel
FV3-JEDI-LM
Increment
ErrorCovariance/
Localization
saber
VarChange/
LinVarChange
Poisson solver

9. Geometry Class: OOPS vs. FV3-JEDI

10. Geometry: Fortran interfaces

11. Geometry Class: Fortran Type

12. Geometry Class: Fortran Methods
Call to dynamical core

13. State and Increment: Methods
random
self_add (+=)
self_schur
self_sub (-=)
self_mul (*=)
axpy_inc
axpy_state
dot_prod
diff_states
getValues_tl
getValues_ad
ug_coord
increment_to_ug
ug_to_increment
dirac
create
delete
zeros
copy
read
write
gpnorm
rms
change_res
axpy
add_increment
getValues
analytic_ic

14. Field Class
The concept of fields is introduced in order to limit duplicate code across sate and increment.
Some model interfaces implement fields at the C++ level, e.g. qg, l95, soca. Some do so at the Fortran level, e.g. fv3-jedi, mpas, lfric.

15. State/Increment variables

16. State/Increment constructor
Incoming vars are decided by the user at run time.
Variables are pre-programmed but not hardwired

17. State/Increment method
Optionally check same list of fields in self and rhs
Loop through all allocated fields. Not dependent on variables chosen.

18. Model class
Factory name

19. Model class
GEOS
NEMSfv3gfs
Dynamical core only
Pseudo model

20. Data flow STATE MODEL MODEL STEP (EXTERNAL) MODEL STATE TRAJ & PPs
Can be pointer or copy. Usually a copy to account for differences in precision

21. LinearModel class Per outer loop Every call CREATE INIT_TL FINALIZE_AD
STEP_TL
STEP_AD
Every call
FINALIZE_TL
INIT_AD
DELETE

22. Variable changes
Incremental hybrid-4DVar involves a number of linear and nonlinear variable transforms:

23. Variable changes Sets of variables:
Background: the variables which you end up with an analysis of. Typically chosen to interact well with the forecast model being restarted.
Control increment: the variables of , chosen based on various considerations.
Model: the variables that the model and linear model need, e.g. staggered winds.
B matrix variables: the variables used in the B matrix, e.g. unbalanced stream function and velocity potential.

24. VarChaC2MFV3JEDI
Increment containing control variables comes in, increment with model variables goes out. The base class handles the allocation and deallocation either side.

25. Get Values

26. OOPS – UFO – IODA – MODEL: the interface advantage
JEDI/UFO introduces standard interfaces between the model and observation worlds.
Observation operators are independent of the model, easy sharing, exchange and comparison.
Observation Locations
Observation space
MODEL
getValues
getValuesTL
getValuesAD
Variables
Observation operator
IODA
OOPS
Observation vector
GeoVals: State values at locations
UFO

27. GetValues
Get Values is the routine by which the model provides the GeoVaLs to the generic part of the observation operator (UFO).
It can be thought of as the grid/model dependent part of the observation operator – providing the variables called for by UFO at the locations of the observations.
Currently it involves some variable changes, though that will be moved. It also contains interpolation from the model native grid to observation locations.
The observations operators to be instantiated are chosen through the YAML, if getValues does not recognize the variable being asked for the system should abort.

28. getValues: Fortran interfaces
Geometry
State/increment
Locations
Variables
Geo Values
Trajectory for TL/AD

29. InterpolatorTraj Trajectory needed for variable change
Interpolation object

30. GetValues: algorithm Compute weights for interpolation
Loop over UFO variables
Select case on variable
Convert variable and prepare interpolation
Loop over levels
Interpolate to locations
end (levels)
End (variables)

31. getValues: prepare state/increment variable
Loop over variables
Flag on whether to interpolate
Set number of levels for variable and interpolation flag.
Transform the variable if need be
Some variables use integration or aren’t float.
ABORT

32. getValues: allocate GeoVaLs
The model has to allocate the GeoVaLs since it knows the number of model levels.

33. getValues: BUMP interpolation weights
Unstructured vector of model grid points.

34. getValues: BUMP interpolation apply
Step 1: Convert to vector. Same structure as grid is organized when passed to BUMP setup.
Call BUMP
Fill up GeoVaLs with locations in current time bin.

35. getValues: BUMP adjoint interpolation
1. Set pointer
2. Apply interpolation weights
3. Convert back to increment.

36. Building an application

37. Building an application driver
Geometry
If an oops branch with a bug is merged and no one is there to compile it, does the bug really exist?
State
Increment
OOPS
Traits/
Factories
Driver
fv3jediVar.cc
fv3jedi_var.x
Model …
oops on its own is just headers requiring a template to be applied to the interface classes and be passed via traits and factories.
ufo/ioda

38. fv3jediVar.cc application driver
Include the model traits
Include the factories
Include the main application
Pass config (YAML)
Initialization step (FMS etc)
Instantiate factories
Create application object
Execute application

39. Run
Subclass can perform any local initialization steps. E.g. initialize library required by FV3
Inheritance from the base class Run
This can do generic initialization such as MPI init and prepare generic monitoring tools.
In turn inherits eckit::Main.
Execute runs the application and diagnostics.

40. Variational.h

41. Run execute Receives and application Calls application execute
Checking of proper run
Output some diagnostics

42. FV3-JEDI Traits OOPS level State.h interface
FV3-JEDI Templates passed in through Traits. Basically just a list of implemented classes.

43. Factory instantiation
Instantiate the change of variable designated by VarChaC2MFV3JEDI. In the YAML we need to call as “Control2Model”
Factory: the class VarChaC2MFV3JEDI is then implemented as normal.
YAML: choose the subclass and the variables to be allocated.

44. C++/Fortran binding

45. Binding: C++ side
GeometryFV3JEDI.cc
GeometryFV3JEDIFortran.h

46. Binding: Fortran side fv3jedi_geom_interface_mod.F90
Access to the object is through a linked list
Integer Key comes in, pointer to an object gets passed.

47. LinkedList inclusion
At the interface_mod level the Linked List is created for the Fortran version of the object.
linkedList_i.f contains the list of objects and linkedList_c.f contains the methods for manipulating and accessing the current object in the linked list.

48. linkedList_i.f
Linked list node is where an object is actually stored in memory. It also contains a pointer to the next element.
Class containing pointer to the head node. Methods for accessing that object.

49. linkedList_c.f: initilaize
If linked list not initialized associate the head node and set the flags.

50. linkedList_c.f: add
Key comes in from OOPS. Adding an object to the linked list so ‘up the counter’ and set the key.
Then allocate the next element. This is the actual allocation of memory for the object.
Associate a pointer to the next element in the linked list.

51. linkedList_c.f: get
Pointer comes in which needs to be associated with the object in the position in the linked list associated with the key.
Do while loop sweeps the linked list until the key matches the point in the linked list.

52. Questions?