EOLDAS_ng The new generation Prof. P. Lewis, Dr. Jose Gomez-Dans, UCL/NCEO

Task 3: EO-LDAS baseline WP3.1: EO-LDAS v1 consolidation o Autumn 2013/workshop o Publicly available software & documentation WP3.2: Integration of enhanced components o Dec/2013 o Integration of other modelling efforts in variational DA

What is EO-LDAS? generic variational DA system, weak or strong constraint different observation operators (RT models) different process models, including regularisation tutorial documentation -- how to do DA with the system Demonstrated so far with optical data Lewis et al (2012), Rem Sens Env Lewis et al (2012), ESA SP

The concept Observations (RT codes) Dynamic Model Prior Define a cost function J as a sum of however many components, but typically: Each component calculates the cost and the partial derivatives of the cost function Use gradient descent minimisation o Need partial derivatives for Observation Operator & Dynamic model Uncertainty is calculated using the Hessian (matrix of 2nd order minimum J, J'

What have we learned with EO-LDAS? v1 design controlled by configuration files o minimise programming requirements o generic structure probably sound o teach users new metalanguage to describe problem Too much flexibility leads to inefficiencies o Difficult to add extra functionality o Difficult to maintain

EO-LDAS v2 1.Assume user knows Python (e.g. I/O, plotting) 2.Streamline system by reducing flexibility 3.Simple components - readily extended 4.Decouple solver from operators a.Essentially, list of J, J' EO-LDAS should be a Python library of state definitions and generic operators, requiring the user to interface to other: ObsOps, Dynamic Models, Data, Plotting, etc

Interactive documentation IPython notebooks to provide interactive documentation on library brary

Faster models through emulation First steps in using Gaussian Processes (GP) Provides a fast version of the code... but also of its partial derivatives! 1. User generates training set with RT model 2. GP is trained to the above 3. This is used in EO-LDAS Much more flexible: o if the model can be emulated, any model can go in o no need to waste time on adjoints o very fast o simplifies library structure very much

Emulator testing Use IPython notebooks to provide the user with interactive documentation on how to do emulation

Spares

Multi-resolution processing

Gotchas Observation Operator gotchas Typical RT codes are science grade software: o Limited support from the authors o Inefficient o Difficult to follow & modify Few RT models have adjoints readily available If the RT model is modified, the adjoint needs to be re-created Dynamic model gotchas All of the above Ended up implemented a linear model: if you can express your model as Ax, then you can slot it in Most dynamic models are highly non-linear, and linearisation requires automatic differentiation So ended up using Tikhonov-type regularisation