1. Current developments in data assimilation for numerical weather prediction in Canada: EnVar, EnKF and 4D-Var
Mark Buehner1, Josée Morneau2 and Cecilien Charette1
1Data Assimilation and Satellite Meteorology Research Section
2Data Assimilation and Quality Control Development Section
January, 2013

2. Contents Background and recent changes to the Canadian NWP systems
The ensemble-variational (EnVar) data assimilation approach
Recent results from using EnVar compared with standard 3D-Var and 4D-Var (but NO comparisons with 4D-Var-Bens or EnKF)
Conclusions and next steps

3. Background
Environment Canada currently has 2 relatively independent state-of-the-art global data assimilation systems
4D-Var (Gauthier et al 2007) and EnKF (Houtekamer et al 2009):
both operational since 2005
both use GEM forecast model and assimilate similar set of observations
current effort towards unifying code of the two systems
4D-Var is used to initialize medium range global deterministic forecasts (GDPS)
EnKF is used to initialize global ensemble forecasts (GEPS)
Intercomparison of approaches and various hybrid configurations was performed in carefully controlled context: similar medium-range forecast quality from EnKF and 4D-Var analyses, 4D-Var-Ben best
Results presented at WMO workshop on intercomparison of 4D-Var and EnKF, Buenos Aires, 2008 (Buehner et al 2010, MWR)

5. Summary of recent changes
Major improvements to DA of global and regional systems
New 10km/4DVar Regional Deterministic Prediction System…
IBM Power7
Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov

6. Improvements to global & regional DA systems (Nov 2011)
New Satellite Data
62 infrared channels from the IASI instrument on board the METOP satellite. - 7 microwave channels from the SSM/IS instrument on board the DMSP F16 satellite. - 1 water vapor channel from the GOES-W, METSAT-1R and both METEOSAT satellites. - Horizontal thinning of all satellite radiance data, previously done at 250 km (except 200 km for SSM/I) was reduced to 150 km, therefore adding much more satellite data to the systems.
Fig: GZ-500hPa anomaly correlations over the S. Hemisphere during the parallel run (Jan-Jun 2011): old vs new global system.
Other Assimilation Changes
Moisture observations measured from properly equipped aircraft (AMDAR) are assimilated. - Unified satellite data bias correction scheme. Reduction of the time period to compute the bias corrections from 15 to 7 days. The same code is used for all radiance data. - Updated version of the RTTOV radiative transfer code for satellite radiance data. - New sea surface temperature analysis on a 0.2° grid.
Amount of data assimilated in the global system
increase from ~1.9 million to ~4.2 million pieces of information per day.

7. New IBM Power7 • System was installed off-site
• Migration of operational jobs
took a year to complete
• Fully operational since early
May 2012
• 2 clusters; 8192 cores each
• ~ 1/2 PFlops peak total
on average, performance gain per CPU about 2.7x compared to P5 CPU

8. Major upgrade of the Regional Deterministic Prediction System (RDPS) (Oct 2012)
Upgrade included:
increase in resolution to 10 km from the previous 15 km
4D-Var data assimilation system replacing the previous 3D-Var
important changes in the physics
Significant improvements in forecasts with most metrics throughout most of the atmosphere, especially:
for the winter season
for the lower portion of the atmosphere and at the surface
Fig: The red line indicates the domain covered by the RDPS. GZ T
Fig: Vertical profiles of bias (dashed lines) and error standard deviation (solid lines) of 48-h forecasts against radiosonde data, over Canada during the winter 2011, from the old (15km) versus new (10km) RDPS.

9. Upcoming NWP operational implementations
Global Deterministic
- resolution: from 33 km to 25 km
- new vertical coordinates / grid + improved physics
- 4D-Var analysis increments: from T108 to T180, ~100 km
Global EPS
- resolution: from 100 km to 66 km
- analysis (EnKF): multi-scale (3x more assimilated data)
Regional EPS
- resolution: from 33 km to 15 km
by Jan 2013
Global Deterministic
- resolution: from 25 km to 15 km
- from lat-lon to Yin-Yang grid
- from 4D-Var to EnVar: main focus of this presentation
- resolution of increments: from 100 km to 50 km
- new sea-ice and land-surface analyses
Global EPS
- resolution: from 66 km to 50 km
High-resolution System
- single grid (2.5-km resolution) covering most of Canada
by the end of 2013

10. GZ T
Major upgrade of the Global Deterministic Prediction System (GDPS) (expected to become operational in early 2012)
Fig: Vertical profiles of bias (dashed lines) and error standard deviation (solid lines) of 120-h forecasts against radiosonde data, over the S. Hemisphere, during the summer 2011, from the old versus GDPS.
horizontal resolution
- from 33km to 25km
- improvements seen in analysis cycle
dynamics
- new vertical coordinate
- new vertical grid (from regular to Charney-Phillips)
- reduction of errors in stratosphere
- noise reduction; improved numerical stability and conservation properties
physics
orographic blocking
- amplification of bulk drag coefficient
- significant reduction of tropospheric errors in winter hemisphere
boundary layer
- turbulent hysteresis effect
- reduction of errors associated with frontal inversions; improvement of upper-air scores
forecast model
outer loop
- from 33km to 25km
- more data (AMSU-A and Aircraft) due to increase of # bins
- all changes contributed to forecast improvements, roughly doubling the gain due to model changes
inner loop
TL / AD
- from 160km to 100km
- t: from 45min (9 bins) to 18min (21 bins)
background error statistics
- from T108 to T180
minimization:# of iterations
- from 55 (30+25) to 65 (35+30)
DA (4Dvar)

11. Upgrade of Global & Regional Ensemble Prediction Systems (expected to become operational in early 2012)
Changes to the Regional EPS
Model component
- horiz. resolution: from 33 to 15km
- vertical levels from 28 to 40
- improved treatment of stochastic physical tendency perturbations to avoid unrealistic precipitation rates
- improved boundary layer parameterization
Assimilation
component
- no changes
(i.e same initial
conditions as the
global EPS)
Fig: Global verification (CRPS error) of temperature at 500hPa against radiosondes, ov OLD versus NEW GEPS, showing a gain in predictability of 12h and plus.
Changes to the Global EnKF and EPS
multi-scale algorithm (horizontal localization varies vertically)
time-step: from 30 to 20min
horiz. resolution: from 100 to 66km
vertical levels: from 58 to 74
topography filter
reduced thinning of observations (2.7 X number of assimilated radiances)
improved dynamics and physics

12. Model Dynamics: New approaches for global grids

13. Proposed Upgrades and Improvements to the MSC Data Processing for Radiosonde and Aircraft Data
Increased volume of data: selection of observations according to model levels
Revised observation error statistics
Revised rejection criteria for radiosonde data based on those used at ECMWF
Horizontal drift of radiosonde balloon taken into account in both data assimilation and verification systems
Bias correction scheme for aircraft temperature reports
wind speed
temperature
proposed for both
radiosonde & aircraft
operational
12h
Impact of proposed changes
General short-range forecast improvements above 500 hPa in both wind and temperature fields
The temperature forecast biases are significantly improved due to the bias correction scheme for aircraft below 200 hPa and to the new rejection criteria for radiosonde humidity data above
48h
Fig: Verification scores against radiosondes over the N. Hemisphere, Jan-Feb 2009 (dash = bias; solid = stde)

14. 2013-2017: Toward a Reorganization of the NWP Suites at Environment Canada
Role of Ensembles…
Perturbed members of the regional
prediction system (RPS)
Regional
EnKF
Perturbed members of the global
prediction system (GPS)
Background error covariances
Global
EnKF
Control member of the regional
prediction system (RPS)
Regional
En-Var ?
Background error covariances
Control member of the global
prediction system (GPS)
High-resolution deterministic
prediction system (HRDPS)
High-res En-Var
Global
En-Var
global system
regional system

15. Ensemble-Variational assimilation: EnVar
EnVar approach is currently being tested in the context of replacing 4D-Var in the operational Global Deterministic Prediction System
EnVar uses a variational assimilation approach in combination with the already available 4D ensemble covariances from the EnKF
By making use of the 4D ensembles, EnVar performs a 4D analysis without the need of the tangent-linear and adjoint of forecast model
Consequently, it is more computationally efficient and easier to maintain/adapt than 4D-Var
Hybrid covariances can be used in EnVar by averaging the ensemble covariances with the static NMC-method covariances
Like 4D-Var, EnVar uses an incremental approach with:
analysis increment at the horizontal/temporal resolution of EnKF ensembles
background state and analysis at the horizontal/temporal resolution of the higher-resolution deterministic forecast model

16. EnVar formulation
In 4D-Var the 3D analysis increment is evolved in time using the TL/AD forecast model (here included in H4D):
In EnVar the background-error covariances and analysed state are explicitly 4-dimensional, resulting in cost function:
Computations involving ensemble-based B4D can be more expensive than with Bnmc depending on ensemble size and spatial/ temporal resolution, but significant parallelization is possible

17. 4D error covariances Temporal covariance evolution (explicit vs
4D error covariances Temporal covariance evolution (explicit vs. implicit evolution)
EnKF and EnVar (4D B matrix):
192 NLM integrations provide 4D background-error covariances
4D-Var:
TL/AD inner loop integrations,
2 outer loop iterations,
then 3h NLM forecast
-3h 0h
+3h
“analysis time”

18. EnVar: a possible replacement of 4D-Var
Overall, EnVar analysis ~6X faster than 4D-Var on half as many cpus, even though higher resolution increments
Wall-clock time of 4D-Var already close to allowable time limit; increasing number of processors has negligible impact
To progress with 4D-Var, significant work would be required to improve scalability of TL/AD versions of forecast model at resolutions and grid configuration used in 4D-Var
Current focus for model is on development of higher-resolution global Yin-Yang configuration that scales well
Decision made to try to replace 4D-Var with more efficient EnVar  if EnVar is at least as good as current 4D-Var

19. Dependencies between global systems
Current system (1-way dependence):
GEPS relies on GDPS to perform quality control (background check) for all observations and bias correction for satellite radiance observations xb
xb, obs xa xb
GDPS:
Bgcheck+BC
4D-Var
GEM (9h fcst)
obs xb xa xb
GEPS:
EnKF
GEM (9h fcst)

20. Dependencies between global systems
Current system (1-way dependence):
With EnVar (2-way dependence):
2-way dependence (EnVar uses EnKF ensemble of background states) increases complexity of overall system  2 systems have to be run simultaneously xb
xb, obs xa xb
GDPS:
Bgcheck+BC
4D-Var
GEM (9h fcst)
obs xb xa xb
GEPS:
EnKF
GEM (9h fcst) xb
xb, obs xa xb
GDPS:
Bgcheck+BC
EnVar
GEM (9h fcst) xb
obs xb xa xb
GEPS:
EnKF
GEM (9h fcst)

21. Dependencies between global systems
Current system (1-way dependence):
With EnVar (2-way dependence):
Another possibility with EnVar (1-way dependence): xb
xb, obs xa xb
GDPS:
Bgcheck+BC
4D-Var
GEM (9h fcst)
obs xb xa xb
GEPS:
EnKF
GEM (9h fcst) xb
xb, obs xa xb
GDPS:
Bgcheck+BC
EnVar
GEM (9h fcst) xb
obs xb xa xb
GEPS:
EnKF
GEM (9h fcst) xb
xb, obs xa xb
GDPS:
Bgcheck+BC
EnVar
GEM (9h fcst) xb xb xa xb
GEPS:
Bgcheck+BC
EnKF
GEM (9h fcst)
xb, obs

22. EnVar formulation: Preconditioning
Preconditioned cost function formulation at Environment Canada:
In EnVar with hybrid covariances, the control vector (x) is composed of 2 vectors:
The analysis increment is computed as (ek is k’th ensemble perturbation divided by sqrt(Nens-1) ): 
Better preconditioned than original “alpha control vector” formulation (with L-1 and 1/β in background term of J)
Most, but maybe not all, applications of the approach use the better preconditioned formulation 

23. Single observation experiments Difference in temporal covariance evolution
EnVar
radiosonde temperature observation at 500hPa
observation at beginning of assimilation window (-3h)
with same B, increments very similar from 4D-Var, EnKF
contours are 500hPa GZ background state at 0h (ci=10m) + +
contour plots at 500 hPa + +

24. Single observation experiments Difference in temporal covariance evolution
EnVar
radiosonde temperature observation at 500hPa
observation at middle of assimilation window (+0h)
with same B, increments very similar from 4D-Var, EnKF
contours are 500hPa GZ background state at 0h (ci=10m) + +
contour plots at 500 hPa + +

25. Single observation experiments Difference in temporal covariance evolution
EnVar
radiosonde temperature observation at 500hPa
observation at end of assimilation window (+3h)
with same B, increments very similar from 4D-Var, EnKF
contours are 500hPa GZ background state at 0h (ci=10m) + +
contour plots at 500 hPa + +

26. Experimental results: Configuration
EnVar tested in comparison with new version of forecast system currently being implemented in operations:
model top at 0.1hPa, 80 levels
model has ~25km grid spacing
4D-Var analysis increments with ~100km grid spacing
EnVar experiments use ensemble members from new configuration of EnKF:
192 members every 60min in 6-hour window
model top at 2hPa, 75 levels
model ~66km grid spacing  EnVar increments ~66km

27. EnVar uses Hybrid Covariance Matrix Model top of EnKF is lower than GDPS
Bens and Bnmc are averaged in troposphere ½ & ½, tapering to 100% Bnmc at and above 6hPa (EnKF model top at 2hPa)
Therefore, EnVar not expected to be better than 3D-Var above ~10-20hPa
Also tested 75% Bens and 25% Bnmc in troposphere, but results slightly worse
Also did preliminary tests with a full outer loop, but degraded the results
pressure
Bens scale factor
Bnmc scale factor
scale factor

28. Forecast Results: EnVar vs
Forecast Results: EnVar vs. 3D-Var and 4D-Var Radiosonde verification scores – 6 weeks, Feb/Mar 2011 U
|U| GZ T
EnVar vs. 3D-Var
120h forecast
North extra-tropics
T-Td

29. Forecast Results: EnVar vs
Forecast Results: EnVar vs. 3D-Var and 4D-Var Radiosonde verification scores – 6 weeks, Feb/Mar 2011 U
|U| U
|U| GZ T GZ T
EnVar vs. 3D-Var
120h forecast
North extra-tropics
EnVar vs. 4D-Var
120h forecast
North extra-tropics
T-Td
T-Td

30. Forecast Results: EnVar vs
Forecast Results: EnVar vs. 3D-Var and 4D-Var Radiosonde verification scores – 6 weeks, Feb/Mar 2011 U
|U| GZ T
EnVar vs. 3D-Var
120h forecast
South extra-tropics
T-Td

31. Forecast Results: EnVar vs
Forecast Results: EnVar vs. 3D-Var and 4D-Var Radiosonde verification scores – 6 weeks, Feb/Mar 2011 U
|U| U
|U| GZ T GZ T
EnVar vs. 3D-Var
120h forecast
South extra-tropics
EnVar vs. 4D-Var
120h forecast
South extra-tropics
T-Td
T-Td

32. Forecast Results: EnVar vs
Forecast Results: EnVar vs. 3D-Var and 4D-Var Radiosonde verification scores – 6 weeks, Feb/Mar 2011 U
|U| GZ T
EnVar vs. 3D-Var
24h forecast
Tropics
T-Td

33. Forecast Results: EnVar vs
Forecast Results: EnVar vs. 3D-Var and 4D-Var Radiosonde verification scores – 6 weeks, Feb/Mar 2011 U
|U| U
|U| GZ T GZ T
EnVar vs. 3D-Var
24h forecast
Tropics
EnVar vs. 4D-Var
24h forecast
Tropics
T-Td
T-Td

34. Forecast Results: EnVar vs
Forecast Results: EnVar vs. 3D-Var and 4D-Var Verification against ERA-Interim analyses – 6 weeks, Feb/Mar 2011
EnVar vs. 3D-Var
48h forecast, global domain
no EnKF
covariances
transition
zone
½ EnKF and
½ NMC
covariances U RH GZ T

35. Forecast Results: EnVar vs
Forecast Results: EnVar vs. 3D-Var and 4D-Var Verification against ERA-Interim analyses – 6 weeks, Feb/Mar 2011
EnVar vs. 3D-Var EnVar vs. 4D-Var
48h forecast, global domain
no EnKF
covariances
no EnKF
covariances
transition
zone
transition
zone
½ EnKF and
½ NMC
covariances
½ EnKF and
½ NMC
covariances U RH U RH GZ T GZ T

36. Forecast Results: EnVar vs
Forecast Results: EnVar vs. 3D-Var and 4D-Var Verification against ERA-Interim analyses – 6 weeks, Feb/Mar 2011
EnVar vs. 3D-Var
120h forecast, global domain
no EnKF
covariances
transition
zone
½ EnKF and
½ NMC
covariances U RH GZ T

37. Forecast Results: EnVar vs
Forecast Results: EnVar vs. 3D-Var and 4D-Var Verification against ERA-Interim analyses – 6 weeks, Feb/Mar 2011
EnVar vs. 3D-Var EnVar vs. 4D-Var
120h forecast, global domain
no EnKF
covariances
no EnKF
covariances
transition
zone
transition
zone
½ EnKF and
½ NMC
covariances
½ EnKF and
½ NMC
covariances U RH U RH GZ T GZ T

38. 500hPa GZ correlation anomaly
Forecast Results: EnVar vs. 3D-Var and 4D-Var Verification against ERA-Interim analyses – 6 weeks, Feb/Mar 2011
North extra-tropics
500hPa GZ correlation anomaly
EnVar vs. 3D-Var EnVar vs. 4D-Var

39. 500hPa GZ correlation anomaly
Forecast Results: EnVar vs. 3D-Var and 4D-Var Verification against ERA-Interim analyses – 6 weeks, Feb/Mar 2011
South extra-tropics
500hPa GZ correlation anomaly
EnVar vs. 3D-Var EnVar vs. 4D-Var
This is the only significant degradation seen vs. 4D-Var in troposphere;
Not in radiosonde scores because it originates from south of 45°S (see next slide)

40. 120h forecast of 500hPa GZ - STDDEV STDDEV - South extra-tropics
Forecast Results: EnVar vs. 3D-Var and 4D-Var Verification against ERA-Interim analyses – 6 weeks, Feb/Mar 2011
120h forecast of 500hPa GZ - STDDEV
120h forecast of 500hPa GZ
STDDEV - South extra-tropics

41. Forecast Results: EnVar vs
Forecast Results: EnVar vs. 3D-Var and 4D-Var Verification against ERA-Interim analyses – 6 weeks, Feb/Mar 2011
Tropics
250hPa U-wind STDDEV
EnVar vs. 3D-Var EnVar vs. 4D-Var

42. 500hPa GZ correlation anomaly
Forecast Results: EnVar vs. 3D-Var and 4D-Var Verification against ERA-Interim analyses – 6 weeks, July-Aug 2011
North extra-tropics
500hPa GZ correlation anomaly
EnVar vs. 3D-Var EnVar vs. 4D-Var

43. 500hPa GZ correlation anomaly
Forecast Results: EnVar vs. 3D-Var and 4D-Var Verification against ERA-Interim analyses – 6 weeks, July-Aug 2011
South extra-tropics
500hPa GZ correlation anomaly
EnVar vs. 3D-Var EnVar vs. 4D-Var

44. Forecast Results: EnVar vs
Forecast Results: EnVar vs. 3D-Var and 4D-Var Verification against ERA-Interim analyses – 6 weeks, July-Aug 2011
Tropics
250hPa U-wind STDDEV
EnVar vs. 3D-Var EnVar vs. 4D-Var

45. Experimental results: 4D-EnVar vs. 3D-EnVar
3D version of EnVar also tested: only uses EnKF flow-dependent ensembles valid at the centre of the 6h assimilation window, instead of every 60 minutes throughout the window
3D-EnVar compared with:
4D-EnVar: impact of 4D vs 3D covariances, and
3D-Var: impact of flow dependent vs stationary (NMC) covariances (both 3D)

46. 500hPa GZ correlation anomaly
Forecast Results: 4D-EnVar vs. 3D-EnVar Verification against ERA-Interim analyses – 4 weeks, Feb 2011
North extra-tropics
500hPa GZ correlation anomaly
4D-EnVar vs. 3D-EnVar D-EnVar vs. 3D-Var

47. 500hPa GZ correlation anomaly
Forecast Results: 4D-EnVar vs. 3D-EnVar Verification against ERA-Interim analyses – 4 weeks, Feb 2011
South extra-tropics
500hPa GZ correlation anomaly
4D-EnVar vs. 3D-EnVar D-EnVar vs. 3D-Var

48. Forecast Results: 4D-EnVar vs
Forecast Results: 4D-EnVar vs. 3D-En-Var Verification against ERA-Interim analyses – 4 weeks, Feb 2011
Tropics
250hPa U-wind STDDEV
4D-EnVar vs. 3D-EnVar D-EnVar vs. 3D-Var

49. Conclusions Comparison of EnVar with 3D-Var and 4D-Var:
EnVar produces similar quality forecasts as 4D-Var below ~20hPa in extra-tropics, significantly improved in tropics
above ~20hPa, scores similar to 3D-Var, worse than 4D-Var; potential benefit from raising EnKF model top to 0.1hPa
EnVar is an attractive alternative to 4D-Var:
like EnKF, uses full nonlinear model dynamics/physics to evolve covariances; no need to maintain TL/AD version of model
instead, makes use of already available 4D ensembles
more computationally efficient and easier to parallelize than 4D-Var for high spatial resolution and large data volumes
computational saving allows increase in analysis resolution and volume of assimilated observations; more computational resources for EnKF and forecasts

50. Next Steps
Finalize testing EnVar with goal of replacing 4D-Var in operational global prediction system during 2013 in combination with other changes:
GEM global model on 15 km Yin-Yang grid
CALDAS: new surface analysis system based on an EnKF
modified satellite radiance bias correction scheme that gives conventional observations more influence on correction
improved use of radiosonde and aircraft data
additional AIRS/IASI channels and modified observation error variances for all radiances
increased resolution of EnKF 66km  50km
Testing of EnVar in regional prediction system as possible replacement of 4D-Var already started