1. Observations Preprocessing Carla Cardinali

2. Observations preprocessing
BUFR
MakeCMA
ECMA
Screening
ODB
CCMA
Minimization
Forecast
MatchUp
ECMA
FeedBack
BUFR

3. Observations preprocessing
ECMA/ODB
CCMA/ODB
Output BUFRs

4. Observations preprocessing
What is ODB?
Developed software at ECMWF to manage large amounts of satellite data. ODB/SQL language is a subset of ANSI/SQL query language. Compiler translates ODB/SQL in C-codes files
How do use it?
Access to ODB database is through Fortran90 modules functions. A ECMA data base contains table with column entries lat, lon, date, time observation type which is used to locate observation at a given moment
Obs. Ident ( sat. id, lat, lon, st.alt., date, time)
Obs. V. (wind, temp.,…per pressure) and (radiances per ch., inst. type)
Various flag: active, blacklisted….
Departures (obs-background, obs-analysis)

5. Observations preprocessing
Incoming Observation The observations arrive at ECMWF through GTS and they are stored in a decoded format in Report Data Base after some rudimentary quality control e.g. observation format and position, climatological and hydrostatic limit and temporal consistency. An observation file suitable for assimilation is created
6-hour Preprocessing Observation Array Format conversions, change of some observed variables (relative humidity from dry and wet bulb temperature) and assignment of observation error statistics

6. Observations Screening
Select the “best” observation At the first trajectory run the model counterparts for all the observations are calculated through the nonlinear observation operators. For the observation screening, the background errors are interpolated to the locations of the observed variables. The "extended" observation array (ECMA) contains observations complemented by the background departures and quality control information for most of the observations This array is stored for later feedback After the screening a "compressed" array is passed to the minimization (CCMA).

7. Independent Observations Screening
The screening logic is to make first those decisions that are not depending on any other ones
Preliminary checks: completeness of the report
Blacklist Data Selection: which observation types, variables, vertical ranges will be used in the assimilation Monthly selection: discarding stations that have been reporting excessively noisy or biased compare to the background field Impact studies.

8. Independent Screening Decision: Conventional Observations
Background Quality Control The BgQC is applied to all the variables that are intended to be used in the assimilation
BgQC of wind observations is done simultaneously for both wind components. For wind direction the error limits of 60, 90 and 120 apply for flags 1,2 and 3 uo ub

9. Independent Screening Decision: Conventional Observations

10. When is not working

11. Independent Screening Decision: Satellite Observation
Bias Correction Bias correction coefficients are recomputed from the past 2/4 weeks of departure statistics. The feedback files are used for monitoring the performance of the observing and assimilation system: e.g. ATOVS and SSMI radiances, SCATT wind.
If the removed bias is a model forecast bias, the subsequent assimilation will enforce it. Usually, only half bias is removed.
Gross check Measured and background brightness temperatures are present for all required channels. Bias correction coefficients, satellite id, and scan position are all valid before proceeding. For all channels a cloud detection is performed
230K
290K

12. Dependent Screening Decision: Satellite Observation
HIRS 3 2 1
land +ocean
Channels below
rejected
SSMI
ocean: LWP=f(ch3/4)
If LWP >  cloud+rain

13. Dependent Screening Decision: Satellite Observation
ocean: LWP=f(ch1,ch2,ch3)
If LWP >  cloud+rain
land: if (ch1-ch3) >  rain  because of large emissivity over land no cloud contamination detection 1 3  Tb
no cloud
cloud
rain
AMSU-A 2 4 5 6

14. Independent Screening Decision: Satellite Observation
Background Quality control: ATOVS, SSMI, METEOSAT Background temperature, specific humidity and ozone profiles are checked to make sure they are close to or within the range for which the radiative transfer model is valid. Temperature is within the range K, specific humidity is positive and not supersaturated and the ozone is within climate extremes.
Radiance at the 2 extreme edge positions of the swath are not used in 4D-Var

15. Independent Screening Decision: Satellite Observation

16. Independent Screening Decision: Satellite Observation

17. Dependent Screening Decision: Conventional Observation
Before performing the dependent screening decisions, the flag information gathered so far is converted into a report status, namely active, passive, rejected or blacklisted i.e the RDB datum flag.
Vertical consistency of multilevel reports Duplicated levels are removed. If some consecutive layers are of suspicious quality they are rejected and for geopotential obs. also the layer above
Removal of duplicated report Search globally for duplicated (AIREP) data. Usually, there are aircraft reports that have the same date/time, roughly the same location and, at least partially, the same data. Usually, the station ID is slightly different.

18. Dependent Screening Decision: Conventional Observation
Redundancy check
Removes redundant SYNOP/PAOB :
co-located reports with the same station ID are searched for and only the closest active report to the analysis time is retained. For simultaneous reports, the one with more active data is retained. The same is done for reports that have equal time difference to the analysis time. Co-located SYNOP (mass obs) is redundant to TEMP (geop.) that is within 50 hPa
Removes redundant SHIP/DRIBU:
moving platforms within a circle of 1 ° are considered as potentially redundant: reports closest to the centre of the screening time with most active date are retained.

19. Dependent Screening Decision: SYNOP
4D-Var Screening for 4D-Var
The effect of the observation screening
on SYNOP surface pressure observations.
Column height gives the number of
observations available, while the shaded
part displays those actually used in the assimilation.
3D-Var Screening for 3D/4D-Var

20. Dependent Screening Decision: Conventional Observation
Level selection for TEMP/PILOT redundancies: for one time window rejection in layers around std-levels according the following priorities
maximum datum flag
time difference to analysis time
distance to the std-level
significant level (turning points of the sounding)
temp over pilot

21. Dependent Screening Decision: Thinning
AIREP thinning
horizontal thinning
from the same platform, a minimum distance between the nearby reports is enforced: Box km
vertical thinning
is performed around the model vertical levels: one aircraft measurement per model level.

22. Dependent Screening Decision: Thinning+
VERT increased number of AIREP
from 15 to 60 vertical levels
+HOR also increased number
of AIREP in horizontal

23. Screening Decision: Thinning
TEMP
GTS
Obs
thin

24. Dependent Screening Decision: Thinning
ATOVS thinning: a repeated scan is performed to get the observation resolution of 70 km (it depends on sensors/channels)
a sea sounding is preferred to a land one
clear sounding to a cloudy one
closeness of observation time to the centre of the screening time window
A second thinning takes place that selects one observation every 140km

25. Dependent Screening Decision: Thinning
The usage of ATOVS reports in the assimilation on the North Eastern Atlantic. Filled rings mark reports contain one or more channels used in the assimilation, whereas the empty rings denote rejected reports. Most of the rejections are due to the horizontal thinning and much less due to the quality reasons.Note that both edges of the swath are rejected.

26. Dependent Screening Decision: Thinning
SCAT:ERS-2
The process is defined with respect to the particular measurement geometry of the instrument. The backscatter data are acquired within individual cells related to a 450 km wide grid with a mesh of 25 km in the across and along track directions. 19 measurement nodes are thus defined across the scatterometer´s swath, numbered from 1 to 19 as the incidence angle increases, while 19 rows are also considered in the along track direction to gather the data in squares of 19 by 19 points. The thinning is then achieved by keeping only every fourth point within these squares. The data are thus used at a resolution of 100 km instead of the original 25 km sampling distance.
Sea-ice contamination < 273K
High wind rejection test > 25m/s
Normalized distance to the cone or wind residual. During the assimilation this quantity is computed and data rejected if a large value is found.

27. Dependent Screening Decision: Thinning
Triplet obs
..        ..
..     ..
..    
……....
check the distance
between observations
1-step
2-step

28. QuikSCAT Quadruplet obs Lars Isaksen Mark Leidner 50 km
Courtesy of Hans Hersbach
Lars Isaksen Mark Leidner
Quadruplet obs

29. Dependent Screening Decision: Thinning
High resolution trajectory
to choose only one

30. Dependent Screening Decision: Thinning
Rej: only 1 beam measur.
active
Minima are not well defined
Var-QC
White areas: insuf. info for MLE inversion e.g rain contamination

31. Screening Statistics Obs. Type Reports Active Passive Rejected
Synop
Airep
Satob
Dribu
Temp
Pilot
Satem
Paob
Scat
Obs. Type
28151
52868
217018
4802
605
805
249279
214
Reports
Active
Passive
Rejected
3933
22899
176518
3094 3 75
193416 76
Blacklisted
100699 7
154867 76
24218
29969
40500
1708
602
730
55863
138

32. After Screening Compression of CMA-file
After screening only 15% of all observations are active
10-20% TOVS left
40% Conventional left
The observation are resorted among the processors for a more optimal load balancing of the parallel computer

33. After Screening Global Array 1 Location ID 2 Time
Parallel computing environment
The observation screening should result in exactly the same selection of observations when different number of processors are used. Independent decision can be made at different processor fully in parallel. But global view of observation array is needed when a dependent decision has to be taken which implies that some communication between the processors is required. The observation array is too large to be copied in each individual processor then only a minimum necessary information is globally communicated
Global Array 1
Location ID 2
Time