1. Comparing Data Denial Trials to FSOI Results
Reduced Russian Radiosonde Reports
James Cotton1, Lawrence Morgan1 and Mary Forsythe1
with Rebecca Reid1, Bruce Ingleby1,2, Mark Rodwell2 & Lars Isaksen2
1Met Office, 2ECMWF

2. Introduction Russian Radiosonde Reduction
Description of Russian reduction
Russian Sonde denial results – ECMWF
What can we learn from FSOI results?
Coordinated data denial study of satellite data and radiosondes
FSOI Comparison to data denial results

3. Russian Radiosonde Network Reduction
The global radiosonde network is known to have large impact on global NWP
The Russian radiosonde program consists of 111 sondes (of ~800 globally)
Over Russia there is little info from aircraft or wind profilers
Satellite data usage limited due to large surface emissivity errors, particularly in winter
Variable quality of Russian sonde reports so exact impact unclear
Mix of types (some not recognised by WMO)
Heights determined by radar - increased uncertainty in upper levels
No wind profilers at all, some aircraft data, mostly near airports
Russian sonde reports are of varying quality, mix of types, including some not recognised by WMO.
Russian sonde heights determined by radar (no pressure sensors) – increases uncertainty in upper levels

4. Russian Radiosonde Network Reduction
In January 2015, the Russian met service, Roshydromet, cut their radiosonde program from two ascents per day to one.
Eastern Russian stations discontinued 00Z ascents,
western stations discontinued 12Z ascents
Monitoring of WMO blocks indicates a reduction from 214 ascents per day in Oct – Dec 2014 to 130 in Jan – Apr 2015 (averaged)
This reduction was expected to have significant impact on forecasts over Russia and potentially globally
Cut was from 1st of Jan, reinstated 1st of April

5. Russian Radiosonde Reduction – Data Denial

6. Russian Radiosonde Network Reduction Data Denial Trials – results at ECMWF
Data denial experiment conducted by ECMWF
Recreate effect of network reduction by blacklisting appropriate ascents in trial and comparing this to a control without the blacklisting
Focus on winter/early spring when snow cover reduces usage of mid-tropospheric information from Satellite microwave channels. (Mid-trop. infrared information not used over land throughout the year).
Trials run by ECMWF – Mark Rodwell, Bruce Ingleby, Lars Isaksen

7. Observed temperature counts per 12hr cycle at 500 hPa (2° gridboxes)
Russian Radiosonde Network Reduction Data Denial Trials – results at ECMWF
(Channel 5)
(Channel 215)
ECMWF
From assimilated data Dec 2014 – Feb 2015
Lack of satellite data over Russia
Aircraft data limited, close to airports
Observed temperature counts per 12hr cycle at 500 hPa (2° gridboxes)
AMSUA channel 5
AIRS channel 215

8. Top: Control RMSE values for Z500 at T+24, T+48 and T+120
Russian Radiosonde Network Reduction Data Denial Trials – results at ECMWF
Top: Control RMSE values for Z500 at T+24, T+48 and T+120
Bottom: Radiosonde cuts appear to lead to deterioration over the North Pacific, increasing with lead-time.
Combined trials of
– ,
ECMWF
Deterioration at T+24 is not negligible – likely to reflect the cumulative effect of reduction in Russian radiosondes over several analysis cycles
~4% increase

9. Russian Radiosonde Network Reduction Data Denial Trials – results at ECMWF
Largest impacts are centred on Russia and the Pacific stormtrack
Overall NH impact scores ~1.5% degradation
Equivalent to ~1/2 year of NWP development
½ year of NWP development statement based upon 10 years of upper-air scores

10. Russian Radiosonde Reduction – FSO Impacts

11. Russian Radiosonde Network Reduction FSOI Results
The Met Office adjoint-based Forecast Sensitivity to Observations (FSO) tool (Lorenc & Marriott, 2014) measures individual observations’ contributions to forecast error reduction
FSO impacts are computed simultaneously for each observation assimilated
Impacts measured using 24-hour energy norm with respect to self-analysis
Global moist energy norm with 150 hPa ceiling
Impact data were gathered for the periods
1 Jan – 31 Mar 2015, the period of reduction in radiosondes over Russia
1 Jan – 31 Mar 2014, for comparison
1 Apr – 30 Jun 2015, for comparison

12. Total FSO Impacts
Total FSO impact of global observations, grouped by observation type, for the period 1 January – 31 March, 2015
TEMP impact similar to Geo AMVs and Aircraft

13. Total FSO Impacts
Total FSO (global) impact of observations over Russia, grouped by observation type, for the period 1 January – 31 March, 2015
TEMP %
SYNOP 26 %
Note that this is with global norm, so impact is global.
“Russia” = area 50N-75 N, 30E-180E

14. Total FSO Impacts
Total FSO (global) impact of observations over Russia, grouped by observation type, for the period 1 January – 31 March, 2014
TEMP %
SYNOP 35 %
In reduction period
- Sonde and SYNOP compare similarly, but proportion of impact lower for both
Larger relative impacts from other observations
Expect Sonde to reduce, but why have SYNOP reduced too?
“Russia” = area 50N-75 N, 30E-180E

15. Mean FSO Impacts Mean Impact per Sounding (J/kg) Jan –Mar 2014
Period of Russian TEMP reduction
Mean Impact per Sounding (J/kg)
Jan –Mar 2014
Jan-Mar 2015
Apr-Jun 2015
TEMP Global
-8.6 x10-4
-8.0 x10-4
-6.7 x10-4
TEMP “Russia”
-8.9 x10-4
-10.9 x10-4
-7.3 x10-4
SYNOP Global
-9.3 x10-6
-8.1 x10-6
-9.2 x10-6
SYNOP “Russia”
-29.7 x10-6
-18.8 x10-6
-30.4 x10-6
METAR Global
-5.9 x10-6
-5.3 x10-6
-5.2 x10-6
META R “Russia”
-28.8 x10-6
-13.4 x10-6
-21.6 x10-6
During normal radiosonde launch periods
- Sondes over Russia have slightly higher mean impact/sounding than average global sonde
During period of Radiosonde reduction
Mean impact/sounding of remaining Russian sondes increases
Mean impact of SYNOP over Russia decreases
Mean impact of METAR over Russia decreases
“Russia” = area 50N-75 N, 30E-180E

16. Fractional Impacts & Numbers
Period of Russian TEMP reduction
“Russia” / Global %
Jan –Mar 2014
Jan-Mar 2015
Apr-Jun 2015
TEMP Number
14.7
9.2
15.4
TEMP Total Impact
15.3
12.5
16.9
SYNOP Number
8.0
8.1
SYNOP Total Impact
25.7
18.5
26.7
METAR Number
1.5
1.7
METAR Total Impact
7.5
3.9
7.0
During normal radiosonde launch periods
Russian sondes makeup 15% of global sondes, and contribute 15-17% of total sonde impact
Russian SYNOP makeup 8% of global SYNOP, and contribute 26-27% of total SYNOP impact
During period of Radiosonde reduction
Russian sondes makeup 9% of global sondes, and contribute 12% to total sonde impact
Russian SYNOP makeup 8% of global SYNOP, but now only contribute 18% of total SYNOP impact

17. Russian Radiosonde Network Reduction Summary
Russian sonde observations are important locally due to lack of other data, and remotely due to the initialisation of the Pacific stormtrack
The average Russian sonde ascent as/more valuable (as measured by FSO impact) than the average global sonde ascent.
During the period of Radiosonde reduction
The mean impact per ascent of the remaining Russian sondes increases (some redistribution) but fractional total impact reduces
Mean impact of SYNOP and METAR surface observations decreases
Degrading one part of the observing system (sonde) appears to harm the impact of another (surface)
Relevance of Russian reduction to larger TEMP/BUFR problems (next slide)

18. Transition to BUFR
Real-time radiosonde reports becoming available in increased vertical resolution, with metadata and precision exceeding that of alphanumeric TAC code.
Needed modernisation of radiosonde reports on global scale. However, change is complex and variety of networks and systems leads to complications
Alphanumeric transmissions are being withdrawn on adhoc basis with no clear timetable, sometimes after transmission of good new BUFR format has begun and sometimes not, sometimes after centres are able to process the new data and sometimes not.
Problems with implementation have led to loss of observing networks, e.g., Met Office temporary loss of ASAPs data assimilation during late 2015
Relevance of Russian reduction to larger TEMP/BUFR problems
Message of comparing network loss to BUFR transition, in danger of losing large numbers of observations for indeterminate periods

19. Coordinated Data Denial Experiments - Parallel Suite 37

20. Observing System Experiments (OSEs)
A coordinated set of OSE’s designed to give us a snapshot of impacts from PS37 observations and to analyse the consistency with Forecast Sensitivity to Observations Impacts (FSOI).
The following set of data denial experiments have been run:
Exp
Data Denied
Expt 1
No IR data (no IASI, CrIS, AIRS, HIRS or SEVIRI)
Expt 2
No MW data (no AMSU/MHS, ATMS, SSMIS, AMSR-2, Saphir, FY-3C)
Expt 3
No MW Humidity (no MHS, ATMS18-22, FY-3C, Saphir, SSMIS 9-11 & 12-16, AMSR-2)
Expt 4
No MW Imagers (no AMSR-2, SSMIS 12-16)
Expt 5
No Adv IR sounder humidity channels (AIRS, CrIS and IASI) and no HIRS 11,12
Expt 6
No AMVs
Expt 7
No GNSSRO
Expt 8
No Scat
Expt 9
No TEMPs
Expt 10
No Ground based GNSS
Baseline is a PS37 N320 control from 12 Nov - 15 Jan 2015/16

21. Impact Scorecards Versus Observations
No IR
No MW
No AMV
No Scat
No GNSSRO
No Sonde NH TR SH
Lead time

22. Impact Scorecards Versus Own Analysis
No IR
No MW
No AMV
No Scat
No GNSSRO
No Sonde

23. Impact Scorecards Versus ECMWF Analysis
No IR
No MW
No AMV
No Scat
No GNSSRO
No Sonde

24. OSE Impact Summary All data denial experiments behave as expected
Complimentarity
Radiances impact H500, PMSL
AMV/Scat winds impact winds (and H500)
Sonde main impact northern hemisphere
Radiances large impact in southern hemisphere
Discrepancy
Scatwinds W850 in tropics, show opposite impact verifying against own analysis (-ive) vs. ECMWF analysis (+ive)
Improved analysis of water vapour to make optimal use of temperature sounding channels

25. FSOI Comparison Forecast Sensitivity to Observations (FSO)
Measures the impact on 24-hour forecast error
Data denial Experiments
Percentage change in T+24 forecast RMS error
Mean of 6 variables
Northern hemisphere: PMSL, Wind 250 hPa
Tropics: Wind 850 hPa, Wind 250 hPa
Southern hemisphere: PMSL, Wind 250 hPa
H500 not considered for days 1-3 due to problem verifying against observations
Apparently large degradations in RMS errors at short range, relaxing to improvements at forecast day 3and beyond. Investigations showed this to be most likely due to the anomalously large role played by biases in the short range, coupled with the intrinsically large uncertainties in sonde geopotential height estimates, most evident at forecast days 1 and 2. .

26. FSOI Fractional Impact
RMSE OBS
RMSE ANL
RMSE EC
1. IR
2. MW
3. AMV
Sonde
5. Scatwind
6. GNSSRO

27. Denial T+24 RMSE vs OBS FSOI RMSE OBS RMSE ANL RMSE EC 1. IR 1. MW
3. AMV
Sonde
4. GNSSRO
5. Scatwind
6. GNSSRO
6. Sonde

28. Denial T+24 RMSE vs ANL
T+24 PMSL RMS error very small number in SH and NH
FSOI
RMSE OBS
RMSE ANL
RMSE EC
1. IR
1. MW
2. MW
2. IR
3. AMV
3. Sonde
Sonde
4. GNSSRO
GNSSRO
5. Scatwind
5. AMV
6. GNSSRO
6. Sonde
6. Scatwind

29. Denial T+24 RMSE vs EC FSOI RMSE OBS RMSE ANL RMSE EC 1. IR 1. MW MW
3. AMV
3. Sonde
Sonde
4. GNSSRO
GNSSRO
4. Sonde
5. Scatwind
5. AMV
5. GNSSRO
6. GNSSRO
6. Sonde
6. Scatwind

30. Conclusions: FSO vs Data Denial
Both FSOI and Data Denial show largest impact from advanced IR sounders and MW
FSOI suggests IR > MW
Denial suggests MW > IR
AMVs and Sonde next important (FSOI and denial), but low Sonde impact verified against observations
GNSSRO shows larger impact with data denial, but relatively low impact with FSO

31. Questions?

32. Global Radiosonde Network
Global radiosonde network has a large impact on global forecast errors (as measured by FSO statistics)
Data denial experiment of total TEMP loss shows reduction in NWP index of 1.35 against observations and 2.26 against analysis
Total loss of TEMP observations results in ~2.3% degradation as measured by forecast RMS

33. Russian Radiosonde Network Reduction Data Denial Trials – results at ECMWF
Reduction in radiosonde observations intended to mimic changes
Based on assimilated data – ,
ECMWF
Top images – Radiosonde counts per 2° grid box, per 12h cycle at 500 +/- hPa
Bottom images – Reduction numbers

34. At longer lead times the effect extends over the North Pacific
Russian Radiosonde Network Reduction Data Denial Trials – results at ECMWF
Top: Control RMSE values for T850 at T+24, Z500 at T+48 and Z200 at T+120
Bottom: Trial results indicate reduced quality of analysis as well as a poorer forecast.
At longer lead times the effect extends over the North Pacific
Combined trials of
– ,
ECMWF
Increase in RMSE values indicates deterioration in fc/analysis quality
~10% increase

35. FSOI Status
Producing Forecast Sensitivity to Observations Impacts (FSOI) from operational suite in NRT since January 2014
Adjoint-based method for estimating observation impact (Lorenc and Marriott, 2013)
Measure the impact on 24-hour forecast error in terms of a global, total (moist) energy norm calculated from the surface up to 150 hPa.
FSOI statistics also produced for latest parallel suite (PS37)
Participating in international FSOI inter-comparison study (Thomas Auligné, Ron Gelaro). Will be discussed at WMO impact meeting in Shanghai, May 2016.
The impact on forecast error due to the assimilation of observations can be measured using the adjoint-based Forecast error Sensitivity to Observations (FSO) method (Langland and Baker, 2004; Lorenc and Marriott, 2013). The adjoint method works by calculating a single, scalar measure of the reduction in forecast error due to each observation. The technique is efficient because all impacts are produced simultaneously and statistics can easily be aggregated in terms of the various satellite and conventional data types.
Lorenc, A. C. and Marriott, R. T. (2013) Forecast sensitivity to observations in the Met Office Global numerical weather prediction system. Q. J. R. Meteorol. Soc., 140, 678, pp

36. Impacts by Category Jan-Jun 2015
Hyperspectral IR = AIRS, IASI, CrIS
ATOVS+ATMS = AMSU-A, AMSU-B/MHS, HIRS, ATMS
Marine = Moored buoys, drifting buoys, ships, platform/rigs
Sondes = TEMP, TEMP SHIP, dropsondes, PILOT
Surface = SYNOP, METAR

37. Impacts by Obstype Jan-Jun 2015

38. Background fits to Adv IR – no MW humidity information
No MW humidity sounding data
Adv IR ‘temperature’ sounding channels
degraded due to degraded humidity fields
No MW imager data
This shows that when we deny humidity sensitive MW obs from the system,  the ‘temperature sounding’ channels of the IR are degraded.
This, we think,  is due to the sensitivity of these channels to water vapour continuum absorption.
The conclusion is:  you need to work on improved analysis of water vapour to make optimal use of temperature sounding channels.

39. New Observations at PS37 January 2016
SAPHIR on Megha-Tropiques
AMSR-2 on GCOM-W1
SSMIS on F17
MWHS-2 on FY-3C
DA changes: Use of correlated errors for CrIS, and VarBC for radiance data..

40. Total FSO Impacts
Total FSO impact of global observations, grouped by observation type, for the period 1 January – 31 March, 2014
TEMP obs are 3rd most important in terms of total impact globally