1. ECMWF/EUMETSAT NWP-SAF Satellite data assimilation Training Course 1 to 4 July 2013

2. ECMWF/EUMETSAT NWP-SAF Satellite data assimilation Training Course The detection and assimilation of clouds in IR radiances

3. Task 1: Overlay clear and cloudy Tropical IASI spectra Duplicate Tropical profile ( tropic.nc > copy_1_tropic.nc) Analyse copy_1_tropic.nc change cloud parameters (2D) Edit / visualise IASI SPECTRUM (with tropic.nc ) Edit / drop IASI SPECTRUM (with copy_1_tropic.nc)

4. Task 1: Overlay clear and cloudy Tropical IASI spectra

5. Option 1: Detect and reject cloud contaminated observations Option 2: Explicitly estimate cloud parameters from the radiances within the data assimilation (simultaneously with T,Q,O3 etc..)

7. Cloud detection methods Simple window channel departure checks Co-located imager checks Pattern recognition algorithms Hybrid systems

8. Cloud detection methods Simple window channel departure checks Co-located imager checks Pattern recognition algorithms Hybrid systems

9. Window channels have the highest sensitivity to cloud

10. Clear population Cold departures indicating cloud contamination in OBS Observed radiance at 11 um minus radiance calculated from background in clear sky

11. Y obs – H(X clear ) > dT threshold

12. …results in a asymmetric population surviving the check …!

13. Cloud detection methods Simple window channel departure checks Co-located imager checks Pattern recognition algorithms Hybrid systems

14. AVHRR imager pixels IASI field of view

15. Homogenous clear Homogenous cloudy Mixed cloud scene

16. AVHRR standard deviation AVHRR mean radiance 220K 250K 280K 300K Scatter stdv versus mean Tb

17. AVHRR standard deviation AVHRR mean radiance 220K 250K 280K 300K Homogenous clear

18. AVHRR standard deviation AVHRR mean radiance 220K 250K 280K 300K Homogenous cloudy

19. AVHRR standard deviation AVHRR mean radiance 220K 250K 280K 300K Mixed scene

20. Cloud detection methods Simple window channel departure checks Co-located imager checks Pattern recognition algorithms Hybrid systems

21. Zoom in …

22. Clear spectrum Cloudy spectrum

23. Break point

24. Cloud at 200hPa Cloud at 500hPa Cloud at 400hPa Zero Cloud Break point depends on cloud height

25. observed-calculated (K) Vertically ranked channel index First filter and find the break point Observed spectrum minus clear sky computed spectrum

26. observed-calculated (K) Vertically ranked channel index First filter and find the break point Observed spectrum minus clear sky computed spectrum

27. observed-calculated (K) Vertically ranked channel index Break point Channels rejected Channels retained

28. CLOUD AIRS channel 226 at 13.5micron (peak about 600hPa) AIRS channel 787 at 11.0 micron (surface sensing window channel) temperature jacobian (K) pressure (hPa) unaffected channels assimilated contaminated channels rejected

29. Cloud detection methods Simple window channel departure checks Co-located imager checks Pattern recognition algorithms Hybrid systems

30. Why do we need Hybrid systems ? Y obs – H(X clear ) > dT But sometimes large errors in the background can lead to: False rejection of a good observation Missed rejection of a bad observation

31. Option 1: Screen the radiance data and reject cloud/rain contaminated observations Option 2: Explicitly estimate cloud parameters from the radiances within the data assimilation (simultaneously with T,Q,O3 etc..)

32. It looks very difficult …why would we try to do this ?

33. CLOUD AIRS channel 226 at 13.5micron (peak about 600hPa) AIRS channel 787 at 11.0 micron (surface sensing window channel) temperature jacobian (K) pressure (hPa) unaffected channels assimilated contaminated channels rejected We have to throw away lots of data !!!

34. Sensitive areas and cloud cover Location of sensitive regions Summer-2001 (no clouds) monthly mean high cloud cover monthly mean low cloud cover sensitivity surviving high cloud cover sensitivity surviving low cloud cover From McNally (2002) QJRMS 128

35. …so what is required to do this ?

36. The cost function J (X) model state observations background error covariance observation* error covariance observation operator (maps the model state to the observation space) If we wish to assimilate cloudy radiance observations …..

37. The cost function J (X) model state must include clouds (clw,cic,cf)

38. Two approaches to assimilate cloud affected infrared radiances Simplified system: very simple cloud representation currently limited to overcast scenes no information on clouds taken from model no back interaction with model via physics Advanced system: very complex cloud representation all cloud conditions treated information on clouds taken from model back interaction with model via physics X=(T,Q,V,ciw,clw,cc) X=(T,Q,V,cp,cf) cp cf

39. The cost function J (X) Background error covariance must include clouds (clw,cic,cf)

40. The cost function J (X) Obervations operator (RT and Model) must include clouds (clw,cic,cf)

41. Potential difficulties The cloud uncertainty in radiance terms may be an order of magnitude larger than the T and Q signal (i.e. 10s of kelvin compared to 0.1s of kelvin) The radiance response to cloud changes is highly non- linear (i.e. H(x) = H x (x)) Errors in background cloud parameters provided by the NWP system may be difficult to quantify and model Conflict between having enough cloud variables for an accurate RT calculation while limiting the number of cloud variables to those that can be uniquely estimated in the analysis from the observations

42. Clear population Cold departures indicating cloud contamination in OBS Observed radiance at 11 um minus radiance calculated from background in clear sky

43. Potential difficulties The cloud uncertainty may be an order of magnitude larger than the T and Q signal (i.e. 10s of kelvin compared to 0.1s of kelvin) The radiance response to cloud changes is highly non- linear (i.e. H(x) = H x (x)) Errors in background cloud parameters provided by the NWP system may be difficult to quantify and model Conflict between having enough cloud variables for an accurate RT calculation while limiting the number of cloud variables to those that can be uniquely estimated in the analysis from the observations

44. surface full cloud at 500hPa dR/dT 500 = 0 dR/dT* = 1 dR/dT 500 = 1 dR/dT* = 0 Clear and Cloudy Jacobians (impact at the cloud top)

45. Potential difficulties The cloud uncertainty may be an order of magnitude larger than the T and Q signal (i.e. 10s of kelvin compared to 0.1s of kelvin) The radiance response to cloud changes is highly non- linear (i.e. H(x) = H x (x)) Errors in background cloud parameters provided by the NWP system may be difficult to quantify and model Conflict between having enough cloud variables for an accurate RT calculation while limiting the number of cloud variables to those that can be uniquely estimated in the analysis from the observations

46. Observed radiance at 11 microns minus radiance calculated from NWP cloud background profile Many clouds with significant radiance signals are accurately represented by the NWP model and RT modelled !

47. Potential difficulties The cloud uncertainty may be an order of magnitude larger than the T and Q signal (i.e. 10s of kelvin compared to 0.1s of kelvin) The radiance response to cloud changes is highly non- linear (i.e. H(x) = H x (x)) Errors in background cloud parameters provided by the NWP system may be difficult to quantify and model Conflict between having enough cloud variables for an accurate RT calculation while limiting the number of cloud variables to those that can be uniquely estimated in the analysis from the observations

48. Choice of cloud parameters and ambiguity with T and Q P n T T P n A very simple cloud model (e.g. single layer grey cloud amount and pressure) should more readily estimated from the data (independently of T and Q), but will make the forward RT calculation very inaccurate in many cloud conditions A more complex cloud model (e.g. cloud liquid and ice at each model level) will allow a more forward RT calculation, but may be difficult to estimate independently of T and Q and may alias into erroneous increments

49. Two approaches to assimilate cloud affected infrared radiances

50. Simplified system: very simple cloud representation currently limited to overcast scenes no information on clouds taken from model no back interaction with model via physics Advanced system: very complex cloud representation all cloud conditions treated information on clouds taken from model back interaction with model via physics X=(T,Q,V,ciw,clw,cc) X=(T,Q,V,cp,cf) cp cf

51. Experience with simplified Cloudy IR Radiance Assimilation ( only overcast conditions used ) X=(T,Q,V,cp,cf) cp cf See McNally (2009) QJRMS 135, 1214-1229 for more details…

52. Why overcast scenes…?

53. The cost function J (X) model state has only one extra variable = cloud top height

54. Why use cloudy radiances only in overcast conditions ? Overcast clouds are least ambiguous in the radiance data Cloud control vector collapses to a single number (cp) Problems with cloud overlap assumptions vanish No cross-talk between cloud and surface variables Termination of jacobians at cloud top provides new high vertical resolution Information on temperature

55. Impact of overcast data on the ECMWF analysis…

56. surface full cloud at 500hPa dR/dT 500 = 0 dR/dT* = 1 dR/dT 500 = 1 dR/dT* = 0 Clear and Cloudy Jacobians (impact at the cloud top)

57. Temperature increments above low clouds Overcast data coverage from HIRS, AIRS and IASI and the estimated cloud top pressure

58. Temperature increments above low clouds Analysis temperature increments at 700hPa Overcast data coverage from HIRS, AIRS and IASI and the estimated cloud top pressure

59. Temperature increments above high clouds Overcast data coverage from HIRS, AIRS and IASI and the estimated cloud top pressure

60. Temperature increments above high clouds Analysis temperature increments at 250hPa Overcast data coverage from HIRS, AIRS and IASI and the estimated cloud top pressure

61. …do these extra increments at the cloud top do any good..?

62. Improved analysis fit to isolated radiosonde observations Monthly averaged RMS temperature increment difference (CLOUDY minus CTRL). Shaded areas indicate a reduction in increments in excess of 0.1K when the cloud radiances are assimilated.

63. …forecast impact...?

64. Cloud obscured singular vector ? In this case the use of overcast observations resulted in analysis differences in an area suggested to be sensitive by the singular vector locations 500hPa temperature analysis difference (K) ? CNTRL CLOUDY Location of leading 500hPa singular vectors SH 500hPa Z Extra overcast data used compared to CTRL

65. Towards an Advanced Cloudy IR Radiance Assimilation

66. (T,Q,V)(T,Q,V) model physics (M) (T,Q,V,ciw,clw,cc) Cloudy (RT) (R cal ) Jo=(R obs -R cal ) Cloudy (RT) * model physics (M*) (T,Q,V,ciw,clw,cc)* (T,Q,V)* (R cal )* Cloudy radiances R cal are simulated via a chain of forward operators (M,RT). The fit of the analysis to the observations is computed (Jo) Jo is minimized by perturbing the analysis variables according to gradients from a chain of adjoint operators (RT*,M*)

67. …not yet… (at least for the infrared)

68. Adjusting analysis variables to fit the cloudy radiance observations

69. End