1. 21-23/04/2015PM27 J-L Vergely, J. Boutin, N. Kolodziejczyk, N. Martin, S. Marchand SMOS RFI/Outlier filtering

2. Historical reminder Filtering strategy at TB level before SSS retrieval. Comparison between SMOS TB and modelled TB at : -individual TB level (5 sigmas filtering) -snapshot level → if a snapshot is considered as contaminated, remove it. Test on the std and presence of outliers. -dwell line level Very difficult because RFI contamination or other contaminations can be very weak. It is not possible to detect/remove all the contaminated TBs. Last year conclusion : paradox → TB does not need to be removed before retrieval. It is easier to detect biased SSS from chi2P index.

3. Historical reminder V5 → TB filtering + chi2P selection V6 → no TB filtering; chi2P selection more efficient The lessons we have learned : 1/ It is more efficient to remove contaminated SSS than to remove contaminated TB. 2/ The TB contamination is, most of the time, not discontinuous. Contamination amplitude from very weak level to very high level. TB contamination with a weak amplitude is not detectable. 3/ Observation : if a dwell line is contaminated with TBs affected by high bias, it is contaminated by TBs affected by low bias. 4/ Land sea and RFI contamination : difficult to disentangle

4. V6 V5 Example of filtering SMOS Ascending dcoast>800km Better RFI/LS sorting

5. Example of filtering : sun contamination CHI2P checked CHI2P not checked V6 No TB outlier filtering V5 TB outlier filtering

6. Objectives Current objectives : to provide the users unbiased SSS using only OTT correction. → to be able to remove biased SSS from indicators like chi2P. New objective ? To correct biases. Correction at TB level (Land Sea correction) or/and at SSS level (L2/L3 correction).

7. Objectives If no chi2P filtering but coastal bias, RFI …etc corrected Finality : to produce SMOS SSS everywhere. SSS filtering from chi2P (and other flags) Finality : to produce unbiased SSS without correction (except OTT …).

8. THE QUESTIONS The change of paradigm requires a change in the strategy To reconsider the filtering at TB level ? Warning : if TBs are too filtered, no more TBs left for the SSS retrieval. Trade off has to be found. 4 cases : - to do nothing, - to filter, - to correct, - to filter + to correct At TB or SSS level ? Find the good strategy according to the grid point position and the time…

9. Objectives Try to map the kind of expected SSS biases and associated strategy. Example of the SM : to map the contaminated TBs/GP. SM dynamic map (monthly updated map). Similar strategy for SSS ? From CESBIO

10. Objectives Correction condition : bias must be stable in time. → verified condition most of the time in full ocean Proposition/question : if bias is stable in time : no TB filtering. if chi2P OK without TB filtering : OTT correction is sufficient ? If SSS still biased, ensure bias stability ? → to be checked from the reprocessing data. Is it possible to filter unstable biased TBs and keep stable biased TB ? SSS v5 (TB filtering) – SSS v6 (no TB filtering) Differences close to the coast, RFI regions, ice. SSS CORRECTION DEPENDS ON TB FILTERING

11. Introduction Diagnostic phase: Bias/outliers at visibilities, TBs or SSS level. Biases could have different origins : difficult to separate the different kind of biases. For example, coast bias and RFI bias are often mixed. Biases have different amplitudes : TB biases not necessary well marked → difficulty to find universal thresholds. Biases are stable or variable : intermittent biases. Spatial variation of the bias : latitudinal, coast …etc.

12. Introduction : biases at TB level Bias classification at TB level. Origin of the contaminations: Sun Approximative modelisation (dielectric constant, roughness model at high/low WS) RFI source detection/mitigation Coast reconstruction problem, polarization mixing, OTT biases. AUX data (GN, WS, SST, …etc) Instrument (calibration, …) Always difficult to find a bias at TB level because radiometric noise of the order of 2K. A bias of 0.1K generates 0.2 psu bias. To find a 0.1K bias, we need 400 TBs !!

13. Introduction : biases at TB level (OTT/LS correction) How to filter TB ? Current filter implementation could be used. The correction Delta TB = mean(Tb_smos – Tb_ref) Expected error on Delta TB for LS contamination : Radiometric Accuracy / sqrt(50) ~ 0.3 K Indicator of the LS stability correction std(Tb_smos – Tb_ref). If the bias is stable std(Tb_smos – Tb_ref) = Radiometric Accuracy. -> if std(Tb_smos – Tb_ref) > threshold : problem in the correction ? Test to be used to filter the TB before the retrieval. -> propagate the error on Delta TB toward the TB covariance matrix before the retrieval. Not needed if std(Tb_smos – Tb_ref)/ra << 1.

14. Introduction : biases at SSS level Bias classification at SSS level: SSS error of 0.7 psu. To reach 0.2 psu, we need 12 SSS. Problem : SSS comes from a TB mixing. SSS bias can carry different kind of biases. SSS bias classification: to define a peculiar strategy for each class of biases. How to make a diagnostic ? From self consistency analysis.

15. Method Outlier detection Use of 4 year SMOS data. Detection a posteriori of the bias classes. It will help us to find way to detect bias during the processing. Coast problem : → mixing of biases coming from reconstruction problem and RFI contamination. Need of an external reference (climatology) ? No. Possibility to do a self consistency analysis.

16. Self consistency analysis Juin 2011 Juin 2012 Juin 2013 Juin 2014 SSS (psu) RELATIVE BIAS ISAS ABSOLUTE BIAS SSS bias depends on the x swath position Estimation of the RELATIVE BIAS using a least square criterion X swath For each grid point

17. Self consistency analysis 32 ascending dwells 32 descending dwells before correction after correction month OUTLIER DETECTION

18. Products (from v550) -Relative corrections to be done at each dwell line and each GP. Outlier flags. -Absolute correction to be done at each GP : average ISAS v6.2 (from MyOcean) on 4 years (07/2010-07/2014). -Corrected SSS on 4 years with a 20 days smoothing length and a 8 days smoothing length. -Std/outlier/chi2 map (spatial) at different scales (50 km, 100 km)

19. Correction from self consistency analysis Relative correction map. Dwell at X_swath=375 km

20. Correction from self consistency analysis FULL OCEAN EXAMPLE CORRECTION LS CORRECTION BIAS CORRECTION : TO BE DONE EVERYWHERE

21. Absolute correction (SMOS – ISAS) : 4 years averaging RFI at least partially corrected Corrected SSS (2/8/2011)Corrected SSS – ISAS (2/8/2011)

22. Outlier detection using 3 sigma filtering (from time series self consistency analysis)

23. 4 years SMOS outliers versus dwell 21/23-04-2015PM 27 If outliers because of SSS natural variability (rain, SSS front …etc → FLAT RESPONSE IS EXPECTED and ASC=DESC.

24. 4 years SMOS outliers versus dwell 21/23-04-2015PM 27 Spatial repartition ASCDESC Larger number of outliers in the north hemisphere for descending orbits

25. Self concistency of dwells asc/desc chi2map_hebd_R1 High chi2 (from self consistency analysis) could be due to : 1/ SSS natural variability on time scale < 1 week 2/ variability of the bias

26. std SSS ISAS versus std SSS SMOS : natural variability 21/23-04-2015PM 27 ISAS SSS natural variability which depends on ARGO coverage… Close to SMOS SSS natural variability. Not very natural… std SSS ISAS (4 years) SMOS (4 years)

27. SSS spatial fluctuations Proposition : if the variations of the SSS spatial standard deviation (the std(std)) is weak, this means that the fluctuations are stable from one month to the other, even if the std is high. Limit : strong natural variability SSS spatial std (R=100km) 2/08/2011SSS spatial std(std) (R=100km) 4 years Without 3 sigma filtering Intermittent RFI

28. SSS bias classification (proposition) Hypothesis : TB correction. OTT correction for full ocean (time dependent) LS correction (time independent) 1/ SSS outliers. SSS after TB bias correction which are sporadically and significantly far from the expected SSS. 2/ bias with small fluctuations. Correction is required and theoretical error < true error. 3/ SSS stable biases : biases well corrected with weak variations absorbed by OTT / LS correction. 4/ intermittent biases : biases which change of amplitude on long period. How to correct them ? Is it possible to distinguish these biases ?

29. SSS outliers. Characteristics : SSS (from corrected TBs) which are sporadically and significantly far from the expected SSS (neighboring SSS in time and in space, climatology). Identification : GP probability to be affected by such outlier. SSS values (in comparison with expected SSS). Possibility to detect them at the end of the retrieval. Tool : Map of outlier probability. Map of mean SSS updated every month. SSS climatology. Processing : 3 sigma detection (~2.5 psu). Attempt a new SSS retrieval after TB filtering ? Flag. Limit : to be detected only if low/medium natural variability.

30. SSS bias with small fluctuations Characteristics : SSS affected by additive noise and bias. Identification : SSS with a larger dynamic than expected. Tool : Map of chi2 time series which shows SSS discrepancies. Processing : to increase the theoretical error. Flag. Where ? Coast bias (mixed with RFI) or AUX data biases (WS, SST …). Limit : Detectable only if low natural variability.

31. SSS stable bias Characteristics : SSS from stable TB corrections Identification : SSS with self consistency signature which shows low discrepancies. Tool : Map of chi2 time series. Processing : check the GP according to the chi2 time series. Where ? Full ocean with well known AUX data, forward model... Limit : Detectable only if low/medium natural variability.

32. SSS intermittent biases Characteristics : biases which change of amplitude on more or less long periods. Difficult to correct them without introducing strong a priori. Map of expected chi2 or chi2P. Identification : variation of the spatial SSS standard deviation. Variations of chi2P or other flags. Processing : change the correction ? Flag. If intermittences with high amplitudes : outlier detection. Where ? RFI biases (north Atlantic, Reunion island), ice boundaries, GN biases, L1 calibration biases (“Eclipse“ biases), RIMPAC …

33. Strategy GP : SMOS TB (dwell) SMOS corrected TB OTT/LS correction SMOS corrected and filtered TB TB filtering (1) Retrieved SSSSSS quality TB filtering (2) SSS estim. SSS flags SSS error OK NOK SMOS corrected and filtered TB SSS flags TB filtering (1) : -Current filtering (L1c flags, footprint …) -Nsig test (basic outlier detection) -std test (on LS correction stability) with threshold SSS quality : -Compare SSS estimation with climatology using SSS natural variability and SSS theoretical error -Check the chi2 time series value : update SSS theoretical error accordingly -Check other indicators (chi2P, …) ? TB filtering (2) : -Use current filter to remove contaminated TBs -New approach to be implemented ?

34. Strategy Impact of the strategy on the different SSS classes. SSS outliers : the SSS outliers are detected from the SSS quality module. The second iteration should allow obtaining better SSS. The number of SSS outliers will decrease. The impact of the correction is uncertain. SSS bias with small fluctuations : the SSS theoretical error should be increased. Done by the SSS quality module. SSS stable bias : TB well corrected (OTT and LS correction) with good indicators from SSS quality module. SSS intermittent biases : the impact of the correction is uncertain. To be flagged. Which indicators (?).

35. Conclusions/propositions Detection of outliers : strategy of filtering. Detection at TB or SSS level. At TB LS correction level : Find a way to extend the correction to the whole map ? Add the information of the std (TB smos – TB model) : –Add a filtering from a threshold applied to the correction stability indicator –Error propagation : no From SSS : Use the probability to have outliers obtained from self-consistency analysis: flagged GP. Strategy should depend on expected natural variability : provide a map. Possibility to compare retrieved SSS with a climatology : if retrieved SSS is significantly different -> add a TB filtering stage and do the SSS retrieval again. Build a dynamic map for the outlier detection. For validation : identify specific area where systematic investigations could be done. North Atlantic, Reunion island, Amazon plume, a region in full ocean (without contamination issue). Process time series for analysis.

36. Extra slides 21/23-04-2015PM 27

37. Max spatial std SSS 21/23-04-2015PM 27

38. SSS self consistency std biais dwell Max -min biais dwell

39. SSS self consistency mean biais dwell

40. Intermittent RFI:determination 21/23-04-2015PM 27 Method : Diagnostic using time series behaviour. If the fluctuations of the spatial SSS encrease statistically in some place : unexpected behaviour. Relative increase : some time series present strong level of fluctuations. Normalisation using the median level of fluctuations.

41. Intermittent RFI :low 21/23-04-2015PM 27 SSS spatial std SSS time std

42. Intermittent RFI 21/23-04-2015PM 27

43. Intermittent RFI 21/23-04-2015PM 27 SSS spatial std SSS time std

44. North Atlantic 21/23-04-2015PM 27