The Aerospace Corporation (Aerospace) Prelaunch Assessment of the Northrop Grumman VIIRS Cloud Mask Thomas Kopp, The Aerospace Corporation Keith Hutchison, Northrop Grumman Andrew Heidinger, NOAA/STAR Richard Frey, University of Wisconsin IGARSS 25 July 2011

2 Meteorological Satellite Systems Outline Definitions of VIIRS Cloud Mask (VCM) contents and validation conditions High level review of the VCM logic Global results with the pre-launch VCM without any tuning Quantitative Improvements Using the Northrop Grumman (NG) tuning tool Methods for evaluating individual granules during Intensive Cal/Val (ICV) of the VCM

3 Meteorological Satellite Systems VCM Contents The VCM itself determines one of four cloud cover conditions for each pixel –Confidently Cloudy –Probably Cloudy –Probably Clear –Confidently Clear All downstream EDR products, except for imagery, require the VCM as an input Downstream products will use either the confidently cloudy or confidently clear condition –The probably clear/cloudy cases account for pixels that are not completely cloud covered but due either to the difficulty of the scene or partial clouds such as cumulus, are not sufficiently clear to reliably determine the conditions at the surface

4 Meteorological Satellite Systems VCM Performance Metrics Probability of Correct Typing (PCT ) − PCT = (1 - Binary Cloud Mask Error) = {1 – [(VCM = conf. clear) & (Truth = conf. cloudy) OR ((VCM = conf. cloudy) & (Truth = conf. clear)]/[total #pixels in each geographic class – PCPC] Cloud Leakage (CL) − CL = [(VCM = conf. clear) & (Truth = conf. cloudy)]/total #pixels in the geographic class False Alarm Rate (FA) − FA = [(VCM = conf. cloudy) & (Truth = conf. clear)]/total #pixels in the geographic class Fraction of Pixels Classified as Probably Clear/Cloudy (FPCPC) –FPCPC = [(VCM = prob. clear) or (VCM = prob. cloudy)]/total #pixels in the geographic class

5 Meteorological Satellite Systems Overview of VCM Approach There are five possible processing paths in the VCM algorithm for the analysis of SDR data collected in daytime conditions Cloud Spectral Tests Used to Determine Cloud Confidence WaterLandDesertCoastSnow 1.M9 (1.38 µm) Reflectance TestXXX (if TPW > 0.25 cm) XX 2. M15-M16 (10.76 – µm) Brightness Temperature Difference (BTD) XXXX 3.Tri-Spectral M14, M15, M16 (8.55, 10.76, µm) BTD Test X 4.M15-M12 ( µm) BTD Test X (if no sun glint) X (if TOC NDVI > 0.2) X (if Lat > 60  or < - 60  ) X (if no sun glint and if TOC NDVI > 0.2) X 5.M12-M13 ( µm) BTD Test X (if –60  < Lat < 60  ) and no sun glint X (if –60  0.2 X (if –60  < Lat < 60  ) 6.M1 (0.412 µm) Reflectance TestX (if –60  < Lat < 60  ) 7.M5 (0.672 µm), M1 (0.412 µm) Reflectance Tests X (M5 if TOC NDVI ≥ 0.2; M1 otherwise) 8.M7 (0.865 µm) Reflectance TestX 9.M7/M5 (0.865 / µm) Reflectance Ratio Test XX (if RefM5 ≥ LD_M5_Gemi Thresh) Cloud Spatial Tests Used to Modify the Final Cloud Confidence Classification WaterLandDesertCoastSnow 10.I5 (11.45 µm) Spatial TestX 11.I2 (0.865 µm) Reflectance TestX

6 Meteorological Satellite Systems Pre-launch, Pre-tuned Global VCM Results VCM version used from 2009 The initial thresholds were used, the VCM for this testing was not tuned Comparisons made with collocated MOD35 C6 cloud mask and CALIOP matchups for comparison – Cloudy for the VCM in this case included probably cloudy pixels – Clear for the VCM in this case included probably clear pixels Compared only 1-km CALIOP segments with either 0% or 100% cloud cover –Resulted in approximately 15 million collocations per month

7 Meteorological Satellite Systems Caveats and Notes to Global VCM Results Results intended to show “where we were” in late 2009 Neither of the two I-band tests could be simulated using the proxy data, a significant source of error that will not be quantified until the post-launch validation of the VCM Thin cirrus has a major impact on the results Analysis limited to near-nadir views (MODIS viewing zenith angle of +/- 20 degrees) Hit rate = (# agree cloud + # agree clear) / total # Hanssen-Kuiper Skill Source (HKSS) = (# agree cloud * # agree clear) – (# disagree cloud * # disagree clear) / (# agree clear + # disagree clear) * (# agree cloud + # disagree cloud) Results follow on the next few slides

8 Meteorological Satellite Systems VCM versus CALIOP, Global Results

9 Meteorological Satellite Systems VCM versus CALIOP, Polar and Non-Polar Results

10 Meteorological Satellite Systems VCM versus CALIOP, Day Results

11 Meteorological Satellite Systems VCM versus CALIOP, Night Results

12 Meteorological Satellite Systems VCM versus CALIOP, Land and Desert Results

13 Meteorological Satellite Systems VCM Designed to Exploit VIIRS 1.38-µm Data MODIS vs VIIRS RSRs MODIS vs VIIRS TOA Radiances VIIRS OOB Response is orders of magnitude less MODIS OOB Response is as large as the in- band response Thin cirrus clouds will be more readily detected with VIIRS data than in MODIS

14 Meteorological Satellite Systems VCM Versus Heritage Performance, COT > 1.0 VCM and heritage performance are comparable when thin cirrus clouds are eliminated from the results

15 Meteorological Satellite Systems One Year Means of Hit Rates and Skill July 2007 – June 2008, Comparison with CALIOP Scene Type Mean VCM Hit Rate Mean MOD35 C6 Hit Rate Mean VCM HKSS Mean MOD35 C6 HKSS Global S-60N Global day Global night S-60N Water day S-60N Water night S-60N Land day S-60N Land night

16 Meteorological Satellite Systems Pre-Launch Tuning Approach Pre-launch tuning is based on 14 granules which employed Global Synthetic Data (GSD) – Of these 14, 11 contained land backgrounds These granules covered each VCM geographic type and ranged from straightforward to difficult scenes GSD provides unique data to set the mid-point thresholds – Typical methods of tuning, using on-orbit sensor data, rely upon 100% cloudy and 100% cloud free distributions – GSD alone allows cloud distributions to be evaluated at the mid-point (50% cloudy) condition GSD allows setting thresholds and then minimize the distance between the confidently cloudy and confidently clear thresholds

17 Meteorological Satellite Systems Advantages Using Global Synthetic Data (GSD) GSD allows testing of the tuning process with proxy data and then apply the procedure to VIIRS-unique data – Tuning process is validated by : – (1) tuning with GSD truth data developed with MODIS Relative Spectral Responses (RSR) – (2) tested in the VCM using MODIS granules – (3) quantitatively evaluated using manually-generated cloud data of the MODIS data – Tuning for VIIRS data is then completed by examining changes in cloud distribution for each test in GSD truth data using the VIIRS RSR

18 Meteorological Satellite Systems Initial VCM Results Showed Following Needs Reduce the number of probably clear and probably cloudy (PCPC) classifications by adjusting the overall cloud confidence threshold Identify tests that generated the highest percentage of false alarms for each VCM background condition and tune the mid-point thresholds (i.e. 50% cloud cover condition) accordingly – only possible with GSD Further reduce the number of PCPC classifications, as necessary, by adjusting the distance between the mid-point thresholds of a given individual cloud test and the low and/or high threshold using cloud distributions in the GSD.

19 Meteorological Satellite Systems Initial Untuned VCM Performance - Land Granule ID _ _ _ _ _ _ _ _ _ _ _ 1840Summary land nPoorQual nCldTruth nClrTruth nConfCldy nConfClr nPrbCldy nPrbClr FalseAlarms Leakage BinaryError FPCPC pFalseAlarms pLeakage pBinaryError PCT

20 Meteorological Satellite Systems Overview of the Pre-Launch Tuning Process Identify the tests causing the largest number of errors Use GSD with MODIS RSRs to generate cloud cover distributions for the cloud detection tests identified above – Generate distributions for 0%, 50%, and 100% cloud cover – Set key mid-point threshold using the 50% cloud cover, then minimize low- and high thresholds Update VCM using these thresholds Execute the updated algorithm on the set of MODIS granules Evaluate the performance using the manually generated cloud masks Assess the changes in performance

21 Meteorological Satellite Systems Example for a Case With Too Many PCPC Pixels MODA Manually-Generated Mask Q thresh = 99% Land - Pre FPCPC0.39 pFalseAlarms0.07 pLeakage0.00 pBinaryError0.07 PCT0.93 Q thresh = 90% Land - Post FPCPC0.14 pFalseAlarms0.05 pLeakage0.01 pBinaryError0.06 PCT0.94

22 Meteorological Satellite Systems Specific Cloud Detection Case, GEMI Test (Land) Changed from 1.95 to 1.87Changed from 1.90 to 1.82Changed from 1.85 to 1.78

23 Meteorological Satellite Systems Quantitative Impacts – GEMI Results M7/M5 Land Summary nPoorQual nCldTruth nClrTruth nConfCldy nConfClr nPrbCldy nPrbClr FalseAlarms Leakage BinaryError FPCPC pFalseAlarms pLeakage pBinaryError PCT Previous untuned results FPCPC pFalseAlarms pLeakage pBinaryError PCT

24 Meteorological Satellite Systems Tests Improved by the Pre-Launch Tuning Effort Reflectance test over desert (M1) Reflectance test over land (M5) Reflectance test over water (M7) Ratio test over land (GEMI) Ratio test over water (M7/M5) Mid-Wave minus long wave infrared over snow (M12 – M15) Mid-Wave infrared difference over snow (M12 – M13)

25 Meteorological Satellite Systems Performance After Daytime Tuning Completed Performance Measure LandOceanDesertSnow PCPC SysSpec % % % % False Alarms SysSpec Leakage (%) SysSpec PCT (%) SysSpec Performance Measure LandOceanDesertSnow PCPC SysSpec % % % % False Alarms SysSpec Leakage (%) SysSpec PCT (%) SysSpec Untuned VCM: March 2010 Tuned VCM: November 2010

26 Meteorological Satellite Systems Tool for Visualization of the VCM The previous analyses reveal quantitative aspects of the VCM, but lack context Historically the capability to visualize the output from each individual cloud detection test has been used operationally at the Air Force Weather Agency Key to a useful visualization are two fundamental factors – It must overlay each test on applicable imagery – It must contain the reflectance/brightness temperatures used within the cloud mask This reveals if any bands have bad or saturated values The visualization should also note if any degraded conditions of note exist in the scene – These include aerosols, sun glint, and shadows The following pair of slides show this capability

27 Meteorological Satellite Systems Aerospace Visualization Tool – Example I

28 Meteorological Satellite Systems Aerospace Visualization Tool – Example 2

29 Meteorological Satellite Systems Conclusion Pre-launch validation of the VCM uses three different approaches to verify the VCM will meet expectations – Large scale quantitative analysis – Small scale quantitative analysis via GSD – Visualization of individual granules with each component cloud detection test Results show promise that the VCM will meet or exceed its requirements Each of these methods will be employed in some form post-launch, though we will no longer need GSD as actual VIIERS data will be available