1. Armando DATA FILTERING PLAN v2 Tucson, AZ 6/30/11

2. Problem Statement QA Flags are not 100% accurate. Therefore depending on only this to filter the information, it results in low quality datasets where: a)Cloud-free pixels were removed (because QA labeled them as clouds). Amount of good data is reduced. b)Cloudy pixels considered as not cloudy. As a result low VI values are introduced into the Time series generating high variations and adding noise. QA is not enough, other elements must be included to enhance the filtering process.

3. Objective To generate an improved filtering algorithm that will use long term average information and heuristic techniques in addition to QA flags to effectively remove bad quality pixels. Adequately label each pixel on a scene according to quality information

4. Flow Chart Process Filtering Phase I: based on QA flags and NDVI START Calculate NDVI Calculate Long Term AVG Cloud Free Pixels Filtering Phase II: QA Flags, NDVI, LTAVG, Heuristics END

5. MODIS Flags and Data VariableData Cloud StateBit 0-1: 00 = Clear 01 = Cloudy 10 = Mixed 11 = not set, assumed clear Cloud Shadow?Bit 2: 1=Yes, 0=No Pixel adjacent to Cloud (Not used)Bit 13: 1=Yes, 0=No Aerosol QuantityBit 6-7: 00 = Climatology 01 = Low 10 = Average 11 = High Snow/IceBit 12: 1=Yes, 0=No Cirrus detected (Not used)Bit 8-9: 00 = None 01 = Small 10 = Average 11 = High View AnglePixel Information, Degrees (Solar Zenith Angle)

6. AVHRR Flags and Data VariableData Cloudy?Bit 1: 1=yes, 0=no Cloud Shadow?Bit 2: 1=yes, 0=no Aerosol QuantityN/A Snow/IceN/A (processing) View AnglePixel Information, Degrees (Solar Zenith Angle, View Zenith Angle)

7. Methodology A Pixel with Excellent quality is defined as a cloud free pixel, low aerosols content and view angle lower than 30 degrees. Calculate NDVI for whole dataset. – NDVI is used to better filter bad quality pixels. For example, there are pixels where QA Flags indicate values are cloudy or not valid data. However when analyzing the RED and NIR reflectance's, the pixel has an excellent quality. If pixels like those are removed, we are eliminating good quality pixels. Filter Dataset based on QA flags and NDVI Time Series. – Information from QA Flags and NDVI is used to determine if the pixel is cloud free or not. Pixels with high NDVI value are retained independently of the QA Flag information. Estimate Long Term Average (Remove lowest pixel value) from the Filtered dataset – Daily long term average is calculated for the excellent quality pixels. Standard deviation is stored too. Filter Input Dataset based on the Long Term Average, QA Flags, NDVI Values and heuristics – Information about the pixel is combined to enhance the classification of it. QA flags information and NDVI values is used to define if the pixel is a valid pixel or not. In addition, Long Term Average information is used to create a low/high boundary. If the pixel falls outside of the boundary, heuristic information is used to decide if the pixel should be used or rejected. – The heuristic approach will analyze the time series of the pixel. It will detect the variation of the pixel with respect of to the local time series segment. Clouds then to lower the NDVI values. If the low value of the pixel is because of noisy information, it will go back to the estimated value and it will be classified as cloudy. However if the pixel remains low for several days, it could because of a seasonal effect like drought, fire, cutting of thress, etc. Therefore this pixels is not considered cloudy. Create a Rank layer to reflect each pixel status. – Once that cloudy pixels have been identified, the rest of the pixels is ranked according to flow chart #2.

8. Decision Tree – Data Ranking Rank=5 IsCloudy Yes No START Cloud Shadow Vz<=30 Rank=1Rank=3 Yes No Rank=2 Data valid Yes No Rank =7 Rank=4 IsSnow Yes Low Aerosol No Yes No Low Aerosol & Vz<30 Rank=2 No Yes CODEDescription 1Ideal Data, Good and Useful 2Good to Marginal 3Marginal to Questionable 4Snow 5Cloud 6Estimated 7NO DATA