Data Quality Screening Service Christopher Lynnes, Bruce Vollmer, Richard Strub, Thomas Hearty Goddard Earth Sciences Data and Information Sciences Center Robert Wolfe, Neal Most, Ali Rezaiyan-Nojani MODIS Adaptive Processing System Peter Fox, Stephan Zednik, Patrick West Tetherless World Constellation, RPI Data Quality: Why So Difficult? Cloud Mask Status Flag 0=Undetermined 1=Determined Cloud Mask Cloudiness Flag 0=Confident cloudy 1=Probably cloudy 2=Probably clear 3=Confident clear Day/Night Flag 0=Night 1=Day Sunglint Flag 0=Yes 1=No Snow/ Ice Flag 0=Yes 1=No Surface Type Flag 0=Ocean, deep lake/river 1=Coast, shallow lake or river 2=Desert 3=Land Bitfield arrangement for Cloud_Mask_SDS variable in atmospheric products from Moderate Resolution Imaging Spectroradiometer Satellite data quality schemes range from simple (left) to fairly complicated (right) Quality level for total precipitable water Hurricane Ike, viewed by the Atmospheric Infrared Sounder (AIRS) Highest pressure of “Best” quality values in moisture profiles Quality flags are also sometimes packed together into bytes Repeat for each user Current user scenarios...Current user scenarios... Nominal scenario  Search for and download data  Locate documentation on handling quality  Read & understand documentation on quality  Write custom routine to filter out bad pixels Equally likely scenario*  Search for and download data  Assume that quality has a negligible effect *Especially outside the core community of experienced researchers Using Bad Quality Data Can Affect Your Results Total Column Precipitable Water Quality BestGoodDo Not Use kg/m 2 The unusually dry pixels on the east side of Hurricane Ike (arrow) have “Do Not Use” quality flags. Visualizations help users see the effect of different quality filters Best Quality Only Best + Good Quality User scenario – Search for data – Default Case: Select science team recommendation for quality screening – Advanced User Case: Select custom quality screening parameters – Download screened data Java implementation supports distribution across multiple sites Screening takes place at the data center where the data reside to avoid excessive data transfers Data Quality Screening Service Implementation How DQSS Works End Product Collaborative Screening?Collaborative Screening? Coming Soon... Screening for other instruments – MODIS, MLS, OMI, HIRDLS More detailed metrics – Basic usage from logs in EMS – Detailed quality criteria selection – Yields (%values retained) – Detailed usage metrics can help science teams tweak algorithms and products An ontology organizes the variations in quality schemes and drives both the selection interface and the Masker algorithm. The ontology-based architecture will make extension to new quality schemes easier to accommodate. user selection Screener Masker Quality Ontology data file screened data file End User quality mask 1.DQSS creates a mask based on user selected criteria and quality fields in the data file, with guidance from a Quality Ontology. 2.DQSS applies the mask to the data fields Output file has the same format and structure as the input file (except for the extra mask and original data fields) Original Data Array: Total Column Precipitable Water Mask Based on User Criteria (Quality level < 2) Good quality data pixels retained DQSS Ontology Funded by NASA ACCESS (Accelerating Collaborative Connections for Earth System Science) 1.Dr. Alice defines one set of screening criteria for using a dataset in a regional water cycle study. 2.She tags it with the terms describing type of study and/or area and saves the criteria with annotations. 3.Dr. Bob is doing a similar study. He browses pre- existing screening criteria and chooses to apply Dr. Alice’s criteria.