1. Regulatory Perspectives on Data Quality Nick Mangus US EPA / OAR / Air Quality System (AQS) Team for the Earth Science Information Partners July 13, 2011 11/18/20151 U.S. Environmental Protection Agency

2. Nightmare Scenario 11/18/20152U.S. Environmental Protection Agency

3. Quality Data is Foundational for Policy First, know the quality Data Quality Objectives –Each purpose has associated DQOs –Prepare to respond to challenges Methods, Operation, Audits, etc. Quality Assurance Project Plans –SLT QAPPs –EPA Regional Review –EPA HQ QAPP Precision and Accuracy tests and reporting (“quality” metadata) Independent performance audits 11/18/20153U.S. Environmental Protection Agency There is nothing either good or bad, but thinking makes it so. - Hamlet

4. The Regulatory Data-Cycle 11/18/2015U.S. Environmental Protection Agency4 Monitor the Air Handle (QA, Flag) Data Acquire Data Report (Load) Data Analyze Regulate Store Disseminate FED SLT NGO

5. Data Management Issues I’m not a quality person I’m in data management: data comes first –Data Quality vs. Quality Data Think food quality vs. quality food Particular issues down the Value Chain –Custody –Movement –Changes –Calculations 11/18/20155U.S. Environmental Protection Agency

6. Custody and Movement Who can change the data? –AQS business model: data is forever owned by the submitter (e.g., not the feds) –When a question / complaint comes in, all we can do is pass it up the chain –We have sufficient metadata to know who’s touched it and (usually) why data is an outlier Movement: recent ETL example of rounding 11/18/20156U.S. Environmental Protection Agency AQS Data Mart ETL We spend a lot of time comparing values: random spot checks

7. Calculations We get hourly ozone data from a monitor We calculate: –Submitted value in standard units of measure –8-hour aggregates –Daily aggregates (of 1-hour and 8-hour values [2], for each standard [3 x ?], in/excluding flagged data [3]) –Quarterly aggregates (ditto) –Annual aggregates(ditto) –3-year aggregates (ditto); substitutions for missing data A lot can go wrong – software QA is essential 11/18/20157U.S. Environmental Protection Agency

8. Changes Regulatory changes drive data changes –Recent examples: Change the number of significant figures carried through calculations Change the substitution routines based on data completeness –Apply retroactively throughout history Analysis artifacts (speciation carbon) Old submittals 11/18/20158U.S. Environmental Protection Agency

9. Summary Quality is expensive / time consuming Pushing issues / metadata back up the chain is an unresolved issue (?) One mistake can tarnish a reputation that took 1,000 correct actions to create Your system is optional We have to work together to keep each other’s systems meaningful and viable 11/18/20159U.S. Environmental Protection Agency You want it bad, you get it bad. - Lillis

10. 11/18/2015U.S. Environmental Protection Agency10 Note for reviewers. The following slides are not intended to be part of the presentation but are “hip pocket” slides intended only to be used in the case of particular questions being asked.

11. AQ Data Chain – One View 11/18/201511U.S. Environmental Protection Agency Disseminate Decide Evaluate Calculate Store Validate Verify Collect Design Purchase Deploy Operate Collect Analyze QA (flag) Report Data Owner (SLT) Data Custodian (EPA)

12. 11/18/201512U.S. Environmental Protection Agency