Precision, Bias, and Total Error (Accuracy) Guest Speaker: Bill Frietsche US EPA

Overall Course Overview: April 7: QA Systems, EPA definitions, PQAOs and common sense – Mike Papp April 14: Routine Quality Control and Data Management (1-pt QC, flow rate, and instrument stability checks) – Travis Maki April 21: Audits Overview (NPAP, PEP, Annual PE, Flow Rate Audits) – Jeremy Howe April 28: Calculating Bias and Precision and AQS reports – Bill Frietsche May 5: 40 CFR 58 App. A- Gaseous Pollutants – Glenn Gehring May 12: 40 CFR 58 App. A- Ozone – Brenda Jarrell May 19: 40 CFR 58 App. A- PM filter and continuous methods –Brandy Toft

…the difference between your answer and the “truth” Error …the difference between your answer and the “truth” Two components of total error (accuracy): Bias (jump) Precision (wiggle)

Precision Error Some imprecision is unavoidable Sometimes up, sometimes down– “random” Difference divided by best estimate of the truth “truth” is: For gas QC checks: known conc. For PM flow rate: audit (known) FR For PM2.5 collocated: their average

1st way of assessing precision: Repeated measurement of the same thing di = the difference between a known value and your value (flow rate, conc., voltage, that should remain the same) Sequence of check (date, time, check number)

2nd way of assessing precision: Pairs of simultaneous measurements (collocated instruments) di (difference between collocated values) date

Bias = calibration error: Can be sudden or due to slow drift over time: di = the difference between the true value (audit value) and the analyzer date

Accuracy = total error Includes both bias and precision At any one time, an audit value could be close to the analyzer value due to precision errors in both Or, audit value could be far from analyzer value due to precision errors adding Audits estimate Accuracy EPA uses audit results to verify that ongoing QC checks do represent total error

Accuracy = total error: Auditor’s see only one day, while your QC checks see overall big picture

Precision and total error both are estimated from the QC checks: 1-pt QC checks no longer called precision checks, because the results are used (by YOU) to calculate both precision and to estimate total error Each check <= 7% is the CRITICAL criteria for each set of data since last passing check The audits basically verify your precision and bias that have been calculated all along from your QC checks (concentration and flow rate, for PM)

QC Checks are used to estimate precision*: Method Pollutants Frequency MQO One-Point QC SO2, NO2, O3, CO Every 2 Weeks O3 : Precision = 7% SO2, NO2, CO : Precision = 10% Flow Rate Verification (QC check) PM10 (lo-Vol) PM2.5 Once every 4 weeks <= 4% of Standard PM10 (high-Vol), TSP Once per quarter <= 7% of Standard Collocated Sampling PM10, TSP, PM2.5 15% of Network Every 12 Days Precision CV= 10% (which means RPD =14%) *and total error, more on this later….

Why is tracking precision important? Estimates your overall system uncertainty, if you were perfectly calibrated Changes over time can point to operator error, lab error, poorly written procedures, equipment/standards going bad Changes can be fixed sooner rather than at audit EPA compares precision by site and agency

di Control charts: Useful to track precision (or bias) over time See this example (and you fill in your values) at: Another example in the DASC tool: di

Control chart in PBDASC tool: The DASC tool automatically writes your values from the columns into the chart: Measured value is your analyzer Audit value is the known di You may want to add dates to the x-axis, or use my example if you want to set up control charts in your program

EPA estimates precision: In their annual QA reports, from your RP transactions in AQS: http://www.epa.gov/ttnamti1/qareport.html http://www.epa.gov/ttnamti1/anlqa.html di

Invalidation criteria for NAAQS: Recommendations in redbook say: Critical criteria invalidate every hour that is not met Operational criteria means something probably wrong, go check it Systematic criteria mean as a set (day? year?) data is not usable for NAAQS, but individual hours or more may be valid

Critical QC invalidation criteria for gas: (NO2 and SO2 are the same) THERE IS A REASON that the checks are no longer called precision checks: the results are used to calculate both precision and bias.

Critical invalidation criteria for PM: Method Pollutants Frequency MQO Flow Rate Verification (QC check) PM10 (lo-Vol), PM2.5 Once every 4 weeks <= 4% of Standard PM10 (high-Vol), TSP Once per quarter <= 7% of Standard

Operational precision invalidation criteria for PM: Method Pollutants Frequency MQO Collocated Sampling PM10, TSP, PM2.5 15% of Network Every 12 Days Precision as CV < = 10% (meaning the relative percent diff must be less than 14% for conc > 3 ug/m3)

Total Error Keep the bias component minimized by calibrating and verifying your equipment against a standard Keep precision component low by consistency You can work to keep bias down, while precision is often out of your control below a certain limit EPA calculates from your RA transactions

Total error requirements for NAAQS are all operational criteria (investigate, but not necessarily invalid): Method Pollutants Frequency MQO Annual Performance Evaluation (Audit) SO2, NO2, O3, CO Once per Year <= 15% for each audit concentration— OPERATIONAL Semi-Annual Flow Rate Audit PM10, PM2.5 Every 6 Months <= 4% of Standard OPERATIONAL PM10 (high-Vol), TSP <= 10% of Standard PM2.5 PEP Program NPAP PM2.5 Quarter Year (see QA Requirements.xls) Bias = 10% -- OPERATIONAL

Bias, precision, and total error: Bias: systematic difference, or “jump” Precision: random error, or “wiggle” Accuracy = total error, a combination of both Simple temp bath illustrations at: http://itep68.itep.nau.edu/itep_downloads/Q A101_Resources/AllDownloadableMovies/

How do I calculate precision and total error? DASC (Data Assessment Statistical Calculator) http://itep68.itep.nau.edu/itep_download s/QA101_Resources/DASC%20EPA%20Prec %20and%20Bias%20Calculator/ AQS: Data Quality Indicators Report (AMP255)

Before we cut to Bill: Minimize bias by regular calibrations both accuracy (total error) and precision are first estimated by di on an ongoing basis, by you Audits verify total error estimates Course website: http://itep68.itep.nau.edu/itep_downloads/Q A101_Resources/ Our emails: Bill Frietsche: frietsche.bill@epamail.epa.gov Melinda.ronca-battista@nau.edu Christopher.lee@nau.edu

AMP255 and other AQS QA tools

Appendix A to Part 58 Table A-2 of App A AQS TTN web site AQSP&A Spreadsheet Discoverer AMTIC web site AQS Helpline AQS AMP255 Data Quality Indicator Report

Define assessments for each criteria pollutant 1 Pt QC check for gases Appendix A to Part 58 Regulations that define QA reporting requirements for criteria pollutants Define assessments for each criteria pollutant 1 Pt QC check for gases Annual performance evaluation for gases Flow rate verification for particulate matter Semi-annual flow rate audit for particulate matter Collocated sampling requirements for particulate matter Pb audit strips for laboratory analysis QA Performance Evaluation Program for PM fine, PM coarse, and Pb Formerly called precision and accuracy data – still use these terms on AQS transactions

This document available at: http://itep68. itep. nau

AQS TTN web site http://www.epa.gov/ttn/airs/airsaqs/ AQSP&A Spreadsheet http://www.epa.gov/ttn/airs/airsaqs/padata/ AQS Discoverer http://www.epa.gov/ttn/airs/airsaqs/aqsdiscover/ AMTIC web site http://www.epa.gov/ttn/amtic/ AQS Help Line EPACALLCENTER@epa.gov 1-866-411-4EPA (4372)

Example run of AMP255 shown onscreen

AQS Precision and Accuracy Conclusions: All QC data helps EPA balance your costs of QC with needed information to protect health Questions to AQS are welcomed EPA committed to improving user friendliness Course website: http://itep68.itep.nau.edu/itep_downloads/Q A101_Resources/ Our emails: Bill Frietsche: frietsche.bill@epamail.epa.gov Melinda.ronca-battista@nau.edu Christopher.lee@nau.edu