Quality Control – Part II Tim Hanley EPA Office of Air Quality Planning and Standards

2 Error …the difference between your answer and the “truth”

3 Continuous PM QC Instrument Stability Checks (EPA calls this general QA in their 1995 memo; see Attachment A in the TAMS template QAPPs) Precision (wiggle) Bias (jump)

4 Instrument Stability Internal diagnostics conducted with each measurement (no error codes) TEOM uses standard filter kit available from dealer BAM uses standard foil or internal source Use kits more frequently at beginning of program, thereafter as necessary and at least twice a year

5 Results of Stability Checks Use dealer’s guidelines Within 2.5% for TEOM standard filter Within 7% for BAM foil Plot results on a control chart (can also plot other parameters on a control chart)

6 Verifying the accuracy of your Transfer Standard Option 1: send standard to manufacturer, who has NIST-traceable equipment; they send you back your standard with certificate of traceability Option 2: use natural standards, such as freezing ice water for temperature, carefully document your calibration according to a standard method

7 Single (One) Point Verifications # Verification occurs when you conduct check using one flow rate or concentration or value # Span is an example of one-point check, meaning spanning from zero up to a range # Verifications DO NOT involve any changes to your equipment

8 Multi-Point Verifications Check of response of instrument to more than one flow rate, concentration, or value Usually zero, a low level, and at the upper end of the range expected to be measured

9 Example of Verification Perform check of equipment Flow rate for PM Concentration for gas Internal mass for TEOM This may be single point or multi-point If within specifications, record and continue

10 Next steps of QC If you are outside limits Review procedures & logs to identify problem Go back and review data—you may have to throw out data to last good check Fix, document and report (when significant)

11 Who can conduct verifications? Routine instrument-stability checks can be conducted by site operator Periodic assessments (may be every 6 months, may be every year) should be conducted by someone OTHER than site operator QAPP specifies how often checks are conducted, by whom, and what to do if results are off

12 Calibrations If results NOT within specs, instrument must be adjusted This adjustment means that response of instrument is changed, which is defined as a CALIBRATION Usually involves a zero and a span After calibrating, check again at a single point

13 Precision Error Unavoidable Sometimes up, sometimes down– “random” Estimated by measuring the same thing several times Minimized by carefully following procedures

14 Precision Collocated analyzers are too expensive, so use repeated measurements of same thing to estimate precision Flow rate is most important parameter for PM Flow rate changes with time—variance is estimate of precision error

15 Estimating Precision in the Field Somehow have to estimate random “wiggle” error Can be done with side-by-side instruments (same sample of air) as with filter-based FRM instruments Can be done with repeated measurements (same sample of air) Verify results are within limits

16 Two Ways to Estimate Precision of Flow Rate At beginning of program, use your external flow rate transfer standard in field, compare flow rate transfer standard’s result with that shown (or set) on analyzer (one-point check) Conduct this check at least every 2 weeks

17 During the First Months Ensure that your external flow rate transfer standard is checked against another flow rate transfer standard at least once (every six months if you want to use second, easier way to assess precision) Record every check of analyzer’s flow rate, verify it is within 4% of external flow rate transfer standard’s flow rate for 3 consecutive biweekly checks

18 Easier Way to Estimate Precision After Several Months of Stable Operation Use “alternative procedure”: Record analyzer’s set point flow rate, assume this is same flow rate that would have been measured by external flow rate transfer standard Record analyzer’s indicated flow rate Calculate relative percent difference, use value as precision error of flow rate (use relative percent difference calculation; see cells B5 – G28 in the Tribal Data Analysis spreadsheet, pg “PM 10 Automated”)

19 Calculating Precision Error Because we don’t know which device is better, no “truth,” so use average as “truth” X = one sampler Y = the other sampler then precision error = ( )

20 Coefficient of Variation (COV) is another statistic to represent imprecision COV = coefficient of variation Where s = the sample standard deviation, or STDEV in Excel d s or

21 Bias Error Minimized by calibrating equipment against a standard Make sure standard is certified against NIST-traceable standard (keep certificate!) Compare this transfer standard to field equipment Measures any bias in your equipment

22 Quantifying Bias Error T = the standard’s result, or “truth” Y = your equipment’s result Your equipment’s bias = T - Y T

23 Definitions EPA calls “accuracy” what others call “total error” Accuracy, or total error, combines both precision and bias errors For continuous PM analyzers, difference between assessing precision and assessing accuracy is that to assess accuracy, flow rate transfer standard used must not be your routine field transfer standard

24 Continuous PM Accuracy Can estimate only accuracy of flow rate, but since this is most important parameter, it’s okay Use external flow rate transfer standard FROM A DIFFERENT CALIBRATION SET to compare indicated (or set) flow rate of analyzer with that shown by transfer standard

25 Accuracy (cont.) Flow rate transfer standard must not be same used to calibrate analyzer (from different calibration set) Audit may be conducted by someone from your tribe, just not routine site operator Conduct this flow rate audit at least once per year for each analyzer

26 Accuracy Calculations Relative percent difference between audit flow rate transfer standard and your analyzer’s flow rate is estimated accuracy of your flow rate Since there is no other (inexpensive) way to estimate accuracy, this is your estimated accuracy, or total error

27 Control Charts Try to keep something steady, but it naturally varies 95% of time within 2 s of average 99.8% of time within 3 s of average red lines within 2 s of average S = sample standard deviation

28 Example Control Chart (see example QAPPs) on Disk