QAPP outline.

QAPP outline

Element 1: Title Page with Approval Signatures
Title of QAPP Name(s) of organizations implementing project Approval personnel Assistance agreement or contract number(s)

Element 2: Table of Contents
List of all required elements and their page numbers Appendices References

Element 3: Distribution List
Lists people who will get original and revised QAPP Everyone who does the work Everyone who manages them Funding agency

4-Organization of Project
Governmental Entities, Contractors, and Key Individuals. Roles and Responsibilities. How often will these be done? How will each person do their job? To whom will they report?

Examples of Agencies Required Agencies Other Agencies
Tribal Environmental Program USEPA Other Agencies TAMS Center State and Local Partners

Possible Contractors Sampling Laboratory Data Analysis QA/QC Audit

Examples of Key Individuals
Required Tribal employees Air Quality Program Manager Quality Assurance Coordinator Optional positions (may be contractors) Environmental Specialist Environmental Technician

Program Manager: Roles & Responsibilities
Oversees monitoring project Prepares or reviews quarterly & annual reports for submittal to EPA Ensures staff is hired and trained

Program Manager: Roles & Responsibilities (cont.)
Prepares & maintains project work plan & budget Communicates with Environmental Director & EPA Project Officer Responsible for approval & modifications of project QAPP

QA Coordinator: Roles & Responsibilities
Prepares or coordinates preparation of QAPPs Reviews and approves corrective actions Conducts system audits

QA Coordinator: Roles & Responsibilities (cont.)
Oversees or conducts method performance audits Prepares QA reports Conducts or oversees data verification, validation and assessment

Environmental Specialist: Roles & Responsibilities
Conducts sample transport, handling & exchange Delivers samples to laboratory (by mail or actual drop off) Signs off on chain of custody forms

Environmental Specialist: Roles & Responsibilities (cont.)
Conducts quarterly calibrations, quarterly audits Records sample information on data forms Reports all aspects of monitoring project to Program Manager

Project Organizational Chart (Example)
John W. Smith Director Navajo Tribe Alexandria Washington Air Quality Program Manager Samuel Vaughn Air Quality Specialist EPA as appropriate Michelle Winston Tom Lamb Air Quality Technician Laboratory Sue Jones QA Officer

5-Project Background History Context
Assume an “ignorant” reader (e.g., member of public)

Why is this work important?
Are there health effects in your community that may result from this problem (asthma, bronchitis)? Reduced visibility?

More reasons why this work is important:
Concern about possible regional transport of pollutant (ozone precursors)? Increased development, more roads, businesses, residents? Concern about children’s exposure?

Summarize Existing Information
Previous results from earlier studies? Results from nearby areas? (if you did not gather the data, you may not be able to use it with your data but it can be useful to you in planning) Results from emissions inventory? Results from compliance monitoring of nearby facilities?

Who are the Decision-makers?
Tribal Council Tribal Environmental Office director EPA Region

Element 6: Project Description
Ondrea Barber Salt River Pima-Maricopa Indian Community

Summarize Purpose of Project
Why are we making these measurements?

Standards What standard will the measurement results be compared against (if applicable)?

Field Work-Summary What kinds of measurements are being made?
What kind of samplers are being used? How many measurements over what time period? Site locations

Field Activities-Summary
Routine field sampling Sample collection Monthly calibrations/audits Instrument maintenance

Laboratory Work How are the samples being analyzed?
state the method can refer to a standard method) Who is doing the analysis?

Schedule 1. Hiring deadlines 2. Training 3. Field measurements
Dry runs with equipment 3. Field measurements 4. Analysis 5. Reporting Activity Frequency Test sampler October 1, 2005 First sampling December 1, 2005

Assessments How will you check on yours and the lab’s work to ensure data is good (summarize)? Who is involved and what are their roles (summarize)?

Records of Assessment Internal assessments (readiness review)
External assessments a. PEP audits for PM2.5, NPAP for others b. Technical Systems Audits

YOUR Assessments of the Analysis Lab
Initial review of their QAPP and calibration certificates when you agree to the contract Onsite visit during the contract Ongoing review of their QC results

Records BRIEF description of project’s records,
Information on where they are stored Ensure that detailed information is in Section 19 What reports are required?

Element 7: Project Quality Objectives
Mathew Plate US EPA Region 9

Project Objectives Why are we making these measurements?
Conformance with NAAQS Obtain baseline data To determine need for additional monitoring Health risk evaluation

Project Objectives What will we do with the results?
Compare with NAAQS Report to community, EPA, health officials

Systematic Planning Required by grant regs: 48 CFR 46
Performance criteria QAPP or equivalent Data assessment Corrective action QA training for management and staff

Why Systematic Planning
Quality is the extent to which our data is sufficient for purposes it is being used We need a process that defines objectives for our monitoring data and ensures that we know when these objectives are met Program objectives should be developed in consultation with decision makers

Decision Maker(s) Those who use data for decisions or conclusions, such as Is a standard violated? Should we take action to improve the air quality? What is the air quality now, so that we will know if it gets worse or better? Should we be taking more measurements?

Who are the Decision Makers for Air Monitoring Data?
Required Decision Makers for EPA grants EPA The Tribe’s Environmental Program Other Decision Makers the Tribe may consider in quality planning State and local organizations Researchers

Types of Objectives Project and Program Objectives
Data Quality Objectives (DQOs) Based on Program Objectives Qualitative and quantitative Measurement Quality Objectives (MQOs) Specific criteria which when met should produce data of acceptable quality Quantitative

Information in DQOs How data will be used Type of data needed
How data should be collected

DQOs Work Backwards... From To Degree of uncertainty you can tolerate
Acceptable degree of uncertainty in each measurement & number of measurements to take To

DQO Functions DQOs Link answers to actual measurements
Set limits on uncertainty so that data produce required uncertainty in the answer

Example "We know that we meet the standard with 80% confidence—this means there is a 20% chance that we could be wrong and we are higher than that standard."

Balancing Cost vs. Degree of Uncertainty
Balances costs of taking many samples with desired uncertainty in Taking many samples with expensive devices yields low decision errors Taking few samples at low cost yields high decision errors Result—may have to change objectives, e.g. minivols to see if you need to monitor and ask for more $

Accuracy DQOs are concerned with determining the accuracy of measurements

Accuracy Accuracy = Total error Includes both bias and precision
Measured by true audit and/or by evaluating method quality objectives

Translating DQOs into Useable Criteria
DQOs should be defined in terms of data quality indicators Criteria set for the data quality indicators are method quality objectives MQOs are set by using empirical data, conservative assumptions, statistical assumptions, and/or common sense.

Data Quality Indicators
These are sometimes Called the PARCCS Precision (P) Bias (A) (bias is sometimes called accuracy) Representativeness (R) Completeness (C) Comparability (C) Detectability (S) (also called sensitivity)

Precision ...how well different measurements of the same thing under prescribed similar conditions agree with each other “Random” component of error— sometimes high, sometimes low

Precision Precision =“wiggle” (variability within many measurements of the same thing) You are trying to estimate the variability within the population of “all” your measurements of the same thing (concentration) Two ways to estimate precision for a single instrument If you have enough equipment, side-by-side, can be two or more devices measuring the same concentration If you have only one continuous instrument, you must estimate precision by how much the measurement fluctuates over time when it is measuring the same concentration?

Precision for Manual Methods
Relative percent difference (RPD) RPD =*difference between two monitors* their average Example: RPD= * *=0.05, or 5%

Precision Exercise for Manual Methods #1

Precision Exercise for Manual Methods #2

Bias A systematic distortion of a measurement process, which causes errors in one direction (i.e., generally positive or generally negative)

Bias Bias = how far from “truth” you are, in terms of percentage
Bias = audit result – your result audit result Expressed as a percentage, so multiply by 100. 0.03 would be 3% 0.11 would be 11%

Bias Exercise #1

Bias Exercise #2

Bias Exercise #3

Representativeness ...a measure of how well your measurements represent the entire population of what you are trying to measure

this value is in instrumentation manuals or laboratory QA plan
Detectability …how low can this be reliably measured with this equipment? Is this low enough to measure trends and evaluate regulatory compliance? this value is in instrumentation manuals or laboratory QA plan

Completeness ... the amount of valid data obtained from a measurement system compared to amount expected to be obtained under correct, normal conditions

Completeness (cont.) Completeness = # results that are usable (valid)
# measurements necessary 80% completeness 8 valid results for Quarter necessary Also can do make-up runs to make up missed days =

Comparability ...measure of confidence with which one data set can be compared to another Beware of apples and oranges in disguise EPA tries to control this by the method designation process

Manual Methods Our precision measured by side-by-side monitors is 10% or less for all checks Our bias measured by independent audits (PEP for PM2.5, NPEP for other criteria pollutants and PM10 flow) is 10% or less for all checks NPEP is mail-in or van program required for all SLAMS and PSD monitors

Automatic (Continuous) Methods
Our precision measured by bi-weekly one-point precision checks (flow rate for PM, span gas for other criteria pollutants) is 15% or less for all checks Our accuracy, measured by independent audits, is 15% or less for all audits

Measurement Quality Objectives (MQOs)
PM2.5 example considering: Bias error Precision error Completeness

EPA’s DQO Standards for the PM2.5 NAAQS
The question: Does this air meet the annual standard of 15 mg/m3 ?

Decision Error Limits EPA decision: acceptable to make correct decision on attainment 95% of time 5% of time, decision may be wrong

EPA Measurement Assumptions
EPA based their calculations on 137 samples in a 3-year period, which is 75% completeness with a 1 in 6 day schedule

MQO Calculations Using acceptable uncertainty and the other assumptions, EPA calculated that each measurement must have a bias and a precision error (MQOs) of 10% or less (See element 7 in the EPA PM2.5 Model QAPP)

MQOs If these MQOs are met, then the conclusion about attainment will be correct 95% of the time (Maybe - - This assumption needs to be verified when the data is assessed and the DQOs are reviewed)

Element 8: Special Training Requirements
Melinda Ronca-Battista ITEP/TAMS

Specialized Training Identify & describe for each person all specialized training/certifications needed How will training be provided? How will training be documented (memos to the file of on-the-job training, personnel files with certificates of training, annual appraisals)?

What Training Is Necessary?
Technical (equipment, QA/QC, etc.) From instrument manufacturer From TAMS Allot TIME allowed/expected for reading manuals, doing on-line courses—40 hrs? Document that you have spent 16 (or more) hours reading the QAPP and understand its requirements, using a sign-off sheet, and this will count as necessary training for anyone working on the project

What Training Is Necessary? (cont.)
Computer use: Word processing Excel (see excellent on-line courses) Access GIS/GPS? On-the-job training/staff orientation Safety courses? (24-hour field safety, first aid/CPR etc.) TAMS AQS training

Examples of Training Options:
EPA (regional, headquarters, OAQPS etc.) ITEP: AIAQTP and TAMS Center California Air Resources Board (CARB) Local or state agencies with whom you have a good relationship borrow their manuals accompany them in the field document this in your personal logbook (# hrs spent, activity, instruments, your initials and date; make photocopy for personnel file)

Examples of Training Options (cont.)
US EPA’s Air Pollution Training Institute (APTI); can view tapes Tribal/state/local agencies University/college courses (college credit, CEUs, etc.) Air & Waste Management Association (

Element 9: Documentation and Records

This section needed to:
Protect legal and financial rights of the agency and persons affected by agency’s activities Ensure that data are legally defensible (e.g., all changes are documented [who/when/why WRITTEN DOWN] )

This section: Element 9 Goes in element 19
Is short—complete data management is described in element 19 Think of data management as: What (requirements) Why Who How Where When Element 9 Goes in element 19 (can be very brief if you reference SOPs!)

“What” This element lists the requirements for the records—what do you require of these records? Records of planning Records of operation Records of data management

Records created during project planning:
Hiring and training records Initial contracts with instrument vendors, contractors Budget records Plans for instrument specs, network, site locations, site visits LIST THESE RECORDS, WHO WILL WRITE/REVISE/APPROVE THEM, HOW OFTEN, WHERE YOU WILL STORE THEM

Records created during project operations:
Site visits Data transfer Audits and assessments QC checks (internal and external) Calibrations LIST THESE RECORDS, WHO WILL WRITE/REVISE/APPROVE THEM, HOW OFTEN, WHERE YOU WILL STORE THEM

Records created during data management:
Download and transfer File naming, moving, pw-protecting, archiving Review, range checks, flagging, calculations Reporting to tribe, EPA LIST THESE RECORDS, WHO WILL WRITE/REVISE/APPROVE THEM, HOW OFTEN, WHERE YOU WILL STORE THEM

A little detail please…
In the text or table, list for each item : Any rules—must be written in pen, dated and initialed Changes must be made so that original data is kept legible Computer files are password-protected for making changes Duplicate copies made and stored where? How long are they kept and who decides to throw them away? File naming conventions?

Requirements for data transfer:
Specify rules for: never over-writing files frequency of downloading files from instrument, backing up files Keeping files on paper (may be summary if very long) and on disk and in PC (designate PC) File structure (see example)

File naming convention example:
All your files follow the same format (example): pppp_ssss_dddddd_nnn_rev# Post this on a wall and changes to it to add fields Where pppp=pollutant Ssss=site name or number Dddddd=date when measurements made Nnn=initials of person saving the file Rev # is the revision number

List all documents: SOPs, QAPP (this one and that of any lab analyzing your samples) Site logbooks, personal logbooks Repair and maintenance records Reports drafted, final, sent out to tribal authorities, EPA Photos of sites Letters from community, EPA, etc. are kept and filed

Data handling procedures:
Example: state that this project will use an Access database for all transcribed or input records and data State that the requirements for data handling are in element 19; this element only lists the records and the requirements for the records

List your plans for reports received and sent out:
Quarterly & Annual Reports submitted to tribe, EPA Quarterly Laboratory Data Report Audit reports Quarterly AIRS-AQS Data submittal to EPA List these in a table

Have all references available in your office
QA documents SOPs Other documents Have all references available in your office

Element 10: Network Design

Rationale for Measurement Location and Frequency
Refer to the data quality objectives (Element 7) Use the objectives when you decide where/how often to monitor Near where people live? Overall community background? Near sources? Discuss purpose of primary & collocated samplers

Design Assumptions Document what you are assuming:
That a source will impact air quality in the monitored area? Weather patterns, road use, community development in the future, whatever Are your locations reasonable in terms of CFRs and other guidance documents (see ITEP CD)

Data Generation/ Collection Design
What type of equipment will be used to generate/collect data? How many will be used? Primary and collocated, meteorological eqmnt Frequency of data collection (24 hours)? Saturdays & Sundays? How many calendar years? Is the monitoring equipment FRM or FEM? If so, what is designation number?

Sampling Network Design
Do you have special purpose samplers (SPMS) measuring for baseline conditions? Do you have a SLAMS-designated site that will provide EPA with national data? Discuss siting requirements (40 CFR Part 58 App. D & E; although these may be impossible to meet so do your best, and confer with your EPA regional office) Monitor location: Roof? Platform? Other? Spatial scale of representativeness (40 CFR Part 58 App. D)

Monitoring Objectives could include:
1. To determine highest concentrations expected to occur in area covered by network 2. To determine representative concentrations in areas of high population density 3. To determine impact on ambient pollution levels of significant sources or source categories

More possible monitoring objectives
4. To determine general background concentration levels 5. To determine extent of regional pollutant transport among populated areas, and in support of secondary standards 6. To determine culture [EPA calls welfare]-related impacts in more rural & remote areas (i.e., visibility impairment, effects on vegetation)

Welfare-related impacts... Micro, Middle, neighborhood
TABLE 1.—RELATIONSHIP AMONG MONITORING OBJECTIVES AND SCALE OF REPRESENTATIVENESS Monitoring Objective Appropriate Siting Scales Population………….. Source impact………. General background… Regional transport….. Welfare-related impacts... Micro, Middle, neighborhood (sometimes urban1) Neighborhood, urban Neighborhood, urban, regional Urban/Regional 1 Urban denotes a geographic scale applicable to both cities and rural areas

Critical / Non-critical Measurements
Critical measurements are required to achieve project objectives or limits on decision errors Field measurement requirements (i.e. ambient temperature, barometric pressure, etc.) -What you would include when submitting data to AQS Non critical measurements are those that are “nice to know” but not make-or-break

“Standard” Measurements
Federal Reference Methods (FRMs) and Federal Equivalent Methods (FEMs) provide standard measurements as required by EPA for comparison to the NAAQS Use of any non-standard measurement method can provide useful information Also pertains to filter-weighing laboratories – EPA requirements must be met

Element 11: Sampling Methods Requirements

Sampling Method Requirements
Requirements, not procedures go in the QAPP Attach your SOPs!

Sampling Equipment, Preservation, & Holding Time Requirements
Requirements for getting samples (data) to lab without losing what you are measuring or making it stink…

Requirements for the Sample
Volume of air or whatever medium you are collecting Object with which you are collecting the medium Data for automatic methods (complete, copied onto floppy, file named correctly, never overwritten)

Requirements for PM2.5 Samplers
Installed & operated with adherence to requirements in 40 CFR Parts 50, 53 and 58; Section 2:12 of EPA QA Handbook; the sampler manufacturer’s operation manual, SOPs, and this QAPP

Sample (Data) Collection Overview
Schedule of receiving filters (data) Schedule of sampling Sample route Describe procedures (may be in SOP) for when retrieving samples (data) do not go as planned

Prevention of Contamination
Requirements for Temperature Humidity Time Integrity Custody Data handling

Sample Preservation Requirements for PM2.5
Filter cassettes stored in filter cassette storage containers Stored with the particulate side up. Temperature (40 CFR Part 50, Appendix L) No direct sunlight or closed-up car during summer Time before sample recovery and time before weighing

Set-Up Requirements When Where How (ref. SOPs) Who Documentation

Sample (or data) Recovery Requirements
When PM2.5 must occur within 96 hours of end of sample period for that filter Where (sites) How (order) Who Documentation

Support Facilities for Sampling Methods
Office, trailer, truck, & cooler must be consistent with requirements for Temperature Humidity Integrity Custody Storage capacity

Field Safety: State that safety comes before getting the sample (data)
Reference health and safety plan Provide training if appropriate

Field Corrective Action
Who is responsible for fixing it? Verifying that it is fixed? Reporting the fix to? Where do they write how they fixed it? When do they have to fix it by? How do you make sure it does not happen again? Where is the documentation stored?

Corrective Action Responsibilities (cont.)
Who makes sure problem was solved Who makes sure it doesn’t happen again who approves changes to sampling locations, personnel, SOPs, QAPP

Element 12: Sample & Data Custody

Sample (Data) Integrity
Packing List Shipping Receipt Annual Shipping Schedule Log Book Other

Sample Custodian Who is responsible for sample or data custody?
Verify sample/data security. Track sample/data storage location.

External Sample Custody Procedure
Formal Chain of Custody Form/Record Dates/Times Signature Pre-sampling Post-sampling

Internal Sample Custody Procedure
Sampling media (filter) receipt Tracked in Logbooks and/or Computers Storage Access Archive Disposal

Data Custody Procedure
Data storage Data transfer Data security Data backup (logger/strip chart)

Element 13: Analytical Methods Requirements

This element is important for comparability:
Requirements of all FRM monitors are the same, so that data from different sites can be compared Performance of all labs meeting these requirements is the same, so that data from different labs can be compared

Summarize method: In one paragraph summarize how the lab or instrument conducts the measurement If FRM or FEM cite the method number List special components, modifications, inlets List requirements for equipment you are using (scales, thermometers)

List in this element: Requirements for how well the lab or instrument performs List analytical methods and equipment

Instrument or lab environment:
Limits for temperature, humidity If the instrument requires a shelter, describe the requirements here If lab or instrument changes their procedures or performance, who approves it? What are the requirements they/it must meet?

Element 14: Quality Control

QC: An Ongoing System Measuring Comparing with MQO
Graphing it Fixing it when needed Everything must be documented and, when significant, reported

Evaluate Where Things Can Go Wrong—and How To Check
Preparing for the field Sampling in the field Analyzing the samples Entering the data Reporting the data

…the difference between your answer and the “truth”
Error …the difference between your answer and the “truth”

Bias Error Minimized by calibrating your equipment against a standard
Make sure the standard has TRACEABILITY to a NIST standard Compare standard to field and/or lab equipment Measures any bias in your equipment

Verifying the Accuracy of Your Transfer Standard
If necessary, send your standard to the manufacturer, who has NIST-traceable equipment, and they send you back your standard with a certificate of traceability If appropriate, use natural standards, such as freezing ice water for temperature, and carefully document your calibration according to a standard method

Single (One) Point Verifications
Single-point verification is when you conduct a check using one flow rate or concentration or value Span is an example of a one-point check Verifications DO NOT involve any changes to your equipment or settings

Multi-Point Verifications
Check of the response of the instrument to more than one flow rate, concentration, or value Usually zero, a low level, and at the upper end of the range expected to be measured

Example of Verification
Perform a check of your equipment: flow rate for PM concentration for gas internal mass for TEOM This may be single-point or multi-point If this is within specifications, record this and continue

Calibrations If results are NOT within specs, then the instrument must be adjusted This adjustment means that the response of the instrument is changed, which is defined as a CALIBRATION Must be multi-point After calibrating, check again at a single point

Example for PM2.5 Corrective Action if the flow rate difference between the transfer standard and the sampler is > 4% Check sampler for internal and external leaks Ensure that temp. and pressure are within normal ranges Run check a 2nd time If still > 4%, perform a multipoint calibration followed by a single-point verification

Who can conduct verifications?
Routine checks of instrument stability can be conducted by the site operator Periodic assessments (may be every six months, may be every year, should be quarterly check of flow rate for PM2.5) should be conducted by someone OTHER than the site operator QAPP specifies how often these checks are conducted, by whom, what to do if results are off

Calculations for Results of Verifications
Consider the standard to be the “ideal truth” for your equipment Difference between the “ideal truth” and your equipment is the instrument error (this may include bias [error either usually high or usually low], and imprecision [wiggle sometimes high and sometimes low]) How is instrument error quantified?

Quantifying instrument error:
Truth = the standard’s result Y = your equipment’s result Your equipment’s error Truth – Y Truth =

Remember... 0.02 = 2 % 0.10 = 10 % Bias calculations exercise

Precision Error Some imprecision is unavoidable
Sometimes up, sometimes down–“random” Estimated by measuring the same thing several times Minimized by carefully following procedures

Two Sources of Precision Error
Field Lab Cows can be problems…. THE COW IS GOING TO KNOCK OVER THE MET TOWER THE MILK WAS SPILLED IN THE LAB

Estimating Precision in the Field
Estimate the random “wiggle” error If you have more than one of the same type of instrument, place side-by-side (measuring the same sample of air within a room or small area) If you only have one instrument make repeated measurements (same sample of air, quickly in time so the air does not “change”) Verify that results are within limits

Estimating Precision in the Lab
Repeated filter weighings OF THE SAME FILTER or standard weight Verify that results are within limits—if results are very different then there is a lot of imprecision and there may be an intermittent power draw, a breeze blowing onto the scale, changes in humidity….

Next steps of QC If you are outside the limits:
Review procedures & logs to identify problem Go back and review data—may have to throw out data to the last good check Fix, document and report (when significant)

Calculating Precision Error
Because we don’t know which device is better, there is no “truth,” so use the average as “truth” X = one sampler Y = the other sampler then precision error is ( ) RPD ADD ABS VALUE (multiplied by 100 to yield a value in percent)

Coefficient of Variation (COV)
COV = coefficient of variation COV = s/(average) Where “s” is the sample standard deviation DEFINE D See the Tribal Data Analysis spreadsheets for example calculations that you can use

Remember... 0.05 = 5% 0.10 = 10% Precision calculations exercises
MOVE TO BEG

Accuracy = Total Error “Accuracy” for EPA means “total error”
Comparison conducted with completely different system can be used to estimate total error For PM2.5, this is a performance evaluation conducted with a side-by-side FRM device, with the filter weighed by a different lab Total error includes both precision and bias errors

Blanks Measure anything that affects the result outside of what you are measuring May make the result greater (contamination) Or decrease the result

Types of Blanks For real-time measurements zero checks display the value with no air Manual methods using filters must use field blanks—accompany “real” samples Labs must measure their own blanks to assess whether there is any contamination in the lab If it is possible that samples get damaged or contaminated during shipping use shipping blanks (trip blanks)

Field Blanks Handled exactly as field samples
Some field blanks go everywhere field samples go With each operator, site, procedure

Control Charts red lines within 2 s of average
S = sample standard deviation red lines within 2 s of average MELINDA WILL DO DEMO OF DRIFT USING CARBON DIOXIDE MONITOR OR…….– HAVE NOT EXCEEDED CRITERIA BUT SHOULD RECALIBRATE Try to keep something steady, but it naturally varies 95% of time within 2 s of average 99.8% of time within 3 s of average

Ondrea Barber Salt River Pima-Maricopa Indian Community
Element 15: Equipment Testing, Inspection, and Maintenance Requirements (General Principles) Ondrea Barber Salt River Pima-Maricopa Indian Community

Purpose of T.I.&M. Increase system reliability Data completeness
Minimize down time Document credibility of data

Ensuring your data meets its quality objectives:
Types of instruments Emphasize those instruments that impact data quality Acceptance testing of equipment Outside person to review final tests Provide a table – referencing SOPs is okay!

Checks Check equipment before you go into field
Check in field before you make measurements Check after you make measurements Ongoing checks and maintenance Checklists! (include in QAPP, even if you know it will be revised)

Personnel Delineate responsibilities Person(s) to do T.I.&M.
Person(s) to order equipment and supplies Person report to for replacement parts or potential problems Person(s) to report problems to and to contact for corrective actions

Procedures Describe how T.I.&M. will be done Schedule of T.I.&M.
Documentation in QAPP appendices (maintenance checklists kept in sampler case and logbook, standard forms with boxes to list values for parameters or check that tubing and wiring is in satisfactory condition) Location/storage of completed checklists Maintenance history – inventory of replacement parts, suppliers, spare parts, other consumables

Visual Inspections Inspect for Damage to monitor
Condition of filter and surroundings (i.e., cleanliness) Consistent power supply so that start/stop times are reliable O-rings in place and not torn Wires and tubes all attached

Inspections Specify what is done in the field or in the office
Be practical Allow for adequate time to do inspections and document their results Take spare parts such as o-rings with you—add this to the checklist

Inspection Considerations
Frequency Inspection parameter – what do you look for? Action if item fails inspection – how do you fix it? Documentation (logbooks for each piece of equipment & inventory of spare parts, oil, etc., instrument or site visit checklists—include in QAPP or reference owner’s manual)

Site-Specific Factors
Temperature Precipitation Wind Curious people Birds, bugs, leaves

Toolbox Screwdrivers (Phillips, flat head, tiny) Wrenches Gloves
Digital multimeter – trouble shoot electrical components Extra batteries – to charge computer board, etc. Duct tape State this in the

Maintenance Spare parts Location of parts Inventory of parts
Where parts or equipment are purchased

Schedule of Maintenance
Field or transfer standard traceability to NIST or certificate of calibration referenced

Cleaning Supplies Ammonia-based general purpose cleaner – clean hub unit Cotton swabs – clean vent tubes Soft bristle brushes – air screen, bug screen, etc.

Cleaning Supplies (cont.)
KimWipes – lint-free Silicone-based lubricant – ensure leak-free fit of rain jar, O-rings, other connections Powder-free rubber gloves – sample handling, transport

Be sure to include: How you conduct T.I.&M. (reference SOPs, pages from owner’s manuals) or briefly summarize Schedule Spare parts Corrective action Checklists, documentation sheets

Element 16: Instrument Calibration

Calibration Calibration is defined as
Comparison of instrument response to a standard and Adjusting response to fall within planned limits (remember that if you just check the response and it is okay then that is a verification)

List Equipment that Requires Calibration
Identify all tools, gauges, instruments, and anything that produces values Make a table that lists the frequency of calibration and how it is to be conducted and who is responsible for ensuring that it gets done

List Equipment that Requires Calibration (cont.)
Limit to equipment you will be responsible for If you use transfer standards (temperature, flow rate, BGI delta-cal) then these must be periodically recalibrated, so that you know they is producing valid results and this is documented

Reference Attached SOPs and...
Describe briefly or reference SOP: How calibration is done When to calibrate Summarize calculations Summarize calibration records (logbooks, forms, reports)

Calibration Standards
Primary standards—keep as the “gold standard” Field, transfer, or working standards are used in the field These apply to flow rate, temp., pressure, etc.

Types of Calibration Multiple point Zero-level
Repetitions at each concentration Always verify stable operation after a calibration by checking at least one point again

Changes to Calibration Schedule
You may have to recalibrate if you: Move, repair, or reassemble equipment If QC checks show degradation If QC checks show great stability, then may not have to recalibrate so soon Change in weather Change in pollutant concentration

Justification for Changes
Documentation—write a memo to the files (see example on CD) External reviewer—get a reality check from another person Periodic verifications of your calibration schedule and procedures

Do Calibrations Yourself?
Requires careful documentation Use standards calibrated by vendor or another certified lab Use these standards to calibrate your instruments

If you need a Laboratory
Get copy of laboratory’s QAPP, include as appendix to your QAPP Conduct tour of facility if possible Communicate regularly with facility personnel Make sure lab documents everything, provides reports

Documents to you from analysis lab:
Copy of their QAPP Copies of their internal performance evaluations within the last year, and throughout the project Agreement that they provide you data on paper and electronically (via ) in excel or whatever format you agree to

Documents to you from the lab that certifies your calibration standards:
Copy of calibration certificate showing what standard they use to calibrate your equipment Traceability of this standard to the National Institute of Standard and Technology (NIST) Agreement that they provide you a calibration certificate and detailed report on paper and electronically (via ) in excel or whatever format you agree to Outline of calibration procedure

Element 17: Inspection and Acceptance for Supplies

Requirements What are the requirements for the equipment / supplies you will use during the project?

Make a List All supplies & consumables that may directly or indirectly affect the quality of the project Filters Hoses Oil Batteries Disks

Provide a Table Listing
Description Vendor Specifications Model number Call the sales rep. and ask them to fax you a list of what parts will be needed

Two tables may be needed:
(1) listing of critical supplies, parts (hoses, filters, disks, etc.) (2) list of their acceptance criteria (diameter/type of hose, type of filter, disks preformatted, whatever is appropriate)

(1) List of Critical Supplies:
Example First Table for PM2.5: Critical Supplies and Consumables (excerpt)

(2) Acceptance Criteria
List criteria for each item Should be consistent with project’s overall technical and quality criteria Should reflect common sense, but need to document the basic requirements, even if they seem obvious to you

(2) Acceptance-Criteria Table
Second Table: Example Acceptance Criteria for Supplies and Consumables

Element 18: Data Acquisition Requirements for Non-Direct Measurements

What is non-direct data?
Any data that you do not gather, such as Weather information Housing information Physical constants Instrument parameters

Examples Chemical and physical properties data Operations manuals
Geographic data – site, boundary conditions, met sites Data from previous studies Meteorological information – U.S. Weather Service Data, wind rose information Census data, housing office data

Be sure to include: Any non-direct data you plan to use
How you plan to use it If you are using historical monitoring data, how you will keep track of which data you gather and which data you have gotten from another source

Why is the QA Significant?
If you plan to use the data with yours, data that you do not gather should: Have undergone QA review Meet quality objectives established for your program This means that you need to get a copy of their QAPP or QA report and compare with this QAPP TRI MODELLING DATA UNCERTAINTY TIME ON WATCHES APPROX IS OKAY, EXACT MEANS A DIFF WALMART THERMOMETERS COMPARED TO NIST—SO HOW DID YOU MEASURE TEMP?

Step 1 – Describe your Requirements:
How you are going to use this data Whether this could change conclusions What are your quality objectives for this data If the data is critical describe some or all QC parameters

Step 2 – Describe the Data Sources:
Nationally recognized source (USGS, NWS, & NIST) Peer Reviewed Source (published in a peer reviewed journal) Monitoring data from another study

Step 3 – Determine if Data Validation will be Performed:
If the data is from a nationally recognized source and is being used for a similar purpose validation is probably not needed If the data has not undergone external review and/or the data will be used to meet a different purpose than intended (when collected) some level of validation is required

Step 4 – Describe Data Validation for data you plan to use but did not gather
Review QC checks Review QA evaluations Compare to other data sets Perform QC measurements Don’t be afraid to discard data that you don’t trust Reference validation in elem. 24

Element 19: Data Management
Melinda Ronca-Battista ITEP/TAMS Ondrea Barber Salt River Pima-Maricopa Indian Community

Six Elements of Data Management
(1) Data processing and transmission (WHAT) (2) Data end use and integrity protection (WHY) (3) Data access (WHO)

Six Elements of Data Management (cont.)
(4) Data dissemination (WHERE) (5) Data storage and retrieval (HOW) (6) Data disposal (WHEN) Document everything!

Describe in the QAPP: Brief description of Data management objectives (all data is retrievable) How you will meet these objectives (no data erased, no files overwritten, use sharpies, date and initial) Describe (briefly) your paper filing system Describe logbooks: personal, instrument Electronic filing system, file naming system

Data Flow Step by step data tracking Who has access to read
Who has access to change data Version tracking of files How changes are approved How file integrity is checked

Files for each instrument and any extra sensors:
Request as much information as possible electronically and on paper from vendors to allow you to make extra copies Maintain file for each instrument with contact information, calibration records, manuals

Material Receipt and Storage
Use logs for everything (see examples in template QAPPs) Use indelible markers Use tags Use designated shelves with labels Use SOP that is POSTED

Types of Field Data Sheets
Site Data Sheets document the site information in site files and in database Sampler run data sheets go into field input info into database Verification data sheets every 4 weeks (1-point flow rate, temp, pressure, time) Internal audit data sheets for the sampler and extra sensors (your own audits)

Before leaving office for the field:
Review the number of each item you will need, and bring backups (use a checklist) Check field data sheet from previous visit to site PM2.5: Ensure that there are filter cassettes for routine, field blanks, and collocated samples PM2.5: Ensure there are enough field transport containers, ice substitutes, max/min thermometers, preprinted mailing labels, if mailing immediately

Write in pen and update the documents:
Continuously update the checklist in pen make photocopy put in “to-do” pile to add the information to the database change the form in the computer if appropriate At the site draw a map on the field data sheet Take photos if possible

Sampler Placement Records
Unobstructed air flow for a minimum of 1 m in all directions Inlet at a height of 2 to 15 m above ground level If collocated with any other PM sampler, the spacing between sampler inlets must be > 1 m for other PM2.5 samplers and > 2 m for PM10 All samplers that are called collocated must be within 4 m Sampler inlet must be level Vertical distance between two inlets < 1 m

Sampler Maintenance and Cleaning Records
Plan all required maintenance 12 months ahead—post 3 months on a calendar Use a checklist that includes items to bring with you and what to do for: Every visit (filter change data sheet) Every month (1/3 day schedule) or quarter (1/6 day) (verification data sheet) Every quarter (flow rate audit data sheet) See examples

Computer Backups Every two weeks or as data are gathered—add “back up today” to calendar Alternate between two sets of backup ZIP or JAZ disks Store backup disks in a relatively fireproof location (another building, garage) Plan for the time and money it takes to save copies of files

Supplementary records must be kept for only 3 years
Chain of custody forms Notebooks Field data sheets

Keep records organized in readiness for an audit
Paper copies must be available to auditor to attach to their reports in the categories of: Management and organization Site information Field operations Raw data Data reporting QA THE BEST WAY TO GET RID OF AN AUDITOR QUICKLY IS TO GIVE HER PAPER COPIES OF EVERYTHING SHE WANTS

Site information files to include:
Site Data Sheets Site maps Site photos Summary of instruments at each site and shelter, trailer information Addresses, names, phone numbers

Field Operations files to include:
Instrument manuals, warranties, calibration certificates in a file for each instrument Standard operating procedures (SOPs) Field notebooks and communications Copies of most recent field sheets Inspection/maintenance records

Raw Data files to include:
Any original data (routine and QC data on disk and paper) Reports from laboratory or external audits Strip charts Disks Supporting data, such as NWS data

Data Reporting files to include:
Internal reports Weekly/monthly summaries Corrective action reports Reports to EPA Copies of presentations to community

QA files to include: For each instrument, copies of instrument check reports (verifications) Calibration reports (multipoint using some standard) Archived control charts QA reports Audit reports and reports on how problems were solved (corrective action reports)

Measurement results should be kept indefinitely
Data must be accessible for 5 years All official reports kept for 5 years Paper copies can be discarded after 5 years and electronic copies archived on disk/CD

Data Transformation & Analysis
Data analysis requirements contained in 40 CFR Part 58, Appendix A Show equations you will use (usually simple relative percent difference) Show examples of charts (use example control chart in the template QAPP)

QAPP outlines plans for data transmittal:
How data gets from equipment into your computer How (who, how often) data is input How (who, how often, how much) someone double checks data entry How data is sent electronically How you will transfer data into AQS

Describe plans for data flagging
Criteria for flagging data Flags may be generated by the instrument Flags may be noted by you on the site data sheet (e.g., high winds) and entered into the database later Flags may be automatically written into a column in excel if values are outside a specified range (see example on CD)

Describe plans for data storage & retrieval
Data archival policies: where, how long, on paper and disk Security of data; locked cabinet; password-protected Database management

Element 20: Assessments & Response
Mathew Plate US EPA Region 9

Corrective Action Designed to identify and correct flaws in your system Anyone can start and document corrective action Documented by a formal process (a review by a supervisor and a memo to the file; see example)

Internal Assessments May be qualitative, such as a review of whether documentation is in order, or technical, checking procedures, and May be conducted by your tribal air organization Someone who has common sense and is technically competent Someone from another program, such as water, solid waste

Internal Assessments, cont.
May be informal and consist of a review between you and your supervisor on progress IF this is to be considered an assessment it must be DOCUMENTED; use a checklist or memo

External Assessments Performed by outsiders such as another tribe, EPA region, or consultant who is technically qualified and Understands project’s QA requirements A real auditor

Types of Assessments Surveillance Technical Systems Audit (TSA)
“Over-the-shoulder” monitoring of project & records; this can be internal or external Technical Systems Audit (TSA) On-site examination of facilities, equipment, personnel, training, procedures, and record keeping, usually conducted by EPA Performance Evaluation (PE) involves numeric comparison of results between auditor’s equipment and yours

Performance Evaluations:
Internal that you conduct as a “test” with a borrowed standard, during an inter-comparison with other agencies—VERY useful, esp. before an external audit External side-by-side on-site with another device independently calibrated Comparison of your results with equipment mailed to you or in an EPA van (NPEP)

Definitions of Assessments
Audits of data quality – qualitative are you working toward your objectives? Do your measurements make sense? Data Quality Assessments – quantitative comparison of results with someone else’s Management Systems Reviews review of QA system; usually conducted by EPA Network Reviews are your locations/instruments appropriate?

All Assessments: Basically compare
What is actually being done in the field and the office Against what is stated in the QAPP and SOPs

Readiness Review—one type of Internal Audit
Conducted before starting routine measurements to assess: “are we ready?” Technical components – equipment Training The report should be approved by an uninvolved person Readiness review can be counted as an internal assessment, IF it is documented and any problems are resolved and documented as well

Technical Systems Audits
Every three years Look at reports, computer files, logbooks, control charts Follow people around Compare what is happening with QAPP and SOPs

Technical Systems Audit (cont.)
Field operations: sampling, shipping QA – corrective action QC – field checks, data flagging, record keeping Data management: security Reporting – accuracy

Audit of Data Quality Evaluating your data before you report it
How data are handled What judgments were made Do your conclusions make sense? Annually and as part of a technical systems audit

Network Review 40 CFR Part 58 Appendices D and E
How well is your network meeting its objectives? How should it be modified? Conducted formally annually, but you should be continually assessing your results

Describe in your QAPP Number, frequency, and types of assessments
People or organizations doing the assessments Schedule Criteria for assessments Reporting and responsibility for follow-up

Element 20 includes a table listing assessments:

Responsibility for Follow-Up and Verification of Corrective Action
CLOSE THE LOOP (fix problems and take action to make sure they do not happen again) And make sure it is documented

Element 21: Reports to Management

Purposes of Reports Communicate Document Track CYA

PM2.5 Reporting Pictures are worth a thousand words— CHART YOUR DATA—by site, by date Flow Rate Audits conducted quarterly by someone other than the routine site operator Results of collocated FRM samplers Results in ranges of concentrations (see tribal data analysis spreadsheet) 40 CFR 58 Appendix A (Section 3.5)

Corrective Action Reports
Ongoing What, where, when, who, how, why Verification of permanent solution

Report Tracking List Those responsible for generating and reviewing reports What is in each report (brief) How often each type of report is issued Who receives each type of report

Element 23: Data Review Methods
Melinda Ronca-Battista

Data Validation Can the data be used for the purpose it is intended?
Is the data invalid or can it be used with qualifications? Is the data generation process likely to produce invalid data in the future?

Data Validation Templates
Developed by OAQPS, EPA Regions and monitoring organizations Three tables generated

Data Validation Tables
Critical – In CFR with acceptance requirements

Operational – In CFR without acceptance criteria or identified in guidance

Systematic – important for correct interpretation of data but do not usually impact validity of sample or group of samples

Template for PM2.5

Template for Ozone

QAPP outline.

Similar presentations

Presentation on theme: "QAPP outline."— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

QAPP outline.

Similar presentations

Presentation on theme: "QAPP outline."— Presentation transcript:

Similar presentations

About project

Feedback