1. Introduction to EPA Quality System Requirements
NOTE: Estimated time for this section (Slides 1-9): 45 minutes + introductions.
DISTRIBUTE the registration sheet. ASK participants to complete the form.
EXPLAIN: This is the "Introduction to EPA Quality System Requirements" course. This course will introduce you to EPA's Quality System and to the requirements of that system.

2. Course Goals At the completion of this course, you will:
Understand EPA's quality system requirements
Understand the roles and responsibilities in implementing a quality system that meets EPA requirements
Be familiar with the basic tools of the EPA Quality System
EXPLAIN: After this course, you will:
Understand EPS's quality system requirements
Understand the roles and responsibilities in implementing a quality system that meets EPA requirements
Be familiar with the basic tools to the EPA Quality System.
DO:
Instructor introductions. Each instructor should introduce himself/herself, giving his/her background and an anecdotal story about the importance of quality systems.
Participant introductions. If the groups is not too large, have students introduce themselves.
DISTRIBUTE: The Pre-Assessment Handout (Handout 1).
EXPLAIN: That the purpose of the pre-assessment is not to test individual knowledge but rather to help the participant focus on areas of the course that are relevant to their Quality System roles and responsibilities.
ALLOW: 5 minutes to complete the pre-assessment.

3. What is a Quality System?
NOTE: For each term, provide the definition and a relevant example.
EXPLAIN: Quality System.
A structured and documented management system describing how and by whom an organization assures quality in its work.
For example, the EPA Quality System defines how we do our work and provides the scientific and technical basis for EPA's decision making process.
EXPLAIN: Quality Management.
The process by which quality system specifications are planned, implemented, and assessed ("making the system work").
Possible examples: strategic planning or long-range budgeting and resource plans.
EXPLAIN: Quality Assurance (QA).
An integrated system of management activities to ensure that a process, item, or service is of the type and quality needed and expected by the user.
Possible example: the requirement for the preparation, review, and approval of sampling and analysis plans.
EXPLAIN: Quality Control (QC).
The overall system of technical activities that compares performance against defined standards to verify that stated requirements are met.
Possible examples: the precision and accuracy measurements required by the sampling and analysis plan.
EXPLAIN: These terms and others that will be used in today's course are included in Handout 2 for future reference.
DISTRIBUTE: Handout 2.

4. Participants in the Quality System
EPA Personnel
Executive Leads/Senior Managers for National Programs
Regional Administrators and Senior Managers
Quality Management Personnel
Agency Managers and Staff
For contracts: Project Officer, Work Assignment Manager, Delivery Order Manager
Non-EPA Personnel
Contractors
Assistance Agreement Partners
Interagency Agreement Partners
NOTE: Emphasize that a consideration for quality must be present throughout the organization.
EXPLAIN: EPA functions are complex and diverse. All levels of management and staff have some role in implementing the EPA Quality System.
EXPLAIN: Quality is everybody's job.

5. NOTE: A full page copy of this slide is included in Handout 3
NOTE: A full page copy of this slide is included in Handout 3. Participants should refer to it for this discussion.
DISTRIBUTE: Handout 3.
EXPLAIN: This is a diagram of the structure of the EPA Quality System.
EXPLAIN: The EPA Quality System is based on consensus standards that embody policies for internal EPA organizations and for external organizations, and on the American National Consensus Standard (ANSI).
EXPLAIN: The EPA Quality policy has 3 functional levels. The basic components of the Quality System are implemented at each level.
EXPLAIN: The policy level establishes the authority and the specifications for the quality system.
EXPLAIN: The system components, which address the planning and implementation of environmental data operations, occur at the organizational level. There are 5 required Quality system components here.
QA System documentation: the requirement to document the system using a QMP or equivalent document.
Annual Review and Planning of QA activities and budget needs: the reporting requirement for the QAARWP or related document.
Systems Assessments which addresses the requirement of assessing the overall system at least annually. These assessments evaluate system effectiveness.
Training and Communication which ensures that the staff are adequately trained to carry out their duties and that the requirements of the Quality System are adequately communicated.
Supporting Systems Elements which includes those items that can directly impact the quality of environmental data operations, although it may not be immediately obvious. This includes procurement and computer hardware and software.
EXPLAIN: At the program level, project implementation occurs. Each component/activity at this level addresses the planning, implementation, or assessment of the environmental data collection activity.

6. Types of Quality Management Tasks
Exclusively EPA functions
Performed by EPA officials such as QA Managers (QAMs) or other designated officials
Discretionary Functions
Performed by either EPA personnel or non-EPA personnel
EXPLAIN: There are many tasks associated with the management of a Quality System.
EXPLAIN: Such tasks can generally be grouped into the categories shown on this slide.
EXPLAIN: Some tasks may only be performed by EPA or other government personnel because of statute or regulation.
NOTE: Define "inherently governmental function or work."

7. Exclusive Quality Management Tasks
Managing and coordinating the Quality System
Quality Management Plan (QMP)
QA Annual Report and Work Plan (QAARWP)
Reviewing and approving procurement and financial assistance documents for QA requirements
Reviewing and approving QA planning documents
Quality Assurance Project Plan (QAPP)
Tracking and reporting Quality System deliverables
CONDUCT: Discussion.
Involve participants in predicting which types of tasks are inherently governmental versus which tasks are discretionary.
For each case, discuss the reason for its categorization.
NOTE: The list of tasks continues on the next slide.

8. Exclusive Quality Management Tasks
Developing guidelines and procedures to implement QA/QC
Managing contractor support work assignments and delivery orders
Planning and conducting management assessments
Planning and conducting technical assessments
Reviewing and approving final reports for quality documentation
NOTE: Continue discussion begun on previous slide.

9. Discretionary Quality Management Tasks
Tasks integral to an effective quality system but not explicitly required by EPA Order include:
Reviewing and assisting in the development and preparation of environmental data collection survey designs
Preparing and presenting quality management information
In technical literature
At meetings and symposia
Performing research relative to quality management issues
EXPLAIN: Note that QA personnel may be regarded as technical consultants within their organization for issues pertaining to QA/QC.
EXPLAIN: We have discussed or defined some common terms related to quality management, reviewed the structure of the EPA Quality System, and discussed some of the primary responsibilities and tasks associated with managing the Quality System.
ASK: Do you have any questions?
EXPLAIN: The next section will discuss the goals and purpose of the Quality System and the regulatory authorities that govern the System.

10. Goals of the EPA Quality System
Protect human health and the environment
Make correct decisions
Conserve/optimize resource use
Ensure that environmental programs and decisions are supported by data of the type and quality needed for their intended use
Estimated time for this section (Slides 10-27): 90 minutes
EXPLAIN: The goals of the EPA Quality System are consistent with the mission of the Agency.
REVIEW: and discuss the goals on the slide.
EXPLAIN: To ensure that the Quality System operates in a manner that supports the Agency's mission, specific requirements have been put in place

11. Graded Approach QA and QC requirements commensurate with:
Importance of work
Available resources
Unique needs of organization
Consequences of potential decision errors
EXPLAIN: When the graded approach is used, Quality System documentation and quality requirements for specific projects are based on the scope of a program and/or the intended use of the data.
EXPLAIN: The Agency's goal is to get data that are adequate for the intended use.
EXPLAIN: Best Quality Data is a Myth.
"Best quality" or "highest quality data" may not exist.
You probably wouldn't recognize best quality data.
You couldn't afford to get such data.
Such data may not be technically available.

12. EPA Quality System Requirements
EPA and non-EPA Organization
American National Standard, ANSI/ASQC E4
EPA Organizations
EPA Order
EPA Order 5360 (Quality Manual)
Contracts Management Manual (EPA Order 1900)
Non-EPA Organizations
Federal Acquisition Regulations (for non-EPA Organizations
EXPLAIN: EPA requirements for the Quality System are defined both for internal Agency organizations and for external organizations who are involved in environmental data operations funded by the Agency.
EXPLAIN: Internal Quality System requirements are defined in ANSI/ASQC E4, EPA Order and 5360, and the Contracts Management Manual.
EXPLAIN: Requirements for external organizations are found in Federal Regulations.

13. EPA Order
EPA Order Policy and Program Requirements for the Mandatory Agency-wide Quality System
Defines Quality System requirements for EPA organizations
Requires participation by every EPA organization collecting and using environmental data for decision making
Requires that the EPA Quality System comply with ANSI/ASQC E4-1994
EXPLAIN: Order establishes the authority for the Quality System.
EXPLAIN: The Order applies to all organizations that are involved in environmental data operations.
EXPLAIN: Remember that the term "environmental data operation" is much broader than sampling and analysis. The term "environmental data operation" includes any measurements or information that describe:
Environmental processes
Location or conditions
Ecological or health effects and consequences
The performance of environmental technology.

14. American National Consensus Standard
ANSI/ASQC E4, Specifications and Guidelines for Quality Systems for Environmental Data Collection and Environmental Technology Programs
American National Consensus Standard providing the basis for planning, implementing, documenting, and assessing a quality system
Includes non-mandatory guidelines for going beyond minimum requirements
Part A: Management Systems
Part B: Environmental Data
Part C: Environmental Technology
EXPLAIN: We mentioned in our earlier discussion that the EPA Quality System is compliant with ANSI/ASQC E4 which is the American National Consensus Standard.
EXPLAIN: This standard provides the basis for planning, implementing, documenting, and assessing quality systems.
EXPLAIN: The standard includes 3 parts. These 3 parts address the minimum specifications for quality systems. These parts are:
Management Systems [Part A]
Environmental Data (collection and evaluation) [Part B]
Environmental Technology (design, construction, and operation) [Part C].
DISTRIBUTE: Handout 4.
EXPLAIN: EPA requirements for the Quality System are specific. The next 2 slides will cover the QA requirements for EPA organizations.

15. Internal EPA Quality System Requirements
An EPA Organizations must:
Assign a Quality Assurance Manager (QAM) independent of the data generation line of management
Document the organization's quality system in a Quality Management Plan (QMP)
Provide sufficient resources to implement the quality system
Submit a QA Annual Report and Work Plan annually
Assess its quality system annually
EXPLAIN: The requirements for EPA organizations are defined in Order
REVIEW and DISCUSS the requirements listed on the slide.
EXPLAIN: (Bullet 1) The primary role of the QAM is to oversee and implement the organization's quality system.
NOTE: Provide the name and location of your organization's Quality Assurance Manager. If you are not addressing a single organization audience, provide a listing of Quality Assurance Managers if available.
EXPLAIN: (Bullet 2) The quality system is defined and documented in the QMP. Each EPA organization involved in environmental data operations must have a QMP.
EXPLAIN: (Bullet 3) Resources must be sufficient to implement the Quality System activities defined in the QMP.
EXPLAIN: (Bullet 4) A summary of those Quality System activities are detailed in the QAARWP (EPA only).
EXPLAIN: (Bullet 5) Order also requires that an organization assess its quality system annually. This refers to internal assessment.

16. Internal EPA Quality System Requirements - Continued
An EPA Organization must:
Ensure that corrective actions from assessments are implemented
Use a systematic planning approach for work
Approve QA Project Plans prior to start of work
Implement Quality System requirements in all EPA funded extramural agreements
Assess the quality of "secondary data"
Provide appropriate training for all management and staff
REVIEW and DISCUSS the requirements listed on the slide.
EXPLAIN: (Bullets 1 and 2) There are also specific requirements for planning, implementing, and assessing environmental data operations. These include systematic planning. The preferred EPA planning process is the DQO process. Other planning processes may be used.
EXPLAIN: (Bullet 3) QAPPs are mandatory. They document the systematic planning process. There may be a few exceptions and these would be detailed in your QMP.
NOTE: Provide examples as appropriate.
NOTE: Explain that QAPPs and DQOs will be discussed in more detail later in the course.
EXPLAIN: (Bullet 5) The Agency routinely uses data generated by other parties. This can be valuable information and an efficient use of existing resources. We must ensure that EPA funded work is supported by an adequate Quality System.
EXPLAIN: (Bullet 5) Specifically, secondary data includes those data that the user was not responsible for collecting or data that was collected for a different purpose than that of the current user. The quality and appropriateness of secondary data must be assessed prior to its use.
EXPLAIN: (Bullet 6) It is important to ensure that training needs are identified and addressed for all levels of staff and management.

17. EPA Quality Manual 5360
Defines procedures for satisfying the internal EPA Quality System Requirements
EXPLAIN: Manual 5360 defines what must be in an EPA QMP, responsibilities for review , approval authorities, etc. These requirements are also defined for EPA QA Project Plans.
EXPLAIN: The Manual also defines what QA functions are inherently governmental and must be performed by EPA personnel.

18. EPA Contracts Management Manual
Requires QA Review Form for contracts, work assignments, delivery orders, and task orders
QA Review Form signed by EPA QA Manager or authorized representative.
EXPLAIN: EPA Contracts Management Manual (CMM) defines Quality System requirements for EPA Project Officers and Work Assignment Managers. It is used only for contracts.
EXPLAIN: The CMM includes the Quality Assurance Review Form (QARF) which is internal to EPA and documents whether QA and QC activities are needed.
EXPLAIN: The QARF is required for contracts, work assignments, delivery orders, and task orders.

19. Extramural Agreements
Extramural Agreements include:
Acquisitions including:
Contracts
Work assignments
Task orders
Technical directives
Financial assistance including:
Cooperative agreements
Grants to state and local governments
Research grants
Grants to non-profit organizations
Interagency agreements
EXPLAIN: A significant number of our environmental data operations are carried out by organizations outside of EPA.
EXPLAIN: There are a number of extramural agreement vehicles.
NOTE: Review those listed on slide.
EXPLAIN: To ensure that the quality and adequacy of products generated from these organizations are consistent with EPA requirements or needs, extramural agreements also have quality assurance requirements.

20. Authorizing Regulations for Quality Requirements
48 CFR Part 46, "Federal Acquisition Regulations"
40 CFR Part 30, "Grants and Agreements with Institutions of Higher Education, Hospitals, and Other Non-Profit Organizations"
40 CFR Part 31, "Uniform Administrative Requirements for Grants and Cooperative Agreements to State and Local Governments"
40 CFR Part 35, "State and Local Assistance"
EPA ICR #0866, OMB # , "QA Specifications and Requirements”
EXPLAIN: These citations are the Code of Federal Register (CFR) requirements you need to be familiar with. They govern the extramural agreements within the Agency.
EXPLAIN: These CFRs can be found on the EPA web site.
EXPLAIN: (Bullet 1) 48 CFR 46 was issued 2/16/99 to require the use of consensus standards for higher level contract quality requirements.
EXPLAIN: (Bullet 2 and 3) EPA has OMB information collection request authority for the collection of QA plans from Part 30 and 31 assistance agreement holders, renewed every 3 years.
DISTRIBUTE: Handout 5.

21. Quality System Requirements for Extramural Agreements
EPA requirements for extramural agreements include:
Conformance to ANSI/ASQC E4-1994
Documentation of the organization's quality system (usually called a Quality Management Plan) which should be approved prior to initiating environmental work, and/or
Documentation of the application of quality assurance and quality control activities to activity-specific efforts (usually called a Quality Assurance Project Plan) which should be approved prior initiating environmental data collection.
EXPLAIN: Each organization receiving EPA funding must have a documented quality system. The plan may have a title other than QMP but it must meet EPA requirements.
EXPLAIN: External organizations must also document the plans for environmental data collection activities. This may be done in QA Project Plans, Sampling and Analysis Plans, or similar documents.
EXPLAIN: That you will now briefly discuss the quality requirements for various types of extramural agreements.

22. EPA Acquisitions 48 CFR Part 46
Proposal from offeror must include:
Quality Management Plan and/or
QA Project Plan and/or
Combination Quality Management Plan/QA Project Plan
QA Project Plan is required for individual projects after award
EXPLAIN: These are the general QA requirements for acquisitions.
EXPLAIN: A QAPP is required for each work assignment and delivery order that requires data collection.
EXPLAIN: If a Program level QAPP (one that details the specific requirements/criteria of the program) or combination QMP/QAPP is included in the proposal, a separate QAPP may not be necessary.

23. Financial Assistance- Non-Profit Group 40 CFR Part 30
Applications must include:
Quality Management Plan or
Combination Quality Management Plan and QA Project Plan
QA Project Plan is required for individual projects after award
EXPLAIN: This category of extramural agreements (Financial Assistance-Non-Profit Group) includes universities, hospitals, volunteer groups, etc.
EXPLAIN: Not all assistance involves RFAs.
EXPLAIN: Applying the graded approach is essential here, especially in very small awards, although the decision to be made, rather than the dollar amount, is the driving factor.

24. Financial Assistance- State and Local Governments 40 CFR Parts 31 and 35
Pre-Award/Application
Quality Management Plan (or Joint Quality Management Plan/QA Project Plan) required
Plan Must demonstrate suitability and effectiveness of Quality System to Award Official
Plan Must demonstrate conformance to ANSI/ASQC E4
Post-Award
QA Project Plans required for individual projects
EXPLAIN: This category of extramural agreements (Financial Assistance-State and Local Governments) includes tribal governments, block grants, and performance partnership agreements/grants.
EMPHASIZE: Data collection must not begin without an approved QAPP!

25. Financial Assistance- Research Grants 40 CFR Part 30
Applications must include:
Quality Management Plan or
Combination Quality Management Plan and QA Project Plan
QA Project Plan is required for individual projects after award
EXPLAIN: This category of extramural agreements (Financial Assistance-Research Grants) includes a special case of non-profit agreements. Note that it has the same CFR citation as Financial Assistance -Non-Profit Group.
EMPHASIZE: (Bullet 2) Data collection may not begin before the QAPP is approved!

26. Interagency Agreements
EPA Quality System requirements cannot be unilaterally imposed (Office of General Counsel opinion)
When EPA provides funding:
Add a special condition to invoke conformance with ANSI/ASQC E4
When another Agency provides funding to EPA:
EPA Quality System requirements are applicable
EXPLAIN: (Bullet 1) Because QA specifications in Interagency Agreements are negotiated, it is important to agree upon and document QA roles and responsibilities.
EXPLAIN: (Bullets 2 and 3) EPA minimum QA requirements apply to EPA environmental data operations regardless of the source of funding.

27. Aids for Implementing the Quality System-Documentation
For EPA, requirements are given in EPA Quality Manual for Environmental Programs (EPA Order 5360)
For holders of extramural agreements, requirements are given in EPA Requirements Documents (QA/R-Series) produced by the Quality Staff
For everyone, non-mandatory guidelines are given in EPA Guidance Documents (QA/G-Series) produced by the Quality Staff
Documents, regulations, and QA contacts are available at
EXPLAIN: There are a number of guidance tools available to help users in implementing a quality system. These tools include the Agency Order and Manual and the EPA QA documents series.
EXPLAIN: Requirements and guidance documents provide practices and tools for planning, implementation, documentation, and assessment of individual quality systems.
EXPLAIN: EPA Requirements and Guidance documents are valid for up to 5 years.
EXPLAIN: After 5 years a document must be reviewed and either:
Reissued without change
Revised or
Withdrawn.
EXPLAIN: When implementing the quality system, it is important to choose an implementation strategy that reflects the unique needs and mission of your organization. For this reason, the Agency advocates using the "graded approach.“
EXPLAIN: We have discussed EPA's Quality System requirements for internal organizations as well as external organizations.
ASK: Do you have any questions?
EXPLAIN: We will now discuss the Quality Management tools used to plan, implement, and assess the Quality System.

28. Quality Management Tools at EPA
Planning: Quality Management Plans
QA Project Plans
Data Quality Objectives Process
Implementation: QA Project Plans
Standard Operating Procedures
QA Annual Report and Work Plans
Assessment: Management Assessments
Technical Assessments
Data Quality Assessment
Data Validation and Verification
Estimated time for this section (Slides 28-56): 2 hours 30 minutes without optional activities
OPTION A Activity: See instructions in the Facilitator Guide. If using this option, add minutes to total class time.
REFER to the slide and EXPLAIN:
There are 3 phases of Quality Management. These are analogous to the "Plan, Do, Check, Act" model from Quality literature:
Planning
Implementation
Assessment.
Tools are available for each phase and we will discuss each one in detail during the remainder of this course.
QAARWPs are only used for EPA's quality system.

29. Quality Management Plans
Purpose: To document how an organization will plan, implement, and assess its Quality System
Responsibility: Senior Management
Documentation: EPA Users: Chapter 3, EPA Quality Manual for Environmental Programs (EPA Order 5360) Extramural Users: EPA Requirements for Quality Management Plans
(QA/R-2)
EXPLAIN: The QMP is REQUIRED by EPA Order and applicable federal regulations.
EXPLAIN: The QMP documents policies on how an organization will plan, implement, and assess its Quality System.
EXPLAIN: This is the review and approval process:
For EPA organizations, QMPs are reviewed by the Quality Staff and approved by the Agency's Senior Mgt. Official for Quality Assurance (Office of Environmental Information).
For extramural organizations, QMPs are reviewed and approved by the responsible EPA Organization.
EXPLAIN: EPA's policy is that QMPs are valid for up to 5 years or for the length of the contract/agreement.
EXPLAIN: State and local governments QMPs are valid for up to 5 years (dependent on Regional or programmatic specifications).
EXPLAIN: QMPs for extramural agreement holders are valid for the length of the agreement.
EXPLAIN: If the organization/operations change (reorganization, for example), the QMP must be revised.
EXPLAIN: Minor changes to EPA programs are noted in the QAARWP.

30. Quality Management Plan General Requirements
The Quality Management Plan must discuss:
Mission and quality policy of the organization
Specific roles and responsibilities with respect to QA and QC activities
Means and structure to assure effective communication
Processes used to plan, implement and assess work and effectiveness for QA and QC activities
Process for continual improvement of the Quality System
EXPLAIN: In general the QMP should reflect the organization's policy and commitment to the Quality System.
EXPLAIN: It should also outline specific roles and responsibilities for implementing and assessing the Quality System.
EXPLAIN: Senior management has specific roles and responsibilities even though they may not be involved in technical aspects of an activity. Examples include:
Budgeting for training and oversight
Ensuring that corrective responses to assessments are implemented
Providing adequate resources to carry out QA and QC activities.
EXPLAIN: Equally important, the QMP must document the process for ensuring that everyone understands the roles, responsibilities, and processes associated with the Quality System.
EXPLAIN: And finally, the QMP documents how an organization gets the job done (implements, plans, and assesses work) and how it takes advantage of opportunities for continuous improvement.

31. Typical Characteristics of a Good QMP
Clearly documented responsibilities for QA Officer/Manager
QA Officer/Manager reports directly, or has unrestricted access, to senior management official
Specific commitment to performing the needed assessments, and a description of the process used
Clearly defined responsibilities for personnel reviewing and approving QAPPs
NOTE: The next 2 slides present general characteristics of an acceptable QMP.
ASK students if they have additional criteria that they would like to add to the list.

32. Typical Characteristics of a Good QMP (continued)
Clearly stated management commitment to QA and specific responsibilities documented
Commitment to a specific process for systematic planning for new data collection activities
Clear process for identifying resources to plan, implement, and assess the effectiveness of the quality system
Process for identifying QA/QC training needs of staff
NOTE: If you choose to conduct the activity, OPTION B, do so after reviewing the entire list.
OPTION B Activity: See instructions in the Facilitator Guide. If using this option, add 45 minutes to total class time.

33. QA Project Plans (QAPPs)
Purpose: To document type and quality of data for environmental decisions; a blueprint for collecting and accessing data
Responsibility: Organization performing activity
Documentation: EPA Users: Chapter 5, EPA Quality Manual for Environmental Programs
(EPA Order 5360) Extramural Users: EPA Requirements for
Quality Assurance Project Plans (QA/R-5)
Everyone: Guidance for Quality Assurance Project Plans (QA/G-5)
EXPLAIN: QAPPs provide project- or task-specific blueprints of how QA and QC are to be applied to environmental data operations. The QAPP provides the mechanism for documenting the results of the planning process (e.g., DQOs).
EXPLAIN: QAPPs are reviewed and approved according to the processes defined in the organization's QMP.
EXPLAIN: The organization performing the activity documented in the QAPP is responsible for:
Preparation
Getting approval
Implementation
Keeping the QAPP current.
EXPLAIN: The EPA project manager is responsible for having a signed and approved QAPP and assuring its implementation.

34. QA Project Plans (QAPPs)
Required planning documents that explain how environmental data collection activities are planned, implemented, documented, and assessed during the life cycle of a specific program, project, or task
QA Project Plans are required when environmental data operations occur for:
Contracts, work assignments, delivery orders
Grants, cooperative agreements
Interagency agreements (when negotiated)
State-EPA agreements
Responses to statutory or regulatory requirements and to consent agreements
EXPLAIN: Emphasize that QAPPs are program, project, or task specific.
EXPLAIN: QAPPs must be:
Approved by EPA prior to the start of data collection
Reviewed at least annually for ongoing projects.
EXPLAIN: QAPPs apply to:
Direct measurements or data generation
Environmental modeling
Compilation of data from literature or electronic media
Data supporting the design, construction, and operation of environmental technology.
EXPLAIN: The content of the QAPP will vary by the type of work being done and the intended use of the data (graded approach).
EXPLAIN that the next slide offers additional information on the scope of information covered in the QAPP.

35. QAPP General Requirements
The QAPP must provide sufficient detail to show that:
The project technical and quality objectives (e.g., DQOs) are identified
The intended measurements or data acquisition methods are appropriate for achieving project objectives
The assessment procedures are sufficient for confirming that data of the type and quality needed are obtained
Data use limitations are defined
ASK participants to consider the purpose and scope of QAPPs. As residents of Cedar Bay (If OPTION A activity was used) or as environmental officials (if OPTION A activity was not used), what types of information would they want to see documented in the QAPP?
REMIND participants that the QAPP is the key document describing the data collection and investigation plan.

36. QMPs vs. QAPPs QMP QAPP QAPP QAPP QAPP
Quality Management Plans reflect activities and policies common to all projects.
Quality Assurance Project Plans reflect specific projects.
QMP
EXPLAIN: Both the QMP and the QAPP are required documents in the EPA Quality System. However, they have very different functions.
EXPLAIN: Remember that the QMP is a document that reflects an organization's policies. That document will set forth the policy which requires the preparation of the QAPP.
EXPLAIN: The QAPP is project specific and defines the procedures for project implementation and assessment. It states, what exactly needs to done and how it will by done.
QAPP
QAPP
QAPP
QAPP

37. Systematic Planning Planning is the key to successful programs
EPA policy requires that all work be planned using a Systematic Planning Process
Quality requirements should be based on a Graded Approach
Effective planning must include all stakeholders (data users, data producers, decision makers) to ensure needs are defined at the outset
Planning must be documented
EXPLAIN: Systematic planning is based on the scientific method. This method identifies and defines the type, quality, and quantity of data needed to satisfy a specified use.
EXPLAIN: You will recall that the graded approach should be applied wherever possible. For example, planning activities for a volunteer monitoring project may not be as complex as planning activities for designing a site characterization study.
EXPLAIN: We must document the planning process to know what we've done and how we did it.
EXPLAIN: Results of systematic planning are usually documented in a QAPP.

38. Systematic Planning Approaches
Several systematic planning approaches are available, including:
Observational Method
DQO Process
EXPLAIN: Systematic planning emphasizes the importance of proper planning and a clear understanding of project goals.
EXPLAIN: It should be clear that the quality of a project is in part determined by using processes that facilitate understanding how decisions will be made and what those decisions are. This is the basis of systematic planning approaches.

39. Common Elements in All Systematic Planning Approaches
Questions to be answered:
Who is making the decision?
Why are data being collected?
What data are needed to make the decision?
Why does the decision maker need that type and quality of data?
How does the decision maker plan to use the data to make a defensible decision?
What are the "measures of success" for the project?
Get only the type, quantity, and quality of data necessary
EXPLAIN: These are some elements that are common to systematic planning approaches.
EXPLAIN: These are the minimum elements that should be addressed in using systematic planning. (Refer to the slide and read the 6 questions.)
EXPLAIN: Systematic planning is often an iterative process. But ultimately, it should improve the effectiveness, efficiency, and defensibility of decisions in a resource-effective manner.

40. Data Quality Objectives (DQO) Process
Purpose:
Responsibility:
Guidance:
Software:
To define type and quality of data that a decision maker needs before carrying out data collection
- Saves time and money
- Doing it right the first time
- Obtaining data sufficient for analysis
Developed by project team of data users and data generators
Guidance for the DQO Process (QA/G-4)
DEFT Software for the DQO Process (QA/G-4D)
EXPLAIN: The DQO process is the Agency's recommended planning approach for selecting between two conditions.
EXPLAIN: The DQO process consists of 7 steps:
State the Problem
Identify the Decision
Identify Inputs to the Decision
Define the Study Boundaries
Develop a Decision Rule
Develop Limits on Decision Errors
Optimize the Design for Obtaining Data.

41. DQO Process
A statistics-based planning process for ensuring that environmental data are of the type, quantity, and quality needed for decision making
Qualitative and quantitative statements derived from each step of the DQO Process define performance criteria
EXPLAIN: Data Quality Objectives (DQOs) are qualitative and quantitative statements derived from the DQO process. DQOs are not statements of laboratory limits for precision and accuracy. Rather, they are statements that result from the steps of the DQO process.
EXPLAIN: These DQO statements clarify the study objective, define the appropriate types of data to collect and the most appropriate conditions from which to collect the data, and specify tolerance limits on the data used to make decision.
NOTE: If you used the Cedar Bay video, ASK students how the DQO process may have helped the Cedar Bay situation.

42. Standard Operating Procedures (SOPs)
Purpose: To document routine technical and administrative activities to ensure
consistency in the quality of the
product
Responsibility: Appropriate technical personnel
working with QA Manager
Documentation: Guidance for the Preparation of Standard
Operating Procedures (QA/G-6)
EXPLAIN: SOPs are routinely used to ensure reproducible results and consistent practices.
EXPLAIN: Review/Approval is handled by the immediate supervisor, as a minimum.
EXPLAIN: Responsibilities for preparation and updates vary among organizations. Policies for SOP preparation, review, approval, and revision should be documented in the QMP.

43. Standard Operating Procedures (SOPs)
Written documents that give precise descriptions of routine procedures
Detail stepwise process for sample collection operations, laboratory analyses, or equipment use
Ensure consistency and conformance with organizational practices
Serve as training aids on methods and instrument
use
Provide ready reference and documentation of
proper procedures
EXPLAIN: Some advantages of Standard Operating Procedures:
Reduce work effort
Reduce errors
Improve data comparability, credibility, and defensibility.
EXPLAIN: When procedures change or new technologies are put into use, SOPs must be revised accordingly.
OPTION C Activity: See instructions in the Facilitator Guide. If using this activity, add 45 minutes to total class time.

44. Quality Assurance Annual Report and Work Plans (QAARWPs)
Purpose: To summarize the results of the implementation of an EPA
organization's Quality System in the
previous fiscal year and to describe
QA activities planned for the
upcoming year
Responsibility: Senior Management
Documentation: EPA Quality Manual for Environmental
Programs (EPA Order 5360)
NOTE: If you are not talking to an EPA audience, don't use Slides
EXPLAIN: QAARWPs are only for EPA; not for extramural projects.
EXPLAIN: QAARWPs are reporting tools that allow the Agency's management to obtain an overview of the Agency wide Quality System status and activities. They are not approved like QMPs, but the reported information supports annual reporting to the Agency's Administrator.

45. Quality Assurance Annual Report and Work Plans (QAARWPs)
QAARWPs are:
Management tools for documenting the past fiscal year's activity and for estimating the workload for the current year
Required by EPA Order
Submitted annually (usually in November) to the Quality Staff, Office of Environmental Information
EXPLAIN: Specifically, information from the QAARWP is used to determine Agency-wide trends and issues.
EXPLAIN: The QAARWP has 2 distinct sections:
A fiscal year activity report
A work plan for the coming year.

46. QAARWP Specifications
QA Annual Report
Management Accomplishments
Innovations implemented
Technical assessments completed
Technical assistance given
Guidance developed or revised
Publications, presentations, awards completed
Management Assessments of Approved Quality System
Assessment made by organization itself
Assessment made by others (i.e., third party)
NOTE: This slide and the next show the elements that are reported in the QAARWP.
NOTE: Briefly discuss the type of information that is provided in each section.

47. QAARWP Specifications
Work Plan
Quality Management Resources
FTE proposed to support quality management implementation
Dollar amount proposed for QA/QC support
Proposed travel funds
Proposed training funds
Activities
Major QA/QC activities expected during the year in:
Training
Assistance
Assessments

48. Quality System Assessment
Management
Technical
Data Quality
EXPLAIN: Assessments are required to ensure that various components of the Quality System are:
Functioning as planned
Operating effectively.
EXPLAIN: This slide lists three classes or categories of assessments. We will discuss each one in the upcoming slides.

49. Management Assessments
Purpose: To determine conformance with an approved QMP and to assess the suitability and effectiveness of its implementation
Responsibility: EPA Managers of individual organizations
EXPLAIN: Every organization should employ some type of management assessment. In most cases, these assessments occur at a frequency specified by the organization or other organizations who may have program level oversight responsibilities.
EXPLAIN: There are two types of management assessments currently used in assessing the EPA Quality System. These are the Quality System Assessment and Management Systems Review processes.

50. Quality System Audit
A documented activity performed to verify, by examination and evaluation of objective evidence, that applicable elements of the quality system are appropriate and have been developed, documented, and effectively implemented in accordance and in conjunction with specified requirements.
Such requirements may be defined by:
EPA Orders
Extramural Agreement Regulations
Approved Quality Management Plans
EXPLAIN: The QSA focuses on a documented quality system and uses specific policy and requirements criteria to determine conformance.
EXPLAIN: The QSA also determines if the quality system has been implemented as prescribed by its documentation; that is, is the organization doing what it said it was going to do in its QMP.

51. Quality System Audit
QSA uses quantitative approach to documented quality systems.
Findings are based on objective evidence.
QSA is a conformance/compliance audit:
Does the quality system conform to specifications?
Does the quality system comply with regulations?
Does the quality systems satisfy the QMP?
QSA does not judge quality of individual data sets.
EXPLAIN: The QSA is an audit in the classic sense. It is quantitative in nature and audits against specified criteria.
EXPLAIN: The QSA does not judge the quality of data sets or the value of results obtained by using the quality system.

52. Management Systems Review
The qualitative assessment of QA and QC practices to establish if they conform to policies and requirements and are adequately implemented to satisfy needs and expectations.
Such policies and requirements may be given by:
EPA Orders
Extramural Agreement Regulations
Approved Quality Management Plans
EXPLAIN: The MSR is a review of current QA and QC practices to determine conformance to policies and requirements.

53. Management Systems Review
Similar, but less quantitative than QSA.
Applies best to situations where the quality system is not well-documented.
Investigative in nature - - seeks to determine what is actually happening.
Interview is primary data collection method.
MSR does not judge the quality of individual data sets.
EXPLAIN: The MSR does not depend on having specific assessment criteria. The MSR is more open-ended. The MSR will employ several techniques associated with audits.
EXPLAIN: The emphasis is on interviews rather than objective evidence to gather information.

54. Technical Assessments
Purpose: To evaluate the implementation of a project
or activity against its defined technical or quality procedures or criteria
Responsibility: Project Managers with the assistance of the appropriate technical personnel, their
EPA Manager, and QA Manager
Documentation: Guidance on Technical Assessments for Environmental Data Operations (QA/G-7)
EXPLAIN: Technical assessments should be applied to projects that involve environmental data operations. While these assessments may involve the review of some aspects of project management, they are technical in nature and focus on the hands-on aspect of how the work is done.

55. Technical Assessments
Technical Assessments are self or independent evaluation processes used to measure the conformance, performance, or effectiveness of systems
Technical Assessments include:
Technical Systems Audits
Readiness Reviews
Surveillances
Performance Evaluations
Audits of Data Quality
Peer Reviews
EXPLAIN: The need for a particular type of assessment must be determined in planning.
EXPLAIN: The purpose/scope of each type of technical assessment using examples that are relevant to the audience. (Bullet 2)
EXPLAIN: The general policy and process for planning, conducting, evaluating, and reporting technical assessments must be documented in the QMP.
EXPLAIN: Specific details on particular technical assessments chosen for a project must be documented in the QAPP.
EXPLAIN: Findings requiring corrective actions must be addressed and resolved in a timely manner.

56. Data Quality Assessment (DQA)
Purpose: To assess type, quantity, and quality of data
- Verifies DQOs
- Develops DQOs not fully developed
- Verifies QAPP components
- Verifies sample collection procedures
Responsibility: Appropriate technical personnel
Documentation: Guidance for DQA: Practical Methods for Data Analysis (QA/G-9)
Data Quality Evaluation Statistical Toolbox (DataQUEST) (QA/G-9D)
EXPLAIN: The DQA is a technical and statistical assessment of data quality.
EXPLAIN: The responsibility for conducting DQAs varies among organizations. In many organizations, management would have little or no involvement at this stage. DQAs are usually done by QA staff and other personnel who are knowledgeable in statistical methods.
EXPLAIN: It is important to note that the DQA process covers all components of decision making including planning, implementation, and data review. It is not a process that is limited to review of analytical data.

57. Data Quality Assessment (DQA)
Determine if environmental data are of the type, quantity, and quality needed
Scientific and statistical evaluation of data
The DQA Process may be performed:
During a project to check the process of data collection
At the end of a project to check if objectives were met
The DQA Process provides a tool for confirming that the systematic planning criteria were met
EXPLAIN: The data quality assessment activities must be determined in planning (scope, frequency, etc.)
EXPLAIN: The general process, frequency, and responsibility must be documented in the QAPP (a statistician may be required).

58. DQA Process - Prospective
What supplemental information is needed?
Are further data really necessary?
Should parts of the DQOs be changed?
Where can the DQO Process be improved?
EXPLAIN: DQAs are not only performed at the conclusion of a project. The process may occur throughout the life of the project. During the project, the questions listed here may be answered by employing the DQA process.

59. DQA Process - Retrospective
Were the data quality objectives achieved?
Were the data quality objectives meaningful?
Were the assumptions viable?
Were the statistical tests powerful enough?
EXPLAIN: At the conclusion of the project, the DQA will provide answers to the questions listed on this slide.
EXPLAIN: We have just discussed the tools used to manage and implement the Quality System. We have defined each tool and discussed its use and some of the roles and responsibilities associated with using these Quality Management tools.
ASK: Do you have any questions?

60. Summary Authorities Internal EPA Policies EPA Order 5360.1
EPA Manual 5360
External Policies
48 CFR 46
40 CFR 30, 31, 35
Estimated time for the summary: 10 minutes
EXPLAIN: This slide and the next summarize all of the EPA Quality Systems requirements that we have covered in this course.
EXPLAIN: We have discussed where the authority for the EPA Quality system requirements can be found, both for the Agency and for those who handle environmental data operations on behalf of the Agency.

61. Summary (continued) Quality System Tools Quality Management Plans
QA Project Plans
Standard Operating Procedures
Systematic Planning and the Data Quality Objectives Process
Assessments
Quality Assurance Annual Report and Work Plan
EXPLAIN: We have also discussed the components of the EPA Quality System and the tools that are used to manage the system.
EXPLAIN: Finally we have discussed the requirements and guidance documents available to you to help you manage your Quality System.
EXPLAIN: Take a moment and revisit your pre-assessment. Have your results changed?
ASK: Are there any questions?

62. Summary (continued) EPA Quality System Documents
EPA Requirements for Quality Management Plans (QA/R-2)
EPA Requirements for Quality Assurance Project Plans (QA/R-5)
Guidance for Quality Assurance Project Plans (QA/G-5)
Guidance for the Data Quality Objectives Process (QA/G-4)
Guidance for the Preparation of Standard Operating Procedures for Quality-Related Documents (QA/G-6)