Presentation on theme: "You Gives Your Money and You Takes Your Chances Grants QA Can Be: A. Fun B. Exasperating C. Rewarding Marcus E. Kantz USEPA Region 2 Air and Water QA Team."— Presentation transcript:
You Gives Your Money and You Takes Your Chances Grants QA Can Be: A. Fun B. Exasperating C. Rewarding Marcus E. Kantz USEPA Region 2 Air and Water QA Team Leader 732-321-6690 Kantz.firstname.lastname@example.org
PART Quiz Who out there knows what the acronym PART stands for in Governmentese ?
PART It stands for: Program Assessment Rating Tool It was developed by OMB to help assess the management and performance of government programs.
PART It is used to evaluate a program’s purpose, design, planning, management, results, and accountability to determine its overall effectiveness, and even to weed out government programs that are not producing tangible, relevant results.
PART Used in EPA’s 2008 Performance and Accountability Report To rate every individual program Red light – Green light (and Blue for “Data not yet available”)
Assessing Program and Project Performance is Here to Stay Equally Clear: Assessing performance and acting on the results is a very “QA Thing to do”. Fits perfectly in EPA’s Plan – Do – Check – Act
Assessing Program and Project Performance is Here to Stay Same as EPA Region 2’s PIE Planning – Implementation – Evaluation
How does this fit into a Grants Talk? OMB says it used PART to evaluate 98% of all US Government Programs. Must include some Grants Programs The great Karnak picks the EPA Pollution Prevention, or P2 Program
What did OMB find out about the P2 grants program? It was “Moderately Effective” –The second highest rating –One of the highest of all EPA ratings Yippee ! Soooooooo… ?
What is P2 about, and why do we care? P2 gives grants to states who help businesses find ways to reduce use and discharge of hazardous chemicals. We care because our P2 POs asked about QA for Environmental Performance Measures for their Grants.
New Requirements (Always Bring Trouble, Right?) P2 Program has new requirements: Assess project effectiveness in terms of Actual Environmental Outcomes. Most P2 “training grants” become “environmental operations” grants So who knew? Practically no one. Some PO’s asked us. We pondered.
It Happened to Fit Into Our Audit Schedule Donna Ringel, Kathryn Seaver and I in the Region 2 QA Office were planning an Assessment of the Water Grants Program. The P2 Program in in one of the Water Branches. We looked at grants in same time period as OMB’s assessment ( not intentionally ).
What did we do in the Audit? Looked at all grants from 4 Water Branches Were the most basic QA measures taken? –Did the PO properly determine if grant involved generation and/or use of Environmental Information? –If so, did PO ensure that QMP or QAPP is in place?
Then what? The POs who did it right got Gold Stars. For POs who did not, we: –Tried to figure out why not. –Wanted to help POs do it better in the future.
What did we find, and why does it matter here? Of the 202 grants, 149 involved collection and/or use of environmental information. Of those 149, for 47 the POs had NOT ensured an appropriate QMP or QAPP approved. That is, One Third of environmental data grants occurred outside the QA System.
How could this happen in the Greatest QA Program in the World? Four Recurring Causes: 1.PO Uncertain about need to apply QA to P2 grants using data ONLY for assessing environmental effectiveness of the grant. 2.PO Uncertain about QA for multi-project grants 3.PO Uncertain about QA for “repeating” grants 4.PO Failed to follow up once determined that QAPP needed
PO Uncertain about QA for P2 Grants using data Only for assessing environmental effectiveness 13 P2 grants: 29% of the 47 grants that “failed” our audit. Pointed to situation under the QA umbrella
How does this become a QA issue? P2 grantees who really do just training, now need to demonstrate how much pollution, sickness, or death this work is project preventing For Credibility: need to show how they figured it out and how reliably Clearly fits in QA Program, whether the grantees are actually measuring something, or estimating, or guessing
The Light Went On We realized that these “naked” performance measures: –Are very real –Are important –Require QA –AND We Had Missed ‘em But we couldn’t do it for them
QA Geek Tip-of-the-Week QA Guys Don’t Decide How Good is Good Enough. The Data Users Decide. QA Guys help them do it. For P2, the Home Office decides. We help them figure out how best to report it. Same is true for ARRA. Programs specify the measures, with influence from above.
The Key: the Graded Aproach Program Managers figure out what they need, and what level of quality they require… QA Practitioners guide them in developing their systematic planning. Using the Graded Approach.
Provide Reassurance Show Project Managers and QA Practitioners the Graded Approach The Level of Sophistication & Rigor proportional to the Complexity & Importance of the Project AND The importance of the QUALITY of the measurements. If an Order of Magnitude is OK, It’s OK
For P2: a mini-QAPP Just the facts And not many of them Tell the POs What You’ll Report, and How You’ll Come Up With It. If the POs (Regions & HQ) say it’s OK, it’s OK
What About the Issues with the Water Grants? Multi-Project Grants:34% of the 47 grants that “failed the Audit Have several interlocking but separate projects on the same “water” Often separate sub-contractors, etc. POs (and grantees) forget to get QAPPs for all of them. Result: projects with no QA Planning or Oversight
The “Repeating Grants” Issued Year after Year, usually with evolving focus and projects. In 38% of the failed audits the POs failed to require review and/or revision of the Old QAPPS or development of new QAPPs Not bureaucratic. Major projects, major evolution, but major planning not happening
PO Knows It But Doesn’t Do It On 10 of the 47 failed grants, PO checked box in IGMS “needs QAPP” but failed to make it happen. IGMS software even sends out reminders and ticklers, but they’re not ticklish. QA and Grants Offices had a plan but didn’t make it sufficiently fail safe.
We Learned a Lot (1) Confirmed that some POs don’t know their jobs, or don’t pay attention Confirmed that isolated environmental perrormance measure projects like P2 are important and need attention (like mini- QAPP) Learned we need to deliver much more focused training.
We Learned a Lot (2) That training needs on QA roles and responsibility needs to be combined with carefully targeting tools (like better utilized IGMS “tickling” capabilities)
QAOs - Wake Up IT IS OUR JOB to make sure that our Project Officers and Project Managers for Grants, Interoffice Agreements and Contracts understand that any and all environmental performance measures are covered by our QA requirements and all require an approved QAPP, utilizing the Graded Approach. Even if they have no other data operations.
Communicate with Funds Recipients If they’re delegated QA responsibility, it’s all their job. But they are unlikely to realize their role Unlikely to understand importance. We need “Terms and Conditions” explaining this to them
Summary Our audit finding woke us up Hope this talk wakes you up. We need to apply Graded Approach to any program or project attempt to demonstrate environmental effectiveness We in QA need more vigilance in making sure our Program Community partners know their responsibilities and perform them properly