1. Applying a Life-Cycle Approach to Information Quality ARRA Performance Data Office of the Chief Financial Officer and the Office of Environmental Information May 2009

2. What are performance data? Data collected/used to monitor and report a program’s accomplishments (its activities, outputs, outcomes) relative to a target Data collected/used to monitor and report a program’s accomplishments (its activities, outputs, outcomes) relative to a target  Performance data can be environmental, administrative (count), activity, survey, you name it

3. ARRA of 2009

4. OMB’s guidance regarding quality assuring ARRA-performance information: “…agencies are responsible for pre- dissemination review of all information that will appear on Recovery.gov. All agencies must ensure all reporting related to ARRA funding is complete and accurate and complies with the agency's Information Quality Act guidelines. Each agency will provide its point-of-contact for quality information on its Recovery.gov page.” “…agencies are responsible for pre- dissemination review of all information that will appear on Recovery.gov. All agencies must ensure all reporting related to ARRA funding is complete and accurate and complies with the agency's Information Quality Act guidelines. Each agency will provide its point-of-contact for quality information on its Recovery.gov page.”

5. Other accountability objectives in ARRA: Public benefits of funds reported clearly, accurately, in timely manner Public benefits of funds reported clearly, accurately, in timely manner Funds used for authorized purpose Funds used for authorized purpose No unnecessary project delays and cost overruns No unnecessary project delays and cost overruns Programs meet their goals and targets, and contribute to improvement of broad economic indicators Programs meet their goals and targets, and contribute to improvement of broad economic indicators

6. OMB Guidance on Performance Measures under ARRA Performance measures are expected quantifiable outcomes…with each outcome supported by a corresponding quantifiable output(s)—to indicate incremental change against present level of performance Performance measures are expected quantifiable outcomes…with each outcome supported by a corresponding quantifiable output(s)—to indicate incremental change against present level of performance Programs must specify results reporting period (e.g., monthly, quarterly), measurement methodology, and how the results will be made accessible to public. Programs must specify results reporting period (e.g., monthly, quarterly), measurement methodology, and how the results will be made accessible to public. Use of existing measures will allow the public to see the marginal performance impact of the Recovery Act investments and will reduce burden. Use of existing measures will allow the public to see the marginal performance impact of the Recovery Act investments and will reduce burden.

7. ARRA Performance Measures— Available in Draft: Developed by Programs receiving ARRA funds Developed by Programs receiving ARRA funds Mix of Outcomes/Outputs[1] Mix of Outcomes/Outputs[1][1] Many are existing planning and annual commitment measures Many are existing planning and annual commitment measures “Green” measures and economic efficiency measures (e.g., jobs created/preserved) “Green” measures and economic efficiency measures (e.g., jobs created/preserved) Measures developed by OIG to detect waste, abuse and fraud Measures developed by OIG to detect waste, abuse and fraud [1][1] Number of existing heavy duty diesel engines (including school bus engines) that have been retrofitted, replaced or retired (DERA); Number of states that have awarded all of the 20% green project reserve (CW SRF); Number of jobs leveraged (Brownfields); Number of criminal, civil and administrative actions (OIG) [1]

8. Performance Measure/Data Quality Plan Performance Measures Development Program performance questions Data/information sources & definitions Acceptance criteria/data quality objectives (award conditions) Data/Info QA Plan Sampling & lab analysis plans/protocols Reporting & record keeping Database design and operation Data input protocol Statistical analysis & presentation Roles & responsibilities Check Check quality processes for adequacy Review oversight results for compliance with QA plans Document issues, problems & deviations Characterize validity of a priori assumptions about measure/data quality Assess implications for intended use of/goals for performance measure/data Identify options to improve performance measure/data quality Act Evaluate measure/data quality options Assess costs & benefits of options Revise measures & QA plans as warranted Do Data/info collection & analysis Note issues/deviations Data recording, storage, access, aggregation, etc. Oversight, corrective actions & implications Analyze & interpret IQG pre-dissemination review Present/communicate

9. Management Action Plan I Explains EPA’s Quality Program’s role under ARRA Explains EPA’s Quality Program’s role under ARRA References existing grants, contracts, and IAs receiving ARRA funds and specifies which have environmental data collection components with Quality Management Plans and/or Quality Assurance Project Plans References existing grants, contracts, and IAs receiving ARRA funds and specifies which have environmental data collection components with Quality Management Plans and/or Quality Assurance Project Plans Good check for ARRA environmental measures that have robust Plan/Do quality assurance Good check for ARRA environmental measures that have robust Plan/Do quality assurance

10. Management Action Plan II Broadens definition of performance data to include administrative data and other types Broadens definition of performance data to include administrative data and other types Lays out “State of QA” for each performance measure Lays out “State of QA” for each performance measure Inventory data standards and definitions, management controls for reporting and oversight, data flow Inventory data standards and definitions, management controls for reporting and oversight, data flow Identifies gaps in QA processes & responsibilities Identifies gaps in QA processes & responsibilities Requires programs to self-certify completeness of data quality processes through a signed Pre-dissemination Review Requires programs to self-certify completeness of data quality processes through a signed Pre-dissemination Review Bottom line purpose (of PDCA, too): To minimize end-of- pipe errors in reporting performance results!

11. Pat’s topics Where we are/were headed when ARRA intervened – larger vision/context Where we are/were headed when ARRA intervened – larger vision/context What MAP II represents compared to the whole effort: limited scope but progress in involving the PM community What MAP II represents compared to the whole effort: limited scope but progress in involving the PM community Progress outside of MAP II Progress outside of MAP II Opportunities for participation Opportunities for participation

12. Performance Measurement Goal – to further QA in all OMB-related performance measures for EPA Goal – to further QA in all OMB-related performance measures for EPA Beyond performance data reporting from ARRA-funded programs Beyond performance data reporting from ARRA-funded programs Broaden to an Agency-wide QA context Broaden to an Agency-wide QA context Plan, Do, Check, Act – quality life cycle Plan, Do, Check, Act – quality life cycle Information Quality Guidelines (IQGs) Information Quality Guidelines (IQGs) CIO 2106 – new Quality Policy CIO 2106 – new Quality Policy

13. ARRA Progress MAP I QA of environmental data in ARRA financial instruments MAP I QA of environmental data in ARRA financial instruments MAP II QA adds performance data and information for ARRA measures MAP II QA adds performance data and information for ARRA measures Communication with and input from: Communication with and input from: NPO & Regional QA communities NPO & Regional QA communities IT/IM community (some affected by ARRA) IT/IM community (some affected by ARRA) Stimulus Steering Committee - planners from 5 program offices Stimulus Steering Committee - planners from 5 program offices

14. What’s NOT in MAP II QA of performance measures in the life cycle framework (ARRA measures are set, ICR issue) QA of performance measures in the life cycle framework (ARRA measures are set, ICR issue) IT/IM QA for all types of data bases IT/IM QA for all types of data bases Enterprise systems (CERCLIS, SDWIS) Enterprise systems (CERCLIS, SDWIS) End-user systems (spreadsheets) End-user systems (spreadsheets) Guidance from IT/IM QA standards references Guidance from IT/IM QA standards references GAO auditing protocols (FISCAM) GAO auditing protocols (FISCAM) Corporate accounting firms Corporate accounting firms OEI policy OEI policy QA review/approval of the procedures QA review/approval of the procedures

15. Performance Measurement QA Plan with stakeholders, internal & external Plan with stakeholders, internal & external Apply the QA graded approach to assure Apply the QA graded approach to assure Adequate QA resources for oversight Adequate QA resources for oversight Performance measure & data are adequate for intended use Performance measure & data are adequate for intended use Document, validate & verify data flow processes Document, validate & verify data flow processes Use life cycle “Check” and “Act” for evolution to environmental outcomes Use life cycle “Check” and “Act” for evolution to environmental outcomes

16. Opportunities for Input IT/IM Managers: Performance data quality expectations for IT/IM Managers: Performance data quality expectations for Large systems Large systems Small spreadsheet scale systems Small spreadsheet scale systems QA Managers: QA adequacy of QA Managers: QA adequacy of Non-environmental data and information used to support the budget process Non-environmental data and information used to support the budget process Environmental data with additional intended uses in performance reporting Environmental data with additional intended uses in performance reporting Performance reporting process Performance reporting process

17. More Opportunities for Management Input Senior managers: views on certification Senior managers: views on certification Program managers Program managers Consult with IT/IM & QA managers on PDCA application to design performance measures more efficiently & apply graded approach Consult with IT/IM & QA managers on PDCA application to design performance measures more efficiently & apply graded approach To ensure performance data are adequate To ensure performance data are adequate To ensure data flow procedures are effective To ensure data flow procedures are effective To ensure verification & validation are transparent and documented To ensure verification & validation are transparent and documented