1. 1 Economic Directorate’s Quality Audit Program Steven S. Klement Office of Statistical Methods and Research for Economic Programs

2. Agenda Origins of the Quality Audit Program (QAP) Scope Current Program Some Results Discussion 2

3. Origins of the QAP Late 1990’s Beginning of Census Bureau Standards 2001Office of Management and Budget (OMB) Quality Guidelines 2004Establishment of Economic Directorate’s quality management program 2006OMB Standards and Guidelines for Statistical Surveys 3

4. Origins of the QAP What form should the Quality Management Program take? Quality Profile vs. Quality Audit Developing the self-assessment tool Administration of the audit process 4

5. Scope There are about 60 programs in the Economic Directorate that are subject to QAP Each program will undergo an audit every five years QAP must average twelve audits per year to meet this goal 5

6. Scope (cont.) The Quality Audit Program (QAP) looks for compliance with the OMB Standards and Guidelines for Statistical Surveys (Sep 2006) –Does the program’s documentation show compliance? –Is there a plan to produce quality estimates? 6

7. Scope (cont.) Auditors do not look in depth to see if execution matches the plan –If the plan calls to compute a variance, the auditor looks to see that there is a variance in the final data product –QAP does not look at the actual computer program which calculates the number 7

8. Conduct of the QAP Currently run by the Lead Auditor for Economic Programs –New Assistant Lead Auditor position just created Overseen by the Research & Methodology (R&M) Assistant Division Chiefs (ADC) Census Bureau’s Quality Improvement Staff assists in training Volunteer auditors from the Divisions within the Economic Directorate 8

9. Timeline Sep Year Prior Set audit scheduleAudit WeekMeet face-to-face to resolve issues Oct Year Prior Assign auditorsFriday Audit wk Draft final report 9 wks priorEngagement notification sent out 1 wk afterFinal report due 7-9 wks priorConduct training2 months afterPAP plan due 4-7 wks priorProgram area fills in checklist 8 months after1 st PAP review 4 wks priorProgram area presentation 20 months after 2 nd PAP review 1-3 wks priorAuditors review checklist Every year after Review PAP until all items complete 9

10. Scheduling A five-year program is put together by the Lead Auditor and reviewed by the Divisions –Includes all programs not already subject to an outside audit –There are limited exceptions –The Division’s comment on schedule and request deferments The Lead Auditor schedules the actual dates –3-4 months out plans dates for next year’s audits –2-3 months out auditors are selected 10

11. Auditor Selection The Divisions within the Econ Directorate submit managers & supervisors names to be auditors The Lead Auditor: –Coordinates schedules –Assigns auditors to each audit. 11

12. Conduct of the Audit Engagement Notification sent nine weeks out Training conducted 7-9 weeks out Checklist is due 4.5 weeks out Program area presentation is four weeks out 12

13. Checklist Example 13 3.1 Data Editing Rating (Auditor) (Auditor) Summary of findings and recommendations {Initials} 1. Agencies must edit data to mitigate or correct detectable errors. Condition: (Program Mgr.) – Objective evidence for existing conditions that supports or refutes compliance with the criteria of the standard.{Initials} Concern: (Auditor) – Cause: The factors responsible for the difference between the condition and criteria. Effect: The consequences (benefits / risks) due to the difference between the condition and criteria. {Initials} Recommendation: (Auditor) – Recommendations to 1) bring the program area in compliance with the standard and for 2) alternative approaches to improve program performance. {Initials} Response: (Program Mgr) – The Program Manager’s position regarding the Auditors concerns or recommendations. {Initials}

14. Conduct of the Audit (cont.) Document review is 1-3 weeks prior –It is the first part of the actual audit –Done by auditors alone On-site week is when auditors meet with program area –Draft findings are discussed and then finalized –Issues are resolved in groups or one-on-one 14

15. Final Report The final report is authored by the Lead Auditor –Reviewed by the auditors –Program Area is given chance to respond After receiving input and completing revisions, report is sent to management 15

16. Program Action Plan (PAP) Conducting an audit is of little value in of itself The value comes in when improvements are made Therefore, the QAP directs that the program area develop a PAP –The PAP is the plan for corrective action –Due two months after the audit –The auditors comment on the feasibility of the PAP to address audit concerns 16

17. Follow Up Finally, progress with the PAP is followed by the Lead Auditor Once the PAP is approved Lead Auditor: –Checks for progress after six months –Continues to check yearly until PAP is completed 17

18. Some Results Nonresponse Bias Study: –It was found that there was a general lack of NR bias studies in the Directorate –Study required if response rate is below 80% –Planning for study required if response rate is near 80% OR falling towards 80% Directorate sponsored a pilot NR study The Directorate created a new team to help programs conduct these studies 18

19. Some Results (cont.) Document storage: –Most programs are well-documented –However, the documentation is scattered Multiple document storage systems in use Many programs did not use any system –For a number of programs, the audit was the first time all documentation was in one location The Directorate will roll out a new centralized easy to use system in summer 2010. 19

20. Some Results (cont.) Holds people accountable for their documentation –It must be accessible and understandable –It must outline processes actually used –Both help in knowledge transfer over time Compliance with standards is brought to the forefront (instead of an afterthought) 20

21. Key Finding Almost everything one program is deficient or needs improvement on… …another program has a best practice for! The question: ‘How much is transferable?’ Auditors take good ideas back to their own programs… …and they share their good ideas with the audited program 21

22. QAP Going Forward Look at ways to spread Best Practices Identify areas that require corporate solutions Help identify areas requiring additional or concentrated resources 22

23. Questions? Thank you for your time Steven S. Klement Lead Auditor for Economic Programs steven.s.klement@census.gov (301)763-6598 23

24. URLs OMB Standards & Guidelines for Statistical Surveys (Sep 2006) http://www.whitehouse.gov/omb/inforeg/statpoli cy/standards_stat_surveys.pdf Guidance on Agency Survey and Statistical Information Collections http://www.whitehouse.gov/omb/inforeg/pmc_su rvey_guidance_2006.pdf 24