Presentation is loading. Please wait.

Presentation is loading. Please wait.

Inspection, Evaluation, and Performance Auditing Presented By George F. Grob Center for Public Program Evaluation To the Association of Inspectors General.

Similar presentations


Presentation on theme: "Inspection, Evaluation, and Performance Auditing Presented By George F. Grob Center for Public Program Evaluation To the Association of Inspectors General."— Presentation transcript:

1 Inspection, Evaluation, and Performance Auditing Presented By George F. Grob Center for Public Program Evaluation To the Association of Inspectors General New Orleans November 2013

2 Center for Public Program Evaluation Essentials of Evaluations and Inspections What are they? Comparison to Audits Other Related Professional Disciplines Evaluation Types and Approaches Analytic Frameworks Possible Organizational Locations Managing an Evaluation Office Quality Standards 2

3 Center for Public Program Evaluation 3 What Is an Evaluation

4 Center for Public Program Evaluation Evaluation involves assessing the strengths and weaknesses of programs, policies, personnel, products, and organizations to improve their effectiveness. 4 Evaluation—What Is That? American Evaluation Association

5 Center for Public Program Evaluation Examples of Government Sponsored Evaluations Early implementation of emergency response systems after 9-11 Effects of programmatic earmarks for international HIV/AIDS relief programs Case study of Freddie Mac’s loss of $1.2 billion from Lehman Brothers bankruptcy Frequency of and criteria for home visits to foster care families Performance standards for homeless youth centers 5

6 Center for Public Program Evaluation More Examples Unlicensed fire sprinkler contractors performing inspections in Chicago Potential use of a fiscal intermediary by the Chicago department of public health Assessment of New Orleans' System of City Courts and Performance Review of New Orleans Traffic Court Evaluation of the City of New Orleans Delinquent Property Tax Collection Program Promising approaches for State and local government enforcement of prohibition on sale of cigarettes to youth Pharmacies with extreme outlier filling of prescriptions for dangerous, addictive drugs 6

7 Center for Public Program Evaluation Websites of OIG Evaluation Reports Federal Agencies City Government Department of Health and Human Services OIG Office of Evaluation and Inspections Department of Commerce OIG Audits and Evaluations Department of State OIG Audits and Program Evaluations OIG Inspections Department of Veterans Affairs OIG Office of Audits and Evaluations Oversight Reports Department of Housing and Urban Development OIG Evaluations and Inspections Federal Housing Finance Agency OIG Audits and Evaluations City of Chicago Audits, Reviews, Advisories City of New Orleans Audits, Inspections, and Evaluations 7 U.S. Congress Government Accountability Office (GAO) Reports and Testimonies Audits, Inspections, and Evaluations

8 Center for Public Program Evaluation 8 Evaluation Standards

9 Center for Public Program Evaluation Association of Inspectors General Council of the Inspectors General on Integrity and Efficiency Government Accountability Office Institute for Internal Auditors American Evaluation Association 9 Government and Professional Standards for Evaluations and Inspections

10 Center for Public Program Evaluation 10 Association of Inspectors General QUALITY STANDARDS FOR INSPECTIONS, EVALUATIONS, AND REVIEWS General Standards  Staff Qualifications  Independence  Due Professional Care  Qualitative Standards  Quality Control  Planning  Data Collection and Analysis  Evidence  Timeliness  Fraud and Other Illegal Acts  Reporting  Confidentiality  Follow-Up

11 Center for Public Program Evaluation CIGIE Standards for Inspection and Evaluation 11 Timeliness Fraud, Illegal Acts, Abuse Reporting Follow-Up Performance Measurement Working Relationships and Communications Competency Independence Quality control Planning Data Collection and Analysis Evidence Records maintenance

12 Center for Public Program Evaluation GAO Standards for Performance Audits Chapter 3: General Standards ▫Independence ▫Professional Judgment ▫Competence ▫Quality Control Chapter 7: Field Work ▫Planning ▫Supervision ▫Evidence ▫Documentation Chapter 8: Reporting ▫Form ▫Contents ▫Quality Elements ▫Report Issuance and Distribution 12

13 Center for Public Program Evaluation Systematic Inquiry Competence Integrity/Honesty Respect for People Responsibilities for General & Public Welfare 13 AEA Guiding Principles for Evaluators

14 Center for Public Program Evaluation Standards for the Professional Practice of Internal Auditing 14 Institute of Internal Auditors https://na.theiia.org/standards-guidance/mandatory- guidance/Pages/Standards.aspx Discusses evaluation of risk and controls, and requires audit results to be based on analysis and evaluation. No “Evaluation” standard as such

15 Center for Public Program Evaluation OIGs may have responsibilities that go beyond audits and investigations. This additional work includes inspecting, evaluating, reviewing, studying, and/or analyzing government operations and programs for the purposes of providing information for decision-making, and of making recommendations to improve programs, policies, or procedures. The objectives of these processes include: providing a source of factual and analytical information, monitoring compliance, measuring performance, and assessing the efficiency and effectiveness of operations. 15 Inspections, Evaluations, and Reviews—AIG Definition Source: Association of Inspectors General

16 Center for Public Program Evaluation Systematic and independent assessments of the design, implementation, and/or results of an Agency’s operations, programs, or policies. They can be used to determine efficiency, effectiveness, impact, and/or sustainability of agency operations, programs, or polices. 16 Inspections and Evaluations—Federal IG Definition Source: Council of the Inspectors General on Integrity and Efficiency

17 Center for Public Program Evaluation They provide information that is timely, credible, and useful for agency managers, policymakers, and others. ▫providing factual and analytical information; ▫measuring performance; ▫determining compliance with applicable law, regulation, and/or policy ▫identifying savings and funds put to better use; ▫sharing best practices or promising approaches; ▫assessing allegations of fraud, waste, abuse, and mismanagement. 17 Inspections and Evaluations—Federal IG Definition Source: Council of the Inspectors General on Integrity and Efficiency

18 Center for Public Program Evaluation 18 The Proving Professions

19 Center for Public Program Evaluation 19 Auditors and Evaluators EvaluationAudit

20 Center for Public Program Evaluation 20 Auditors and Evaluators EvaluationAudit

21 Center for Public Program Evaluation 21 Auditors and Evaluators EvaluationAudit Performance Audit Performance Evaluation Financial Audits Attestation Audits Non-Audit Services Issue Analysis Inspections Reviews A Multitude of Analysis Types

22 Center for Public Program Evaluation 22 Other Professional Disciplines Evaluators/Inspectors Come From Many Professions Evaluation Inspection Audit Social Science Economics Operations Research Analysis Policy Analysis Public Administration Program Specialties ▫Health Care ▫Housing ▫Defense ▫Trade ▫International Assistance

23 Center for Public Program Evaluation 23 Legal Analysis Inspections Economics Evaluation Operations Research Applied Social Research Psychology Statistics Public Administration

24 Center for Public Program Evaluation 24 Legal Analysis Inspections Economics Evaluation Operations Research Applied Social Research Psychology Statistics Public Administration

25 Center for Public Program Evaluation What About Investigators? 25 EvaluatorsInvestigators ?

26 Center for Public Program Evaluation Useful Interplay 26 Investigators Evaluators Fraud leads Fraud trends Tech assist

27 Center for Public Program Evaluation 27 Evaluation Types and Approaches

28 Center for Public Program Evaluation Evaluation Types and Approaches There are many legitimate and useful forms of evaluations Each has its purpose. The methodological approach must be adapted to it Generally accepted professional guidelines are available to guide the work. 28

29 Center for Public Program Evaluation 29 As A Program Matures The Evaluation Questions Change How is the implementation going? Any Issues? Are service providers in compliance with program rules? Are some service providers doing better than others? How so? Are the required services being delivered? Are impacted citizens or businesses satisfied with program results? How do the services stack up to goals and standards? Have conditions improved as intended?

30 Center for Public Program Evaluation 30 And So Do the Analytic Approaches and Methods Used Some inspections and evaluations need to be done fast Others need more time Some require more resources than others All must be done professionally Following are some evaluation and inspections types commonly used today.

31 Center for Public Program Evaluation 31 Inspection/Evaluation Types Compliance Review Performance Review Landscape Survey Early Implementation Reviews Impact Assessment Report Card Process Review

32 Center for Public Program Evaluation 32 More Inspection/Evaluation Types “Just the Facts” Review Report from “On the Ground” “Try it out” Reviews Client Satisfaction Surveys Before and After Assessment Control System Survey Program Logic Model

33 Center for Public Program Evaluation 33 Analytic Frameworks

34 Center for Public Program Evaluation 34 The Standard IG Analytic Model FINDINGS -- Criteria -- Condition -- Effect -- Cause RECOMMENDATIONS

35 Center for Public Program Evaluation What the Federal IG Blue Book Says Center for Public Program Evaluation 35 “Findings often have been regarded as containing the elements of criteria, condition, effect, and, when problems are found, cause. However, the elements needed for a finding depend entirely on the objectives of the inspection. Thus, a finding or set of findings is complete to the extent that the inspection objectives are satisfied and the report clearly relates those objectives to the applicable elements of a finding.”

36 Center for Public Program Evaluation Two Analytic Frameworks Performance AuditPerformance Evaluation Findings ▫Criteria ▫Condition ▫Effect ▫Cause Evidence ▫Physical ▫Documentary ▫Testimonial ▫Analytic Recommendations Objective Questions to Be Answered Analytic Approach ▫Mixed Methods ▫Qualitative ▫Quantitative Findings Recommendations 36

37 Center for Public Program Evaluation 37 METHODOLOGIES

38 Center for Public Program Evaluation 38 The Fundamentals The Correct Methods to Use Are the Ones Most Compatible With the Purpose of the Study Always Use More Than One Method ▫At least one quantitative ▫At least one qualitative

39 Center for Public Program Evaluation 39 Quantitative and Qualitative Methods Quantitative methods involve things you can count ▫Even if they are subjective answers to a closed ended questionnaire Qualitative methods are applied to the interpretation of statements ▫Especially from interviews ▫But also from observations and in documents Reviews are far more compelling if both methods are used ▫Otherwise people will say: “Numbers always lie” or “They always say that”

40 Center for Public Program Evaluation 40 The Most Common Methods Used by Evaluators Background Search On-Site Inspections Review of Administrative Files Sampling Data Collection Instruments Surveys Focus Groups

41 Center for Public Program Evaluation 41 Some Exercises

42 Center for Public Program Evaluation You Be the Evaluator Early implementation of emergency response systems after 9-11 Frequency of and criteria for home visits to foster care families Pharmacies with extreme outlier filling of prescriptions for dangerous, addictive drugs 42

43 Center for Public Program Evaluation The Quick Start 1. State the purpose 2. Ask two or three key questions 3. Choose the methods for each question  Quantitative  Qualitative 4. Conduct the Field Work 5. Analyze the data 6. Construct Findings and Recommendations 7. Draft the report Center for Public Program Evaluation 43

44 Center for Public Program Evaluation Discussion 1. Would your approach work as either an audit or evaluation? 2. Who will use the findings and recommendations? 3. How will you know that the recommendations have been acted upon? 4. What can you expect will be the result? 44

45 Center for Public Program Evaluation 45 Organizing and Managing the Evaluation Function

46 Center for Public Program Evaluation Possible Organizational Location 46 Inspector General Evaluation AuditInvestigation Evaluation

47 Center for Public Program Evaluation Managing an Evaluation Office 47 Work Planning Operating Procedures Quality Control Quality Assurance Professional Development These are generally similar to audit practices Take Them Seriously !

48 Center for Public Program Evaluation Training Priorities for Evaluators Highest PriorityOther Worthwhile Skills General Evaluation Concepts Writing to the Point Survey Design Interviewing Descriptive Statistics Sampling Spread Sheets Graphs Focus Groups Data Base Development Randomized Control Trials Team Leadership Programmatic Knowledge 48

49 Center for Public Program Evaluation Work Planning Combine Audit and Evaluation Focus on risk and return on investment Use a Tactical Approach ▫To systematically fill gaps ▫To develop staff capacity ▫To keep up the attention on important areas 49

50 Center for Public Program Evaluation 50 Program Area A Program Area B Program Area C Program Area D A strategic plan might consist of several families of program studies, some in the early stages, some further along, even in final stages

51 Center for Public Program Evaluation 51 Legal Analysis Inspections Economics Evaluation And Audit Operations Research Applied Social Research Psychology Statistics Public Administration Listen to What They Have to Say

52 Center for Public Program Evaluation 52 Good Luck!

53 Center for Public Program Evaluation George F. Grob Center for Public Program Evaluation


Download ppt "Inspection, Evaluation, and Performance Auditing Presented By George F. Grob Center for Public Program Evaluation To the Association of Inspectors General."

Similar presentations


Ads by Google