Presentation on theme: "Get Results From Your Performance Audits. Session Objectives This session is intended to help you: 1. Identify and select high impact performance audits."— Presentation transcript:
Session Objectives This session is intended to help you: 1. Identify and select high impact performance audits. 2. Perform preliminary reviews that enhance audit effectiveness and efficiency with: ◦ The audit team having an understanding of the program or entity being audited. ◦ A prioritized assessment of audit risk and audit opportunity relative to the program or entity. ◦ A basis for developing audit objectives or terminating the audit.
Session Objectives This session is intended to help you: 3. Prepare preliminary review documents that present a persuasive rationale for proceeding with or terminating a performance audit. 4. Develop well-defined audit objectives to achieve a focused audit effort for maximum audit efficiency. 5. Understand how group activity techniques add value to performance auditing.
What Is a Performance Audit? A Performance Audit is an analysis that answers a question. The question is: “What Is?” Which is then compared to: “What Should Be?”
Yellow Book Description Performance audits are defined as audits that provide findings or conclusions based on an evaluation of sufficient, appropriate evidence against criteria. Performance audits provide objective analysis to assist management and those charged with governance and oversight in using the information to improve program performance and operations, reduce costs, facilitate decision making by parties with responsibility to oversee or initiate corrective action, and contribute to public accountability. (2.10)
Examples of High Impact Performance Audit Findings Unemployment Insurance Agency Business Application Modernization (BAM) Project Pharmaceutical Costs
Unemployment Insurance Benefit Overpayments and Nonmonetary Eligibility Determinations We estimated that UIA failed to identify and pursue recovery of overpayments of up to $72.5 million: Overpayments - Claimant wage and benefit cross match process = $17.9 million Cross-program benefit offset process = $12.1 million Verification of claimant identity = $8.2 million Deceased claimants = $0.4 million Untimely nonmonetary eligibility determinations = $26.0 million Ineligible alien claimants = $7.9 million Total = $72.5 million
Unemployment Insurance Benefit Overpayments and Nonmonetary Eligibility Determinations We estimated that UIA did not assess fraud-related penalties of between $120.0 million and $236.6 million: Unassessed Penalties - Classification of claimants’ misrepresentations = $81.5 - $191.8 million Claimant wage and benefit cross match process = $37.8 – $43.4 million Deceased claimants = $0.7 – $1.4 million Total = $120.0 - $236.6 million
Business Application Modernization (BAM) Project Enforcement of Contract Provisions - The Department should have collected liquidated damages resulting from the development contractor failing to meet the agreed-upon implementation dates for BAM. The development contract specified that liquidated damages would be assessed at $200,000 per month if the contractor failed to meet the agreed-upon implementation dates for Phases 3A, 3B, and 3C. The development contractor missed the initial implementation date of August 2007 for Phase 3A (renamed as Release 1). As a result, we calculated that the development contractor owed the State liquidated damages of $12.5 million.
Prisoner Pharmaceutical Costs Potential Savings Based on Seroquel and Risperdal Prescription Data January through July 2010 Percentage reduction in Seroquel10%25%33%50%75%100% Average monthly number of Seroquel scripts2,386 Potential reduction in number of Seroquel scripts2395977871,1931,7902,386 Average monthly cost of Seroquel script $ 335 Projected reduction in Seroquel cost $ 80,065 $ 199,995 $ 263,645 $ 399,655 $ 599,650 $ 799,310 Average monthly cost of Risperdal script $ 38 Projected increase in Risperdal cost $ 9,082 $ 22,686 $ 29,906 $ 45,334 $ 68,020 $ 90,668 Average monthly reduction in cost $ 70,983 $ 177,309 $ 233,739 $ 354,321 $ 531,630 $ 708,642 Annualized potential savings $851,796 $2,127,708 $2,804,868 $4,251,852 $6,379,560 $8,503,704
Prisoner Pharmaceutical Costs PHARMACEUTICAL COSTS Department of Corrections (DOC) Comparison of Michigan's and Other States' Average Atypical Antipsychotic Pharmaceutical Costs Per Prisoner Per Month For the Period February 2010 through July 2010 Note: The chart includes 10 states whose departments of corrections contract with PharmaCorr, LLC, for statewide pharmacy services. As noted in Finding 1, about 80% of Michgan's atypical antipsychotic medications prescribed for DOC prisoners are written by the Department of Community Health and contracted psychiatrists.
How to Identify and Select High Impact Performance Audits? Start with a Risk Ranking Qualitative Quantitative Program/System
Qualitative Risk Ranking Is there a risk to: Public Health, Safety, Protection Service Delivery State Security Information Security & Privacy Economic Growth Education Conservation & Protection of Natural Resources Commercial or Public Transportation
Qualitative Risk Ranking Things to consider: Public Health, Safety, &Protection Injury or loss to life, care & protection of vulnerable citizens, privacy or citizen rights, food safety, agriculture Service Delivery Services not factored into other categories (examples, state park reservation system, hunting permits, licensing), Medicaid payments provided to doctors, eligibility determination, family reunification services, mental health services, etc. State Security Homeland security, emergency preparedness (state & citizen), customs & borders, national guard, escape, recurring offender, parole violations, and MSP. Information Security & Privacy Protection of IT resources, data security
Qualitative Risk Ranking Things to consider: Economic Growth Michigan economy growth, job increase, business increase Education K - 12, higher-ed, adult ed, early childhood, special ed, laws & regulations, school aid, school improvement grants Conservation and Protection of Natural Resources Air, land, waste, water, forests, wildlife Commercial or Public Transportation Roads, rail, public transportation, bridges, ferries, aeronautics
Qualitative Risk Ranking 0 = Not Relevant 1 = Low Relevance 2 = Moderate Relevance 3 = High Relevance
Quantitative Risk Ranking Is there a risk of: State Revenue Sources Not Being Realized or Collected Ineffective and/or Inefficient Use of State Resources Improper Payments (State or Federal) Contingencies of Potential Liabilities Value/Condition of Major State Assets
Quantitative Risk Ranking Things to consider: Value of Major Assets Need for replacement, need for repairs, and need for maintenance using State resources Examples of Assets: infrastructure, bridges, roads, property, IT equipment State Revenue Sources Not Being Realized or Collected Failure to collect tax revenues or fees, incorrect federal match, underreporting of accounts receivable Contingencies or Potential Liabilities Financial obligation, debt, claim
Quantitative Risk Ranking Things to consider: Ineffective and/or Inefficient Use of State Resources Not seeking federal grants, higher priced contracts, use of overtime, staffing allocations, potential privatization, combined departmental operations, program does not function as intended, program provides no value to State or stakeholders, ineffective program Improper Payments (State and Federal) Payments to ineligible recipient, payments for ineligible service, duplicate payments, payments for services not received, wasted payments that should not have been made, payments of incorrect amount (overpayment & underpayment)
Quantitative Risk Ranking 0 = Not Relevant 1 = Less than $5 million 2 = $5 million - $10 million 3 = $10 million - $50 million 4 = Over $50 million
Program/System Level Risk Ranking Is there a risk of: Unauthorized Access of Loss of Sensitive Information Persons/Clients at Risk if Program Fails to Provide Services Internal Monitoring/Oversight Reduced Confidence in Government Legislative/Media Interest Material Control Weaknesses Reliability of External Data
Program/System Level Risk Ranking Things to consider: Reliability of External Data For example - daycare billings from providers, data from school districts, Medicaid provider billings. Lack of oversight of third party administrators. Program Environment Unreliable decision-making data, competent staff, management oversight, tone at the top, high staff turnover, limited resources, segregation of duties, limited evaluation of program data Internal Monitoring/Oversight Functioning internal audit
Program/System Level Risk Ranking Things to consider: Persons/Clients at risk of program fails to provide services Clients not receiving assistance, medical treatment, or needed services; persons harmed, neglected, or killed because of failure to keep safe or protect Unauthorized access or loss of sensitive information Confidentiality of information, potential impact to organizational operations, assets, or individuals Reduced confidence in government Past program failure
Program/System Level Risk Ranking Things to consider: Legislative/Media Interest Recent or current legislative interest Material Control Weaknesses Lack of prior audit finding correction, demonstration to resolve material control weaknesses, progress to strengthen controls to address material control weaknesses Other Unusual risks such as fraud, significant safety risks, other extreme high risks not fully measured elsewhere in model
Program/System Level Risk Ranking 0 = Not Applicable 1 = Low Risk 2 = Medium Risk 3 = High Risk
Audit Risk Ranges High Risk Audit = 77 and Up Moderate Risk Audit = 39-76 Low Risk Audit = 0-38
How do I begin a Performance Audit? How do I begin a Performance Audit? To determine the SCOPE of your performance audit, start by gathering INFORMATION and obtaining an understanding of the program or entity.
Exercise 1 As auditors, we are used to being observant. But just How Observant Are You? 1. On a standard traffic light, is green on the top or bottom? 2. When you walk, does your left arm swing with your right or left leg? 3. On the U.S. flag, is the top stripe red or white? 4. Which way does water go down the drain: clockwise or counter-clockwise? 5. Which way do fans rotate? 6. Whose face is on a dime? 7. Do books have even numbered pages on the right or left side? 8. How many curves are there in a standard paper clip? 9. How many lug nuts are on a standard car wheel? Great Job!!
Determine the Scope and Gather Information During the Preliminary Review The primary goal during the preliminary review is to gain the best understanding of the agency/program and its processes. This will allow the audit team to identify deficiencies and risks, and plan the audit scope and methodology.
Preliminary Review Process Obtain an overall understanding of the audited activity. Quickly gather information without detailed verification. Gain an understanding of internal control. Identify audit risk and audit opportunity.
Preliminary Review Process Audit Risk The degree to which activities are exposed to the potential for ineffective or inefficient use of resources; the inappropriate disclosure of data; or to forms of fraud, waste, and abuse which could result from inadequate controls.
Preliminary Review Process Audit Opportunity The potential for developing audit findings related to effectiveness and efficiency of a program.
Preliminary Review Process Preliminary Review Considerations What are the program’s mandates, responsibilities, functions and activities? Which functions are most critical to the overall operation of the program? What are the risks related to the program? What responsibilities involve/create the most problems? What changes would make your program more effective in accomplishing its designated responsibilities? Who are the program’s clients/customers? Does program management have any concerns that should be considered? Have recent reorganizations occurred (downsizing)?
Preliminary Review Process Preliminary Review Procedures/Methodologies Interview program personnel. Perform analytical reviews of expenditures, revenue, and data pertinent to the program’s mission. Analyze prior audit results of audits of similar programs in other states. Review other external or internal audits. Review program mission, goals, objectives, policies, & procedures. Obtain organization charts and review staffing levels, including the number and type of employees. Research professional literature. Tour facilities, including off-site locations. Review other states’ similar programs and their mission, goals, objectives, and requirements. Obtain norms and standards pertinent to the program’s operations.
Preliminary Review Summary Preliminary Reviews enhance audit effectiveness and efficiency with: The audit team having an understanding of the program or entity being audited. A prioritized assessment of audit risk and audit opportunity relative to the program being audited. A basis for developing audit objectives. Achievable audit budgets.
Developing Audit Objectives What is the audit going to accomplish? Specific audit objectives are critical to the process of auditing smarter and focusing our audit efforts. Without precisely stated objectives, the risk is that the audit work will not produce the desired results. Therefore, it is important to define the issue, problem, or concern that the audit is to examine.
Types of Objectives Effectiveness ◦ Success in achieving mission and goals Efficiency ◦ Achieving the most outputs and outcomes practical with the minimum amount of resources.
Audit Objective Characteristics Clarity ◦ Objectives should be stated clearly to inform the reader what aspects of the program, activity, or function are being assessed. Neutrality ◦ Objectives should be stated in neutral terms so that the reader understands that we gathered and analyzed data without bias.
Writing Solid Audit Objectives Objectives should be: ◦ Specific ◦ Focused on one topic Objectives should not be: ◦ Broad ◦ Compound
Writing Solid Audit Objectives Ask Yourself: ◦ Will I be able to conclude on this objective? ◦ Will I have multiple conclusions ? Effective at one thing; not effective at another thing. ◦ Will my conclusion make sense? ◦ Is this something we even want to conclude on? We may not want to conclude that tax collections were accurate, but we would conclude that their efforts to collect taxes were effective.
Audit Objectives - Example 1 To assess the effectiveness of the Department's efforts to establish performance measures, identify best practices, and evaluate services provided to adults transitioning from foster care to self- sufficiency.
Audit Objectives - Example 1 Final Report Version To assess the effectiveness of DHS's efforts to evaluate services provided to youth transitioning from foster care to self-sufficiency. To assess the effectiveness of DHS's efforts to ensure the propriety and reasonableness of discretionary payments and post-secondary educational assistance payments made on behalf of youth transitioning from foster care to self-sufficiency. To assess the effectiveness of DHS's efforts to ensure that training for caseworkers is tailored to address youth transitioning from foster care to self-sufficiency.
Audit Objectives - Example 2 To assess the effectiveness of the Correctional Facility's efforts to comply with policies and procedures.
Audit Objectives - Example 2 Final Report Version To assess the effectiveness of DOC's efforts to comply with selected policies and procedures related to safety and security at the St. Louis Correctional Facility.
Audit Objectives – Example 3 To assess the effectiveness of the Bridges information system.
Audit Objectives – Example 3 Final Report Version To assess the effectiveness of DHS and DTMB's efforts to implement change controls over the Bridges application and data. To assess the effectiveness of DHS and DTMB's efforts to implement controls to ensure the accuracy, completeness, and timeliness of Bridges interfaces.
Audit Objective Summary Well-defined audit objectives achieve a focused audit effort and maximum audit efficiency.
Referent Group Meetings Objectives Documents to Bring Topics Discussed
Referent Group Meetings Help decide the scope of the audit. Objectives of a referent group meeting: Use a risk-based approach to select activities & programs for audit Agree upon audit focus, scope, and methodology Bring people together Maximize the use of limited resources Produce significant savings in audit hours
Referent Group Meeting Who Attends: Audit Supervisor Audit Team Primary Audit Division Administrator and/or Audit Manager Concurring Audit Division Administrator and/or Audit Manager Performance Audit Coordinator Deputy Auditor General State Relations Officer
Referent Group Meeting What happens during the meeting? Bring several people together (brainstorming). Decide on focus and scope of audit. Discuss audit methodology. Discuss identified risks and probable audit findings. Review the wording of the audit objectives. Strive to include only high impact areas. Review budget proposals. Alleviate fears of missing an issue. Discuss fraud risk factors. Produce significant savings in terms of audit hours.
Documents Used at the Referent Group Meeting Narrative document that includes Description of Agency/Program Preliminary Objectives Primary Questions that Objectives Will Answer Audit Budget Proposed Budget Hours Estimated Fieldwork Completion Date Estimated Staff Release Dates
Documents Used at the Referent Group Meeting Audit Approach Matrix Preliminary Audit Objectives Known Findings Potential Findings Preliminary Testing Performed Proposed Audit Methodology Audit Objectives Memo
Other Topics Discussed at the Referent Meeting Efficiency Aspects of the Audit Identify “efficiency” related issues (dollar savings) Develop “efficiency” objective (if warranted) Legislative Interests/Concerns Audit-related interests/concerns communicated by legislators and/or their staff
Other Topics Discussed at the Referent Meeting IT Systems Are there any significant information systems related to the audit objectives? Do we need an objective regarding the reliability and integrity of data within significant systems? Need for IT auditor assistance? Potential for “Data Mining” (downloads, extractions, and analyses)? Fraud Risk Factors (SAS 99 Compliance) Discuss significant results from “Fraud Brainstorming”
Yellow Book Changes Related to Performance Audits (from 2011 Government Auditing Standards) The discussion of validity as an aspect of the quality of evidence has been revised to indicate that it is the extent to which evidence is a meaningful or reasonable basis for measuring what is being evaluated. In other words, validity refers to the extent to which evidence represents what it is purported to represent. (6.60b) The discussion of the sufficiency and appropriateness of computer- processed information now indicates that the assessment of the sufficiency and appropriateness of computer-processed information includes considerations regarding the completeness and accuracy of the data for the intended purposes. (6.66)
Yellow Book Changes Related to Performance Audits (from 2011 Government Auditing Standards) The auditor’s responsibilities for communicating identified internal control deficiencies that are not significant to the audit have been clarified. Related documentation requirements and those related to noncompliance with provisions of contracts or grant agreements or abuse that are not significant to the audit have been removed. (7.19, 7.22) The fraud reporting requirement is now limited to occurrences that are significant within the context of the audit objectives (7.21), with a requirement to communicate in writing other instances of fraud that warrant the attention of those charged with governance. (7.22)
Yellow Book Changes Related to Performance Audits (from 2011 Government Auditing Standards) The requirement that audit organizations develop policies to address requests by outside parties to obtain access to audit documentation has been removed. (2007 GAGAS, 7.84) Early communication of deficiencies has been added as a consideration auditors may follow in the course of the performance audit. (6.78)
You Too Can Get Results From Your Performance Audits by – Conducting a risk assessment to select high impact audits Conducting a preliminary review to establish scope Developing clear audit objectives Brainstorming with a team