AAA Monitoring Framework Check roster to see if SUA staff are participating Elizabeth O’Neill, Program Evaluator State of Oregon DHS APD State Unit on Aging March 2014

Overview Purpose of initiating monitoring efforts State assumptions Central questions Evaluation vs. monitoring Monitoring framework adapted from GAO Monitoring next steps: instructions & effort

Purpose of Monitoring Create consistent framework for quality assurance with local variation Share best practices statewide Identify & correct areas of weakness Comply with federal and state regulations; satisfy previous and future audits Strengthen service delivery Provide historical context for previous monitoring efforts by SUA

Assumptions

Assumptions SUA assumes AAAs have in-depth, expert knowledge of the national and state requirements. SUA assumes that AAAs are doing regular, thorough monitoring of their program and operational systems. SUA assumes AAAs are operating under the practices they believe to be the most efficient and equitable. SUA believes that AAA are actively engaged in sharing quality assurance practices, though this process could be substantiated. SUA believes that we should be able to share program quality information with legislators and other stakeholders in a more substantive way. Sorry for the text heavy slide, we just feel the content is critical.

Assumptions continued SUA believes that standards may be strengthened and efficiencies gleaned through ongoing monitoring efforts. SUA believes the statewide aging network needs a more systemic monitoring system. SUA believes that a self-reported monitoring system will produce factual data and honest self-reflection. SUA believes we can leverage our small capacity through AAA self monitoring and make the process more efficient for AAAs. Systemic monitoring system includes the refinement or development of standards, trainings on quality control, and how to implement continuous quality improvement. Monitoring system will consist of SUA conducting periodic onsite and desk reviews to learn mechanics of program successes, assist with areas of development, and verify reported numbers.

Monitoring Intent Program Standards Fiscal Business & Operations This monitoring effort is specific to program standards. Fiscal monitoring occurs with monthly reconciliation of your invoice, and NAPIS data verification for dollars to service units. It is not part of this assessment. Monitoring related to AAA business and operations is captured in the Area Plan and also not part of these assessments.

Central Questions Do the reviewed programs meet national and state standards? How are specific AAAs meeting program standards? What are local and statewide trends? What are the best practices? How do AAAs propose continuing areas that are working well and fixing areas that are not meeting standards? What can the SUA do to better support AAAs in meeting standards? Each is local and statewide Meet standards to provide quality services

Understanding Evaluation vs. Monitoring See GAO handout. Monitoring (forest) is checking on overall process, procedures, compliance, communication. Monitoring answers if the programs are meeting standards. Information gleamed may help inform evaluation design, or help to serve as early indicators of success/failure. Program Evaluation (trees) seeks to answer how well the program is working and why, not just if it met its objectives. Program Evaluation is often more in-depth and not ongoing.

Elements of a Finding Criteria Effect Cause Condition Recommendations What should be Effect What is the impact Cause Why the condition happened Condition What is The U.S. Government Accountability Office (GAO) developed the Government Auditing Standards known as the Yellow Book. The Elements of a Finding, also referred to as the Five Findings, is the evaluation model used in the Government Auditing Standards. The Five Findings is the basis of reporting for the GAO and all state and local auditing entities. I find the framework useful to break down the mystique of evaluation and I think it will allow AAAs to accurately reflect on their situation through self and peer monitoring. Recommendations What will address the cause

Finding Finding is the compilation of collecting specific data, analyzing it and determining the condition. A finding is dependent on evaluation objectives. A finding can be positive or negative in this context. A finding is the result of a logical pulling together of information (data gathering), processing the information (data analysis), arriving at conclusions on the basis of the sum of the information about an organization, program, activity, function, condition, or other matter which was analyzed or evaluated and considered to be of interest, concern, or of use to the client.” Institute of Internal Auditors, Puget Sound Chapter 2004

Elements of a Finding Criteria Effect Cause Condition Recommendations What should be Effect What is the impact Cause Why the condition happened Condition What is The U.S. Government Accountability Office (GAO) developed the Government Auditing Standards known as the Yellow Book. The Elements of a Finding, also referred to as the Five Findings, is the evaluation model used in the Government Auditing Standards. The Five Findings is the basis of reporting for the GAO and all state and local auditing entities. I find the framework useful to break down the mystique of evaluation and I think it will allow AAAs to accurately reflect on their situation through self and peer monitoring. Recommendations What will address the cause

Criteria “What should be?” Provides information so that the user will be able to determine what is required, desired, or expected from the program or operation. Criteria is often program standards or rules. The criteria are easier to understand when stated fairly, explicitly, and completely. For this monitoring effort, the criteria is the OAA guidelines with any state interpretations. For nutrition and legal services, the standards are posted in the instructions documents.

Elements of a Finding Criteria Effect Cause Condition Recommendations What should be Effect What is the impact Cause Why the condition happened Condition What is The U.S. Government Accountability Office (GAO) developed the Government Auditing Standards known as the Yellow Book. The Elements of a Finding, also referred to as the Five Findings, is the evaluation model used in the Government Auditing Standards. The Five Findings is the basis of program evaluation reporting for the GAO and all state and local auditing entities. I find the framework useful to break down the mystique of evaluation and I think it will allow AAAs to accurately reflect on their situation. Recommendations What will address the cause

Condition “What is?” Condition provides evidence on what is found regarding the actual situation. Reporting the scope or extent of the condition allows the user to gain an accurate perspective about program status. The monitoring process will reveal the information about the central questions. Information in the assessment forms will provide information on the condition for the specific program areas.

Elements of a Finding Criteria Effect Cause Condition Recommendations What should be Effect What is the impact Cause Why the condition happened Condition What is Recommendations What will address the cause

Cause “Why the condition happened?” Cause provides persuasive evidence on the factor(s) responsible for the difference between condition & criteria. Cause explains why the difference between the ideal & actual circumstances exist.

Elements of a Finding Criteria Effect Cause Condition Recommendations What should be Effect What is the impact Cause Why the condition happened Condition What is Recommendations What will address the cause

Effect Difference between “what is” & “what should be?” Effect provides a clear, logical link to establish the impact of the difference between condition & criteria. Effect is easier to understand when it is stated clearly, concisely, and if possible, quantifiable.

Elements of a Finding Criteria Effect Cause Condition Recommendations What should be Effect What is the impact Cause Why the condition happened Condition What is Recommendations What will address the cause

Recommendations What can be perpetuated to replicate success? What actions are needed to correct the cause? Recommendations are designed to replicate success and remedy areas of weakness. There are no inherent negative intentions for the Five Findings, or for the specified recommendations. Recommendations address how to align criteria and condition and should directly address the cause.

Elements of a Finding: SUA’s Model Criteria What should be State or Federal program guidelines & contractual requirements Effect What is the impact Cause Why the condition happened Reflection from AAA Reflection from AAA Condition What is Results of monitoring efforts as demonstrated in the assessments This slide explains how the GAO model applies to the monitoring effort of SUA and AAAs. Criteria “gold” standard Green are the action areas for AAAs in the assessment forms. Recommendations What will address the cause Designed by AAA Followed up and celebrated by SUA

Monitoring Instructions Also contains monitoring responsibilities for SUA and AAAs Ramifications for lack of compliance Links to program standards for OPI, Family Caregiver Support Program and Health Promotion. Assessment tools for OPI, FCSP & Health Promotion due to SUA by 5/23/2014.

Monitoring Effort All monitoring efforts will be done by the reporting AAA for 2013-14. A small number of programs will pilot peer monitoring in 2014-15 where AAAs will partner and conduct an assessment for one another. Both self-monitoring and peer-monitoring will be verified by SUA through abbreviated on-site reviews. Peer monitoring will foster greater shared understanding between AAAs and help us identify successes. Onsite reviews done by AAA liaison, and SUA program expert for training.

Location of Monitoring Materials SUA website  Area Agency on Aging business  AAA Self Monitoring http://www.oregon.gov/dhs/spwpd/pages/sua/info-aaa.aspx This webinar will also be posted at the SUA website.

Resources GAO Yellow Book http://www.gao.gov/yellowbook/overview GAO Designing Evaluations http://www.gao.gov/products/GAO-12-208G GAO Performance Management and Evaluation http://www.gao.gov/assets/80/77277.pdf Institute of Internal Auditors http://www.theiia.org/intAuditor/back-to-basics/2013/report-audit-findings/ AARP Public Policy Institute’s focus on Livable Communities http://www.aarp.org/research/ppi/liv-com/ Program Evaluation and Practice: A Comprehensive Guide; Mertens and Wilson Improving Performance in Service Organizations: How to Implement a Lean Transformation; Miller, Bogatova, Carnohan

Elizabeth O’Neill, Program Evaluator State of Oregon DHS APD State Unit on Aging elizabeth.a.oneill@state.or.us