1. AAA Monitoring Framework
Check roster to see if SUA staff are participating
Elizabeth O’Neill, Program Evaluator
State of Oregon DHS APD State Unit on Aging
March 2014

2. Overview Purpose of initiating monitoring efforts State assumptions
Central questions
Evaluation vs. monitoring
Monitoring framework adapted from GAO
Monitoring next steps: instructions & effort

3. Purpose of Monitoring
Create consistent framework for quality assurance with local variation
Share best practices statewide
Identify & correct areas of weakness
Comply with federal and state regulations; satisfy previous and future audits
Strengthen service delivery
Provide historical context for previous monitoring efforts by SUA

4. Assumptions

5. Assumptions
SUA assumes AAAs have in-depth, expert knowledge of the national and state requirements.
SUA assumes that AAAs are doing regular, thorough monitoring of their program and operational systems.
SUA assumes AAAs are operating under the practices they believe to be the most efficient and equitable.
SUA believes that AAA are actively engaged in sharing quality assurance practices, though this process could be substantiated.
SUA believes that we should be able to share program quality information with legislators and other stakeholders in a more substantive way.
Sorry for the text heavy slide, we just feel the content is critical.

6. Assumptions continued
SUA believes that standards may be strengthened and efficiencies gleaned through ongoing monitoring efforts.
SUA believes the statewide aging network needs a more systemic monitoring system.
SUA believes that a self-reported monitoring system will produce factual data and honest self-reflection.
SUA believes we can leverage our small capacity through AAA self monitoring and make the process more efficient for AAAs.
Systemic monitoring system includes the refinement or development of standards, trainings on quality control, and how to implement continuous quality improvement.
Monitoring system will consist of SUA conducting periodic onsite and desk reviews to learn mechanics of program successes, assist with areas of development, and verify reported numbers.

7. Monitoring Intent Program Standards Fiscal Business & Operations
This monitoring effort is specific to program standards.
Fiscal monitoring occurs with monthly reconciliation of your invoice, and NAPIS data verification for dollars to service units. It is not part of this assessment.
Monitoring related to AAA business and operations is captured in the Area Plan and also not part of these assessments.

8. Central Questions
Do the reviewed programs meet national and state standards?
How are specific AAAs meeting program standards?
What are local and statewide trends?
What are the best practices?
How do AAAs propose continuing areas that are working well and fixing areas that are not meeting standards?
What can the SUA do to better support AAAs in meeting standards?
Each is local and statewide
Meet standards to provide quality services

9. Understanding Evaluation vs. Monitoring
See GAO handout. Monitoring (forest) is checking on overall process, procedures, compliance, communication. Monitoring answers if the programs are meeting standards. Information gleamed may help inform evaluation design, or help to serve as early indicators of success/failure. Program Evaluation (trees) seeks to answer how well the program is working and why, not just if it met its objectives. Program Evaluation is often more in-depth and not ongoing.

10. Elements of a Finding Criteria Effect Cause Condition Recommendations
What should be
Effect
What is the impact
Cause
Why the condition happened
Condition
What is
The U.S. Government Accountability Office (GAO) developed the Government Auditing Standards known as the Yellow Book. The Elements of a Finding, also referred to as the Five Findings, is the evaluation model used in the Government Auditing Standards.
The Five Findings is the basis of reporting for the GAO and all state and local auditing entities. I find the framework useful to break down the mystique of evaluation and I think it will allow AAAs to accurately reflect on their situation through self and peer monitoring.
Recommendations
What will address the cause

11. Finding
Finding is the compilation of collecting specific data, analyzing it and determining the condition.
A finding is dependent on evaluation objectives.
A finding can be positive or negative in this context.
A finding is the result of a logical pulling together of information (data gathering), processing the information (data analysis), arriving at conclusions on the basis of the sum of the information about an organization, program, activity, function, condition, or other matter which was analyzed or evaluated and considered to be of interest, concern, or of use to the client.” Institute of Internal Auditors, Puget Sound Chapter 2004

12. Elements of a Finding Criteria Effect Cause Condition Recommendations
What should be
Effect
What is the impact
Cause
Why the condition happened
Condition
What is
The U.S. Government Accountability Office (GAO) developed the Government Auditing Standards known as the Yellow Book. The Elements of a Finding, also referred to as the Five Findings, is the evaluation model used in the Government Auditing Standards. The Five Findings is the basis of reporting for the GAO and all state and local auditing entities. I find the framework useful to break down the mystique of evaluation and I think it will allow AAAs to accurately reflect on their situation through self and peer monitoring.
Recommendations
What will address the cause

13. Criteria “What should be?”
Provides information so that the user will be able to determine what is required, desired, or expected from the program or operation.
Criteria is often program standards or rules.
The criteria are easier to understand when stated fairly, explicitly, and completely.
For this monitoring effort, the criteria is the OAA guidelines with any state interpretations. For nutrition and legal services, the standards are posted in the instructions documents.

14. Elements of a Finding Criteria Effect Cause Condition Recommendations
What should be
Effect
What is the impact
Cause
Why the condition happened
Condition
What is
The U.S. Government Accountability Office (GAO) developed the Government Auditing Standards known as the Yellow Book. The Elements of a Finding, also referred to as the Five Findings, is the evaluation model used in the Government Auditing Standards. The Five Findings is the basis of program evaluation reporting for the GAO and all state and local auditing entities. I find the framework useful to break down the mystique of evaluation and I think it will allow AAAs to accurately reflect on their situation.
Recommendations
What will address the cause

15. Condition “What is?”
Condition provides evidence on what is found regarding the actual situation.
Reporting the scope or extent of the condition allows the user to gain an accurate perspective about program status.
The monitoring process will reveal the information about the central questions.
Information in the assessment forms will provide information on the condition for the specific program areas.

16. Elements of a Finding Criteria Effect Cause Condition Recommendations
What should be
Effect
What is the impact
Cause
Why the condition happened
Condition
What is
Recommendations
What will address the cause

17. Cause “Why the condition happened?”
Cause provides persuasive evidence on the factor(s) responsible for the difference between condition & criteria.
Cause explains why the difference between the ideal & actual circumstances exist.

18. Elements of a Finding Criteria Effect Cause Condition Recommendations
What should be
Effect
What is the impact
Cause
Why the condition happened
Condition
What is
Recommendations
What will address the cause

19. Effect Difference between “what is” & “what should be?”
Effect provides a clear, logical link to establish the impact of the difference between condition & criteria.
Effect is easier to understand when it is stated clearly, concisely, and if possible, quantifiable.

20. Elements of a Finding Criteria Effect Cause Condition Recommendations
What should be
Effect
What is the impact
Cause
Why the condition happened
Condition
What is
Recommendations
What will address the cause

21. Recommendations
What can be perpetuated to replicate success? What actions are needed to correct the cause?
Recommendations are designed to replicate success and remedy areas of weakness. There are no inherent negative intentions for the Five Findings, or for the specified recommendations.
Recommendations address how to align criteria and condition and should directly address the cause.

22. Elements of a Finding: SUA’s Model
Criteria
What should be
State or Federal program guidelines & contractual requirements
Effect
What is the impact
Cause
Why the condition happened
Reflection from AAA
Reflection from AAA
Condition
What is
Results of monitoring efforts as demonstrated in the assessments
This slide explains how the GAO model applies to the monitoring effort of SUA and AAAs.
Criteria “gold” standard
Green are the action areas for AAAs in the assessment forms.
Recommendations
What will address the cause
Designed by AAA
Followed up and celebrated by SUA

23. Monitoring Instructions
Also contains monitoring responsibilities for SUA and AAAs
Ramifications for lack of compliance
Links to program standards for OPI, Family Caregiver Support Program and Health Promotion.
Assessment tools for OPI, FCSP & Health Promotion due to SUA by 5/23/2014.

24. Monitoring Effort
All monitoring efforts will be done by the reporting AAA for
A small number of programs will pilot peer monitoring in where AAAs will partner and conduct an assessment for one another.
Both self-monitoring and peer-monitoring will be verified by SUA through abbreviated on-site reviews.
Peer monitoring will foster greater shared understanding between AAAs and help us identify successes.
Onsite reviews done by AAA liaison, and SUA program expert for training.

25. Location of Monitoring Materials
SUA website 
Area Agency on Aging business 
AAA Self Monitoring
This webinar will also be posted at the SUA website.

26. Resources GAO Yellow Book http://www.gao.gov/yellowbook/overview
GAO Designing Evaluations
GAO Performance Management and Evaluation
Institute of Internal Auditors
AARP Public Policy Institute’s focus on Livable Communities
Program Evaluation and Practice: A Comprehensive Guide; Mertens and Wilson
Improving Performance in Service Organizations: How to Implement a Lean Transformation; Miller, Bogatova, Carnohan

27. Elizabeth O’Neill, Program Evaluator
State of Oregon DHS APD State Unit on Aging