1. Statewide AT Program Leaders Symposium Using Data and the Program Review Manual for Systematic Program Improvement May 17, 2011 Sara Sack--Kansas

2. Using Data & The Review Manual for Program Evaluation  Kansas has a “thing” about data—validity and reliability are extremely important  “Collecting good quality data plays a vital role in supplying objective information for the problems under study so that some analytical understanding of the problems and hence solutions can be obtained.”  “Making decision on the basis of poor quality data is risky and may lead to disastrous results, as the situation may be distorted and hence all subsequent analyses and decision making will rest on a shaky ground.” (Hong Kong Polytechnic University)

3. Using Data & The Review Manual for Program Evaluation  Data quality involves ensuring the accuracy, timeliness, completeness and consistency of data used by an organization while also making sure that everyone who uses the data has a common understanding of what the data represents. (Andrew Greenyer)  Kansas ATK staff achieve 85% reliability on 75 item coding task before being allowed to enter data into database.

4. Analyzing AT Data: Management  Weekly management level meeting with rotating agenda  Review grants & contract obligation & performance  AT Access Site Data Review  Collaborations & Partnerships  Communication & Public Awareness  Electronic database (and entry required by the 5 th of the month for all subcontractors) makes data pull easy  Admin level reports provide data on their AT Site’s performance: completed goals by AT category, age, county, and outcomes collection within a specified date range  Super user level has access to all reports  60:40 performance easy to obtain

5. ATK – 60:40 Split

6. Southeast AT Access Site Parsons (SKIL)

7. Analyzing AT Data: All Staff  All ATK staff meet every other month generally face to face (webinars in the winter) to obtain training and discuss data and performance  So….you would think we would have program evaluation pretty well covered, right?

8. Utilizing the Review Manual for Program Evaluation  Review process requires the program to compare 3 years of data. ATK’s data review generally compared performance to the previous year or to the same quarter of the previous year  Review process also requires programs to review their financial data for a three year period.  Looking at a three year period provided additional, important information

9. ATK-KEE REUTILIZATION DONATIONS 2007-20082008-20092009-2010 Medicaid32 $73,884 52 $64,786 70 $128,839 Public/private670 $622,243 852 $866,665 716 $851,388 KRS (low vision) 7 $8,100 702 $696,127 904 $931,451 793 $988,327

10. REUTILIZATION AT Category2008 N = 620 2009 N = 678 2010 N = 698 Vision2 – 3,0001 – 2,1008 – 12,480 Hearing000 Speech Comm.3 – 20,1005 – 26,7905 – 12,030 Learning, Cognition 3 – 3,6003 – 2,5080 Mobility, Seating288 – 416,793281 – 538,705337 – 637,912 Daily Living308 – 221,053382 – 288,128345 – 275,246 Environmental Adapts 01 – 3,2000 Vehicle Mods.000 Computers & Related 15 – 10,3345 – 10,0403 – 2,336 Recreation, Sports 1 – 1,00000 TOTAL$675,880$871,471$940,004

11. What New Information Did We Gain From This Three Year Analysis?  In the area of Demonstration, we saw that our efforts to educate policymakers about the value of demonstration really were beginning to take hold and pay off  In Device Loan we were reminded of our efforts to “act super frugally” and invest savings into equipment purchases  AT Access Sites were reminded of the positive effects of specific Public Awareness Activities and vowed to try some of those efforts again

12. What New Information Did We Gain From This Three Year Analysis?  In Financial Loan of our AFP, the three year scan highlighted shifts in applications by equipment categories more clearly than just the two year scan  The funding support service that ATK provides as part of the State Financing showed major shifts in funders’ support for various technologies

15. What New Information Did We Gain From The Analysis of Three Years of Financial Data?  In the breakout and analysis of cost allocations trends could be identified and, as with other sections, staff recalled particularly strong efforts and activities.  In the area of Transition, the three year view of data showed how much was accomplished with very little money

16. Program Improvement Initiatives Based on Self-Evaluation & Program Review  Reviewers are most likely in the audience so we don’t want to tip our hand but….  Continue to refine our online database—more automated features so staff have more time for consumers  Continue to analyze use of time—productivity is high so want to increase focus on “difficult to solve” situations  As our state moves increasingly to a “community based service model” we want to always be a part of the community solution

17. Program Improvement Initiatives Based on Self-Evaluation & Program Review  Conduct a detailed data analysis regarding our Public Awareness efforts and figure out if they are advancing our community based service efforts and our efforts to reach consumers that are in areas that receive fewer services  Work with the data to see if we can determine if consumers are utilizing a continuum of services

18.  Questions? Contact: Sara Sack ssack@ku.edu