1. Workshop #6: Measuring Access to Justice: Why is it important and how are we going to do just that? Society of Ontario Adjudicators and Regulators Annual Conference November 6, 2014 Dr. Keiko Kuji-Shikatani, C.E. Research, Evaluation and Capacity Building Branch Student Achievement Division, Ministry of Education

2. You’ve got to know where you are if you want to get to where you want to get to. Google Maps - ©2012 Google Keiko.Kuji-Shikatani@ontario.ca SOAR’s Annual Conference Workshop #6 Measuring Access to Justice

3. For vision-and values-driven social innovators highest form of accountability is internal Seriousness of inquiry and resulting learning constitutes accountability: Are we walking the talk? Are we being true to our vision? Are we dealing with reality? Are we connecting the dots between here-and-now reality and our vision? And how do we know? What are we observing that’s different, that’s emerging? (Patton 2010) Keiko.Kuji-Shikatani@ontario.ca SOAR’s Annual Conference Workshop #6 Measuring Access to Justice

4. Logic Models Defined Logic models are visual methods of presenting an idea. Logic Models: Offer a way to describe and share an understanding of relationships among elements necessary to operate a program or change effort Describe a bounded project or initiative: both what is planned (the doing) and what results are expected (the getting) Likened to a picture map of how the program theoretically works to achieve benefits for participants - often expressed as brief diagrams or flow charts Logic Models provide a clear roadmap to a specified end. Knowlton & Phillips (2012) The Logic Model Guidebook – Better Strategies for Great Results, Treasury Board of Canada (2001) p.11 Keiko.Kuji-Shikatani@ontario.ca SOAR’s Annual Conference Workshop #6 Measuring Access to Justice

5. Uses of Logic Models Logic models serve as a tool with multiple uses: To clarify for managers and staff the linkages between activities, outputs and the expected outcomes of the policy, program or initiative. In so doing, it will serve to clarify and distinguish the expected immediate, intermediate and ultimate outcomes To communicate externally about the rationale, activities and expected results of the policy, program or initiative To test whether the policy, program or initiative “ makes sense ” from a logical perspective To provide the fundamental backdrop on which the performance measurement and evaluation strategies are based (i.e., determining what would constitute success) Treasury Board of Canada, 2001, p.11 Keiko.Kuji-Shikatani@ontario.ca SOAR’s Annual Conference Workshop #6 Measuring Access to Justice

6. 6 Two Types of Logic Models: One Logic Theory of Change Logic Model and Program Logic Models differ by level of detail and use but represent the same logic: A Theory of Change Model is a general representation of how you believe change will occur A Program Logic Model details: –Resources –Planned activities –Outputs –Outcomes over time that reflect intended results Knowlton & Phillips (2008) The Logic Model Guidebook – Better Strategies for Great Results Keiko.Kuji-Shikatani@ontario.ca SOAR’s Annual Conference Workshop #6 Measuring Access to Justice

7. We believe that … Why do you need to do this? How do you know? High level descriptions of the evidence/evaluation (including needs assessment) and research, practice, and theory. Define the assumptions that support the specified strategies that inform choices Keiko.Kuji-Shikatani@ontario.ca SOAR’s Annual Conference Workshop #6 Measuring Access to Justice

8. … will get us there. Where do you want to get to?  High level descriptions, specifying the intended results Keiko.Kuji-Shikatani@ontario.ca SOAR’s Annual Conference Workshop #6 Measuring Access to Justice

9. … doing this … How would you summarize what you are doing to get there?  High level descriptions of how intended changes will occur  Name the activities and strategies that will deliver your intended results Keiko.Kuji-Shikatani@ontario.ca SOAR’s Annual Conference Workshop #6 Measuring Access to Justice

10. Discuss with your elbow partner Try the elevator conversation test We believe that…. doing this …. will get us there Keiko.Kuji-Shikatani@ontario.ca SOAR’s Annual Conference Workshop #6 Measuring Access to Justice

11. Discuss with your elbow partner: articulate where you want to get to Objectives: What do you want to achieve? What change(s) will occur? Describe stepwise series of outcomes Keiko.Kuji-Shikatani@ontario.ca SOAR’s Annual Conference Workshop #6 Measuring Access to Justice

12. Discuss with your elbow partner: articulate the doing Name all the activities needed to generate the outcomes. Define the resources/inputs that link directly to and will "supply" the activities. Identify the outputs that reflect the accomplishment of activities. (Knowlton & Phillips, 2012) Keiko.Kuji-Shikatani@ontario.ca SOAR’s Annual Conference Workshop #6 Measuring Access to Justice

13. Discuss with your elbow partner: What would tell you that you are getting there? How do you know?  Logic models shows when, where and how to look for information most needed to manage the program and determine its effectiveness. Keiko.Kuji-Shikatani@ontario.ca SOAR’s Annual Conference Workshop #6 Measuring Access to Justice

14. Context Challenges Common Pitfalls: blind spots (unintentional omissions & errors out of habits) & myths (e.g. access = use, knowledge = action, activities = outcomes, send = communications) Logic, scale (of effort) and specificity (discrete work seeking near-term outcomes) Politics, persuasion (grandiose claims?) and perception (shared-view vs. truth/reality) (Knowlton, L., & Phillips, C. C. (2012). The Logic Model Guidebook - Better Strategies for Great Results. Thousand Oaks, CA: Sage) Keiko.Kuji-Shikatani@ontario.ca SOAR’s Annual Conference Workshop #6 Measuring Access to Justice

15. Testing model quality - SMART SMART tests outcomes Specific: what to do is clear enough to act-on and is connected to outcomes Measurable: the content can be both quantified and qualified Action oriented: the content is selected to provoke change in awareness, knowledge, skill and behavior Realistic: the content is both plausible and feasible Timed: the content specifies a duration and illustrates the time-dependent sequence of outcomes for progress toward results Knowlton & Phillips (2012) Keiko.Kuji-Shikatani@ontario.ca SOAR’s Annual Conference Workshop #6 Measuring Access to Justice

16. Testing model quality - FIT FIT tests activities Frequency of occurrence Intensity of strength of the given effort Targeted at a specified market or audience The treatment (or intervention) needs to be both appropriate (the right choice) and adequate (the right dose) for the resolution of problems or generating a change in awareness, knowledge, skill or behaviour. Knowlton & Phillips (2012) Keiko.Kuji-Shikatani@ontario.ca SOAR’s Annual Conference Workshop #6 Measuring Access to Justice